Kafka check - first pass

[conorbranagan]

Uses the standard JMX format to pull in metrics from Kafka.

We're missing metrics by topic which is currently a limitation of the JMX configuration. I think we'll have to add something comparable to how we post-process cassandra metrics (https://github.com/DataDog/jmxfetch/blob/master/src/main/java/org/datadog/jmxfetch/Reporter.java#L94-L104) because the Kafka metrics have the topic name in them. I think we'd also want to limit to a list of topics to handle the case where a person has hundreds of topics.

Here's how a per-topic metric looks in JMX as a bean:

kafka.server":type="BrokerTopicMetrics",name="check_runs-FailedFetchRequestsPerSec"

[remh]

Maybe you should use the is_jmx flag in the init_config. I think that's what we should use for newer jmx checks.

[conorbranagan]

Where is that read?

[remh]

Starting from 4.1.0:
https://github.com/DataDog/dd-agent/blob/master/jmxfetch.py#L151-L152

For 4.0.x:
https://github.com/DataDog/dd-agent/blob/4.0.1/jmxfetch.py#L104-L105

[clofresh]

what unit is kafka.net.bytes_in/kafka.net.bytes_out? I'm getting really low values (< 1)

[remh]

Nitpick, you should add the instances section on top of the file so we don't have to scroll all the way down to configure a check.

[clofresh]

@conorbranagan @remh Added a kafka_consumer check which queries kafka and zookeeper for their offsets and computes lag: 9675070

[remh]

defaultdict is not in python 2.4, you should import from compat:

from compat import defaultdict

[clofresh]

Hmm, the libraries that this check depends on are extremely incompatible with 2.4, how should we handle that?

[remh]

two options:

• Wait for the self contained agent
• Catch the python version and raise an exception if < 2.6

[remh]

@clofresh @conorbranagan That's great stuff!
Can you fix the according to the different comments so it will be part of the next release ?

[remh]

Thanks!