Add OTel java agent with JMX Metric Insights to kafka (#654)

* Add otel agent with JMX metric insights to kafka

Signed-off-by: svrnm <neumanns@cisco.com>

* Increase memory limit for kafka

* Remove pb files

Signed-off-by: svrnm <neumanns@cisco.com>

* Add docs for kafka service

Signed-off-by: svrnm <neumanns@cisco.com>

* Revert removal of demo.proto

Signed-off-by: svrnm <neumanns@cisco.com>

* Update CHANGELOG

Signed-off-by: svrnm <neumanns@cisco.com>

* Fix markdownlint error

Signed-off-by: svrnm <neumanns@cisco.com>

Signed-off-by: svrnm <neumanns@cisco.com>
Co-authored-by: Carter Socha <43380952+cartersocha@users.noreply.github.com>
Co-authored-by: Pierre Tessier <pierre@pierretessier.com>

CHANGELOG.md

@@ -158,6 +158,8 @@ significant modifications will be credited to OpenTelemetry Authors.
 ([#628](https://github.com/open-telemetry/opentelemetry-demo/pull/628))
 * Update `quoteservice` to use opentelemetry-php beta release
 ([#644](https://github.com/open-telemetry/opentelemetry-demo/pull/644))
+* Add OTel java agent with JMX Metric Insights to kafka
+([#654](https://github.com/open-telemetry/opentelemetry-demo/pull/654))
 * Add resource detectors to payment service
 ([#651](https://github.com/open-telemetry/opentelemetry-demo/pull/651))
 * Add resource detectors to frontend service

docker-compose.yml

@@ -537,10 +537,14 @@ services:
     deploy:
       resources:
         limits:
-          memory: 600M
+          memory: 800M
     restart: always
     environment:
-      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
+      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
+      - OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
+      - OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
+      - OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE
+      - OTEL_SERVICE_NAME=kafka
     logging: *logging
     healthcheck:
       test: nc -z kafka 9092

docs/services/kafka.md

@@ -0,0 +1,21 @@
+# Kafka
+
+This is used as a message queue service to connect the checkout service with
+the accounting and fraud detection services.
+
+[Kafka service source](../../src/kafka/)
+
+## Auto-instrumentation
+
+This service relies on the OpenTelemetry Java Agent and the built in
+[JMX Metric Insight Module](https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/jmx-metrics/javaagent)
+to capture [kafka broker metrics](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation/jmx-metrics/javaagent/kafka-broker.md)
+and send them off to the collector via OTLP.
+
+The agent is passed into the process using the `-javaagent` command line
+argument. Command line arguments are added through the `KAFKA_OPTS`
+in the `Dockerfile`.
+
+```dockerfile
+ENV KAFKA_OPTS="-javaagent:/tmp/opentelemetry-javaagent.jar -Dotel.jmx.target.system=kafka-broker"
+```

src/kafka/Dockerfile

@@ -1,5 +1,10 @@
 FROM confluentinc/cp-kafka:latest-ubi8
 
+USER root
+ARG version=1.21.0
+ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v$version/opentelemetry-javaagent.jar /tmp/opentelemetry-javaagent.jar
+RUN chmod go+r /tmp/opentelemetry-javaagent.jar
+
 USER appuser
 WORKDIR /tmp
 COPY ./src/kafka/clusterID clusterID
@@ -18,5 +23,6 @@ ENV KAFKA_METADATA_LOG_MAX_RECORD_BYTES_BETWEEN_SNAPSHOTS=2800
 ENV KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
 ENV KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
 ENV KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1
+ENV KAFKA_OPTS="-javaagent:/tmp/opentelemetry-javaagent.jar -Dotel.jmx.target.system=kafka-broker"
 
 ENTRYPOINT ["/bin/sh", "-c", "ls -lh /tmp && /tmp/update_run.sh && /etc/confluent/docker/run"]