[Chore] Add e2e test case for OTEL Kafka receiver and exporter. #2274
Conversation
|
I'm not sure whether the use case you're testing against is in scope for the operator. Don't get me wrong, this seems like a useful test if you wanted to verify end-to-end that sending telemetry through Kafka in K8s works as expected. But the operator has nothing to do with Kafka directly, so I'm not convinced we should be on the hook for failures in this kind of test. WDYT @open-telemetry/operator-approvers ? |
|
adding to the SIG agenda for tomorrow |
|
cc @pavolloffay @iblancasa, this test case pertains to the Kafka support we're implementing. Given that the Kafka receiver and exporter are part of opentelemetry-collector-contrib, I concur with @swiatekm-sumo: this test case falls outside the scope of operator. It's worth noting that this test case is part of the e2e-openshift test suite and isn't executed within the project's CI pipeline but will be executed by OpenShift CI. Nevertheless, it serves as a valuable check for Kafka support, helping us identify any potential regressions in the feature. I'd appreciate hearing your thoughts on this. BTW I can move it to https://github.com/openshift/distributed-tracing-qe and run it with the OCP CI jobs, if the SIG decides not to include it in the project repository. |
|
What is the expectation of the Operator maintainers if and when this test case fails? |
|
@bryan-aguilar The e2e-openshift test suite is not run as part of the operator repo's CI. The test cases are OpenShift specific and the Operator maintainers wouldn't need to act upon any failures. If there is any regression in this Kafka feature, we need to reach out in https://github.com/open-telemetry/opentelemetry-collector-contrib to get it fixed. |
|
@pavolloffay @jaronoff97 Do we have any update from SIG, is it fine if we merge this in the e2e-openshift test suite ? |
|
I don't mind merging this as it is tested in a separate OpenShift CI. However I would prefer to keep the test scoped to only Kafka - do not use Jaeger and prometheus. |
|
@pavolloffay Updated the test and removed dependency on Jaeger and Prometheus. |
|
| LABEL_SELECTOR="app.kubernetes.io/instance=kuttl-kafka.kafka-receiver" | ||
|
|
||
| # Define the search strings | ||
| SEARCH_STRING1="Name : lets-go" |
There was a problem hiding this comment.
How is the assertion working? Does it check the kafka stdout that contains traces? If that is the case is there a way to control attributes/service name/operation name of trace generator and use the same string in assertion?
There was a problem hiding this comment.
Its from the OTEL collector which has kafka receiver and exporting the traces via the debug exporter. https://github.com/open-telemetry/opentelemetry-operator/pull/2274/files#diff-712c49955ef631e30d16b4371d67b239c8793bba5aa500cbbe4d694fcebcf8d9R14 The script searches for the receiver otel pod and check the logs which the debug exporter has generated for the presence of traces.
Telemetrygen --> OTEL with OTLP receiver and Kafka exporter -->> Kafka instance -->> OTEL with Kafka receiver and debug exporter -->> Script checks the pods logs.
With this we do not have to ship the traces to a telemetry store and can test both the Kafka receiver and exporter in test case.
There was a problem hiding this comment.
If we want we can check other attributes.
There was a problem hiding this comment.
The format is not json for the debug exporter. https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/debugexporter/README.md#detailed-verbosity It would be nice if it can print json.
There was a problem hiding this comment.
The approach is fine but I would prefer to assert on the attributes or service name that is explicitly set in the telemetrygen
--telemetry-attributes map[string]string Custom telemetry attributes to use. The value is expected in the format "key=\"value\"". Flag may be repeated to set multiple attributes (e.g --telemetry-attributes "key1=\"value1\"" --telemetry-attributes "key2=\"value2\"")
--service string Service name to use (default "telemetrygen")
There was a problem hiding this comment.
Ok, I'll change the assert and update the PR.
There was a problem hiding this comment.
Updated and added the following attribute and service name to Telemetrygen:
- "--otlp-attributes=test=\"kuttl-kafka\""
- "--service=\"kafka\""
And check the same from the assert script:
logger.go:42: 08:17:54 | kafka/5- | running command: [sh -c ./tests/e2e-openshift/kafka/check_traces.sh]
logger.go:42: 08:17:54 | kafka/5- | "-> service.name: Str("kafka")" found in kafka-receiver-collector-6c8857c54c-mf6xx
logger.go:42: 08:17:54 | kafka/5- | "-> test: Str(kuttl-kafka)" found in kafka-receiver-collector-6c8857c54c-mf6xx
logger.go:42: 08:17:54 | kafka/5- | Traces with service name Kafka and attribute test=kuttl-kafka found.
logger.go:42: 08:17:54 | kafka/5- | test step completed 5-
Testing:
This e2e case tests the Kafka receiver and exporter.