Replacing the okhttp based sender with the JDK http one in the OTLP exporter

[ppatierno]

Hi all,
we are using OpenTelemetry in our Strimzi projects and by default the otlp exporter.
The problem we have is about the okttp + kotlin dependencies it brings because of the usage of the okttp based sender.
I came across this issue #5351 about adding an "experimental" JDK HTTP based sender which would be great for us to be used, getting rid of the okttp + kotlin dependency.
What I can't understand from that issue and corresponding merged PR is the status of the art for it.
If it's usable, it's not officially supported yet, if there is a way to swap it with the default okhttp one.
Anyone providing more information about this?
Maybe @jack-berg I saw you mostly interested and involved on this?

[jack-berg]

Hi @ppatierno! That PR was merged on July 6th and first available in the July 1.28.0 release. Its usable, but is still relatively new (as indicated by its "alpha" status). Please let us know if you find any issues!

I'm working on documenting the different OTLP exporter options - when / how to use them - but not quite there yet. This [comment
(https://github.com/open-telemetry/opentelemetry.io/pull/3144#discussion_r1287632380) describes the situation at a high level. And here's an example of how you would use the jdk version via gradle:

  implementation('io.opentelemetry:opentelemetry-exporter-otlp') {
    exclude group: 'io.opentelemetry', module: 'opentelemetry-exporter-sender-okhttp'
  }
  implementation('io.opentelemetry:opentelemetry-exporter-sender-jdk')

[ppatierno]

@jack-berg thanks for your reply! Just to be sure I got it right how to use it ...
I guess your example above is about an application (using Gradle) which excludes the okhttp sender and add the dependency to the jdk http based one. In our case we are using Maven but of course I could do the same.
The question is ... are opentelemetry-exporter-sender-okhttp and opentelemetry-exporter-sender-jdk available as artifacts in Maven Central? (starting from 1.28.0 release).

[jack-berg]

The question is ... are opentelemetry-exporter-sender-okhttp and opentelemetry-exporter-sender-jdk available as artifacts in Maven Central? (starting from 1.28.0 release).

Yes, starting with 1.28.0:

• https://repo1.maven.org/maven2/io/opentelemetry/opentelemetry-exporter-sender-jdk/
• https://repo1.maven.org/maven2/io/opentelemetry/opentelemetry-exporter-sender-okhttp/

[ppatierno]

Thanks @jack-berg, we'll consider to use it or not taking into account it's alpha and it's really something new (July was almost ... yesterday :-))
Anyway thanks for explanation!

[fax4ever]

Hi @jack-berg. I want try to do the same for Infinispan.
Trying to replace okhttp based sender with the JDK http one in the OTLP exporter using OpenTelemetry SDK Autoconfigure. I get this error saying that No GrpcSenderProvider found on classpath:

09:28:03,288 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:423)
09:28:03,289 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at org.infinispan.server.core.telemetry.TelemetryServiceFactory.construct(TelemetryServiceFactory.java:37)
09:28:03,289 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doInstantiateWrapper(BasicComponentRegistryImpl.java:188)
09:28:03,290 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at org.infinispan.factories.impl.BasicComponentRegistryImpl.instantiateWrapper(BasicComponentRegistryImpl.java:178)
09:28:03,291 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at org.infinispan.factories.impl.BasicComponentRegistryImpl.getComponent0(BasicComponentRegistryImpl.java:143)
09:28:03,291 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at org.infinispan.factories.impl.BasicComponentRegistryImpl.getComponent(BasicComponentRegistryImpl.java:62)
09:28:03,292 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at org.infinispan.factories.AbstractComponentRegistry.getComponent(AbstractComponentRegistry.java:193)
09:28:03,292 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at org.infinispan.factories.AbstractComponentRegistry.getComponent(AbstractComponentRegistry.java:179)
09:28:03,293 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at org.infinispan.rest.RestServer.startInternal(RestServer.java:128)
09:28:03,293 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at org.infinispan.server.core.AbstractProtocolServer.start(AbstractProtocolServer.java:119)
09:28:03,294 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at org.infinispan.server.SecurityActions.lambda$startProtocolServer$2(SecurityActions.java:60)
09:28:03,294 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at org.infinispan.security.Security.doPrivileged(Security.java:49)
09:28:03,294 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at org.infinispan.server.SecurityActions.startProtocolServer(SecurityActions.java:59)
09:28:03,295 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at org.infinispan.server.Server.lambda$run$3(Server.java:467)
09:28:03,295 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
09:28:03,296 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
09:28:03,296 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
09:28:03,297 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
09:28:03,297 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
09:28:03,298 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
09:28:03,299 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
09:28:03,299 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
09:28:03,300 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
09:28:03,300 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
09:28:03,301 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: Caused by: java.lang.IllegalStateException: No GrpcSenderProvider found on classpath. Please add dependency on opentelemetry-exporter-sender-okhttp or opentelemetry-exporter-sender-grpc-upstream
09:28:03,302 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at io.opentelemetry.exporter.internal.grpc.GrpcExporterBuilder.resolveGrpcSenderProvider(GrpcExporterBuilder.java:229)
09:28:03,303 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at io.opentelemetry.exporter.internal.grpc.GrpcExporterBuilder.build(GrpcExporterBuilder.java:156)
09:28:03,303 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporterBuilder.build(OtlpGrpcSpanExporterBuilder.java:189)
09:28:03,304 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at io.opentelemetry.exporter.otlp.internal.OtlpSpanExporterProvider.createExporter(OtlpSpanExporterProvider.java:61)
09:28:03,305 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at io.opentelemetry.sdk.autoconfigure.internal.SpiHelper.lambda$loadConfigurable$0(SpiHelper.java:60)
09:28:03,305 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at io.opentelemetry.sdk.autoconfigure.internal.NamedSpiManager.tryLoadImplementationForName(NamedSpiManager.java:51)
09:28:03,306 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
09:28:03,307 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at io.opentelemetry.sdk.autoconfigure.internal.NamedSpiManager.getByName(NamedSpiManager.java:41)
09:28:03,308 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at io.opentelemetry.sdk.autoconfigure.SpanExporterConfiguration.configureExporter(SpanExporterConfiguration.java:91)
09:28:03,308 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at io.opentelemetry.sdk.autoconfigure.SpanExporterConfiguration.configureSpanExporters(SpanExporterConfiguration.java:67)
09:28:03,310 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at io.opentelemetry.sdk.autoconfigure.TracerProviderConfiguration.configureTracerProvider(TracerProviderConfiguration.java:53)
09:28:03,310 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:356)
09:28:03,311 INFO  [o.i.CONTAINER] [o.i.s.f.RequestTracingIT#0]STDOUT: 	... 23 more

[laurit]

jdk http client based sender does not support grpc, only http/protobuf

[fax4ever]

makes sense thank you

[fax4ever]

Also without using the Autoconfigure module, if I try to manually build a span exporter:

OtlpGrpcSpanExporter.builder().setEndpoint("http://localhost:4317").build();

I get the error:

java.lang.IllegalStateException: No GrpcSenderProvider found on classpath. Please add dependency on opentelemetry-exporter-sender-okhttp or opentelemetry-exporter-sender-grpc-upstream

	at io.opentelemetry.exporter.internal.grpc.GrpcExporterBuilder.resolveGrpcSenderProvider(GrpcExporterBuilder.java:229)
	at io.opentelemetry.exporter.internal.grpc.GrpcExporterBuilder.build(GrpcExporterBuilder.java:156)
	at io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporterBuilder.build(OtlpGrpcSpanExporterBuilder.java:189)

[fax4ever]

Do I need to add opentelemetry-exporter-sender-grpc-upstream if I want to use the grpc protocol?

[jack-berg]

The senders which support gRPC are the default included opentelemetry-exporter-sender-okhttp or opentelemetry-exporter-sender-grpc-managed-channel. If you want to use opentelemetry-exporter-sender-grpc-managed-channel, you should:

• Exclude the opentelemetry-exporter-sender-okhttp as a dependency. Its possible to have both opentelemetry-exporter-sender-okhttp and opentelemetry-exporter-sender-grpc-managed-channel present, but then you need to specify which of the two is used by setting io.opentelemetry.exporter.internal.grpc.GrpcSenderProvider to the FQCN of the one you wish to use.
• Call OtlpGrpc{Signal}ExporterBuilder#setChannel(..). If using with autoconfigure, follow the instructions in this comment to override the autoconfigured exporter with an alternative which has had setChannel invoked.

[fax4ever]

makes sense thank you!

[ppatierno]

@fax4ever because we (from the Strimzi maintainers team) are interested in this for our components as well, did you have it working by excluding okhttp and using the Java SDK client? I am curious to know if it's good enough to be used and we should start to investigate on it. Thanks!

[fax4ever]

Hi @ppatierno. Here you can find the solution we applied to Infinispan. infinispan/infinispan#11474. We made the choice of not exporting the tracing data with gRPC, but only with http/protobuf, all perfectly working with the SDK autoconfigure module. Because in order to export the tracing using gRPC the Java SDK client is not enough, we had to add all gPRC libraries to the Infinispan server and we didn't want that.

[ppatierno]

@fax4ever thanks for the update! While it's a good news, on the other hand it would mean changes on the user's backend side if they have the gRPC port enabled for getting telemetry data. Going with the Java SDK would mean forcing them to move from the gRPC to the HTTP port. It would be a kind of breaking change but it's clear that there is no real workaround because of Java SDK not being enough for gRPC.