Unable to get traces exported out of the demo app

[MrHadiSatrio]

Hello! First of all, thank you for publishing this project. It will undoubtedly help many Android developers improve their application's health and performance.

Unfortunately, I'm unable to export traces from the demo app, even though I have Jaeger and the collector running on my machine. Logcat consistently reports the following error:

Failed to export logs. The request could not be executed. Full error message: unexpected end of stream on http://10.0.2.2:4318/... 
java.io.IOException: unexpected end of stream on http://10.0.2.2:4318/...
	at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:210)
	(...)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:637)
	at java.lang.Thread.run(Thread.java:1012)
Caused by: java.io.EOFException: \n not found: limit=0 content=…
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:335)
	(...)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:637) 
	at java.lang.Thread.run(Thread.java:1012) 

I've verified using nc that port 4318 and 4317 is accessible. Did I miss any configuration steps for Jaeger or the collector that might be causing this issue?

[LikeTheSalad]

Hi! It seems to be an issue with Jaeger, because I was also having this issue until I locally rolled back Jaeger's version from 1.59 to 1.58 here. Said version was recently set to 1.59.

Maybe something about Jaeger's config has changed recently, do you know anything about this @breedx-splk ?

[breedx-splk]

Yeah, this is called out in the release notes: https://github.com/jaegertracing/jaeger/releases/tag/v1.59.0

I opened #508 to fix this....just remember to docker compose build to get the update.

[julian-sanz-code-seat]

Hi @LikeTheSalad and @breedx-splk! why do you refer to jaeger when @MrHadiSatrio are talking about demo-app and sending traces to collector??
I have the same issue. I guess that demo-app are trying to send traces and logs to collector through http://10.0.2.2:4318 address but it could not "reach out" collector endpoints or something like that.

[julian-sanz-code-seat]

I think I have just solved the problem adding otlp protocols endpoints in collector.yaml like this:
receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318

after that, down and up the docker compose again and jaeger started to show demo app info.

[breedx-splk]

why do you refer to jaeger when @MrHadiSatrio are talking about demo-app and sending traces to collector

@julian-sanz-code-seat well jaeger was also mentioned in the original issue description, but ok. Above, I linked to the release notes for the latest jaeger, which this project uses, which details the root cause of this problem. To quote, from the jaeger release notes linked above:

The OTEL Collector upgrade brought in a change where OTLP receivers started listening on localhost instead of 0.0.0.0 as before. As a result, when running in container environment the endpoints are likely unreachable from other containers (Issue jaegertracing/jaeger#5737). The fix will be available in the next release. Meanwhile, the workaround is to instruct Jaeger to listen on 0.0.0.0, as in this fix:
- COLLECTOR_OTLP_GRPC_HOST_PORT=0.0.0.0:4317
- COLLECTOR_OTLP_HTTP_HOST_PORT=0.0.0.0:4318

This is why the workaround you mentioned above works. The root cause is that the collector only listens on localhost by default now, which mean that jaeger (when running on a separate host like in our demo app docker compose env) can't just talk to it by service name (because it resolves to a different internal IP address). That's also why changing the config to listen on all network interfaces causes things to start working.

Thanks again for following up, and please reopen if things are still a problem for you.

[MrHadiSatrio]

@breedx-splk, @LikeTheSalad, thanks for the replies but I'm still facing a similar issue on my machine. Even downgrading Jaeger to 1.58 didn't work out. :/

[MrHadiSatrio]

Looks like there is a change on the collector side as well.

collector-1  | 2024-08-08T15:14:35.456Z	info	localhostgate/featuregate.go:63	The default endpoints for all servers in components have changed to use localhost instead of 0.0.0.0. Disable the feature gate to temporarily revert to the previous default.	{"feature gate ID": "component.UseLocalHostAsDefaultHost"}

I'm able to see the traces post disabling said feature gate.

Alternatively, explicitly specifying the endpoints on collector.yaml also did the trick.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318