Cross-Component Telemetry Correlation for the Java SDK

[grlima]

We want to support automatic telemetry correlation across components (e.g. services) in a composite application running Java.

Proposed Solution
We propose to design and implement a similar solution to what is available in other SDK’s, making use of fields in the telemetry data model and HTTP headers to propagate correlation ID’s among the different layers and components.

For .NET, issue was discussed here: microsoft/ApplicationInsights-dotnet#47

HTTP Headers
X-ms-request-id: the ID of the parent telemetry
X-ms-request-root-id: the ID of the logical operation
Request-Id: if present, supersedes the above 2 headers.
Correlation-Context: if exists, the key-value pairs here go into custom properties for the request telemetry.

Telemetry Fields
Id: unique identifier for each request and dependency telemetry item.

OperationId: identifier for the logical operation. Value needs to be set from Request-Id or x-ms-request-root-id

ParentId: identifies telemetry that generated the current sub-operation. In case of dependency telemetry, the parent is the request that originated it (within the same service/component). In case of request telemetry, the parent is the dependency that triggered the request. In this case, the dependency belongs to a client component and, therefore, the value needs to be set from the x-ms-request-id header (Request-Id if present)

[SergeyKanzhelev]

Headers X-ms-request-id and X-ms-request-root-id are deprecated. Please do not use them for the fresh implementation. They can cause problems with Azure Storage REST API calls.

Make sure to follow specs for Request-Id to extract the OperationId field correctly. Also note that Id may be stored in Correlation-Context field according to the spec.

Also please support Request-Context header for http request and response. It is widely used for app-id propagation. See http://apmtips.com/blog/2017/10/18/two-types-of-correlation/ and https://docs.microsoft.com/en-us/azure/application-insights/application-insights-correlation

Beware that there will be a HTTP protocol change some time soon to align with other tracing vendors. However it will be great to keep back compat with the existing protocol so this work will not go to waste.

What's the plan to propagate correlation context from requests to dependencies? Do you use Tls? Will you provide helper methods to set a context on a thread after async thread switch?

[gavlyukovskiy]

BTW there is project called Spring Cloud Sleuth that is developed to solve same problem.
It is very extendable so that most probably you can write different exporter (Tracer) to the Application Insights. But at least you can look how they propagate TraceId (X-ms-request-root-id) and SpanId(X-ms-request-id) to the different components including http requests, thread executor tasks, rabbitmq messages, etc..

[SergeyKanzhelev]

@dhaval24 promised to look at Zipkin closer. In fact integration work has started already openzipkin-attic/zipkin-azure#27 . Terminology mapping described here: https://docs.microsoft.com/en-us/Azure/application-insights/application-insights-correlation#open-tracing-and-application-insights

Please note that x-ms- headers were deprecated. New headers defined in https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.DiagnosticSource/src/HttpCorrelationProtocol.md should be used

[dhaval24]

@SergeyKanzhelev thanks for reminding me about Zipkins. I will definitely take a look at this once I have some time.

[leikelman]

Is this issue/thread related to this article? https://docs.microsoft.com/en-us/azure/application-insights/app-insights-monitor-multi-role-apps

We have a microservice project and custom logging that we have added via this SDK to Application Insights. The challenge we are having is that we are not seeing a way to visualize all of the microservices (APIs in our case) in a single Application Insights instance. Based on the article above, it appears the way to do this is via the multi-role strategy.

Is this functionality/properties exposed via the Java SDK?

[grlima]

@leikelman - yes, this is the work which will enable what the article talks about (for Java components). Regarding properties, you can set the role_name like so: requestTelemetry.getContext().getDevice().setRoleName("FrontEnd"); Same goes for dependency telemetry (RemoteDependencyTelemetry class).

[leikelman]

@grlima Thank you. I have tested this with a process where API #1 is calling API #2 and I can see both APIs in the Application Map with their individual dependencies. The only part that is missing is that the link between API #1 and API #2 is not present. Is there a mechanism to link/set API #2 as a dependency for API #1? Right now they are showing up as individual applications in the map and not linked.

[dhaval24]

@leikelman I believe this is the purpose of this feature. Having correlation between cross application so AppMap can show the linkage but I might be mistaken. @grlima to comment on this and correct me if I am wrong. Also just in case if you have glanced through #469 , #468 and would be interested in filling out a short survey, would be great,

[leikelman]

@dhaval24 with the current version, I can show multiple applications in one AppMap, but can't seem to set the dependencies between the two (or more) applications. I see this has been moved to release 1.0.11, so expecting the capability will be available once 1.0.11 is released.

[dhaval24]

@leikelman yes hopefully in the next release we should have this feature. Please make sure to check it out once it is out.