Using Java Agent with AI SDK

[wcislo]

What we have:

Right now we are using AI SDK v 2.6.3 for tracking HTTP operations and remote dependencies. We use it in multi-tenant system, and it is important for us to incorporate additional information about tenant (which is passed via request headers) in our telemetry data.

1. For incoming requests, we use a filter:

protected void doFilterInternal(...) {
        ...
        RequestTelemetry requestTelemetry = ThreadContext.getRequestTelemetryContext().getHttpRequestTelemetry();
        requestTelemetry.getProperties().putIfAbsent("TENANT", tenantHolder.getTenant());
        requestTelemetry.setName(tenantHolder.getTenant() + " " + requestTelemetry.getName());
        filterChain.doFilter(httpServletRequest, httpServletResponse);
    }

What we achive is visible distinction between requests for tenants, and ability to filter telemetry for them.

2. We also achieved something similiar for remote dependencies by manually sending telemetry data via RemoteDependencyTelemetry and TelemetryClient. We are able to produce names with tenant information, full URL, time measurments and it basically works perfectly fine

What we want to achieve:

We want to keep all of above functionalities, but also have automatic telemetry collected for JMS, Redis, Postgresql etc. so we tried to use SKD with Java Agent v 3.1.0

Problems:

1. Despite https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent#override-the-request-telemetry-name-using-the-2x-sdk , this solution doesn't work for us. For above code getRequestTelemetryContext() always returns another instance, HttpRequestTelemetry is not populated and code is not having any effect on collected telemetry data. When debugging, invoking requestTelemetry.getName() after filterChain returns desired name.
2. Our custom telemetry is not correlated with auto-collected, so requests are duplicated, and we couldn't find a way to disable auto collection of only remote dependencies, or to somehow explicitly correlate auto and manual telemetry.

In short:

Basically what we want is to preserve HTTP telemetry names in form:

<tenant_name> <http_method> <endpoint/url>
e.g.: Tenant_123 GET /api/v1/get-tenant-info/{tenant_id}

and what we have is:
GET /api/v1/get-tenant-info/{tenant_id} - for incoming requests

HTTP GET - for outgoing request (+ duplicated manual telemetry)

We use:

Spring 4.3.30, applicationinsights-web 2.6.3, Application Insights Java Agent 3.1.0

Thanks in advance for any suggestions.

[wcislo]

I tried to convert as much code to configuration (span processors) as I could, but again I encountered problems:

1. https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-sampling-overrides#common-span-attributes lists some common attributes - where I can find all of them? I tried https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/http.md, as the documentation seems to be copied from there, but only some of the properties in telemetry processors work (I didn't check all of them, but those I've checked seem to correlate with aforementioned some common attributes)
2. Is there any way to add custom properties to auto-collected remote dependency telemetry via SDK? It would be perfect if https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent#add-request-custom-dimensions-using-the-2x-sdk would be propagated also to outgoing requests/queries etc. (it seems to affect only operations/incoming requests)

Any help would be appreciated.

[trask]

hey @wcislo! I'm having a hard time following exactly what's working and what's not for you. Would it be possible for you to create a small repro to demonstrate? I think that would help us to guide you to a solution.

[wcislo]

1. Auto-collected operations:
   Use this code:

 RequestTelemetry requestTelemetry = ThreadContext.getRequestTelemetryContext().getHttpRequestTelemetry(); 
 requestTelemetry.getProperties().putIfAbsent("CP", someDynamicProperty); 
 requestTelemetry.setName(someDynamicProperty + " " + requestTelemetry.getName()); // automatically generated name is ok, I just want to add a prefix to it (dynamically set, based e.g. on HTTP header value)

With pure SDK v2.6.3: This code works, name is modified and visible in Azure Portal
With SDK v2.6.3 + Java Agent 3.1.0: property is set, but name do not change (it's the same as if requestTelemetry.setName was not ivoked at all)

2. Remote HTTP dependencies:
   We use this code:

 RemoteDependencyTelemetry remoteDependencyTelemetry = new RemoteDependencyTelemetry();
remoteDependencyTelemetry.setName(someDynamicProperty + request.getMethod() + request.getPath());
telemetryClient.trackDependency(remoteDependencyTelemetry);

This works perfectly fine with SDK + Java Agent, but the thing is that I can't find a way to correlate it with auto-collected telemetry, so it gets duplicated (I can see same remote dependency request twice, mine and auto-collected). I want to find a way to either correlate them, or disable auto-collected remote dependencies, or modify auto-collected telemetry with custom attribute.

What we want
In other words, we want to have name displayed in form:

<some custom property> <http method> <http path>

where custom property must be set dynamically.

I think it should be possible to achieve through telemetry processors e.g.:

{
...
	"type": "span",
	"name": {
		"fromAttributes": [
			"CP",  <- custom dynamic property set via requestTelemetry.getProperties().putIfAbsent("CP", "Dynamic property");
			"http.method",
			"http.url
		],
		"separator": " "
	}
...
}

above example works, but there are two problems:

1. I can't find full list of all attributes that can be used here anywhere. I've tried to use some from opentelemetry docs but with no success - the ones that I used in above example work, but for example http.target or http.route doesn't
2. I can't figure out a way to add dynamic custom property to auto-collected remote dependency telemetry. If I could do it I would remove manual telemetry and avoid duplication. It would be nice if there was some InheritableThreadLocal variable, preferably map, that would add custom properties to any dependency by given thread/child thread -ThreadContext.getRequestTelemetryContext() seems to work only for operations (incoming requests)