[Azure Monitor] Unable to track incoming requests to FastAPI end point using azure-monitor-opentelemetry-exporter

[gautam-ergo]

• azure-monitor-opentelemetry-exporter
• Package Version - azure-monitor-opentelemetry-exporter = {version = "^1.0.0-beta.4", allow-prereleases = true}
• Operating System - macOS Mojave v10.14.6
• Python Version - v3.9.0

Describe the bug
We are using FastAPI in our production and successfully implemented log exporting using Azure log handers to Azure App Insights.
Currently we are not able to track incoming requests to the api endpoint made by the .NET client but those api POST calls are shown as outbound requests in the .NET application map.

But those outbound requests made by the .NET client are not being captured as incoming requests in the api telemetry
instrumented using ASGI and Azure exporter open telemetry

To Reproduce
Attached document contains more info
otel.docx

Expected behavior
Ability to track incoming requests

Screenshots
Attached document with screen grabs

Additional context

As part of the performance tracking only the healthcheck endpoint is being displayed

kindly advise

[mccoyp]

Hi @gautam-ergo, thank you for opening this issue! We'll investigate ASAP.

[lzchen]

@gautam-ergo
Are you using the fast-api instrumentation?

[gautam-ergo]

yes @lzchen

current stack:
opentelemetry-instrumentation-fastapi = "^0.19b0"
azure-monitor-opentelemetry-exporter = "^1.0.0-beta.4"
python = "^3.9"
fastapi = "^0.63.0"

[lzchen]

@gautam-ergo
Can you try to print out your spans generated to the console using the consoleexporter to see if you are collecting the telemetry properly? Please provide a screenshot of your output.

[gautam-ergo]

Thanks @lzchen
Made use of console exporter as you suggested.
Attached file contains all the events generated by the telemetry package for a single POST request.

As seen in the screenshot above, there are 4 events being generated, collated all the details in this file,
azure.txt

Please let me know if you need more info

[lzchen]

@gautam-ergo
Looks like you are collecting the spans properly, so it must be something related to the azure exporter itself. Just to verify, can you show a screenshot of the "requests" table in the "logs" blade after you make this call with the azure exporter? This is to make sure you are actually not receiving any incoming requests telemetry.

[gautam-ergo]

hey @lzchen, not sure if this is what you expected,

Following graph is for the request count from the request blade you requested, spread across 30min bins and the flat line corresponds to 360

Background: the flat line corresponds to request made to /health end point from within the pod in which the api is hosted, which is continuously polled at 5 seconds which translates to 12 calls per minute and 360 calls per 30 mins

Since you said the telemetry collector is doing its job, I created a sample python client and made calls to the endpoint and could see a spike in request count as shown below.

If you could help me understand, when I created a python client with the same namespace as that of the api, I was able to track the incoming requests but actual scenario is that a .NET client residing in a different app insight namespace is making the calls to this api which are not getting tracked.

From the above two graphs, it is clear that only the /health endpoint is being tracked whereas the calls made by the backend client to the api url is not showing up.

Kindly advise.

[lzchen]

I created a sample python client and made calls to the endpoint and could see a spike in request count as shown below.

Is this sample python client using FastAPI, instrumented with opentelemetry-fastapi-instrumentation and you are using the AzureSpanExporter? It would be useful if you can paste your code in which you do all of this. If yes to all three, then we can isolate the issue as an either configuration issue that relates to app insights itself, not the exporter or opentelemetry.

First of all, what do you mean by different app insight namespace ? Is it a separate app insights instance? To see a correlated trace tree, you must be sending telemetry to the same instance of appinsights (the same instrumentation_key must be used from all your sources of telemetry).

[gautam-ergo]

Thanks @lzchen ,

Yes, the .NET backend client that makes the calls to api endpoint is using a different instrumentation key.

API receiving these calls are hosted using a different instrumentation key.

Intention was to capture those outbound calls (from a different instrumentation key) as incoming requests

This is the trace information from the .NET client, which is using a different key; these are shown as outbound POST calls in its respective application map

This is the URI that is captured when made from the same app instance as that of api using the python client

Was under the assumption that since the endpoint ('/api/v1/metrics/track' ) being tracked is the same, telemetry would pick up inbound requests as well.

considering that .NET client is on a different instance, what do you recommend (different cloud roles? namespaces ?)

Kindly advise

[lzchen]

considering that .NET client is on a different instance, what do you recommend (different cloud roles? namespaces ?)

To my knowledge, telemetry captured must be under the same instrumentation key to be traced together in the same application map. There might be some other solutions but I don't have the expertise for this as it is not an SDK or exporter issue. Might I suggest filing a ticket about this through azure support?

[gautam-ergo]

Would that be a different repo, can you please point me towards the appropriate service repo.

Thanks for your time and inputs @lzchen

[lzchen]

@gautam-ergo
Sorry for the confusion, I've done some more digging and this functionality does not exist at all, so creating a new ticket wouldn't be much help for you. The only way you would be able to see a connected application map is if your telemetry is all sending to the same place (instrumentation key).

[lzchen]

Closing this issue for now. Please feel free to reopen if any additional details. @gautam-ergo