Sampling affects the accuracy of performance metrics

[avereshchak]

Component(s)

exporter/azuremonitor

Describe the issue you're reporting

I'd like to see correct numbers of operations/calls on Application Map and Performance/Failure blades even if most of telemetry was aggressively sampled out by collector.

E.g. an application is handing 1000 requests and makes 1000 SQL queries to accomplish that. OTEL Collector is using the following sampling configuration to retain only 1% of telemetry:

processors:
  batch:
  probabilistic_sampler:
    sampling_percentage: 1

As an outcome, the Application Map is showing only 10 operations which were sampled in even if there was much more (1000 in my example).

Both Application Insights SDK for .NET and Azure.Monitor.OpenTelemetry.Exporter are providing that capability. Could the exporter/azuremonitor benefit from #34272 to provide similar experience?

[github-actions]

Pinging code owners:

• exporter/azuremonitor: @pcwiese

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[frzifus]

I assume thats something azure monitoring specific? Not sure what metrics azure expects. But you may want to generate your metrics before applying some kind of sampling.

You may want to look into the spanmetrics or count connector.

[avereshchak]

I assume thats something azure monitoring specific? Not sure what metrics azure expects.

Yes, these are AzureMonitor-specific metrics.

But you may want to generate your metrics before applying some kind of sampling.
You may want to look into the spanmetrics or count connector.

I'm too lazy and would prefer having it out of the box. :) At least it is already available in Application Insights SDK and its successor Azure.Monitor.OpenTelemetry.Exporter.

For example, Application Insights SDK has an AutocollectedMetricsExtraction which produces metrics similar to the one below:

{
    "data": {
        "baseData": {
            "metrics": [
                {
                    "count": 1000,
                    "kind": "Aggregation",
                    "max": 30.798,
                    "min": 10.1023,
                    "name": "Server response time",
                    "stdDev": 5.14446362344043,
                    "value": 23805.666699999987
                }
            ],
            "properties": {
                "DeveloperMode": "true",
                "Request.Success": "True",
                "_MS.AggregationIntervalMs": "59000",
                "_MS.IsAutocollected": "True",
                "_MS.MetricId": "requests/duration",
                "cloud/roleInstance": "*******",
                "cloud/roleName": "Unknown",
                "operation/synthetic": "False",
                "request/performanceBucket": "<250ms",
                "request/resultCode": "Unknown"
            },
            "ver": 2
        },
        "baseType": "MetricData"
    },
    "name": "AppMetrics",
    "tags": {
        "ai.application.ver": "1.0.0.0",
        "ai.cloud.roleInstance": "*******",
        "ai.internal.nodeName": "*******",
        "ai.internal.sdkVersion": "m-agg2c:2.22.0-997"
    },
    "time": "2024-12-09T10:40:02.0000000Z"
}

I guess these metrics are then used for building and rendering the Application Map and other GUI. I also thinking it doesn't depend on #34272 as I initially thought.