Configuration to disable specific dependencies

[martupeppe]

Hi,
I just configured my Java application on-prem installing the Java Codeless Agent applicationinsights-agent-3.0.0-PREVIEW.5.jar.

I figured out my application is logging milions of dependencies and I would like to disable them, but I think there is no way to disable only dependencies because as far as I understood the dependencies are auto-tracked.

I would like to have a dedicated configuration to avoid to send dependencies to Application Insights.

Thanks,
Giuseppe

[MS-jgol]

@martupeppe
Here are the configuration options we have available for Java agent:
https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-config

What dependencies are excessive? You may greatly benefit from enabling sampling.

[martupeppe]

Hi,
Thanks for the Answer I knew that link and it seems there is no way to limit the jdbc dependencies which are causing cost issues.

Thanks

[MS-jgol]

@martupeppe Do you think sampling may work for you?

[martupeppe]

No I don't think so. The customer would like to have all the requests.

Another solution would be avoid to send requests from a specific operationname so that the related dependencies would be filtered out.

Any suggestion?

Giuseppe

[martupeppe]

@MS-jgol any suggestion?

[martupeppe]

@MS-jgol no ideas? Should I switch to java sdk to disable dependencies?

[martupeppe]

@MS-jgol looking at the code I think it should be fine amending te BeforeAgentInstaller.java to read the configuration e set a Global Property. In order to avoid to send dependency telemetry, the ByteCodeUtilImpl.java file should be changed and specifically the method trackDependency should evaluate the Global property previously set and log only if the configuration has been enabled. The same mechanism is used for Sampling just to mange the sampling percentage. Do you think it is feasible?

[MS-jgol]

@martupeppe
We will add disabling certain dependencies in our GA version, which will be in the first week of November. Hopefully this timeline will work for you.

[martupeppe]

Hi,
This is a great news.
How it will be possible to disable the dependencies? And what type of dependencies will be disabled?

Thanks

[MS-jgol]

@martupeppe JDBC dependencies definitely. I do not have an insights in how this will be disabled, most like through configuration, and the idea is to be able to also be disable dependencies that may raise cost concerns.

[martupeppe]

Thanks!!!!

[trask]

hey @martupeppe! can you expand on this part?

Another solution would be avoid to send requests from a specific operationname so that the related dependencies would be filtered out.

[martupeppe]

Sure... it would be nice if there was configuration where it's possibile to insert a specific url (or regex) to be filrered out and not sent to application insights.

Is it clear enough?

Thanks,
Giuseppe

[trask]

Are these health-check urls? Or just really noisy requests?
Would this solution be better in your scenario compared to disabling jdbc dependencies across all requests?

Thanks!

[martupeppe]

Disabling JDBC dependencies is what I looked for so it's fine having this option in the next release (do you have an official date?).

The request url filter was an alternative for noisy requests that anyway could be useful anyway (next release?).

Giuseppe

[trask]

Yeah, the use case for url filter, particularly for health checks, is something I've heard from a few folks. But I'm still not completely understanding the use case for disabling jdbc dependencies, which is why I'm curious if the url filter is more applicable for your case, or maybe you can provide more details about your use case?

[martupeppe]

Disabling the jdbc dependencies comes from the scenario where we enabled the applocation insights codeless agent for a java application exposed over internet. The first day we ingested 80gb with of course high costs. Analizyng the ingested data we have seen that more than the 80% of the data was dependencies. Given we can t disable the dependencies, we currently configured a daily cap and a 25% sampling.

Is the use case clear?

Giuseppe

[trask]

Is the jdbc dependency data not useful? Sampling generally works well to control cost, and with sampling you can still see when there are issues with your application <-> database interactions.

[martupeppe]

The jdbc data are useful but the business prefers having the full incoming requested hence with no sampling.

Can you confirm the jdbc dependencies can be disabled in the next agent release in november?

Giuseppe

[trask]

Got it, thanks for the details.

Yes, jdbc dependencies will be able to be disabled 👍

[martupeppe]

Perfect.

Last question.... do you have a release date?

Thanks,
Giuseppe

[MS-jgol]

@martupeppe The next release is GA - first week of November.
Could you please tell us the name of your company? If you wish, send me an email to jgol@microsoft.com
I would love to also discuss a couple of other potential usecases uf possible.

[martupeppe]

@MS-jgol could you please use my private email for now (giuseppe.marturano@gmail.com)? I will be changing company in the next month and I prefer to provide you with the new company email. The company where I will be working starting from the 23rd of November is Avanade (you should know it ;) ).

Thanks!

[martupeppe]

@trask I just saw the 3.0.0 release has been released but I expected the feature to enable/disable the jdbc dependencies via configuration.

Am I wrong?

Giuseppe

[trask]

Ya sorry just not documented yet, I believe this should work:

{
  "instrumentation": {
    "jdbc": {
      "enabled": false
    }
  }
}

[pistolero38]

Hi !

we have the same problem as @martupeppe : we want to disable the jdbc and redis logs because our dependencies are too big and the customer obviously wants to lower the costs.

I'm using version 3.0.0 of the agent, I put the same configuration as before:

{
    "sampling": {
        "percentage": 100
    },
    "instrumentation": {
        "logging": {
            "level": "INFO"
        },
        "jdbc": {
                "enabled": false
        }
    },
    "heartbeat": {
        "intervalSeconds: 60
    }
}

But I still see the SQL dependency logs in the AppInsights. What's wrong? If I want to do the same thing to disable the redis logs, what configuration should I set ?

Thanks in advance!

[MS-jgol]

@pistolero38
Do you want to disable all redis logs?

[pistolero38]

Yes both redis and SQL dependencies logs

[gunsliver]

Hi can you also provide the way to disable Kafka?

I think it would be great if there is a way to pass property or environment variable to opentelemerty such as OTEL_TRACE_CLASSES_EXCLUDE or otel.integration.[id].enabled * ota in this case

[trask]

hi @gunsliver, do you want to completely disable kafka telemetry, and distributed trace propagation across kafka?

or is there a specific piece that is generating useless noise? thx

[gunsliver]

hi @gunsliver, do you want to completely disable kafka telemetry, and distributed trace propagation across kafka?

or is there a specific piece that is generating useless noise? thx

Hey @trask, thanks for super quick response. I am looking forward to completely disable kafka telemetry.

[LumirH]

I have the same problem as @pistolero38 -- even though I have disabled JDBC instrumentation using recommended config, I still see a lot of SQL dependency traces such as...

2020-12-22 10:10:47.403Z DEBUG c.m.a.agent.Exporter - exporting span: SpanWrapper{delegate=RecordEventsReadableSpan{traceId=XXXXX, spanId=YYYY, parentSpanId=ZZZZ, name=.....

[LumirH]

@trask -- I tested 3.0.1-beta2 -- the fix works well.
Thank you.