ITelemetryProcessor does not appear to be supported in Function Apps

[jeffhollan]

From @StevenTaylor on October 5, 2018 5:50

Documentation (https://docs.microsoft.com/en-us/azure/application-insights/app-insights-api-filtering-sampling) suggests that ITelemetryProcessor can be used to filter\sample the telemetry before it is sent App Insights.

But the implementation code is never called in our Function App.

The code to set up the builder is called, which looks like:

var builder = TelemetryConfiguration.Active.TelemetryProcessorChainBuilder; builder.Use((next) => new AppInsightsFilter(next)); builder.Build();

But the 'Process' method implemented in the class is never called, the class looks like:

` class AppInsightsFilter : ITelemetryProcessor
{
private ITelemetryProcessor Next { get; set; }

    // Link processors to each other in a chain.
    public AppInsightsFilter(ITelemetryProcessor next)
    {
        this.Next = next;
    }

    public void Process(ITelemetry item)
    {
        // To filter out an item, just return
        if (!OKtoSend(item)) { return; }

        // Modify the item if required
        //ModifyItem(item);

        this.Next.Process(item);
    }

    private bool OKtoSend(ITelemetry item)
    {            
        // try something!
        return (DateTime.Now.Second <= 20); // only send for the first 20 seconds of a minute
    }
}`

PS. We have tried setting config values SamplingPercentage (App Insights config) and maxTelemetryItemsPerSecond (host.json) as low as possible to reduce the telemetry data, but there is still too much telemetry.

Copied from original issue: Azure/Azure-Functions#981

[jeffhollan]

From @roryprimrose on November 2, 2018 4:13

This is a big issue for me as well. The amount of trace records being written to AI is large and increases the cost. Being able to filter out the unnecessary entries would save a lot of bandwidth and cost.

[jeffhollan]

Curious on if work around dependency injection ( #3736 ) will help with this or anything else? @brettsam as well as he has tons of AI experience

[roryprimrose]

My scenario is that I want to remove trace messages written to AI, primarily from the storage extension while allowing custom metrics to be sent to AI. I've tried just about everything to get a resolution to this without success. I'm also using DI for my C# functions.

Here are some of the things I've found:

• I get a TelemetryClient from the host in an extension at the first available opportunity. I modify it as per the code from @StevenTaylor. I found that the client did appear to have the processors registered, but the code was never executed.

• I tried disabling AI logging Information entries in config

  "Logging": {
    "LogLevel": {
      "Default": "Information"
    },
    "ApplicationInsights": {
      "LogLevel": {
        "Default": "Warning"
      }
    }
  }

This wiped out all Information tracing to AI. I would prefer to be more targeted than that, but it solved the first problem. Unfortunately it also wiped out reporting of customMetrics to AI. I suspect this is because customMetric records are written with a trace level of Information. This was a little weird though because on reflection I realised I was sending custom metrics directly from the TelemetryClient rather than via any injected ILogger.

• I thought about the configuration above and tried using a new blank TelemetryClient for reporting custom metrics. I hoped this would not pick up the logging configuration filter. Weirdly, my new TelemetryClient also could not report custom metrics with the above configuration in play.

[brettsam]

This is a great scenario so it'd be interesting to walk through it here. and I can build these up on the wiki to help everyone else, too.

I'm guessing that the processor isn't working b/c the processor chain gets built here: https://github.com/Azure/azure-webjobs-sdk/blob/dev/src/Microsoft.Azure.WebJobs.Logging.ApplicationInsights/Extensions/ApplicationInsightsServiceCollectionExtensions.cs#L270. Once that happens, I believe it's immutable. If we wanted to allow custom processors, we'd need to expose some other builder mechanism that allowed you to plug in before we build. But we may be able to solve this with filters.

What are the specific messages that you're trying to silence? You can filter anything based on the category (or even the category prefix). So if you know what you want to let through and what you want to prevent, we should be able to craft the right filter. To see the category of a particular message, check the customDimensions.Category value in App Insights Analytics

[roryprimrose]

@brettsam Yes, the categories between traces and customMetrics are different. I'll try using a config filter against the traces category.

[roryprimrose]

@brettsam Nope, sorry, I was right the first time. My custom metrics, written directly via TelemetryClient, are being recorded with the same category used to write trace records from the queue trigger.

I have a queue trigger AnalyticsHintsProcessor which logs with the category Function.AnalyticsHintsProcessor. I also have a timer trigger AnalyticsScavenger which logs with the category Function.AnalyticsScavenger.

These are the customMetrics that are recorded for both scenarios.

These are the trace records written by the triggers.

So I can't use a logging config filter to wipe out the trace records for the storage triggers without also wiping out the custom metrics.

Why is TelemetryClient coupled to ILogger configuration filters?

[roryprimrose]

The OOTB metrics for the function are the ones that use a different category being Host.Aggregator.

[brettsam]

Can you share how you're logging your metrics? We've added a LogMetric() extension to ILogger that log metrics to App Insights with a Category of Function.FunctionName.User -- which is how we indicate logging coming from within your code. You could then filter based on that, which would remove all of those host logs you're seeing (the ones without .User)

[roryprimrose]

I'm not using the ILogger.LogMetric extension because I wanted to do pre-aggregation of my custom metrics. I'm using some code that was originally on MSDN somewhere (for the life of me I can't find it now). The logic calculates aggregated metric data for a period of time before sending the metrics to AI.

For example, I have some processes where the metric data can be aggregated to one or five minute blocks of data where hundreds of individual metrics could be tracked within those periods. This saves a lot of raw metrics being sent to AI.

As far as getting the TelemetryClient in this scenario, this is how I hook it up in a function. This is part of my custom dependency injection configuration.

[assembly: WebJobsStartup(typeof(DependencyInjectionStartup))]

public class DependencyInjectionStartup : IWebJobsStartup
{
    public void Configure(IWebJobsBuilder builder)
    {
        builder.AddExtension<InjectExtensionConfigProvider>();
    }
}

public class InjectExtensionConfigProvider : IExtensionConfigProvider
{
    private readonly InjectBindingProvider _bindingProvider;

    public InjectExtensionConfigProvider(TelemetryClient telemetryClient, ILoggerFactory loggerFactory)
    {
        Ensure.Any.IsNotNull(telemetryClient, nameof(telemetryClient));
        Ensure.Any.IsNotNull(loggerFactory, nameof(loggerFactory));

        _bindingProvider = new InjectBindingProvider(telemetryClient, loggerFactory);
    }

    public void Initialize(ExtensionConfigContext context)
    {
        context.AddBindingRule<InjectAttribute>().Bind(_bindingProvider);
    }
}

The InjectBindingProvider class holds an Autofac container which is configured with the TelemetryClient and ILoggerFactory values. My custom metric aggregation logic is created by the container using the TelemetryClient.

[michaeldaw]

I have a related problem so I thought I'd write about it here rather than make a new issue.
In my function app, there are many calls to dependent services. These calls are all tracked as Dependencies in Application Insights. The vast majority of data in my Application Insights instances are from successful dependency calls. The cost of tracking these calls has become substantial.

I've tried implementing the ITelemetryProcess interface to filter them out, and then later found here that this doesn't work.

Is there some other way to disable Dependency Tracking in Application Insights for Function Apps? Currently, my two options are 1) Pay a substantial amount of money to needlessly track dependency successes, or 2) get rid of Application Insights.

[KonnectSalon]

I am also having this same problem. I have been trying to filter our Dependency Tracking as it's costing a large sum of money for data I don't really need.

[jamesdheadup]

@michaeldaw hit the nail on the head. We're having the exact same issue and would love to be able to reduce our AI bill by filtering out records. Particularly filtering out successful dependency calls would be a big positive.

For now we're going to have to replace our logging implementation to write useful data to another logging service until this gets some traction.

[jeffhollan]

@brettsam do you have a good feel for what the right feature / fix is that could resolve this? Seems to be a fairly common type ask and not sure I'm clear on exactly if just a doc gap, feature gap, or something that could be solved with other in flight work.

[brettsam]

I apologize for the delay -- I lost track of this issue. Looping in @lmolkova for her thoughts as well.

For the core issue filed -- registering a custom ITelemetryProcessor should be do-able and fairly easy. You could register one or more with DI and we could stick it at the end of the processor chain, after all of our built-in processors run. That shouldn't be too difficult and as long as we clearly document where it runs in the chain, I think it makes sense. It would allow a lot of what I've listed out below to be done in a custom way by anyone.

There's a couple of other issues brought up that are good as well:

• @roryprimrose: looking back up at this comment, I think I see the issue. You're using the same TelemetryClient that we create via DI, which runs a custom ITelemetryInitializer. This will grab the current logging scope and apply it to the telemetry being logged. Because you're doing this in a background task, that likely explains why you're getting an incorrect category. If you, instead, created your own TelemetryClient there (using the same instrumentation key -- which can be pulled from an injected IConfiguration) -- you should get category-less metrics that you can log however you want.
• Although not directly brought up, it may make sense to log metrics with a different category -- like Function.FunctionName.Metric to allow for more granular filtering. They currently get lumped in as Function.FunctionName.User, which is workable but may not give enough switches for everyone.
• @michaeldaw / @KonnectSalon / @jamesdheadup: Can you share what the customDimentions.Category is for your dependencies that you're seeing? I'm going through how we "tag" DependencyTelemetry right now and seeing a few things we can improve (I'll move these to a separate issue but wanted to enumerate them here):
  • If the dependency fails, we don't flip the LogLevel to Error, which we should. That would allow you to filter and only show failed dependencies.
  • We probably should add a blanket on/off for dependency auto-tracking.
  • There are scenarios where the category will be Host.Bindings and scenarios where the category will be Function.FunctionName. We should try to reconcile this for better filtering.
  • We should think about whether we want to have some threshold for slow dependencies as well -- i.e. only log (as Warning?) ones that take longer than 1 second (configurable).

[michaeldaw]

@brettsam For me, the category seems to always be Function.FunctionName. I don't see any categorized as Host.Bindings.

[lmolkova]

Here is a code sample that demonstrates how to add a processor to the chain. We should probably simplify this and enable the same approach as we have for TelemetryIntializers

using System.Linq;
using FunctionApp12;
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Hosting;
using Microsoft.Extensions.DependencyInjection;

[assembly: WebJobsStartup(typeof(Startup))]

namespace FunctionApp12
{
    public class Startup : IWebJobsStartup
    {
        public void Configure(IWebJobsBuilder builder)
        {
            var configDescriptor = builder.Services.SingleOrDefault(tc => tc.ServiceType == typeof(TelemetryConfiguration));
            if (configDescriptor?.ImplementationFactory == null)
                return;
            
            var implFactory = configDescriptor.ImplementationFactory;

            builder.Services.Remove(configDescriptor);
            builder.Services.AddSingleton(provider =>
            {
                if (!(implFactory.Invoke(provider) is TelemetryConfiguration config))
                    return null;

                config.TelemetryProcessorChainBuilder.Use(next => new AiErrorFilteringProcessor(next));
                config.TelemetryProcessorChainBuilder.Build();

                return config;
            });
        }
    }

    internal class AiErrorFilteringProcessor : ITelemetryProcessor
    {
        private readonly ITelemetryProcessor _next;
        public AiErrorFilteringProcessor(ITelemetryProcessor next)
        {
            _next = next;
        }

        public void Process(ITelemetry item)
        {
            if (!(item is TraceTelemetry trace && trace.Message == "AI (Internal): Current Activity is null for event = 'System.Net.Http.HttpRequestOut.Stop'"))
            {
                _next.Process(item);
            }
        }
    }
}

• If the dependency fails, we don't flip the LogLevel to Error, which we should. That would allow you to filter and only show failed dependencies.

Agree on this, but the failure criteria problem would immediately arise after that (are 404, 401, 409, 429 failures).

Normally we approach this problem by sampling: you either want everything for the particular transaction to be sampled in or nothing at all.
If you don't track a dependency call, other features may be broken like end-to-end transaction viewer or app map. Even though dependency call is successful, you might still be interested in what happened for the whole transaction.

So I'd suggest to carefully weight everything before deciding to disable successful dependency tracking.

[michaeldaw]

Please factor the following while weighing the pros and cons of tracking successful dependencies:

These are screenshots of the Application Insights instance and cost analysis for a particular Function App and related storage account in our system.

Above is a capture of the default 24-hour period search for the service in question. You can see that dependency tracking accounts for 1.4 million items, while Trace, Request, and Exception account for 40K, 18K, and 1.9K (really must look in to those exceptions), respectively. Dependency events account for approximately 96% of all events.

This is the cost projection of the Application Insights instance. As before, the image shows that "REMOTEDEPENDENCY" events make up for the vast majority of recorded events.

Finally, the above screenshot is a filtered selection from the "Cost by Resource" view, showing the cost of the Function App, Storage Accounts, and Application Insights instances in question. The cost of the Application Insights instance is 1252% that of the cost of the Function App it is monitoring.

These costs are woefully unsustainable for us. Of late, my decision to use Function Apps, which I'd touted to my colleagues as extremely cost effective, is being called in to question by my teammates and superiors. Application Insights has been an invaluable tool for diagnostics and troubleshooting. I'd liken using a Function App without an associated Application Insights instance to flying by the seat of my pants. That said, I will eventually have to choose to either stop using Application Insights, or stop using Function Apps. I'm sure I'm not the only one who would really appreciate if the Functions and Application Insights teams could find a solution by which that choice doesn't have to me made.

[brettsam]

What @lmolkova has above will work -- it lets you grab the TelemetryConfiguration that we generate and append an ITelemetryProcessor to it. From there, you can filter out any DependencyTelemetry however you want. Thanks for putting that together @lmolkova! That at least gives folks an escape hatch while we work through the design here. We'll keep this issue open and work to come up with better configuration and extensibility for this.

And thanks a ton @michaeldaw (and others) -- having all this data makes it really easy to see how important this is.

A couple of other notes to anyone trying the DI approach above (neither of these will be required long-term, but are right now):

• Make sure to reference Microsoft.ApplicationInsights 2.7.2 in your project. Anything newer will end up with type mismatches.
• Add a direct nuget reference to version 1.0.2 of https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator. It fixes an issue when targeting .NET Core 2.1+ and using WebJobsStartup.

[michaeldaw]

@brettsam @lmolkova Thanks for looking in to this, everyone. I've implemented the code @lmolkova posted above.
Strangely, the code seems to work as expected when I debug the function app locally, but the filtering does not occur when I publish the app to Azure. I was hoping someone might have some suggestions as to why.
I know there are Application Insights configuration options available in the host.json file, but I don't have anything there except the "version": "2.0" statement. I also do not have an ApplicationInsights.config file.
Is there something else I might be missing?

Here's my implementation of ITelemetryProcessor specifically for filtering out DependencyTelemetry items:

    internal class AiDependencyFilter : ITelemetryProcessor
    {
        private readonly ITelemetryProcessor _next;
        public AiDependencyFilter(ITelemetryProcessor next)
        {
            _next = next;
        }

        public void Process(ITelemetry item)
        {
            var request = item as DependencyTelemetry;

            if (request?.Name != null)
            {
                return;
            }

            _next.Process(item);
        }
    }

[brettsam]

What version of the host are you running locally? Can you check the bin\extensions.json file both locally and in Azure?

Azure Functions Core Tools (2.4.379 Commit hash: ab2c4db3b43f9662b82494800dd770698788bf2d)
Function Runtime Version: 2.0.12285.0

[michaeldaw]

I wasn't aware of the extensions.json file, but it looks like that was the problem.
Here's the local extensions.json file. The "Startup" class mentioned is the one I added that replicates the functionality @lmolkova mentioned.

{
  "extensions":[
    { "name": "DurableTask", "typeName":"Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskWebJobsStartup, Microsoft.Azure.WebJobs.Extensions.DurableTask, Version=1.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289"},
    { "name": "AzureStorage", "typeName":"Microsoft.Azure.WebJobs.Extensions.Storage.AzureStorageWebJobsStartup, Microsoft.Azure.WebJobs.Extensions.Storage, Version=3.0.2.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"},
    { "name": "Startup", "typeName":"Scadavantage.Startup, Scadavantage.Data, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"}
  ]
}

The file in Azure did not have that last line mentioning the Startup class. I deploy to this Function App from Azure DevOps. Something there must ignore the file when deploying. I can look further in to that. I added the last line manually using Kudu.

With that last line in place, it seems to work! I'm not seeing Dependency calls in the either the Live Metrics Stream, or when I do a search for events. What a relief!

I wanted to mention that with this filter in place, the Live Metrics view seems to be adversely affected. I no longer see vales in the Request Rate or Request Duration graphs in the Live Metrics view. Could this be related to the concerns raised by @lmolkova about related events being affected by the absence of dependency events? Requests and traces still appear fine in the Search section.

In any event, I'm very much willing to live with a diminished experience with the Live Metrics Stream if it means so drastically reducing our costs. Thank you very much to everyone who's been looking in to this! It's a huge help.

Not sure if it still matters, but here's my local host version:

Azure Functions Core Tools (2.3.199 Commit hash: fdf734b09806be822e7d946fe17928b419d8a289)
Function Runtime Version: 2.0.12246.0

[brettsam]

Can you share the package references in your .csproj? And what is your TargetFramework? I'm seeing several reports of build servers incorrectly generating the extensions.csproj so I'm trying to narrow it down.

[michaeldaw]

Absolutely. The target framework is .NET Core 2.1.
Here are the package references from the Function App:

  <ItemGroup>
    <PackageReference Include="Microsoft.ApplicationInsights" Version="2.7.2" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="1.7.1" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="3.0.2" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator" Version="1.0.2" />
    <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="2.1.1" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="1.0.24" />
    <PackageReference Include="Scadavantage.Common.Core" Version="1.0.0-prerelease067" />
    <PackageReference Include="Twilio" Version="5.26.1" />
  </ItemGroup>

It mentions a "scadavantage.common.core". Here are the package references from that package:

  <ItemGroup>
    <PackageReference Include="Microsoft.ApplicationInsights" Version="2.7.2" />
    <PackageReference Include="Microsoft.AspNetCore" Version="2.1.7" />
    <PackageReference Include="Microsoft.Azure.DocumentDB.Core" Version="2.2.2" />
    <PackageReference Include="System.Security.Cryptography.Algorithms" Version="4.3.1" />
    <PackageReference Include="WindowsAzure.Storage" Version="9.3.3" />
  </ItemGroup>

Let me know if I can help further.

[brettsam]

Thanks that helped confirm the problem. I just opened this issue (feel free to comment over there so we don't derail this issue :-)) Azure/azure-functions-vs-build-sdk#277

[lmolkova]

@michaeldaw

thanks for the great write up! Now I understand your motivation for disabling dependencies better.
This is something we should eventually handle in the ApplicationInsights SDK (for Functions and any other applications) and we need to find the right approach for it. I'll start this discussion internally.

Not tracking successful calls to the majority of bindings (e.g. tables) is reasonable and would not create any obvious issues with UX as long as these dependencies are leaf nodes.

Not tracking http calls to your own services or some output bindings (e.g queues) would break end-to-end tracing. Think about transaction traces as a tree. If one node is missing, reference is lost and instead of one tree, we now have two. We still know they are correlated, but causation is lost. In some cases, it is still good enough and for sure much better than the costs associated with redundant data.

As @brettsam mentioned (Azure/azure-webjobs-sdk#2123), we'll provide better configuration to turn off the collection.

I'll also check why sampling was not useful for original issue author, maybe something is broken here:

PS. We have tried setting config values SamplingPercentage (App Insights config) and maxTelemetryItemsPerSecond (host.json) as low as possible to reduce the telemetry data, but there is still too much telemetry.

[michaeldaw]

@lmolkova: thank you for explaining this. In our case, the functions in question are HTTP-triggered. The dependencies in questions are, for the most part, calls to tables using the .net storage API. It sounds like these kinds of calls would fall under the second part of your explanation (http calls, etc.).

I've modified the original AiDependencyFilter class that I posted earlier to allow failed dependency calls through the filter. I've only tested it a little bit, but it seems to restore the end-to-end tracing at least for those function calls. This is entirely sufficient for my purposes. It's really only when the calls fail that I'm interested in the tracing.

Thank you for your help.

[vitalybibikov]

The same here, TelemetryConfiguration is always null for 3.0.7

[brettsam]

@vitalybibikov, some initial questions:

1. Are you seeing this locally or in production?
2. Do you have APPINSIGHTS_INSTRUMENTATIONKEY set? Even locally -- setting it to some dummy string (like "abc") will cause the TelemetryConfiguration to be registered.

If both of those are yes, can you share a sample application that reproduces this? Even a csproj file may be enough.

[NateB2]

Adding the "APPINSIGHTS_INSTRUMENTATIONKEY" fixed it for me. I only previously tested it locally, didn't test it in Azure.

[vitalybibikov]

I've checked it out, when APPINSIGHTS_INSTRUMENTATIONKEY is set,
instance is no longer null.

Is it reflected in docs? if it's not, maybe it should be reflected, as it's not quite obvious.

Thanks.

[kamilzzz]

What's the current state of this issue?

I've tried code posted by @michaeldaw and my custom TelemetryProcessor was indeed called but then Live Metrics in the portal are broken. Now they are showing only CPU/Memory usage. Cannot see traces/dependencies anymore (they are sent for sure because after a few minutes I can see them in the Performance -> Dependencies UI).

[luthus]

Has there been any progress on fixing this issue? I'm still seeing the issue with Azure Functions V3.

[luthus]

This change appears to resolve the issue:
luthus/azure-webjobs-sdk@3137c5c

With this change we should be able to use builder.Services.AddApplicationInsightsTelemetryProcessor() in the same way that we do for ASP.NET Core.

[jschieck]

@luthus's solution above is the correct one, but if you don't want to have to fork the web jobs sdk, you can get it working correctly in durable functions (or azure functions in general) WITHOUT breaking live metrics dependency and error logging like @kamilzzz saw and I did as well with @lmolkova's solution.

Adding the ITelemetryModule and inserting your ITelemetryProcessors into the builder there works as expected. This gets around builder.Services.AddApplicationInsightsTelemetryProcessor() not working as expected.

// startup.cs
builder.Services.AddSingleton<ITelemetryModule, MyCustomTelemetryModule>();
builder.Services.AddApplicationInsightsTelemetry(Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY"));

// custom module
public class MyCustomTelemetryModule : ITelemetryModule
{
	public void Initialize(TelemetryConfiguration configuration)
	{
		// add custom processors
		configuration.TelemetryProcessorChainBuilder.Use(next => new MyCustomTelemetryProcessor(next));
		configuration.TelemetryProcessorChainBuilder.Build();
	}
}

// custom processor
public class MyCustomTelemetryProcessor : ITelemetryProcessor
{
	private readonly ITelemetryProcessor _next;

	public MyCustomTelemetryProcessor(ITelemetryProcessor next)
	{
		_next = next;
	}

	public void Process(ITelemetry item)
	{
                bool myCustomSkipTelemetry = false;
		if (myCustomSkipTelemetry)
			return;
		_next.Process(item);
	}
}

[luthus]

@jschieck that works as a workaround for now.

[DarinMacRae]

@jschieck This is good!

I just want to clarify something though.

The docs specifically say not to add AddApplicationInsightsTelemetry...

here: https://docs.microsoft.com/en-us/azure/azure-functions/functions-dotnet-dependency-injection#logging-services

...but in your testing you found that it is ok to do so when customizing the processors?

[jschieck]

@jschieck This is good!
I just want to clarify something though.
The docs specifically say not to add AddApplicationInsightsTelemetry...
here: https://docs.microsoft.com/en-us/azure/azure-functions/functions-dotnet-dependency-injection#logging-services
...but in your testing you found that it is ok to do so when customizing the processors?

The app we're doing this in runs in a kubernetes cluster not default azure functions so I'm not sure if different rules apply. But we are still getting all of the built in ILogger/app insights functionality from the functions host runtime without any side effects (that we've noticed)

Also I haven't tested in a normal function app if builder.Services.AddSingleton<ITelemetryModule, MyCustomTelemetryModule>(); is all you would need to get it working

[gabrielweyer]

@jschieck I tested your code sample in a "normal" Azure Function. I can confirm that the telemetry initializers and telemetry processors are called as expected (whereas telemetry processors are not being called for request telemetry items using the previous solution posted in this thread).

The drawback of your approach is that telemetry processors are being called for dependencies that are internal to the Azure Functions SDK. These dependencies are being discarded beforehand using the approach I linked to above. Example of such internal dependencies are retrieving new messages for the Service Bus trigger or the blob used for locking. In my sample Function App with a single Service Bus trigger, 135 "internal" dependencies were recorded in 60 seconds. These dependencies do end up being discarded before being ingested by Application Insights but that is still a lot of telemetry going through the processors.

[ssa3512]

Would it be possible to get the PR for this (Azure/azure-webjobs-sdk#2657) reviewed and merged so users can start utilizing custom telemetry processors? I am in the process of trying to stand up a new functions app that is registering very large numbers of dependency telemetry so utilizing a custom filter to exclude this is a must have.

[piotrzbyszynski]

So 2 and a half years have passed since original issue was created. As I understand it is still not resolved, just workarounds?

[piotrzbyszynski]

So I have to filter out an exception, a false positive (for details see Azure/azure-webjobs-sdk#2612, the issue is completely ignored for over half a year now), filtering this exception is a workaround all together, since this exception should not be logged at all... But anyways...

I can not use category filtering because exception is thrown by Entity Framework code, the category is Microsoft.EntityFrameworkCore.Update and I want to log this exceptions except exceptions with specific message.

I developed a telemetry processor and added it using custom ITelemetryModule approach, and it seems to work locally. Unfortunately when deployed to Azure exceptions are not being filtered. I use Microsoft.ApplicationInsights.AspNetCore version 2.15 and I can not downgrade it to 2.6.1 as suggested here because I also use EnableSqlCommandTextInstrumentation configuration

@brettsam @lmolkova the issue is not fixed for over 2 years now. This issue makes us loose money on AI fees and even more money on developing and researching workarounds. Could we please have this issue fixed once and for all? Pretty please?

[luthus]

@brettsam @lmolkova can the team take a look at this PR? Azure/azure-webjobs-sdk#2657

[piotrzbyszynski]

One more thing regarding category filtering. I have a function called let's say MyFunction, triggered by service bus message. The default category for this function is Function.MyFunction. To this category belongs tons of noise I do not care about like:

Executing 'MyFunction' (Reason='(null)', Id=ee1c2eb3-73ec-417b-a552-70cbab784d17)

or

Trigger Details: MessageId: XYZ, SequenceNumber: 123, DeliveryCount: 6, EnqueuedTimeUtc: 2021-06-01T11:27:42.2850000Z, LockedUntilUtc: 9999-12-31T22:59:59.9999999Z, SessionId: ASD

But also, inside MyFunction I create custom dependency like so:

using (var operation = _telemetryClient.StartOperation(new DependencyTelemetry()
            {
                Name = "My function",
                Type = "MyFunctionOperation",
            }))

My question is: how can I assign custom category to this dependency, because it has Function.MyFunction category assigned by default, and I can not see a way to change that. That means I can not filter out Function.MyFunction category using host.json. Also all DB and HTTP calls have Function.MyFunction category assigned (and I want it to be logged). Any workaround or preferably legitimate solution other then telemetry processors that do not work @brettsam @lmolkova?

[piotrzbyszynski]

Another month has passed without even single response from MS... 🤦‍♂️

Any of you guys can recommend some good Application Insights replacement?

[gabrielweyer]

I came up with a solution that calls telemetry processors on request telemetry items while re-using the existing processor chain (so that the Functions internal dependencies are discarded before hitting the user's telemetry processors). I was hoping that Azure Functions v4 would support telemetry processors but it does not (at least for the in-process hosting model). v4 emits a bit more telemetry and Azure.Core emits a lot more telemetry.

I understand that using telemetry processors can skew the statistics and that sampling is the preferred approach but sometimes I need to discard a specific telemetry item and I'm happy for sampling to apply to everything else. Without any configuration some Functions will incur an Application Insights bill 10 times bigger than the compute bill!

Adding telemetry processors support to Azure Functions would make it a more compelling offering. Currently we have to take into consideration the additional Application Insights spending.

The solution I came up with has been inspired by the other workarounds posted on this thread. I created a library out of it which hopefully will be easy enough to integrate into any Functions App. The code is a bit too long to post here, the interesting part is the extension method configuring Application Insights.

Using this customisation:

• You can add any numbers of telemetry processors in the same way you would add them in ASP.NET Core
• When APPINSIGHTS_INSTRUMENTATIONKEY / APPLICATIONINSIGHTS_CONNECTION_STRING is not present, I register a no-op TelemetryConfiguration so that it can be resolved by the container (no more exception when Application Insights is not configured)
• Most of the "cruft" telemetry emitted by the runtime is discarded

The project comes with a couple of sample Functions so that you can see how it behaves.

Hopefully this helps someone else, it has helped us reduce our Application Insights spending while retaining the ability to troubleshoot issues.

[mrebuffet]

We are also impacted by this issue and have tried several options to fix this behaviour.
Our Functions are sitting behind FrontDoor and we are using the built-in Health check option to probe our services.
Since Azure has a significant amount of Point of Presence around the world (which is great), this results in almost 200k logs per day per backend pool on the lowest setting (probing at a 255 second interval).

Using the built-in filtering option, we were only able to hide the traces (Executing 'HealthProbe' / Executed 'Health Probe') but couldn't find any out of the box way to filter out the ~70k requests generated in the process.

@gabrielweyer solution is indeed allowing us to filter out all the unwanted requests but forces us to import a new package and fix our custom implementation or bring in a lot of code that may break with future releases.

Before we spend more time looking into this and implementing an alternative, is there any update on this issue or future plans to allow developers to filter out requests by name?
Is this active PR still being looked at? Azure/azure-webjobs-sdk#2657

[gha-zund]

We struggle with very high cost for application insights and the solutions and workaround proposed for this issue would help us to reduce the cost and still be able to use application insights. In my opinion, filtering should be possible with configuration, without need to add code which has a high chance to break with future SDK or Function runtime releases...

The fact that this issue is open for almost four and a half year now will have some weight in a decision for another monitoring solution...

Additionally, I would expect to learn all this things I can read in this thread in the documentation without having to read through a GitHub-issue!

[drilko]

What is the status on this? It will soon be 5 years since this was opened. Please provide some feedback.

[JackD111]

I've encountered the same issue when first using isolated Azure Functions. The host manages the RequestTelemetry sent to Application Insights, and there's no way to control those requests from the worker (without completely disabling request telemetry data). Therefore, I developed a custom worker extension to address this.

The repository is available here and is also published as a NuGet package that can be referenced in your isolated azure function project.

This worker extension introduces custom HTTP trigger function bindings that incorporate a custom ITelemetryProcessor. This processor can:

1. Change the RequestTelemetry.Success property to false if the trigger response yields an unsuccessful status code.
2. Discard successful RequestTelemetry events

Discarding successful request events has been a primary objective, significantly reducing our Azure analytics ingestion costs. These costs were primarily driven by front door health probes, which generated gigabytes of data daily.

This should assist others looking to optimize their Azure analytics expenses.

[pregress]

@JackD111 thanks for saving the day your nuget package is exactly what we needed

[federiconintex]

@JackD111 I am facing the same issue (RequestTelemetries being handled by host only) and had a look at your project. I tried to replicate it to adapt it to my requirements but I had issues with the extension startup not being invoked.

1. Must the extension provide custom bindings to be invoked? Is it possible in your knowledge to have the extension code take over the TelemetryConfiguration without having to define those?
2. Do you have some source material to learn more about taking over host functionalities? I am struggling to find any good docs