[BUG] Azure Function with UserAssigned ManagedIdentity has a 16% chance to result in Azure.Identity.CredentialUnavailableException

[nols-neulsen]

Library name and version

Azure.Identity 1.12.0

Describe the bug

I have a Windows hosted Function App (Consumption plan) with a single HTTP trigger function.
This function will initialize an ArmClient, using ManagedIdentityCredential, to spawn Container App Jobs.
From a test (902 invocations) this function only succeeds 84% of the time, the other 16% fails due to Azure.Identity.CredentialUnavailableException.
Running locally, everything works 100% of the time if I provide a AzureCliCredential, VisualStudioCredential (with Sync is active) seems to also not work all the time.

Function App:

• net8.0
• dotnet-isolated
• AZURE_CLIENT_ID = obfuscated
• ...

Packages:
<PackageReference Include="Azure.Identity" Version="1.12.0" />
<PackageReference Include="Azure.ResourceManager.AppContainers" Version="1.1.1" />
<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.22.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.2.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.3.2" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.2" />
<PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.2.0" />

User Assigned Managed Identity role assignments:

"assignableScopes": [
    "/subscriptions/<obfuscated>",
    "/subscriptions/<obfuscated>/resourceGroups/<obfuscated>"
],
"permissions": [
    {
        "actions": [
            "Microsoft.Resources/subscriptions/read",
            "Microsoft.Resources/subscriptions/resourceGroups/read",
            "microsoft.app/jobs/read",
            "microsoft.app/jobs/stop/action",
            "microsoft.app/jobs/start/action"
        ],
        "notActions": [],
        "dataActions": [],
        "notDataActions": []
    }
]

Code:

    try
    {
        var userManagedIdentityId = Environment.GetEnvironmentVariable("AZURE_CLIENT_ID"); ArgumentException.ThrowIfNullOrEmpty(userManagedIdentityId);
        var resourceIdString = Environment.GetEnvironmentVariable(...); ArgumentException.ThrowIfNullOrEmpty(resourceIdString);
        var environment = Environment.GetEnvironmentVariable("Environment");
        
        ...
        var resourceId = new ResourceIdentifier(resourceIdString);
        var subscriptionId = resourceId.SubscriptionId;

        ArmClient armClient;
        switch (environment)
        {
            case "NPRD":
                ...
                armClient = new ArmClient(new ManagedIdentityCredential(userManagedIdentityId), subscriptionId);
                break;
            case "CN":
                ...
                armClient = new ArmClient(
                    new ManagedIdentityCredential(userManagedIdentityId, new TokenCredentialOptions { AuthorityHost = AzureAuthorityHosts.AzureChina }), 
                    subscriptionId, 
                    new ArmClientOptions { Environment = ArmEnvironment.AzureChina });
                break;
            default:
                ...
                armClient = new ArmClient(new DefaultAzureCredential(), subscriptionId); // Pick any available credential, info https://learn.microsoft.com/en-us/dotnet/api/azure.identity.defaultazurecredential?view=azure-dotnet
                break;
        }

        var caj = armClient.GetContainerAppJobResource(resourceId);
        ...
        var template = ...;
        await caj.StartAsync(Azure.WaitUntil.Started, template);

        return ...;
    }
    catch (Exception ex)
    {
        _logger.LogInformation("{Message}", ex.Message);
        _logger.LogInformation("{StackTrace}", ex.StackTrace);
        throw;
    }
}

Stacktrace:

Azure.Identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable. Multiple attempts failed to obtain a token from the managed identity endpoint.
 ---> System.AggregateException: Retry failed after 6 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80))
 ---> Azure.RequestFailedException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)
 ---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)
 ---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async)
   --- End of inner exception stack trace ---
   at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async)
   at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   --- End of inner exception stack trace ---
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
   at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
 ---> (Inner Exception #1) Azure.RequestFailedException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)
 ---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)
 ---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async)
   --- End of inner exception stack trace ---
   at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async)
   at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)<---

 ---> (Inner Exception #2) Azure.RequestFailedException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)
 ---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by i

Expected behavior

Retrieving the credential succeeds 100%

Actual behavior

In 16% of the cases the execution fails due to Azure.Identity.CredentialUnavailableException

Reproduction Steps

Hosting info and code provided in bug description

Environment

Not relevant for this bug: for the buggy VisualStudioCredential, my VS version: VS 22 17.10

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jsquire]

@nols-neulsen: Thanks for reaching out and we regret that you're experiencing difficulties. The error indicates that the local managed identity endpoint on the host is unavailable or inaccessible to HTTP traffic when the application starts running and the Identity library attempts to acquire a token. This is not something that the Azure.Identity library or the application has insight into nor influence over.

This requires investigation of the Azure Functions host environment. I've transferred this to the Azure Functions host repository as #10238 to ensure that the Functions team has visibility and can provide assistance.