"409 The specified container already exists." #2166

tymtam2 · 2022-02-21T07:59:32Z

In my Azure Function which doesn't have any storage account code I'm getting the following warning in Application Insights.

Error response [15fd74...ec3601] 409 The specified container already exists. (00.0s)
Server:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id:b72b1...f5-26564f000000
x-ms-client-request-id:15fd74...0ec3601
x-ms-version:2020-08-04
x-ms-error-code:ContainerAlreadyExists
Date:Mon, 21 Feb 2022 07:33:56 GMT
Content-Length:230
Content-Type:application/xml

I think it's reasonable to assume this is something to do with the storage account that the Azure Functions use to record their state etc.

SDK version azurefunctions: 4.1.3.17473

Is there a way to debug and/or resolve the issue?

The text was updated successfully, but these errors were encountered:

domasd · 2022-02-23T09:32:05Z

We are experiencing the same issue on some of our v4 functions.

From what we investigated it appears that this warning (or similar one 404 The specified container does not exist.) are being spit by function sdk, but in the same time, by default, they are supressed and if you have default host.json in place, you shouldn't see them.

Our assumption is that we are seeing these warnings because we override default log level.
So currently we are just filtering the category Azure.Core with Error severity and it fixes the issue.

Just for reference, our host.json looks like this:

{
  "version": "2.0",
  "logging": {
    "logLevel": {
      "default": "Warning",
      "Custom": "Information",
      "Function": "Information", // dependency telemetry, used to analyzing dependencies and their performance
      "Host.Results": "Information", // request telemetry, used for analyzing execution performance
      "Azure.Core":  "Error" // suppressing sdk blob warnings 
    }
  }
}

v-bbalaiagar · 2022-02-24T09:31:48Z

Hi @pragnagopa , Could you please look into this issue

pragnagopa · 2022-02-24T20:24:45Z

Tagging @AnatoliB @gzuber for triage

tymtam2 · 2022-03-01T07:29:42Z

Just adding more info. The problem seem to be coming in batches of 4 issues:

Error response [06ac509e-...] 409 The specified container already exists. (00.1s)
Server:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id:e483b8d7-...
x-ms-client-request-id:06ac509e-...
x-ms-version:2020-08-04
x-ms-error-code:ContainerAlreadyExists
Date:Tue, 01 Mar 2022 07:11:44 GMT
Content-Length:230
Content-Type:application/xml

Error response [4c57b299-...] 404 The specified blob does not exist. (00.0s)
Server:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id:80603de1-...
x-ms-client-request-id:4c57b299-...
x-ms-version:2020-10-02
x-ms-error-code:BlobNotFound
Date:Tue, 01 Mar 2022 07:11:14 GMT
Content-Length:215
Content-Type:application/xml

Error response [9d09446f-...] 404 The specified blob does not exist. (00.0s)
Transfer-Encoding:chunked
Server:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id:80603da6-...
x-ms-client-request-id:9d09446f-...
x-ms-version:2020-10-02
x-ms-error-code:BlobNotFound
Date:Tue, 01 Mar 2022 07:11:14 GMT

Error response [b873e541-...] 409 The specified container already exists. (00.3s)
Server:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id:80603cc6-...
x-ms-client-request-id:b873e541-...
x-ms-version:2020-10-02
x-ms-error-code:ContainerAlreadyExists
Date:Tue, 01 Mar 2022 07:11:13 GMT
Content-Length:230
Content-Type:application/xml

AnatoliB · 2022-03-03T01:35:50Z

I think Application Insights is just a messenger here. Something in the host process is trying to access a blob, and this attempt is captured, but I don't see any indication that this blob is accessed because of Application Insights.

Does this repro with the simplest "hello world" app with a single HTTP trigger?

tymtam2 · 2022-03-03T02:34:28Z

Category for these isssues is Azure.Core.1.

As I was setting up trying the out-of-the-box http trigger function I realised that this issue is only reported by the functions with [EventHubTrigger]. I can see now that the functions that only have a http trigger do not report these 4 issues.

AnatoliB · 2022-03-05T01:46:59Z

@tymtam2 Do you know which version of the Event Hubs extension you are using?

tymtam2 · 2022-03-07T06:48:17Z

Here are all the referenced packages

<PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <AzureFunctionsVersion>v4</AzureFunctionsVersion>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Dapper" Version="2.0.123" />
    <PackageReference Include="Microsoft.ApplicationInsights" Version="2.20.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="5.0.0" />
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
    <PackageReference Include="Microsoft.CodeAnalysis.NetAnalyzers" Version="6.0.0">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="4.1.0" />
    <PackageReference Include="StyleCop.Analyzers" Version="1.2.0-beta.406">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="System.Data.SqlClient" Version="4.8.3" />
  </ItemGroup>

I get the 4 issues when running locally (not just when deployed) and in the bin folder the version of
Azure.Messaging.EventHubs.dll is 5.6.2
and Microsoft.Azure.WebJobs.Extensions.EventHubs.dll is 5.0.0.

As an experiment I added an explicit dependency on Azure.Messaging.EventHubs 5.7.0-beta.3 but after doing that the function reports on startup:
Microsoft.Azure.WebJobs.Host: Error indexing method '(name_here)'. Microsoft.Azure.WebJobs.Host: Multiple properties named 'ContentType' found in type 'EventData'..

AnatoliB · 2022-03-07T07:34:30Z

@alrod Track2 extension issue?

alrod · 2022-03-09T00:41:43Z

@tymtam2,
can you please try to use "Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="4.3.1"?
do you see the "409 The specified container already exists." in this case?

tymtam2 · 2022-03-09T09:16:50Z

I've tried "Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="4.3.1" and I don't see the warnings.

GuiGimenez · 2022-04-13T01:45:52Z

This occurs with [ServiceBusTrigger] as well

mail4hafij · 2022-05-11T14:00:16Z

I had this bug

Multiple properties named 'ContentType' found in type 'EventData'.

Problem: I had Microsoft.Azure.WebJobs.Extensions.EventHubs 4.2 that comes with VS template when creating function app (iothub trigger). It does not work. I had to install Azure.Messaging.EventHubs.Processor. Then it works just fine. But if I update the Microsoft.Azure.WebJobs.Extensions.EventHubs to latest version then it complains about Microsoft.Azure.EventHubs which is deprecated and needed to be replaced with with Azure.Messaging.EventHubs. Finally you have produced the error.

Fix: I have not updated the Microsoft.Azure.WebJobs.Extensions.EventHubs 4.2 and installed Azure.Messaging.EventHubs.Processor. Now its behaving properly.

Concern: I am sure this is not a proper fix but I could not solve it any other way. Please share your solution if you find any. The Microsoft.Azure.EventHubs is deprecated. It seems to be Azure.Messaging.EventHubs is buggy.

jawinnerjr · 2022-05-12T17:34:29Z

Any resolution on this. Using VS 2022 MAC RC2 on M1. Worked fine until Microsoft.Azure.Webjobs.Extensions.EventHubs updated to 5.1.0. Rolling back however does not resolve the issue. This is specifically an issue with the event hub trigger in Azure functions ~4 / .NET 6.

[2022-05-12T17:32:08.551Z] Microsoft.Azure.WebJobs.Host: Error indexing method ''. Microsoft.Azure.WebJobs.Host: Multiple properties named 'ContentType' found in type 'EventData'.
[2022-05-12T17:32:08.600Z] Error indexing method ''
[2022-05-12T17:32:08.600Z] Microsoft.Azure.WebJobs.Host: Error indexing method ''. Microsoft.Azure.WebJobs.Host: Multiple properties named 'ContentType' found in type 'EventData'.
[2022-05-12T17:32:08.601Z] Function '' failed indexing and will be disabled.

When deployed to Azure the functions load and execute properly. This issue appears to be limited to the Visual Studio environment only.

wkoeter · 2022-05-25T10:20:18Z

I'm also seeing this warning on startup with [TimerTrigger] on .NET 6 with dotnet-isolated.

[2022-05-25T10:17:21.413Z] Error response [26c59ccf-e824-4fb0-a586-9cd340504cba] 409 The specified container already exists. (00.0s)
[2022-05-25T10:17:21.417Z] Server:Azurite-Blob/3.14.1
[2022-05-25T10:17:21.419Z] x-ms-error-code:ContainerAlreadyExists
[2022-05-25T10:17:21.420Z] x-ms-request-id:277e9701-a041-4f85-802d-234f3d0c8caa
[2022-05-25T10:17:21.421Z] Date:Wed, 25 May 2022 10:17:21 GMT
[2022-05-25T10:17:21.423Z] Connection:keep-alive
[2022-05-25T10:17:21.424Z] Keep-Alive:REDACTED
[2022-05-25T10:17:21.426Z] Transfer-Encoding:chunked
[2022-05-25T10:17:21.427Z] Content-Type:application/xml

Seems to me like the Core is trying to do a CreateIfNotExists somewhere internally or something because TimerTrigger also uses blob storage under the hood, and that is shown in our logs.

The timers run fine by the way.

michaeldaw · 2022-06-18T04:46:38Z

No doubt a manifestation of this issue, which hasn't been fixed for 10 years 😖
Container.CreateIfNotExist throws exception if container exists

If anyone gets assigned to this (and I hope someone does), make sure to check if the container exists before trying to create it. CreateIfNotExist throws an exception if the container exists. If that sounds absurd, it's because it is.

klenium · 2022-07-17T11:31:16Z

I'm getting this error after upgrading a project/host from .NET Core 3.1 / AF 3 to .NET 6 / AF 4. I'm not using any event* triggers, only http/timer/durableTask.

brennfoster · 2022-07-21T15:45:55Z

My http triggers work fine but my event hub triggers are all failing with the errors described in this thread.

net6
func v4
Microsoft.Azure.Functions.Worker.Extensions.EventHubs 5.1.0

brennfoster · 2022-08-17T18:42:07Z

I just realized that I was not awaiting the host.RunAsync() command in Program.cs and that's why this was failing for me.

xsurfer · 2022-10-25T23:06:08Z

is there any update on this issue? I am facing the same issue while using net6 and function app v4. Thanks

SaveliyKolesnikov · 2022-10-27T14:15:33Z

This occurs with [ServiceBusTrigger] after upgrading from .NET Core 3.1 / AF 3 to .NET 6 / AF 4.

jameswoodley · 2023-01-12T19:07:58Z

I am seeing this behavior with [ServiceBusTrigger] and [TimerTrigger]

luisgmgouveia · 2023-01-13T09:11:52Z

Our assumption is that we are seeing these warnings because we override default log level. (@domasd)

I would just like to say that @domasd assumption seems absolutely correct. My application is in production for 6 months and I've never came across this warning before I customized my host.json. Once I've changed the loglevel, I've started seeing this warning.

MSIT-BPSC-IOT · 2023-02-09T04:49:55Z

Does this require any code changes? Even with below values in host.json (V4 function), the issue still persists

{
"version": "2.0",
"functionTimeout": "00:55:00",
"logging": {
"logLevel": {
"default": "Warning",
"Custom": "Information",
"Function": "Information",
"Host.Results": "Information",
"Azure.Core": "Error",
"Azure.Messaging.ServiceBus": "Error",
"Azure.Storage.Blobs": "Error",
"Microsoft.Azure.WebJobs.Extensions.Storage": "Error",
"Microsoft.Azure.WebJobs.Extensions.ServiceBus": "Error"
}
}
}

xlight05 · 2023-02-09T06:56:38Z

Getting the same issue for http trigger in prod. For some reason, only in Linux. Works just fine in windows.
host json

{
  "version": "2.0",
  "extensions": {
    "http": {
      "routePrefix": ""
    }
  },
  "customHandler": {
    "description": {
      "defaultExecutablePath": "java",
      "defaultWorkerPath": "multi_out.jar",
      "arguments": [
        "-jar"
      ]
    },
    "enableForwardingHttpRequest": false
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[3.*, 4.0.0)"
  }
}

@pragnagopa @alrod

hhao01-becls · 2023-04-03T03:11:14Z

If you use Serilog as the logging provider, you can add the following app setting to ignore the warning logs.

{
  "Serilog": {
    "MinimumLevel": {
      "Override": {
        "Azure.Core": "Error"
      }
    }
  }
}

But the countermeasure will ignore other warnings.

jaschrep-msft · 2023-04-04T16:14:12Z

This looks to be the same issue as tracked here Azure/azure-sdk-for-net#21511

MarGraz · 2023-05-05T11:44:14Z

Same problem here, I'm using .net 6 and Azure Function v. 4.

Those are the dependencies in my project:

    <PackageReference Include="Azure.Extensions.AspNetCore.Configuration.Secrets" Version="1.2.2" />
    <PackageReference Include="Azure.Identity" Version="1.8.2" />
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="5.1.2" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="4.2.0" />

The exception seems to be called by:

BlobContainerClient.CreateIfNotExists ---> BlobContainerClient.Create

it seems to happen in my implementation when I use:

await blobContainerClient.CreateIfNotExistsAsync(PublicAccessType.BlobContainer);

I see this error in Azure Application Insights:

Azure.RequestFailedException: The specified container already exists.
RequestId:7aadc04c-601e-0051-2726-7fab43000000
Time:2023-05-05T07:54:33.5483399Z
Status: 409 (The specified container already exists.)
ErrorCode: ContainerAlreadyExists

Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>ContainerAlreadyExists</Code><Message>The specified container already exists.
RequestId:7aadc04c-601e-0051-2726-7fab43000000
Time:2023-05-05T07:54:33.5483399Z</Message></Error>

Headers:
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 7aadc04c-601e-0051-2726-7fab43000000
x-ms-client-request-id: 45bb8263-7256-4297-9f81-12c74c6ecb1d
x-ms-version: 2022-11-02
x-ms-error-code: ContainerAlreadyExists
Date: Fri, 05 May 2023 07:54:32 GMT
Content-Length: 230
Content-Type: application/xml

   at Azure.Storage.Blobs.ContainerRestClient.Create(Nullable`1 timeout, IDictionary`2 metadata, Nullable`1 access, String defaultEncryptionScope, Nullable`1 preventEncryptionScopeOverride, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.BlobContainerClient.CreateInternal(PublicAccessType publicAccessType, IDictionary`2 metadata, BlobContainerEncryptionScopeOptions encryptionScopeOptions, Boolean async, CancellationToken cancellationToken, String operationName)

Do you know if it is available a fix?

Thank you

MarGraz · 2023-05-05T12:10:59Z

I found this conversation on stackoverflow that seems interesting.

c0rn3liusha11 · 2023-05-16T19:01:48Z

While suppressing the logging of this by only letting Error level messages through from Azure.Core works, it also means that any other actually valid warning messages will get missed.
It seems like the AZF SDK should be updated to change these messages to "Info" or "Debug" level if they are actually not actionable.

MarGraz · 2023-05-16T19:49:36Z

@c0rn3liusha11 thank you for your reply, I prefer to avoid suppress the log messages. I opened a new issue here and MS is checking the problem.

mtrmmsoftware · 2024-01-17T12:46:46Z

Having the same issue even if the "Azure.Core": "Error" is set in the host.json, using the Microsoft.Azure.Functions.Worker.Extensions.Storage nuget package with version 6.2.0

Any update on this issue?

restfulhead · 2024-01-29T20:48:13Z

Seems like we're now impacted by this since January 18. Now we have hundreds of these warning messages in our log. Suppressing warnings can't be the real solution.

Environment details

Azure Function Runtime: ~4 (Node.js)
Environment: Windows

  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[3.3.0, 4.0.0)"
  },

jasonvangundy · 2024-01-31T15:21:27Z

This just started happening to me and my team as well. We have deployed many nodejs function apps over the last year and we've never seen this. The one I am working on locally right now (as of yesterday) just started to have this issue. Happens using Azurite and a real Azure storage account.

Environment details

Nodejs
Environment: VSCode on Mac M1

Azure Functions Core Tools
Core Tools Version:       4.0.5455 Commit hash: N/A  (64-bit)
Function Runtime Version: 4.27.5.21554

phazlett · 2024-05-06T15:35:52Z

This is also a problem for me, but only for HTTP triggered functions.

drilko · 2024-06-18T12:49:48Z

Hi, I'm getting thousands of these logs as well. Any update?

scottcollins · 2024-08-28T16:53:59Z

Is there any resolution to this yet? I am using .net 8 isolated process and I am seeing it with [TimerTrigger]?

Purgator · 2024-10-16T13:29:53Z

I have the exact same problem with TimeTrigger.

yarmoliq · 2024-10-30T16:11:35Z

Same

NicolaiSattler · 2024-12-16T19:37:40Z

Still an issue for TimeTrigger and QueueTrigger.

tymtam2 added the bug label Feb 21, 2022

v-bbalaiagar self-assigned this Feb 22, 2022

This comment was marked as outdated.

Sign in to view

pragnagopa unassigned v-bbalaiagar Feb 24, 2022

MarGraz mentioned this issue May 5, 2023

Microsoft.Azure.WebJobs.Extensions.Storage version 5.1.2 and Azure Functions v. 4.2.0, error: "The specified container already exists", status 409 #2370

Open

liliankasem mentioned this issue Sep 20, 2023

Check if a container or table exists before trying to create it Azure/azure-functions-host#9555

Merged

8 tasks

"409 The specified container already exists." #2166

"409 The specified container already exists." #2166

Comments

tymtam2 commented Feb 21, 2022 • edited Loading

domasd commented Feb 23, 2022

This comment was marked as outdated.

v-bbalaiagar commented Feb 24, 2022

pragnagopa commented Feb 24, 2022

tymtam2 commented Mar 1, 2022

AnatoliB commented Mar 3, 2022

tymtam2 commented Mar 3, 2022

AnatoliB commented Mar 5, 2022

tymtam2 commented Mar 7, 2022

AnatoliB commented Mar 7, 2022

alrod commented Mar 9, 2022

tymtam2 commented Mar 9, 2022

GuiGimenez commented Apr 13, 2022

mail4hafij commented May 11, 2022 • edited Loading

jawinnerjr commented May 12, 2022 • edited Loading

wkoeter commented May 25, 2022 • edited Loading

michaeldaw commented Jun 18, 2022

klenium commented Jul 17, 2022

brennfoster commented Jul 21, 2022

brennfoster commented Aug 17, 2022

xsurfer commented Oct 25, 2022

SaveliyKolesnikov commented Oct 27, 2022

jameswoodley commented Jan 12, 2023

luisgmgouveia commented Jan 13, 2023

MSIT-BPSC-IOT commented Feb 9, 2023

xlight05 commented Feb 9, 2023 • edited Loading

hhao01-becls commented Apr 3, 2023 • edited Loading

jaschrep-msft commented Apr 4, 2023

MarGraz commented May 5, 2023 • edited Loading

MarGraz commented May 5, 2023

c0rn3liusha11 commented May 16, 2023

MarGraz commented May 16, 2023

mtrmmsoftware commented Jan 17, 2024

restfulhead commented Jan 29, 2024

jasonvangundy commented Jan 31, 2024 • edited Loading

phazlett commented May 6, 2024 • edited Loading

drilko commented Jun 18, 2024

scottcollins commented Aug 28, 2024

Purgator commented Oct 16, 2024

yarmoliq commented Oct 30, 2024

NicolaiSattler commented Dec 16, 2024

tymtam2 commented Feb 21, 2022 •

edited

Loading

mail4hafij commented May 11, 2022 •

edited

Loading

jawinnerjr commented May 12, 2022 •

edited

Loading

wkoeter commented May 25, 2022 •

edited

Loading

xlight05 commented Feb 9, 2023 •

edited

Loading

hhao01-becls commented Apr 3, 2023 •

edited

Loading

MarGraz commented May 5, 2023 •

edited

Loading

jasonvangundy commented Jan 31, 2024 •

edited

Loading

phazlett commented May 6, 2024 •

edited

Loading