Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bicep stopped working on azure-cli:latest Docker image (2.64.0) #29828

Open
dallmair opened this issue Sep 3, 2024 · 15 comments
Open

Bicep stopped working on azure-cli:latest Docker image (2.64.0) #29828

dallmair opened this issue Sep 3, 2024 · 15 comments
Assignees
Labels
Auto-Assign Auto assign by bot Azure Deployments az deployment/bicep/stack/deployment-scripts/ts/group export bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported Issues that are reported by GitHub users external to the Azure organization. Packaging/Mariner Service Attention This issue is responsible by Azure service team.

Comments

@dallmair
Copy link

dallmair commented Sep 3, 2024

Describe the bug

We're using Bicep to deploy our Azure resources. That worked fine up to Azure CLI 2.63.0 Docker image, but since today with 2.64.0 it fails. Please note we're using azure-cli:latest, so we expected some of our own stuff to break when moving from Alpine to CBL Mariner 2.0. However, we did not expect Bicep to break.

Related command

az deployment sub create --subscription $SUBSCRIPTION_ID --name $NAME --location "North Europe" --template-file ./main.bicep

Errors

ERROR: Process terminated. Couldn't find a valid ICU package installed on the system. Please install libicu (or icu-libs) using your package manager and try again. Alternatively you can set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support. Please see https://aka.ms/dotnet-missing-libicu for more information.
   at System.Environment.FailFast(System.String)
   at System.Globalization.GlobalizationMode+Settings..cctor()
   at System.Globalization.GlobalizationMode+Settings.get_Invariant()
   at System.Globalization.GlobalizationMode.get_Invariant()
   at System.Globalization.CultureData.CreateCultureWithInvariantData()
   at System.Globalization.CultureData.get_Invariant()
   at System.Globalization.CultureInfo..cctor()
   at System.Globalization.CultureInfo.get_InvariantCulture()
   at System.Globalization.NumberFormatInfo.get_InvariantInfo()
   at System.Reflection.AssemblyNameParser.ParseVersion(System.String)
   at System.Reflection.AssemblyNameParser.Parse()
   at System.Reflection.AssemblyNameParser.Parse(System.String)
   at System.Reflection.AssemblyName..ctor(System.String)
   at System.Reflection.TypeNameParser.ResolveAssembly(System.String)
   at System.Reflection.TypeNameParser.GetType(System.String, System.ReadOnlySpan`1<System.String>, System.String)
   at System.Reflection.TypeNameParser+NamespaceTypeName.ResolveType(System.Reflection.TypeNameParser ByRef, System.String)
   at System.Reflection.TypeNameParser.Parse()
   at System.Reflection.TypeNameParser.GetTypeReferencedByCustomAttribute(System.String, System.Reflection.RuntimeModule)
   at System.Reflection.CustomAttributeTypedArgument.ResolveType(System.Reflection.RuntimeModule, System.String)
   at System.Reflection.CustomAttributeTypedArgument..ctor(System.Reflection.RuntimeModule, System.Reflection.CustomAttributeEncodedArgument)
   at System.Reflection.RuntimeCustomAttributeData.get_ConstructorArguments()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.GetDynamicallyAccessedMemberTypes(System.Type)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.ValidateTrimmingAnnotations(System.Type, System.Type[], System.Type, System.Type[])
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.Populate()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory..ctor(System.Collections.Generic.ICollection`1<Microsoft.Extensions.DependencyInjection.ServiceDescriptor>)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider..ctor(System.Collections.Generic.ICollection`1<Microsoft.Extensions.DependencyInjection.ServiceDescriptor>, Microsoft.Extensions.DependencyInjection.ServiceProviderOptions)
   at Microsoft.Extensions.DependencyInjection.ServiceCollectionContainerBuilderExtensions.BuildServiceProvider(Microsoft.Extensions.DependencyInjection.IServiceCollection, Microsoft.Extensions.DependencyInjection.ServiceProviderOptions)
   at Microsoft.Extensions.DependencyInjection.ServiceCollectionContainerBuilderExtensions.BuildServiceProvider(Microsoft.Extensions.DependencyInjection.IServiceCollection)
   at Microsoft.Extensions.Logging.LoggerFactory.Create(System.Action`1<Microsoft.Extensions.Logging.ILoggingBuilder>)
   at Bicep.Cli.Program.CreateLoggerFactory(Bicep.Cli.IOContext)
   at Bicep.Cli.Program.ConfigureServices(Bicep.Cli.IOContext)
   at Bicep.Cli.Program..ctor(Bicep.Cli.IOContext, System.Action`1<Microsoft.Extensions.DependencyInjection.IServiceCollection>)
   at Bicep.Cli.Program+<>c__DisplayClass3_0+<<Main>b__0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Bicep.Cli.Program+<>c__DisplayClass3_0+<<Main>b__0>d, bicep, Version=0.29.0.0, Culture=neutral, PublicKeyToken=null]](<<Main>b__0>d ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Bicep.Cli.Program+<>c__DisplayClass3_0+<<Main>b__0>d, bicep, Version=0.29.0.0, Culture=neutral, PublicKeyToken=null]](<<Main>b__0>d ByRef)
   at Bicep.Cli.Program+<>c__DisplayClass3_0.<Main>b__0(System.Threading.CancellationToken)
   at Bicep.Cli.Program+<RunWithCancellationAsync>d__6.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Bicep.Cli.Program+<RunWithCancellationAsync>d__6, bicep, Version=0.29.0.0, Culture=neutral, PublicKeyToken=null]](<RunWithCancellationAsync>d__6 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Bicep.Cli.Program+<RunWithCancellationAsync>d__6, bicep, Version=0.29.0.0, Culture=neutral, PublicKeyToken=null]](<RunWithCancellationAsync>d__6 ByRef)
   at Bicep.Cli.Program.RunWithCancellationAsync(System.Func`2<System.Threading.CancellationToken,System.Threading.Tasks.Task`1<Int32>>)
   at Bicep.Cli.Program+<Main>d__3.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Bicep.Cli.Program+<Main>d__3, bicep, Version=0.29.0.0, Culture=neutral, PublicKeyToken=null]](<Main>d__3 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Bicep.Cli.Program+<Main>d__3, bicep, Version=0.29.0.0, Culture=neutral, PublicKeyToken=null]](<Main>d__3 ByRef)
   at Bicep.Cli.Program.Main(System.String[])
   at Bicep.Cli.Program.<Main>(System.String[])

Issue script & Debug output

Not related to Python part of az, but the Bicep extension.

Expected behavior

Successful deployment. AFAIK, .NET sets globalization invariant mode in their Alpine images, but obviously this is not set in the azure-cli images based on cbl-mariner-2.0 and affects Bicep.

Environment Summary

azure-cli                         2.64.0

core                              2.64.0
telemetry                          1.1.0

Dependencies:
msal                              1.30.0
azure-mgmt-resource               23.1.1

Python location '/usr/bin/python3.9'
Extensions directory '/root/.azure/cliextensions'

Python (Linux) 3.9.19 (main, Aug 23 2024, 00:07:48)
[GCC 11.2.0]

Legal docs and information: aka.ms/AzureCliLegal


Your CLI is up-to-date.

Additional context

No response

@dallmair dallmair added the bug This issue requires a change to an existing behavior in the product in order to be resolved. label Sep 3, 2024
@microsoft-github-policy-service microsoft-github-policy-service bot added customer-reported Issues that are reported by GitHub users external to the Azure organization. Auto-Assign Auto assign by bot Azure Deployments az deployment/bicep/stack/deployment-scripts/ts/group export Service Attention This issue is responsible by Azure service team. labels Sep 3, 2024
@yonzhan
Copy link
Collaborator

yonzhan commented Sep 3, 2024

Thank you for opening this issue, we will look into it.

@rubenmamo
Copy link

I have the same issue when deploying with the azure/cli action. This was working fine up till yesterday and broke today.

@yonzhan
Copy link
Collaborator

yonzhan commented Sep 3, 2024

As announced in https://learn.microsoft.com/en-us/cli/azure/run-azure-cli-docker, the image base of Azure CLI 2.64.0 has changed from Alpine to CBL Mariner 2.0. This change may introduce some breaking changes that could cause failures in bicep. We will investigate this issue during our work hours.

@pschonefeld-sot
Copy link

still broken for me...
image

@jiasli
Copy link
Member

jiasli commented Sep 4, 2024

I am able to repro with the below GitHub action task:

    - name: Azure CLI script
      uses: azure/cli@v2
      with:
        azcliversion: 2.64.0
        inlineScript: |
          set -x
          az bicep install
          az bicep version

Output:

+ az bicep install
WARNING: The configuration value of bicep.use_binary_from_path has been set to 'false'.
Installing Bicep CLI v0.29.47...
Successfully installed Bicep CLI to "/root/.azure/bin/bicep".

+ az bicep version
ERROR: Process terminated. Couldn't find a valid ICU package installed on the system. Please install libicu (or icu-libs) using your package manager and try again. Alternatively you can set the configuration flag System.Globalization.Invariant to true if you want to run with no globalization support. Please see https://aka.ms/dotnet-missing-libicu for more information.
   at System.Environment.FailFast(System.String)
   at System.Globalization.GlobalizationMode+Settings..cctor()
   at System.Globalization.GlobalizationMode+Settings.get_Invariant()
   at System.Globalization.GlobalizationMode.get_Invariant()
   at System.Globalization.CultureData.CreateCultureWithInvariantData()
   at System.Globalization.CultureData.get_Invariant()
   at System.Globalization.CultureInfo..cctor()
   at System.Globalization.CultureInfo.get_InvariantCulture()
   at System.Globalization.NumberFormatInfo.get_InvariantInfo()
   at System.Reflection.AssemblyNameParser.ParseVersion(System.String)
   at System.Reflection.AssemblyNameParser.Parse()
   at System.Reflection.AssemblyNameParser.Parse(System.String)
   at System.Reflection.AssemblyName..ctor(System.String)
   at System.Reflection.TypeNameParser.ResolveAssembly(System.String)
   at System.Reflection.TypeNameParser.GetType(System.String, System.ReadOnlySpan`1<System.String>, System.String)
   at System.Reflection.TypeNameParser+NamespaceTypeName.ResolveType(System.Reflection.TypeNameParser ByRef, System.String)
   at System.Reflection.TypeNameParser.Parse()
   at System.Reflection.TypeNameParser.GetTypeReferencedByCustomAttribute(System.String, System.Reflection.RuntimeModule)
   at System.Reflection.CustomAttributeTypedArgument.ResolveType(System.Reflection.RuntimeModule, System.String)
   at System.Reflection.CustomAttributeTypedArgument..ctor(System.Reflection.RuntimeModule, System.Reflection.CustomAttributeEncodedArgument)
   at System.Reflection.RuntimeCustomAttributeData.get_ConstructorArguments()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.GetDynamicallyAccessedMemberTypes(System.Type)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.ValidateTrimmingAnnotations(System.Type, System.Type[], System.Type, System.Type[])
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory.Populate()
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteFactory..ctor(System.Collections.Generic.ICollection`1<Microsoft.Extensions.DependencyInjection.ServiceDescriptor>)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider..ctor(System.Collections.Generic.ICollection`1<Microsoft.Extensions.DependencyInjection.ServiceDescriptor>, Microsoft.Extensions.DependencyInjection.ServiceProviderOptions)
   at Microsoft.Extensions.DependencyInjection.ServiceCollectionContainerBuilderExtensions.BuildServiceProvider(Microsoft.Extensions.DependencyInjection.IServiceCollection, Microsoft.Extensions.DependencyInjection.ServiceProviderOptions)
   at Microsoft.Extensions.DependencyInjection.ServiceCollectionContainerBuilderExtensions.BuildServiceProvider(Microsoft.Extensions.DependencyInjection.IServiceCollection)
   at Microsoft.Extensions.Logging.LoggerFactory.Create(System.Action`1<Microsoft.Extensions.Logging.ILoggingBuilder>)
   at Bicep.Cli.Program.CreateLoggerFactory(Bicep.Cli.IOContext)
   at Bicep.Cli.Program.ConfigureServices(Bicep.Cli.IOContext)
   at Bicep.Cli.Program..ctor(Bicep.Cli.IOContext, System.Action`1<Microsoft.Extensions.DependencyInjection.IServiceCollection>)
   at Bicep.Cli.Program+<>c__DisplayClass3_0+<<Main>b__0>d.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Bicep.Cli.Program+<>c__DisplayClass3_0+<<Main>b__0>d, bicep, Version=0.[29](https://github.com/jiasli/github-action-test/actions/runs/10697597733/job/29655128999#step:3:30).0.0, Culture=neutral, PublicKeyToken=null]](<<Main>b__0>d ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Bicep.Cli.Program+<>c__DisplayClass3_0+<<Main>b__0>d, bicep, Version=0.29.0.0, Culture=neutral, PublicKeyToken=null]](<<Main>b__0>d ByRef)
   at Bicep.Cli.Program+<>c__DisplayClass3_0.<Main>b__0(System.Threading.CancellationToken)
   at Bicep.Cli.Program+<RunWithCancellationAsync>d__6.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Bicep.Cli.Program+<RunWithCancellationAsync>d__6, bicep, Version=0.29.0.0, Culture=neutral, PublicKeyToken=null]](<RunWithCancellationAsync>d__6 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int[32](https://github.com/jiasli/github-action-test/actions/runs/10697597733/job/29655128999#step:3:33), System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Bicep.Cli.Program+<RunWithCancellationAsync>d__6, bicep, Version=0.29.0.0, Culture=neutral, PublicKeyToken=null]](<RunWithCancellationAsync>d__6 ByRef)
   at Bicep.Cli.Program.RunWithCancellationAsync(System.Func`2<System.Threading.CancellationToken,System.Threading.Tasks.Task`1<Int32>>)
   at Bicep.Cli.Program+<Main>d__3.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Bicep.Cli.Program+<Main>d__3, bicep, Version=0.29.0.0, Culture=neutral, PublicKeyToken=null]](<Main>d__3 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Bicep.Cli.Program+<Main>d__3, bicep, Version=0.29.0.0, Culture=neutral, PublicKeyToken=null]](<Main>d__3 ByRef)
   at Bicep.Cli.Program.Main(System.String[])
   at Bicep.Cli.Program.<Main>(System.String[])

I am working with Bicep and Mariner teams internally on this issue and will share the update here.

@jiasli
Copy link
Member

jiasli commented Sep 4, 2024

Workaround: For now, please use the 2.63.0 tag.

@anthony-c-martin
Copy link
Member

anthony-c-martin commented Sep 4, 2024

This feels related to Azure/bicep#1938. For anyone able to repro this - could you try setting the following environmental variable before running Bicep?

export DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1

In @jiasli's example this would be:

    - name: Azure CLI script
      uses: azure/cli@v2
      with:
        azcliversion: 2.64.0
        inlineScript: |
          set -x
          export DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1
          az bicep install
          az bicep version

@dallmair
Copy link
Author

dallmair commented Sep 4, 2024

Yep, this is the setting that should probably be added to the cbl-mariner2.0 based container images of Azure CLI. The Docker images published by the .NET team have this set for the Alpine-based images, but not for the CBL-Mariner-based images.

And it works, here in a manually spun up container:

root [ / ]# export DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1
root [ / ]# az bicep install
Installing Bicep CLI v0.29.47...
The configuration value of bicep.use_binary_from_path has been set to 'false'.
Successfully installed Bicep CLI to "/root/.azure/bin/bicep".
root [ / ]# az bicep version
Bicep CLI version 0.29.47 (132ade51bc)

root [ / ]# az version
{
  "azure-cli": "2.64.0",
  "azure-cli-core": "2.64.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {}
}
root [ / ]#

@shenglol
Copy link
Contributor

shenglol commented Sep 4, 2024

Yep, this is the setting that should probably be added to the cbl-mariner2.0 based container images of Azure CLI. The Docker images published by the .NET team have this set for the Alpine-based images, but not for the CBL-Mariner-based images.

And it works, here in a manually spun up container:

root [ / ]# export DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1
root [ / ]# az bicep install
Installing Bicep CLI v0.29.47...
The configuration value of bicep.use_binary_from_path has been set to 'false'.
Successfully installed Bicep CLI to "/root/.azure/bin/bicep".
root [ / ]# az bicep version
Bicep CLI version 0.29.47 (132ade51bc)

root [ / ]# az version
{
  "azure-cli": "2.64.0",
  "azure-cli-core": "2.64.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {}
}
root [ / ]#

Bicep actually works with the Mariner-based dotnet image because it includes icu as a dependency: https://github.com/dotnet/dotnet-docker/blob/d90d458deada9057d7889f76d58fc0a7194a0c06/src/runtime-deps/8.0/cbl-mariner2.0/amd64/Dockerfile#L16. I’m curious why the environment variable was set for the Alpine-based dotnet image, given that Globalization Invariant isn’t a default setting in .NET core.

@jiasli Would it be possible to include these .NET dependencies in the Azure CLI image build: https://github.com/dotnet/dotnet-docker/blob/d90d458deada9057d7889f76d58fc0a7194a0c06/src/runtime-deps/8.0/cbl-mariner2.0/amd64/Dockerfile#L11-L22?

@pschonefeld-sot
Copy link

this works, ty

export DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1

@jiasli
Copy link
Member

jiasli commented Sep 5, 2024

I can confirm manually installing icu or libicu package also solves the issue.

tdnf install --assumeyes icu

tdnf shows the icu package takes 33.29M installed size and this will increase the docker image size:

# tdnf install icu
Loaded plugin: tdnfrepogpgcheck

Installing:
icu                      x86_64             68.2.0.9-1.cm2           mariner-official-base  33.29M            13.24M

Total installed size:  33.29M
Total download size:  13.24M

This will make #7387 worse.

I am considering if icu can be dynamically installed when installing the bicep executable.

The Alpine-based image installs icu-libs in the build process:

RUN --mount=type=bind,target=/azure-cli,source=./,rw apk add --no-cache ca-certificates bash bash-completion libintl icu-libs libc6-compat jq openssh-keygen \

@dallmair
Copy link
Author

dallmair commented Sep 6, 2024

I’m curious why the environment variable was set for the Alpine-based dotnet image, given that Globalization Invariant isn’t a default setting in .NET core.

Alpine is about size, and not having ICU in those containers makes a huge difference. For the same reason, the .NET team considered making globalization-invariant mode the default in all other Docker images, but ultimately decided against it as the value proposition is different for the other distros. Also, from that issue:

  • Alpine got added because it offered a very different set of characteristics than Debian. It also got added in a time when we started to receive vulnerability scans with vulnerabilities in .NET Core Linux images. So, Alpine was an important choice for size and security (and user asks exactly aligned with that).

@davidfowl
Copy link
Member

cc @richlander

@pamelafox
Copy link
Member

@anthony-c-martin That env variable worked for our workflows: https://github.com/Azure-Samples/azure-search-openai-demo/actions/runs/10819953360/job/30018949127?pr=1964
However, we have the same workflow in dozens of repositories in Azure-Samples. I would like to avoid having to update all of them if possible. :)

@anthony-c-martin
Copy link
Member

anthony-c-martin commented Sep 12, 2024

@pamelafox agreed, I was just looking to get more info on the problem. Having AzureCLI set this env var on the container feels like a viable longer-term solution.

EDIT: I think #29897 should address this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Auto-Assign Auto assign by bot Azure Deployments az deployment/bicep/stack/deployment-scripts/ts/group export bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported Issues that are reported by GitHub users external to the Azure organization. Packaging/Mariner Service Attention This issue is responsible by Azure service team.
Projects
None yet
Development

No branches or pull requests

10 participants