ERROR: 'charmap' codec can't encode characters in position 4790-4801: character maps to <undefined> #21409

doggy8088 · 2022-02-23T11:50:26Z

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Describe the bug

Recently, my script run into this problem. My az script is run on Azure Pipelines. Here is my PowerShell script on Windows Server 2019 agent:

az deployment sub what-if `
    --name "main$(Get-Date -Format "yyyyMMddHHmmss")" `
    --location 'japaneast' `
    --template-file 'main.bicep' `
    --parameters  `
        "env=$(env.name)" `
        "projectName=$(projectName)" `
        "dbAdminAccount=$(sql.serveradmin.name)" `
        "dbAdminPwd=$(sql.serveradmin.password)"

Here is the error output:

ERROR: 'charmap' codec can't encode characters in position 4790-4801: character maps to <undefined>

For the following command, it works.

az deployment sub validate `
    --name "main$(Get-Date -Format "yyyyMMddHHmmss")" `
    --location 'japaneast' `
    --template-file 'main.bicep' `
    --parameters  `
        "env=$(env.name)" `
        "projectName=$(projectName)" `
        "dbAdminAccount=$(sql.serveradmin.name)" `
        "dbAdminPwd=$(sql.serveradmin.password)"

So the error only shows when running az deployment sub what-if command.

To Reproduce

Expected behavior

My script is running for more than 1 month. It works before. I found it's broken today.

Environment summary

C:\Windows\system32\cmd.exe /D /S /C ""C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az.cmd" --version"
azure-cli                         2.32.0 *
WARNING: You have 2 updates available. Consider updating your CLI installation with 'az upgrade'


core                              2.32.0 *
Please let us know how we are doing: https://aka.ms/azureclihats
telemetry                          1.0.6
and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy

Extensions:
azure-devops                      0.22.0

Dependencies:
msal                              1.16.0
azure-mgmt-resource               20.0.0

Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Program Files\Common Files\AzureCliExtensionDirectory'

Python (Windows) 3.8.9 (tags/v3.8.9:a743f81, Apr  6 2021, 13:22:56) [MSC v.1928 32 bit (Intel)]

Legal docs and information: aka.ms/AzureCliLegal

Additional context

The text was updated successfully, but these errors were encountered:

yonzhan · 2022-02-23T12:25:16Z

ARM

zhoxing-ms · 2022-02-23T13:14:25Z

So the error only shows when running az deployment sub what-if command.

@shenglol Could you please help look at the issue related to what-if?

ghost · 2022-02-23T13:40:44Z

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @armleads-azure.

Issue Details

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Describe the bug

Recently, my script run into this problem. My az script is run on Azure Pipelines. Here is my PowerShell script on Windows Server 2019 agent:

az deployment sub what-if `
    --name "main$(Get-Date -Format "yyyyMMddHHmmss")" `
    --location 'japaneast' `
    --template-file 'main.bicep' `
    --parameters  `
        "env=$(env.name)" `
        "projectName=$(projectName)" `
        "dbAdminAccount=$(sql.serveradmin.name)" `
        "dbAdminPwd=$(sql.serveradmin.password)"

Here is the error output:

ERROR: 'charmap' codec can't encode characters in position 4790-4801: character maps to <undefined>

For the following command, it works.

az deployment sub validate `
    --name "main$(Get-Date -Format "yyyyMMddHHmmss")" `
    --location 'japaneast' `
    --template-file 'main.bicep' `
    --parameters  `
        "env=$(env.name)" `
        "projectName=$(projectName)" `
        "dbAdminAccount=$(sql.serveradmin.name)" `
        "dbAdminPwd=$(sql.serveradmin.password)"

So the error only shows when running az deployment sub what-if command.

To Reproduce

Expected behavior

My script is running for more than 1 month. It works before. I found it's broken today.

Environment summary

C:\Windows\system32\cmd.exe /D /S /C ""C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az.cmd" --version"
azure-cli                         2.32.0 *
WARNING: You have 2 updates available. Consider updating your CLI installation with 'az upgrade'


core                              2.32.0 *
Please let us know how we are doing: https://aka.ms/azureclihats
telemetry                          1.0.6
and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy

Extensions:
azure-devops                      0.22.0

Dependencies:
msal                              1.16.0
azure-mgmt-resource               20.0.0

Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Program Files\Common Files\AzureCliExtensionDirectory'

Python (Windows) 3.8.9 (tags/v3.8.9:a743f81, Apr  6 2021, 13:22:56) [MSC v.1928 32 bit (Intel)]

Legal docs and information: aka.ms/AzureCliLegal

Additional context

Author:	doggy8088
Assignees:	zhoxing-ms
Labels:	`Service Attention`, `ARM`, `customer-reported`, `Auto-Assign`
Milestone:	Backlog

shenglol · 2022-02-23T18:07:48Z

@doggy8088 Could you attach the Bicep file?

jiasli · 2022-02-24T04:03:05Z

We previously worked on a similar issue #17994 and that issue has been solved long ago. @doggy8088 could you also run the command with --debug flag and share the output? It will show the call stack of the error. Thanks.

ian90911 · 2022-02-24T07:48:18Z

We previously worked on a similar issue #17994 and that issue has been solved long ago. @doggy8088 could you also run the command with --debug flag and share the output? It will show the call stack of the error. Thanks.

@jiasli Hi, I'm @doggy8088 's coworker, I added --debug flag and here's Azure devops' output log:
issues21409-bicep what if err log.txt (file deleted)

jiasli · 2022-02-24T09:19:22Z

Thanks @ian90911 for the debug log which is extremely helpful.

Judging by the log:

DEBUG: cli.azure.cli.core.util: Traceback (most recent call last):
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 692, in _run_job
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 328, in __call__
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/resource/custom.py", line 815, in what_if_deploy_arm_template_at_subscription_scope
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/resource/custom.py", line 835, in _what_if_deploy_arm_template_at_subscription_scope_core
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/resource/custom.py", line 930, in _what_if_deploy_arm_template_core
  File "encodings\cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode characters in position 4665-4676: character maps to <undefined>

The failed code is at

azure-cli/src/azure-cli/azure/cli/command_modules/resource/custom.py

Line 930 in 6756801

print(format_what_if_operation_result(what_if_result, cli_ctx.enable_color))

Apparently print throws this error.

Also from the log, I noticed you have a resource name with Unicode characters:

"resourceId":"/subscriptions/***/resourceGroups/em-s/providers/microsoft.insights/activityLogAlerts/\u65e5\u672c\u6771\u5340\u57df\u7684\u670d\u52d9\u5065\u5eb7\u8b66\u793a"

I used Python to decode it to

python -c "print('\u65e5\u672c\u6771\u5340\u57df\u7684\u670d\u52d9\u5065\u5eb7\u8b66\u793a')"
日本東區域的服務健康警示

I tested without Azure CLI but only with Python on Windows agent and reproduced this issue.

jobs:
- job: TestAzureCLITaskUbuntu
  pool:
    vmImage: 'ubuntu-latest'
  steps:
  - task: AzureCLI@2
    displayName: Azure CLI
    inputs:
      azureSubscription: ...
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        python3 -c "import sys; print(sys.stdout.encoding); print('汉字')"
- job: TestAzureCLITaskWindows
  pool:
    vmImage: 'windows-latest'
  steps:
  - task: AzureCLI@2
    displayName: Azure CLI
    inputs:
      azureSubscription: ...
      scriptType: batch
      scriptLocation: inlineScript
      inlineScript: |
        "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe" -c "import sys; print(sys.stdout.encoding); print('汉字')"

The Ubuntu agent gives output

utf-8
汉字

The Windows agent gives output

cp1252
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "encodings\cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-1: character maps to <undefined>

Solution

For end user, the are 2 possible solutions:

Use a Ubuntu agent on which stdout is UTF-8.
Remove Unicode characters in the resource name since Azure DevOps's stdout's encoding is cp1252 which is not able to handle Unicode. For example, you may rename it to japan-north-service-health-alert.

For DevOps, Windows agent should use UTF-8 as stdout's encoding. We will internally work with Azure DevOps team on this topic.

jiasli · 2022-02-24T09:41:41Z

@shenglol, I saw this line was submitted by you in #12942:

azure-cli/src/azure-cli/azure/cli/command_modules/resource/custom.py

Line 930 in 6756801

print(format_what_if_operation_result(what_if_result, cli_ctx.enable_color))

Another less optimal solution I can think of is to discard Unicode characters if UnicodeEncodeError happens, like Knack does:

https://github.com/microsoft/knack/blob/d407b81d451d36c6ca1baeb3cd067a3f1275cb85/knack/output.py#L154-L158

        except UnicodeEncodeError:
            logger.warning("Unable to encode the output with %s encoding. Unsupported characters are discarded.",
                           out_file.encoding)
            print(output.encode('ascii', 'ignore').decode('utf-8', 'ignore'),
                  file=out_file, end='')

But this is certainly not a good solution as the resource name will be lost and the output will become useless.

doggy8088 · 2022-02-24T11:22:58Z

@jiasli I'm just curious why my script works for months. Only failed recently. I was running on Windows for a while.

doggy8088 · 2022-02-24T16:32:15Z

@jiasli I confirmed the ubuntu-latest agent works. Thanks.

jiasli · 2022-02-25T02:25:43Z

@jiasli Jiashuo Li FTE I'm just curious why my script works for months. Only failed recently. I was running on Windows for a while.

I think this is because previously you don't have Chinese characters in the output of the command az deployment sub what-if.

Anyway, please use ubuntu-latest to avoid hitting Unicode character issues in Azure DevOps.

ghost added customer-reported Issues that are reported by GitHub users external to the Azure organization. Auto-Assign Auto assign by bot ARM az resource/group/lock/tag/deployment/policy/managementapp/account management-group labels Feb 23, 2022

ghost assigned zhoxing-ms Feb 23, 2022

ghost added this to the Backlog milestone Feb 23, 2022

ghost added CXP Attention This issue is handled by CXP team. SQL az sql Upgrade az upgrade labels Feb 23, 2022

ghost assigned jiasli Feb 23, 2022

yonzhan removed SQL az sql Upgrade az upgrade labels Feb 23, 2022

yonzhan unassigned jiasli Feb 23, 2022

yonzhan removed this from the Backlog milestone Feb 23, 2022

yonzhan removed the CXP Attention This issue is handled by CXP team. label Feb 23, 2022

yonzhan added this to the Backlog milestone Feb 23, 2022

yonzhan added the Service Attention This issue is responsible by Azure service team. label Feb 23, 2022

yonzhan removed this from the Backlog milestone Feb 23, 2022

jiasli self-assigned this Feb 24, 2022

bebound mentioned this issue Jul 10, 2023

ERROR: 'charmap' codec can't encode character '\u2388' in position 198 #26843

Closed

eyast mentioned this issue Jul 4, 2024

UnicodeEncodeError when pushing graphrag:backend image to Azure Container Registry (ACR) Azure-Samples/graphrag-accelerator#45

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ERROR: 'charmap' codec can't encode characters in position 4790-4801: character maps to <undefined> #21409

ERROR: 'charmap' codec can't encode characters in position 4790-4801: character maps to <undefined> #21409

doggy8088 commented Feb 23, 2022

`az feedback` auto-generates most of the information requested below, as of CLI version 2.0.62

yonzhan commented Feb 23, 2022

zhoxing-ms commented Feb 23, 2022

ghost commented Feb 23, 2022

`az feedback` auto-generates most of the information requested below, as of CLI version 2.0.62

shenglol commented Feb 23, 2022

jiasli commented Feb 24, 2022 •

edited

Loading

ian90911 commented Feb 24, 2022 •

edited

Loading

jiasli commented Feb 24, 2022

jiasli commented Feb 24, 2022

doggy8088 commented Feb 24, 2022

doggy8088 commented Feb 24, 2022 •

edited

Loading

jiasli commented Feb 25, 2022

ERROR: 'charmap' codec can't encode characters in position 4790-4801: character maps to <undefined> #21409

ERROR: 'charmap' codec can't encode characters in position 4790-4801: character maps to <undefined> #21409

Comments

doggy8088 commented Feb 23, 2022

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

yonzhan commented Feb 23, 2022

zhoxing-ms commented Feb 23, 2022

ghost commented Feb 23, 2022

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

shenglol commented Feb 23, 2022

jiasli commented Feb 24, 2022 • edited Loading

ian90911 commented Feb 24, 2022 • edited Loading

jiasli commented Feb 24, 2022

Solution

jiasli commented Feb 24, 2022

doggy8088 commented Feb 24, 2022

doggy8088 commented Feb 24, 2022 • edited Loading

jiasli commented Feb 25, 2022

`az feedback` auto-generates most of the information requested below, as of CLI version 2.0.62

`az feedback` auto-generates most of the information requested below, as of CLI version 2.0.62

jiasli commented Feb 24, 2022 •

edited

Loading

ian90911 commented Feb 24, 2022 •

edited

Loading

doggy8088 commented Feb 24, 2022 •

edited

Loading