ufbt no longer properly deploys SDK on Windows Systems with non-Latin/Cyrillic system language

[Alex4386]

Describe the bug.

Running ufbt no longer properly deploys SDK on Windows system

Reproduction

1. Open Command Prompt
2. (Optional) If you have latest firmware SDK installed, clean it via ufbt clean
3. Try setting up SDK via running ufbt
4. The ufbt errors out with following error specified in Logs

Target

N/A

Logs

Raw:

C:\Users\alex4386\git\f0-template>ufbt
10:42:53.953 [I] Deploying SDK for f7
10:42:53.953 [I] Fetching version info for UpdateChannel.RELEASE from https://update.flipperzero.one/firmware/directory.json
10:42:54.564 [I] Using version: 1.1.2
10:42:54.565 [I] uFBT SDK dir: C:\Users\alex4386\.ufbt\current
10:42:57.428 [I] Deploying SDK
10:42:58.024 [I] SDK deployed.
Downloading Windows toolchain..done!


    디렉터리: C:\Users\alex4386\.ufbt


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----      4357-11-30  오전 10:43                toolchain
Extracting Windows toolchain..An error occurred
"2"개의 인수가 있는 "ExtractToDirectory"을(를) 호출하는 동안 예외가 발생했습니다. "경로에 잘못된 문자가 있습니다."
Please close VSCode and any other programs that may be using the toolchain and try again.
C:\Users\alex4386\AppData\Local\Programs\Python\Python311\python.exe: No module named SCons

Translation of output of windows-toolchain-download.ps1#L49.

While calling "ExtractToDirectory" with "2" arguments, Following exception has occurred:
"The path has invalid characters."

Anything else?

Something to note:

• The Windows installation of this machine has language set to Korean. So, please note the CJK quirks (e.g. WideChar, MultiByte nonsense on Windows)
• This does not reproduce on *NIX Machines (i.e. Linux, macOS, etc.)
  • Possible *NIX path (that's incompatible with NTFS Path) included in ZIP Archive?
• Partial decompression does occur and ~/.ufbt/current contains some contents of the ZIP archive decompressed.

[Alex4386]

I have implemented the following using .NET 6.0 since Powershell uses underlying C# APIs for handling Windows stuff, but calling the same API programmatically didn't cause the issue:

Will try with lower .NET version since I'm using Powershell included with Windows 10 build 19041 right now. The CLRVersion of my powershell is 4.0.30319.42000

I think the System.IO.Compression::ZIPFile.ExtractToDirectory implementation differs by the .NET Runtime version and reproduces only on lower powershell versions (i.e. pre-Powershell 7 era) that uses .NET Framework as their implementation, will look into it.

[Alex4386]

I can successfully reproduce with targetting .NET Framework 4.7.2. Currently ZipFile.ExtractToDirectory raises ArgumentException in mscorlib.dll!System.Security.Permissions.FileIOPermission.EmulateFileIOPermissionChecks(string fullPath) while it checks for HasWildCardCharacters.

According to Microsoft's ReferenceSource, This should only check for * and ? in the path.
which doesn't seem to be like one. 🤔

Here is the debugger catch screenshot if you want to do some checks:

[Alex4386]

Found the culprit!

It WAS an Encoding issue.

Due to cyrillic alphabet on /openocd/scripts/target/1986ве1т.cfg, My system "which is configured to use CP949 (Korean CodePage)", by default processed it as a cyrillic alphabet as a malformed Korean alphabet.

Since UTF-8 is enabled by default of the .NET 6+, So this didn't cause issue with Powershell 7+.
The fix will be setting UTF8 manually on Powershell call, so I will make a PR very quick with some little check on my side. Thanks.

• The Windows installation of this machine has language set to Korean. So, please note the CJK quirks (e.g. WideChar, MultiByte nonsense on Windows)

Well, This really was a fore-shadowing, isn't it.

[Alex4386]

I have verified that ufbt with patched windows-toolchain-download.ps1 now works with the patch I've PR-ed at #4036. Please give it a look. Thanks.