UnicodeDecodeError: 'gbk' codec can't decode byte 0xa2 in position 4094106: illegal multibyte sequence

[saierXP]

Godot version

4.1.beta2

godot-cpp version

4.1.beta(be25a50)

System information

windows 10 amd 2500u

Issue description

Python:3.9.0 Scons:4.4.0
When compiling the latest version, there will be the following errors.

UnicodeDecodeError: 'gbk' codec can't decode byte 0xa2 in position 4094106: illegal multibyte sequence:
  File "D:\Temp\godot\bug\godot-cpp\godot-cpp\SConstruct", line 232:
    bindings = env.GenerateBindings(
  File "d:\programlanguage\python\python39\lib\site-packages\SCons\Environment.py", line 439:
    return super().__call__(target, source, *args, **kw)
  File "d:\programlanguage\python\python39\lib\site-packages\SCons\Util\envs.py", line 242:
    return self.method(*nargs, **kwargs)
  File "d:\programlanguage\python\python39\lib\site-packages\SCons\Builder.py", line 663:
    return self._execute(env, target, source, OverrideWarner(kw), ekw)
  File "d:\programlanguage\python\python39\lib\site-packages\SCons\Builder.py", line 569:
    tlist, slist = self._create_nodes(env, target, source)
  File "d:\programlanguage\python\python39\lib\site-packages\SCons\Builder.py", line 531:
    target, source = self.emitter(target=tlist, source=slist, env=env)
  File "D:\Temp\godot\bug\godot-cpp\godot-cpp\binding_generator.py", line 146:
    files = [env.File(f) for f in get_file_list(str(source[0]), target[0].abspath, True, True)]
  File "D:\Temp\godot\bug\godot-cpp\godot-cpp\binding_generator.py", line 79:
    api = json.load(api_file)
  File "d:\programlanguage\python\python39\lib\json\__init__.py", line 293:
    return loads(fp.read(),

When I used git bisect, I found that the error originated from this commit.

Because the default value of push_list(godotengine/godot#75017) has this symbol Bullet 2022 • , but gbk does not have it.

Run the python code local.getdefaultlocale(), it returns 'zh_CN', 'cp936', so open the file and read it in gbk encoding by default instead of utf8.

This method can override the default function.

import _locale
_locale._getdefaultlocale = (lambda *args: ['en_US', 'UTF-8'])

Another way to fix the problem is by setting environment variables set PYTHONUTF8=1

The last one is to specify the encoding as utf-8 when opening the file

def get_file_list(api_filepath, output_dir, headers=False, sources=False):
    api = {}
    files = []
    with open(api_filepath, encoding='utf-8') as api_file:
        api = json.load(api_file)

Steps to reproduce

If your system's default encoding is not cp936, you can set it with the following code.

import _locale
_locale._getdefaultlocale = (lambda *args: ['zh_CN', 'cp936'])

Then compile the code and you can see a similar error.

Minimal reproduction project

N/A

[bruvzg]

import _locale
_locale._getdefaultlocale = (lambda *args: ['zh_CN', 'cp936'])

This is doing absolutely nothing, open seems to always use UTF-8 regardless of system locale, not sure why your python setup acts differently. But I guess adding encoding to open won't hurt (it's done in the main repo, and we expect everything to be UTF-8).

[luo-l]

run "set PYTHONUTF8=1" in CMD before you run python script, it will force python use utf-8 mode.

---

source
https://peps.python.org/pep-0540/ PEP 540 – Add a new UTF-8 Mode