Patch pybi sysconfig to workaround distutils/setuptools bugs relocation bugs?

[njsmith]

Currently, distutils/setuptools are buggy when run inside a relocatable Python. For example, pypa/setuptools#3786, though there may be others.

Ultimately we should fix distutils/setuptools. And as a temporary workaround, posy's PEP 517 frontend currently monkeypatches distutils to paper over that specific bug. But in the discourse thread, @mattip mentioned another possible workaround: changing the _sysconfig_*.py files that are shipped inside the relocatable python distributions. This is attractive because it would work for anyone who uses our pybis; not just posy. And apparently pypy has experience with this that we might be able to steal? Matti, maybe you can explain more about how pypy does it?

[mattip]

As part of the unpacking process, PyPy can call python -m sysconfig --generate-posix-vars which will generate a file in a setuptools-compatible build directory like 'build/lib.linux-x86_64-3.9/_sysconfigdata__linux_x86_64-linux-gnu.py'. Then this file is copied into the stdlib directory. PyPy will not read the Makefile (which it does not use) to get these values, rather it will take them from the current sysconfig set. This solves the problem of location-specific-values in the file. However it leaves the problem of where to get the rest of the values. Over time PyPy has collected a minimum set of values that must appear in _sysconfig_*.py to support the build tools that use it (CMake, meson, cross-compile, ...). Since PyPy is not used on as many exotic platforms as CPython, it can hard-code some of the values. This is what running that command generates on a PyPy on my linux x86_64 system (this is about the minimum set of values needed for a posix system):

# system configuration generated and used by the sysconfig module
build_time_vars = {'ABIFLAGS': '',
 'AR': 'ar',
 'ARFLAGS': 'rc',
 'CC': 'gcc -pthread',
 'CCSHARED': '-fPIC',
 'CFLAGS': '-DNDEBUG -O2',
 'CONFINCLUDEPY': '/home/matti/oss/pypy3.9-HEAD/include/pypy3.9',
 'CXX': 'g++ -pthread',
 'EXE': '',
 'EXT_SUFFIX': '.pypy39-pp73-x86_64-linux-gnu.so',
 'GNULD': 'yes',
 'INCLUDEPY': '/home/matti/oss/pypy3.9-HEAD/include/pypy3.9',
 'LDFLAGS': '-Wl,-Bsymbolic-functions',
 'LDLIBRARY': 'libpypy3.9-c.so',
 'LDSHARED': 'gcc -pthread -shared -Wl,-Bsymbolic-functions',
 'LDVERSION': '3.9',
 'LIBDIR': '/home/matti/oss/pypy3.9-HEAD/bin',
 'MULTIARCH': 'x86_64-linux-gnu',
 'OPT': '-DNDEBUG -O2',
 'Py_DEBUG': 0,
 'Py_ENABLE_SHARED': 0,
 'SHLIB_SUFFIX': '.so',
 'SIZEOF_VOID_P': 8,
 'SO': '.pypy39-pp73-x86_64-linux-gnu.so',
 'SOABI': 'pypy39-pp73',
 'TZPATH': '/home/matti/oss/pypy3.9-HEAD/share/zoneinfo:/home/matti/oss/pypy3.9-HEAD/lib/zoneinfo:/home/matti/oss/pypy3.9-HEAD/share/lib/zoneinfo:/home/matti/oss/pypy3.9-HEAD/../etc/zoneinfo:/usr/share/zoneinfo:/usr/lib/zoneinfo:/usr/share/lib/zoneinfo:/etc/zoneinfo',
 'VERSION': '3.9'}

Additionally, PyPy extends the command line interface of python -m sysconfig --generate-posix-vars by modifying sysconfig.py to allow including additional key,value pairs. This is used to add HOST_GNU_TYPE which is needed for cross-compile. On CPython this can be obtained from the Makefile. So on my system I would run

pypy -m sysconfig --generate-posix-vars HOST_GNU_TYPE x86_64-pc-linux-gnu

[mattip]

PyPy has the luxury of shipping its own stdlib, so it can be lazy about patching the upstream (CPython) stdlib with changes it made. Conda does not have that luxury, so it has a patching mechanism to download the CPython sources and patch them. Maybe posy could patch pybi where needed.

[njsmith]

Oh yeah, I was wondering what conda did! Looks like it's.... here?

https://github.com/conda-forge/python-feedstock/blob/21038d707640da7b5da152a5239e199c720f0574/recipe/build_base.sh#L450

or maybe here?

https://github.com/conda-forge/python-feedstock/blob/21038d707640da7b5da152a5239e199c720f0574/recipe/build_base.sh#L386

Maybe I should just look at the package instead of deciphering build scripts.

...oh fun, latest miniconda installer gives errors on Ubuntu because it has bashisms in a #!/bin/sh, what the heck:

./Miniconda3-latest-Linux-x86_64.sh: 438: [[: not found

Installing * environment...

./Miniconda3-latest-Linux-x86_64.sh: 444: [[: not found

CondaFileIOError: '/home/njs/miniconda3/pkgs/envs/*/env.txt'. [Errno 2] No such file or directory: '/home/njs/miniconda3/pkgs/envs/*/env.txt'

It also has a symlink from lib/python3.1 -> lib/python3.10? That's weird.

~/miniconda3/lib 
❯ ls -ld python*
lrwxrwxrwx  1 njs njs  10 Feb  2 00:39 python3.1 -> python3.10/
drwxrwxr-x 36 njs njs 13k Feb  2 00:39 python3.10/

Well... anyway... it looks like they rely on conda's trick where it can rewrite files after installing to substitute in the installation path. Which is cool and all, but it would be Extremely Nice™ if pybis could be relocatable without that extra step, and it seems pretty viable given that sysconfig gets to execute arbitrary python code at runtime. So I guess conda is a dead end.

[h-vetinari]

It also has a symlink from lib/python3.1 -> lib/python3.10? That's weird.

I'd have to look up the details, but basically that's a disgusting hack that became necessary for reasons™ after python's minor version gained a second digit.

[mattip]

There are two parts to this issue:

• calling sysconfig to regenerate the file,
• how to patch sysconfig to do this.

The regeneration is available when calling python -m sysconfig --generate-posix-vars, plus minus details. I don't know where would be the right place to inject this into the workflow.

It seems like the viable alternatives for patching sysconfig.py are to submit an upstream PR (problematic for older Pythons) or by some magic done when packaging.

[eli-schwartz]

Possibly relevant: python/cpython#99942