setup.py based installation can mix universal and CPython ABI

[steve-s]

When installing a HPy extension by using python setup.py --hpy-abi=ABI install, the build directory is not cleared between builds. If the user installs the extension with different ABIs and does not clean the build directory in between, then both myextension.hpy.so and myextension.cpython-38-...so are installed and the later takes precedence when the extension is loaded. Unless one knows how HPy works internally figuring out what's going on and why universal mode does not work can be bad first user experience.

[antocuni]

Good catch.
I think that the proper solution is to teach distutils to use a different build directory depending on the selected hpy abi. On my machine, if you build a non-hpy module, you get the two folloing directories:

$ ls build/
lib.linux-x86_64-3.8/  temp.linux-x86_64-3.8/

Probabily, we want to use something like lib.linux-x86_64-3.8-hpy-universal or similar when --hpy-abi=universal.

Moreover, we have another related problem. If you have only hpy_ext_modules, all the files are put in build/lib instead of build/lib.linux-x86_64-3.8: this is because distutils explicitly check the presence of self.distribution.ext_modules to determine whether it's a "pure" o "plat" build:
https://github.com/python/cpython/blob/eb0004c27163ec089201c81eb9ece470700be6c7/Lib/distutils/command/build.py#L101-L108

            if self.distribution.ext_modules:
                self.build_lib = self.build_platlib
            else:
                self.build_lib = self.build_purelib

I don't know whether this might cause problems or not, to be honest.

I thought it was a quick fix and I've started a branch, but it seems more messy than expected.

[antocuni]

Moreover, we have another related problem. If you have only hpy_ext_modules, all the files are put in build/lib instead of build/lib.linux-x86_64-3.8

An update on this. It seems to depend on the version of setuptools:

• newer versions of setuptools automatically monkey-patch distutils, so that import distutils gets an improved version (without the bug)
• older versions of setuptools seems to NOT automatically monkey-patch, so that hpy patches the stdlib version of distutils, which have bugs. This happens by default if you e.g. you do python3.8 -m venv.

I think that hpy.devel should check that import distutils gives the one provided by setuptools and fail early if it doesn't.

[hodgestar]

Failing with older setuptools makes a lot of sense -- that one can obtain two possible versions of distutils already causes other difficulties too and the Python distutils is deprecated and scheduled for removal in any case.