Python packages are being installed to local/lib/python*/dist-packages due to tweaks to sysconfig.py in OS (Ubuntu 22.04)

[anselmicz]

Problem description

Hi. Recently, one of my machines started experiencing problems with installing EasyBuild via the bootstrap method.

== 2022-11-26 22:18:10,235 easyblock.py:3774 INFO Running method sanity_check_step part of step sanitycheck
== 2022-11-26 22:18:10,281 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/eb_tmp/local/lib/python3.10/dist-packages/easybuild/base/exceptions.py:124 in __init__): Failed to isolate installed egg dir for easybuild-framework (at easybuild/eb_tmp/local/lib/python3.10/dist-packages/easybuild/easyblocks/e/easybuildmeta.py:235 in sanity_check_step)

I use a custom build script I wrote based on the official instructions in the Lua, Lmod and EasyBuild documentations:
https://github.com/anselmicz/easybuild-installer/blob/main/easybuild.sh

My current builds fail during the sanity check phase, I'm enclosing the build log, environment at the time of installation, and trace output as a gist:
https://gist.github.com/anselmicz/72d1a0239cf67b685ed878aef2f61359

The error itself comes from https://github.com/boegel/easybuild-easyblocks/blob/develop/easybuild/easyblocks/e/easybuildmeta.py#L235

In a Slack conversation, @ocaisa mentioned that the error may be somehow connected to
#2805

System

OS
Debian bookworm/sid

Custom dependencies

Lua/5.1.4.9
Lmod/8.5

System packages

python3/3.10.6
python3-pip/22.3
wget/1.21.3
gcc/12.2.0
make/4.3
rsync/3.2.6
tcl-dev/8.6.11
libreadline-dev/8.2
libibverbs-dev/43.0
xdot/1.2

Python packages

setuptools/65.6.3
python-graph-core/1.8.2
python-graph-dot/1.8.2
archspec/0.2.0
autopep8/2.0.0
GitPython/3.1.29
pep8/1.7.1
pycodestyle/2.10.0
Rich/12.6.0

Do you have any thoughts on this? I tried lowering both the temporary and to-be-installed versions of EasyBuild, but the issue persists both on 4.6.1 and 4.6.0, and I can't try 4.5.4 because then the setuptools and setup.py problems arise.

[boegel]

@anselmicz Can you share the contents of /apps/all/EasyBuild/4.6.2, in particular the contents of the lib*/python*/site-packages subdirectory that should be there?

[anselmicz]

I have also tried lowering the setuptools and pip versions to no avail.

@boegel I have updated the gist with a directory tree of the /apps/all/EasyBuild/4.6.2

lib/python3.10/site-packages is empty, however, everything seems to install under /apps/all/EasyBuild/4.6.2/local, including lib... this is probably the distribution difference, but here the path is .../lib/python3.10/dist-packages.

Same goes for the temporary EasyBuild installation, instead of installing under the prefix, /tmp/$USER/eb_tmp, the actual location of bin, lib, etc. is under /tmp/$USER/eb_tmp/local

I remember dealing with this problem before, but the issue has always been exclusively with the temporary installation of EasyBuild that could have always been fixed by accommodating the PYTHONPATH, i. e. setting export PYTHONPATH=$(/bin/ls -rtd -1 $EB_TMPDIR/local/lib*/python*/dist-packages | tail -1):$PYTHONPATH.

[crubb]

Here is a Dockerfile to reproduce the error using Ubuntu 22.04 LTS:

FROM ubuntu:jammy-20221130
#FROM ubuntu:focal-20221130

ARG DEBIAN_FRONTEND="noninteractive"

RUN set -eux \
  && apt-get update -qq \
  && apt-get -y upgrade \
  && apt-get install -y -q --no-install-recommends \
      lmod \
      python3-pip \
  && apt-get clean \
  && python3 -m pip install --ignore-installed --prefix /tmp/eb_tmp easybuild \
  && rm -rf /tmp/hsperfdata* /var/*/apt/*/partial /var/lib/apt/lists/* /var/log/apt/term*

RUN set -x \
  && export LOGNAME=root \
  && export PATH=$(find /tmp/eb_tmp -type d -name bin):$PATH \
  && export PYTHONPATH=$(find /tmp/eb_tmp/ -type d -name *-packages | tail -1) \
  && export EB_PYTHON=python3 \
  && . /etc/profile.d/lmod.sh \
  && eb --allow-use-as-root-and-accept-consequences --install-latest-eb-release -d --prefix /tmp/easybuild \
  ;  cat /tmp/eb-*/easybuild-EasyBuild-4.6.2-*.log

ENTRYPOINT ["/bin/bash"]

When you docker build the same thing using FROM ubuntu:focal-20221130 it works. So there has to be something different with Ubuntu 22.04 LTS, which I haven't figured out, yet. One of the obvious things, as mentioned: jammy installs everything in /tmp/eb_tmp/local/, while focal installs directly into /tmp/eb_tmp.

@anselmicz, is it the latest Ubuntu LTS for you, too?

Any ideas? Is there a fix or config change needed in Ubuntu, or with EasyBuild?

[anselmicz]

@crubb my report deals with Debian Unstable

I confirm I can reproduce the error in jammy. However, this is not at all that surprising, because, based on my understanding of Ubuntu versioning, jammy is based on Debian bookworm (current sid / unstable). It also makes sense that it might work correctly in focal, given it's based on bullseye where it works too (at least for me).

[crubb]

root cause of the problems seems to be the following: https://discuss.python.org/t/linux-distro-patches-to-sysconfig-are-changing-pip-install-prefix-outside-virtual-environments/18240/14

Essentially, some distros patched Python's sysconfig to change the default installation scheme from posix_prefix to posix_local.

I found a bug report for Ubuntu, which essentially sits dormant after more or less stating that it's a design choice: https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1967920

[boegel]

@crubb I can see how that affect the installation of EasyBuild with EasyBuild (since that's done with the system Python), but how does that affect installing Python packages with a Python that was installed with EasyBuild?!

[anselmicz]

@crubb Good catch! This (temporary) workaround seems to work for me:

--- /lib/python3.10/sysconfig.py.orig	2022-12-13 17:38:06.221735514 +0100
+++ /lib/python3.10/sysconfig.py	2022-12-13 17:40:28.549111500 +0100
@@ -271,7 +271,7 @@
         if deb_build in ('deb', 'deb_system'):
             prefix_scheme = 'deb_system'
         else:
-            prefix_scheme = 'posix_local'
+            prefix_scheme = 'posix_prefix'

     return {
         'prefix': prefix_scheme,

@boegel, looking at the links @crubb provided, it seems like in Debian, sysconfig.py got some extreme mods between the versions:
https://lists.debian.org/debian-python/2022/03/msg00039.html.
So far this seems to be distro specific, so the EasyBuild Python should be unaffected.

[crubb]

I actually borrowed from Debian and stole their DEB_PYTHON_INSTALL_LAYOUT environment variable. Set it to eb, and it uses posix_prefix as the prefix_scheme:

--- sysconfig.py-orig   2022-12-13 21:37:25.489016711 +0000
+++ sysconfig.py        2022-12-13 21:37:52.088975055 +0000
@@ -270,6 +270,8 @@
         deb_build = os.environ.get('DEB_PYTHON_INSTALL_LAYOUT', 'posix_local')
         if deb_build in ('deb', 'deb_system'):
             prefix_scheme = 'deb_system'
+        elif deb_build in ('eb'):
+            prefix_scheme = 'posix_prefix'
         else:
             prefix_scheme = 'posix_local'

[SebastianAchilles]

I just run into the same problem while generating a test-report on a ubuntu-22.04 container (recipe, image @ ghcr.io ) for EasyBuild-4.7.0.eb: easybuilders/easybuild-easyconfigs#17065 (comment)

With the described workaround I was able to build it successfully.

[boegel]

Did some more digging, found some more info:

• "sysconfig default scheme change in Python 3.10" (on debian-python mailing list);
• In Fix the install scheme to posix_prefix when --prefix is passed pypa/pip#10978 it's highlighted that this problem isn't specific to Debian/Ubuntu at all, it also affects Fedora 37 (so soon also RHEL & friends?), or even Fedora 36 (based on this and this), but there they didn't even bother to introduce a custom installation scheme like Debian/Ubuntu did with posix_local (which is not standard in Python, see https://docs.python.org/3/library/sysconfig.html#installation-paths), they just fiddled with the default posix_prefix instead, making it harder to detect;
  • this seems to suggest the change was rolled back in Fedora though...
• Spack has been dealing with this too, see:
  • Installation issue: meson 1.1.0 binary goes into local/bin/ instead of bin/ and dependent builds fail spack/spack#37632

[boegel]

Updates to the PythonPackage easyblock are implemented in #2977 to deal with this, using that EasyBuild installs without problems on Ubuntu 22.04.

[NicolasMelot]

@boegel redirected me here about the problem on Ubuntu 22.04. I could not follow easybuild installation guide on Installing EasyBuild tutorial. The procedure fails at step 2 with the error:

build failed (first 300 chars): Failed to isolate installed egg dir for easybuild-framework (took 1 min 10 secs)

and no other obvious information. The solution discussed here seem to be used with the flag --include-easyblocks-from-pr 2977, as in:

eb --include-easyblocks-from-pr 2977 --install-latest-eb-release --prefix /tmp/easybuild

and it worked immediately as expected on my system.

[boegel]

@akesandgren mentioned that he was hitting some problems when testing #2977, so I'm re-opening this issue to try and figure out what was overlooked (I haven't been able to reproduce this problem):

== 2023-08-24 16:23:42,987 pythonpackage.py:285 INFO Subdirectories of /scratch/ebtest4/software/EasyBuild/4.8.0/local: ['lib']
== 2023-08-24 16:23:42,988 build_log.py:171 ERROR EasyBuild crashed with an error (at easybuild/base/exceptions.py:126 in __init__): Failed to install EasyBuild packages: [Errno 22] Invalid argument: '/scratch/ebtest4/software/EasyBuild/4.8.0/lib' (at easybuild/easyblocks/e/easybuildmeta.py:155 in install_step)

[boegel]

I did a quick debugging session on this with @akesandgren, and it's now clear why he was hitting the Invalid argument error, which is triggered by EasyBuild trying to symlink <prefix>/local/lib to <prefix>/lib while <prefix>/lib already exists.

In @akesandgren's Ubuntu 22.04 container, Python 3.10 is installed, but pip is not.
The EasyBuildMeta easyblock automatically enables installing EasyBuild with pip install --prefix ... for sufficiently recent versions of EasyBuild, if pip is installed (and recent enough). Since that condition is not met, the EasyBuildMeta easyblock falls back to using python setup.py install --prefix ....

Apparently whether the active Python naming scheme of the system Python is posix_prefix (the default) or posix_local (the custom one in Ubuntu 22.04) only matters when installing Python packages with pip install --prefix ... (or python3 -m pip install --prefix ...), not when using python3 setup.py install --prefix ....

The changes in #2977 make PythonPackage prepare an empty <prefix>/local/python*/site-packages directory, based on the active Python naming scheme being posix_local, but the installation is actually done in <prefix>/local/python*/site-packages (as it should be!).
PythonPackage.install_step then tries to symlink local/lib to lib after installing the easybuild-framework package, which fails because <prefix>/lib exists already.

[boegel]

#2988 should fix the issue that @akesandgren observed...