pip install --prefix='' workaround seems incompatible with pip >= 21.3 #74405

henryptung · 2022-07-04T10:50:12Z

The problem

Seeing errors like the following when updating Home Assistant on Arch:

Jul 04 03:07:03 mediaserver hass[1089]: 2022-07-04 03:07:03 ERROR (SyncWorker_2) [homeassistant.util.package] Unable to install package sqlalchemy==1.4.37: WARNING: No metadata found in ./deps/lib/python3.10/site-packages
Jul 04 03:07:03 mediaserver hass[1089]: ERROR: Could not install packages due to an OSError: [Errno 30] Read-only file system: '/lib/python3.10/site-packages/sqlalchemy/processors.py'
Jul 04 03:07:04 mediaserver hass[1089]: 2022-07-04 03:07:04 ERROR (SyncWorker_1) [homeassistant.util.package] Unable to install package sqlalchemy==1.4.37: WARNING: No metadata found in ./deps/lib/python3.10/site-packages
Jul 04 03:07:04 mediaserver hass[1089]: ERROR: Could not install packages due to an OSError: [Errno 30] Read-only file system: '/lib/python3.10/site-packages/sqlalchemy/processors.py'
Jul 04 03:07:06 mediaserver hass[1089]: 2022-07-04 03:07:06 ERROR (SyncWorker_4) [homeassistant.util.package] Unable to install package sqlalchemy==1.4.37: WARNING: No metadata found in ./deps/lib/python3.10/site-packages
Jul 04 03:07:06 mediaserver hass[1089]: ERROR: Could not install packages due to an OSError: [Errno 30] Read-only file system: '/lib/python3.10/site-packages/sqlalchemy/processors.py'
Jul 04 03:07:06 mediaserver hass[1089]: 2022-07-04 03:07:06 ERROR (MainThread) [homeassistant.setup] Setup failed for recorder: Requirements for recorder not found: ['sqlalchemy==1.4.37'].

This looks like a manifestation of the behavior mentioned in this comment: https://stackoverflow.com/questions/4495120/combine-user-with-prefix-error-with-setup-py-install/4495175#comment32831892_4495175

Commenting out this line resolved the issue and allowed dependencies to be installed in deps as intended. Not clear what exactly in the environment is causing this interaction, but that "workaround" seems quite old at this point and it looks like --prefix= is being treated as "use / as prefix" instead of "unset prefix" somehow.

Note also that the SO question has been updated with a different argument to pip, --install-option="--prefix=", rather than --prefix= as is done in package.py.

What version of Home Assistant Core has the issue?

2022.6.7

What was the last working version of Home Assistant Core?

2022.4.6

What type of installation are you running?

Home Assistant Core

Integration causing the issue

recorder

Link to integration documentation on our website

https://www.home-assistant.io/integrations/recorder/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

OS: Arch Linux
Architecture: x86_64
Python version: core/python 3.10.5-1

Downstream bug report: https://bugs.archlinux.org/task/75204

Maybe with newer (setuptools-based) pip, the entire PYTHONUSERBASE={target} --user --prefix= runaround can just be replaced with --prefix={target}?

The text was updated successfully, but these errors were encountered:

probot-home-assistant · 2022-07-04T10:50:22Z

recorder documentation
recorder source
_{^{(message by IssueLinks)}}

probot-home-assistant · 2022-07-04T10:50:25Z

Hey there @home-assistant/core, mind taking a look at this issue as it has been labeled with an integration (recorder) you are listed as a code owner for? Thanks!
_{^{(message by CodeOwnersMention)}}

henryptung · 2022-07-04T11:33:55Z

Note: Can also confirm that python-pip 21.0-1 did not manifest the error, but python-pip 22.1.2-1 does:

$ PYTHONUSERBASE=$PWD/deps python -m pip install --user --prefix='' sqlalchemy==1.4.37
Collecting sqlalchemy==1.4.37
  Using cached SQLAlchemy-1.4.37-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
Requirement already satisfied: greenlet!=0.4.17 in /usr/lib/python3.10/site-packages (from sqlalchemy==1.4.37) (1.1.2)
Installing collected packages: sqlalchemy
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/lib/python3.10/site-packages/sqlalchemy/processors.py'
Check the permissions.

henryptung · 2022-07-04T11:42:38Z

Hard for me to test different globally-installed pip versions, but some fairly suggestive output from testing with pip 21.1 in a virtualenv (21.0 doesn't report these):

(venv) $ PYTHONUSERBASE=$PWD/deps python -m pip install --prefix='' sqlalchemy==1.4.37
Collecting sqlalchemy==1.4.37
  Using cached SQLAlchemy-1.4.37-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
Requirement already satisfied: greenlet!=0.4.17 in ./lib/python3.10/site-packages (from sqlalchemy==1.4.37) (1.1.2)
Installing collected packages: sqlalchemy
  WARNING: Value for scheme.platlib does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
  distutils: /home/htung/venv/lib/python3.10/site-packages
  sysconfig: /lib/python3.10/site-packages
  WARNING: Value for scheme.purelib does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
  distutils: /home/htung/venv/lib/python3.10/site-packages
  sysconfig: /lib/python3.10/site-packages
  WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
  distutils: /home/htung/venv/include/site/python3.10/sqlalchemy
  sysconfig: include/site/python3.10/sqlalchemy
  WARNING: Value for scheme.scripts does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
  distutils: /home/htung/venv/bin
  sysconfig: /bin
  WARNING: Value for scheme.data does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
  distutils: /home/htung/venv
  sysconfig: .
  WARNING: Additional context:
  user = False
  home = None
  root = None
  prefix = ''

and pip 21.3 outright fails:

(venv) $ PYTHONUSERBASE=$PWD/deps python -m pip install --prefix='' sqlalchemy==1.4.37
Collecting sqlalchemy==1.4.37
  Using cached SQLAlchemy-1.4.37-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
Requirement already satisfied: greenlet!=0.4.17 in ./lib/python3.10/site-packages (from sqlalchemy==1.4.37) (1.1.2)
Installing collected packages: sqlalchemy
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/lib/python3.10/site-packages/sqlalchemy/processors.py'
Check the permissions.

https://peps.python.org/pep-0632/ seems relevant, as it's taking place as part of Python 3.10:

The distutils module has for a long time recommended using the setuptools package instead. Setuptools has recently integrated a complete copy of distutils and is no longer dependent on the standard library. Pip has been silently replacing distutils with setuptools when installing packages for a long time already, and the distutils documentation has stated that it is being phased out since 2014 (or earlier). It is time to remove it from the standard library.

synthead · 2022-07-06T03:28:54Z

Looks like the previous workaround was implemented as part of #64068.

See https://github.com/home-assistant/core/pull/64068/files#diff-f840671acc5fc415be942524b75af58e6ff5bd746e70aa9a57913bc8964f073cL92-R94.

Thank you so much for filing this issue, by the way!

synthead · 2022-07-06T03:55:31Z

If anyone is attempting to build HA for Arch Linux, here's a Gist with the --prefix= workaround removed. This makes HA build and run on Arch.

https://gist.github.com/synthead/819fa9bad91f9ba905125b2c7f34937d

henryptung · 2022-07-10T19:16:27Z

Note: using --prefix to directly specify target dir doesn't look like a good idea right now, because of pypa/pip#10978 (i.e. even though https://docs.python.org/3/install/index.html#alternate-installation-the-user-scheme and https://docs.python.org/3/install/index.html#alternate-installation-unix-the-prefix-scheme indicate the user and prefix schemes SHOULD be identical on Unix systems, they currently aren't on Debian because it's choosing the wrong scheme).

henryptung · 2022-07-10T19:49:04Z

Confirmed that adding /usr/lib/python3.X/distutils/distutils.cfg does not disrupt user installs via pip, even from legacy packages that only use distutils, on pip 20.x:

user@debian11:~/data_profiler$ python3 -m pip --version
pip 20.3.4 from /usr/lib/python3/dist-packages/pip (python 3.9)
user@debian11:~/data_profiler$ cat /usr/lib/python3.9/distutils/distutils.cfg 
[install]
prefix=/usr/local
user@debian11:~/data_profiler$ find ~/test2
find: ‘/home/user/test2’: No such file or directory
user@debian11:~/data_profiler$ PYTHONUSERBASE=$HOME/test2 python3 setup.py install --user
running install
error: can't combine user with prefix, exec_prefix/home, or install_(plat)base
user@debian11:~/data_profiler$ PYTHONUSERBASE=$HOME/test2 python3 -m pip install --user .
Processing /home/user/data_profiler
Building wheels for collected packages: data-profiler
  Building wheel for data-profiler (setup.py) ... done
  Created wheel for data-profiler: filename=data_profiler-1.0.0+4.gea60a9d.dirty-cp39-cp39-linux_x86_64.whl size=53077 sha256=4267dfd9b45ea40b8b50847948fe16f62b74abcb69f5da575f1b46815c501f61
  Stored in directory: /tmp/pip-ephem-wheel-cache-i_agkcvg/wheels/1f/5c/57/ad78dc9d996f978504e2aed24eb618c0d82fbe52ae54ee7c1e
Successfully built data-profiler
Installing collected packages: data-profiler
Successfully installed data-profiler-1.0.0+4.gea60a9d.dirty
user@debian11:~/data_profiler$ find ~/test2
/home/user/test2
/home/user/test2/bin
/home/user/test2/bin/profiler
/home/user/test2/lib
/home/user/test2/lib/python3.9
/home/user/test2/lib/python3.9/site-packages
/home/user/test2/lib/python3.9/site-packages/data_profiler
/home/user/test2/lib/python3.9/site-packages/data_profiler/profile.py
/home/user/test2/lib/python3.9/site-packages/data_profiler/prof.cpython-39-x86_64-linux-gnu.so
/home/user/test2/lib/python3.9/site-packages/data_profiler/__pycache__
/home/user/test2/lib/python3.9/site-packages/data_profiler/__pycache__/profile.cpython-39.pyc
/home/user/test2/lib/python3.9/site-packages/data_profiler/__pycache__/__init__.cpython-39.pyc
/home/user/test2/lib/python3.9/site-packages/data_profiler/__pycache__/pstats.cpython-39.pyc
/home/user/test2/lib/python3.9/site-packages/data_profiler/__pycache__/display.cpython-39.pyc
/home/user/test2/lib/python3.9/site-packages/data_profiler/__pycache__/_version.cpython-39.pyc
/home/user/test2/lib/python3.9/site-packages/data_profiler/display.py
/home/user/test2/lib/python3.9/site-packages/data_profiler/__init__.py
/home/user/test2/lib/python3.9/site-packages/data_profiler/snakeviz
/home/user/test2/lib/python3.9/site-packages/data_profiler/snakeviz/style.css
/home/user/test2/lib/python3.9/site-packages/data_profiler/snakeviz/template.jinja
/home/user/test2/lib/python3.9/site-packages/data_profiler/_version.py
/home/user/test2/lib/python3.9/site-packages/data_profiler/pstats.py
/home/user/test2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info
/home/user/test2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/METADATA
/home/user/test2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/REQUESTED
/home/user/test2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/RECORD
/home/user/test2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/direct_url.json
/home/user/test2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/LICENSE
/home/user/test2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/INSTALLER
/home/user/test2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/WHEEL
/home/user/test2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/top_level.txt
user@debian11:~/data_profiler$

Note that the config is in the correct location, as it does have effect on non-user installs when set to a non-default value:

user@debian11:~/data_profiler$ cat /usr/lib/python3.9/distutils/distutils.cfg 
[install]
prefix=/usr/local2
user@debian11:~/data_profiler$ find /usr/local2
find: ‘/usr/local2’: No such file or directory
user@debian11:~/data_profiler$ sudo python3 -m pip install .
Processing /home/user/data_profiler
Building wheels for collected packages: data-profiler
  Building wheel for data-profiler (setup.py) ... done
  Created wheel for data-profiler: filename=data_profiler-1.0.0+4.gea60a9d.dirty-cp39-cp39-linux_x86_64.whl size=53069 sha256=b7321a37e0b63fdf453b09215e2dc8808f8663f768dac5e237295a2fcc20a16c
  Stored in directory: /tmp/pip-ephem-wheel-cache-1hhgoidc/wheels/1f/5c/57/ad78dc9d996f978504e2aed24eb618c0d82fbe52ae54ee7c1e
Successfully built data-profiler
Installing collected packages: data-profiler
Successfully installed data-profiler-1.0.0+4.gea60a9d.dirty
user@debian11:~/data_profiler$ find /usr/local2
/usr/local2
/usr/local2/bin
/usr/local2/bin/profiler
/usr/local2/lib
/usr/local2/lib/python3.9
/usr/local2/lib/python3.9/site-packages
/usr/local2/lib/python3.9/site-packages/data_profiler
/usr/local2/lib/python3.9/site-packages/data_profiler/profile.py
/usr/local2/lib/python3.9/site-packages/data_profiler/prof.cpython-39-x86_64-linux-gnu.so
/usr/local2/lib/python3.9/site-packages/data_profiler/__pycache__
/usr/local2/lib/python3.9/site-packages/data_profiler/__pycache__/profile.cpython-39.pyc
/usr/local2/lib/python3.9/site-packages/data_profiler/__pycache__/__init__.cpython-39.pyc
/usr/local2/lib/python3.9/site-packages/data_profiler/__pycache__/pstats.cpython-39.pyc
/usr/local2/lib/python3.9/site-packages/data_profiler/__pycache__/display.cpython-39.pyc
/usr/local2/lib/python3.9/site-packages/data_profiler/__pycache__/_version.cpython-39.pyc
/usr/local2/lib/python3.9/site-packages/data_profiler/display.py
/usr/local2/lib/python3.9/site-packages/data_profiler/__init__.py
/usr/local2/lib/python3.9/site-packages/data_profiler/snakeviz
/usr/local2/lib/python3.9/site-packages/data_profiler/snakeviz/style.css
/usr/local2/lib/python3.9/site-packages/data_profiler/snakeviz/template.jinja
/usr/local2/lib/python3.9/site-packages/data_profiler/_version.py
/usr/local2/lib/python3.9/site-packages/data_profiler/pstats.py
/usr/local2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info
/usr/local2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/METADATA
/usr/local2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/REQUESTED
/usr/local2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/RECORD
/usr/local2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/direct_url.json
/usr/local2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/LICENSE
/usr/local2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/INSTALLER
/usr/local2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/WHEEL
/usr/local2/lib/python3.9/site-packages/data_profiler-1.0.0+4.gea60a9d.dirty.dist-info/top_level.txt

Given Home Assistant already requires pip >= 21.0, going to submit a PR for removing the --prefix workaround.

Remove --prefix workaround See discussion in #74405. This workaround is no longer needed on pip >= 21.0 and actively causes problems for pip >= 21.3.

…oken ref: home-assistant/core#74405

probot-home-assistant bot added the integration: recorder label Jul 4, 2022

henryptung changed the title ~~pip install --prefix workaround seems harmful on Arch?~~ pip install --prefix workaround seems incompatible with pip >= 21.3 Jul 4, 2022

henryptung changed the title ~~pip install --prefix workaround seems incompatible with pip >= 21.3~~ pip install --prefix='' workaround seems incompatible with pip >= 21.3 Jul 4, 2022

icaruseffect mentioned this issue Jul 9, 2022

Custom integrations with requirements fail to start with 2022.7.X #74700

Closed

henryptung mentioned this issue Jul 10, 2022

Remove pip --prefix workaround #74922

Merged

23 tasks

frenck closed this as completed in #74922 Jul 11, 2022

frenck pushed a commit that referenced this issue Jul 11, 2022

Remove pip --prefix workaround (#74922)

7d27dad

Remove --prefix workaround See discussion in #74405. This workaround is no longer needed on pip >= 21.0 and actively causes problems for pip >= 21.3.

balloob pushed a commit that referenced this issue Jul 13, 2022

Remove pip --prefix workaround (#74922)

3f53022

Remove --prefix workaround See discussion in #74405. This workaround is no longer needed on pip >= 21.0 and actively causes problems for pip >= 21.3.

archlinux-github pushed a commit to archlinux/aur that referenced this issue Jul 29, 2022

added temp workaround for pip installation in home assistant being br…

384b877

…oken ref: home-assistant/core#74405

github-actions bot locked and limited conversation to collaborators Aug 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pip install --prefix='' workaround seems incompatible with pip >= 21.3 #74405

pip install --prefix='' workaround seems incompatible with pip >= 21.3 #74405

henryptung commented Jul 4, 2022 •

edited

Loading

probot-home-assistant bot commented Jul 4, 2022

probot-home-assistant bot commented Jul 4, 2022

henryptung commented Jul 4, 2022 •

edited

Loading

henryptung commented Jul 4, 2022 •

edited

Loading

synthead commented Jul 6, 2022

synthead commented Jul 6, 2022

henryptung commented Jul 10, 2022 •

edited

Loading

henryptung commented Jul 10, 2022 •

edited

Loading

pip install --prefix='' workaround seems incompatible with pip >= 21.3 #74405

pip install --prefix='' workaround seems incompatible with pip >= 21.3 #74405

Comments

henryptung commented Jul 4, 2022 • edited Loading

The problem

What version of Home Assistant Core has the issue?

What was the last working version of Home Assistant Core?

What type of installation are you running?

Integration causing the issue

Link to integration documentation on our website

Diagnostics information

Example YAML snippet

Anything in the logs that might be useful for us?

Additional information

probot-home-assistant bot commented Jul 4, 2022

probot-home-assistant bot commented Jul 4, 2022

henryptung commented Jul 4, 2022 • edited Loading

henryptung commented Jul 4, 2022 • edited Loading

synthead commented Jul 6, 2022

synthead commented Jul 6, 2022

henryptung commented Jul 10, 2022 • edited Loading

henryptung commented Jul 10, 2022 • edited Loading

henryptung commented Jul 4, 2022 •

edited

Loading

henryptung commented Jul 4, 2022 •

edited

Loading

henryptung commented Jul 4, 2022 •

edited

Loading

henryptung commented Jul 10, 2022 •

edited

Loading

henryptung commented Jul 10, 2022 •

edited

Loading