Install globally with --user

[raxod502]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

I am using Poetry to install project dependencies into a Docker container where I do not have root access. Since using a virtualenv would be unnecessary (there is only ever one project in this container) and a waste of time (this is done as part of a web service aiming at speed), I configure settings.virtualenvs.create to false. However, this causes Poetry to try to install the packages system-globally, which fails because I don't have root access in the container.

I would like a way to tell Poetry to use pip install --user instead of just pip install to install packages when virtualenvs are disabled.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Closing this issue automatically because it has not had any activity since it has been marked as stale. If you think it is still relevant and should be addressed, feel free to open a new one.

[richin13]

I picked up this issue and started working on an implementation

Can somebody re-open the issue please? cc: @sdispater

[finswimmer]

Hey @richin13 ,

thanks a lot for your contribution! I reopened this issue and assigned you for seeing that this is in progress.

As this is a new feature @sdispater has to decide if it gets included. But it looks like a useful thing to me 👍

fin swimmer

[richin13]

Awesome, thank you @finswimmer

[abn]

Shouldn't the following just work?

pip install --user /path/to/poetry-project

This should result in your project being built in an isolated environment without development requirements and pip installing the built package respecting the --user flag.

[harrybiddle]

@richin13 did your submission go anywhere or was it ignored? Currently my workaround is:

pip install --user --requirement <(poetry export --format requirements.txt)

However, this doesn't give me the project itself installed into the user.

I also tried setting PIP_USER=1 hoping that it would get passed through poetry down to the underlying pip command, but no joy.

[abn]

@harrybiddle why not let pip install the project itself as suggested above?

@raxod502 @richin13 I beleive this behaviour is dependent on the underlying pip version (ie. the pip provided in your container). This often times (if ensured via python -m ensure pip) tends to be older versions.I beleive since 20.0, pip defaults to installing to user site when not root. Trying a pip install --upgrade pip prior to doing poetry install might do the trick here.

[harrybiddle]

Hey @abn, thanks for the help, I really appreciate it.

I just gave this a go and it worked, except for two things: (1) my dev dependencies didn't get installed, and (2) I think your command installs the dependencies in the pyproject.toml rather than the pinned dependencies in poetry.lock, is that correct?

I couldn't see an option to pip install to pick up dev dependencies from pyproject.toml, and I played around with its --constraint option, but could not get it to work. In the end I came up with this command (pip 20.2.2, poetry 1.0.10):

pip install --requirement <(poetry export --dev --format requirements.txt)
pip install --no-deps .

This works where poetry config virtualenvs.create false && poetry install fails on Ubuntu 18.04 (it fails uninstalling system packages).

[tbrlpld]

@raxod502 What is the reason you don't want to install the dependencies "system wide" on the container? It would just require to do the USER switch after the dependencies have been installed.

[raxod502]

At the time, it was because I was working with a system where the dependencies were installed at runtime (in response to incoming requests), not as part of the Dockerfile build. Therefore, any solution that required modifying the base image wasn't ideal.

I believe it would be possible to work around the problem by creating a virtualenv in the base image and hardcoding some environment variables (or initializing them as part of the ENTRYPOINT script), but these solutions just didn't seem as elegant as installing with --user, which is already supported directly by Pip.

[tbrlpld]

Oh interesting.

[kristjanvalur]

Found this here issue when looking for the equivalent of poetry install --user.
The reason is that I am working in a VSCode development container, and want to avoid the unnecessary fuss of creating a virtualenv, thus poetry config virtualenv.create false. using sudo poetry install doesn't feel quite right.

It would even be better to just configure this, somehow, so that my development container could have these settings here:

poetry config virtualenv.create false
poetry config install.user true

or similar.

It's perfectly possible to create a virtualenv inside the development container, but this is really quite redundant, because in most cases, there is only ever the single environment inside it. This saves a bunch of steps and indirections.

[neersighted]

Please see #6398 -- virtual environments are not really a layer of 'fuss', but the way you keep your Python peas and potatoes from touching. They're closer to a node_modules folder or similar in the Python world.

I'm mostly of the opinion that installing to site.getuserbase() is basically the same as a virtual environment, but with site-packages leaking in. The only advantage is not having to prefix commands in the container with the virtual environment, but there are plenty of other patterns that avoid that (see #6397 (comment) for instance).

This is difficult/unlikely to be supported in the medium term as we are planning to migrate from pip to performing installs ourselves -- and thus PIP_USER and similar will not work with the new installer. It's possible we could support this in the new installer, but I find that this might be useful for completeness (there are situations where virtualenvs.create false is probably what you do want, but they are rare), but it's mostly a footgun that few people will find real value in (mostly, it will cause them headaches).

[kristjanvalur]

Thanks for your comment.
When working with containers, turning off virtualenvs is sometimes definitely what one wants. there are no separate peas and potatoes, I control all python packages installed. For production containers, making them as tight as possible is sometimes preferable, hence instaling without virtualenv is what one regularly does.
This is also not a problem when building containers, because one can simply use the proper UID when doing so.
It is slightly less convenient when working in a development container because then one currently has to sudo.

Containers can provide exactly the same kind of application isolation as virtualenvs do, and so using one within the other is often quite superfluous.

Of course, Poetry is great with virtualenvs and dependency management and thus our preferred tool. People on mac/linux, for example, won't use the dev container, but just the virtualenv. People on Windows may elect to use a dev container, because that gives them a linux environment. And when building production containers, we install directly into the root.

No big deal, just wanted to throw this out there :)

[neersighted]

I'm still asking the question of why -- you don't control every Python package in the root, as you are subject to packages that come from the operating system. If your base image does something odd, you will be exposed to packages outside your control in the system prefix. This is not a hypothetical -- Ubuntu ships Python code in dist-packages with version numbers incompatible with standard tooling, causing Poetry to choke on their version numbers.

It's not superfluous to isolate Poetry from your code, and your code from the operating system -- you of course are free to do what you want, but it's an incredible burden to try to support every quirky patch a linux distribution ships, when a virtual environment prevents 90% of their upstream-incompatible changes to Python from affecting your project. I have to stress that if you give up letting Poetry manage your environment exclusively (by having Poetry operate in a virtual environment that only it installs packages to), you do so at your own risk and with breakage being likely.

[kristjanvalur]

I've been a core developer of CPython since long before virtualenvs were invented and so I´m well aware of their utility.
But as I say, containers can solve the isolation problem just as well. A typical deployment container's baseline is very bare bones and comes without any of the baggage of a distro. A Python container contains no python components other than the standard lib.

As a developer, I elect to install my locked dependencies in to the root of the deployment container, because there is nothing to isolate them from, and there is no virtualenv to initialize at runtime. And as explained, this already is trivial to do, by disabling virtualenv generation for Poetry and using the uid of root, as one does during Dockerfile execution.

As a developer, I also control the contents of my development container. It also is very slim and has no distro baggage. I'd like the simplicity of installing my locked dependencies directly and not into a virtualenv. It is a safety belt, a tool that I might elect not to use, as an adult, in the safety of my own dev container.

But allow me to reiterate, it is no fuss. I was simply assuming that it was possible and looking for the knob to turn, because already, the option is in place not create a virtualenv, and so it looks like an obvious extension: If not using a virtualenv, install into the user folder rather than the root. Poetry is an awesome tool even if it doesn't respond to every whim of a cranky old developer. Cheers!

[neersighted]

It's technically been there on and off (as we have used pip to implement installation -- in 1.2 I don't think just setting PIP_USER will work however), but we plan to migrate to build + installer longer-term and thus will be unlikely to re-implement --user ourselves unless there is a very compelling reason.

[kristjanvalur]

Right. Didn't realize there was anything to implement, as such, since the user folder has been part of Python since before importlib (IIRC). And also, didn't know that it was even possible to go without pip. But never mind, Python packaging has always been its own incredibly mysterious and complex part of the ecosystem, one that I have steered (mostly) clear of. Keep up the good work!

[nottrobin]

I'm still asking the question of why -- you don't control every Python package in the root, as you are subject to packages that come from the operating system. If your base image does something odd, you will be exposed to packages outside your control in the system prefix. This is not a hypothetical -- Ubuntu ships Python code in dist-packages with version numbers incompatible with standard tooling, causing Poetry to choke on their version numbers.

It's not superfluous to isolate Poetry from your code, and your code from the operating system -- you of course are free to do what you want, but it's an incredible burden to try to support every quirky patch a linux distribution ships, when a virtual environment prevents 90% of their upstream-incompatible changes to Python from affecting your project. I have to stress that if you give up letting Poetry manage your environment exclusively (by having Poetry operate in a virtual environment that only it installs packages to), you do so at your own risk and with breakage being likely.

I came to this bug looking for a solution for the following scenario. Although as I'll get to, it turns out I think we can actually get away without --user in my case.

We create Docker images of our Python websites to push to Kubernetes. We naturally want these images to be as lean as possible so deployment and any dynamic replacement done by Kubernetes itself are as quick as possible. We'd also like them to be as straightforwardly implemented as possible so they're easier to reason about and fix. Nonetheless, it would be a fantastic improvement for us to be able to benefit from Poetry's lockfile functionality.

I accept that using a Python virtualenv isn't much of an overhead. But in this case is should genuinely be unnecessary, as we start from a very stripped down base-layer version of Ubuntu, which doesn't even have Python installed, and then we explicitly choose what goes in.

We currently use pip install --user in a build step within our Dockerfile, so that we can encapsulate all the site's dependencies in /root/.local/lib/ for copying into our final image. So we were hoping to simply do poetry install --user to replicate this same functionality.

However it turns out that we don't really need to do that because in fact the actual dpkg system modules get installed into /usr/lib/, whereas poetry install (with virtualenvs.create false) will install them into /usr/local/lib/. Hence in our case we can simply use (in this case) /usr/local/lib/python3.10/dist-packages/ to encapsulate our dependencies.

[kristjanvalur]

Yes. When building deployment images, this is exactly that which we do ourselves. It's easy because at build-time we are UID==0.
My use case was more about development time, when you have a similarly clean container to work in, but you don't want to be root when installing dependencies, and may want to skip the step of activating a virtualenv. No biggie, but would be nice :)

[Matesanz]

I think it would be useful to reconsider this issue in order to enhance the relationship between poetry and devcontainers as stated by #1214 (comment). 🙏

[sryabkov]

There is another use case where this feature would be useful: extending a pre-existing image where pip install was done to the user location. An example is adding custom code to the stock airflow image.

[DougPlumley]

I think it would be useful to reconsider this issue in order to enhance the relationship between poetry and devcontainers as stated by #1214 (comment). 🙏

I landed in this issue for this exact reason, most of my devcontainers I run as a non-privileged user, but for poetry managed projects I'm finding I need to run as root so I don't need to jump through a bunch of hoops just to run poetry install.

[hberntsen]

For a devcontainer you could pre-create the virtual environment that will be used. I.e.:

ENV VIRTUAL_ENV="/opt/venv"
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
RUN python3 -m venv $VIRTUAL_ENV && \
  # Not strictly required, just to set PS1 and a deactivate function for root and users created after this command
  echo ". ${VIRTUAL_ENV}/bin/activate" >> /etc/bash.bashrc && \
  echo ". ${VIRTUAL_ENV}/bin/activate" >> /etc/skel/.bashrc

And ensure that your user in the container has write access to $VIRTUAL_ENV. Poetry commands after this will use the venv

[fjmacagno]

Seconded, this would be a nice feature to have, I also want to be able to extend an existing pre-built python installation at runtime so that i do not have to reinstall all 400+ packages into a new virtualenv.

[YodaEmbedding]

My use-case is a bit more mundane. I just want to have a global poetry.lock managed installation for throwaway scripts or a REPL for rough calculations/experimentation which is not associated with a specific project.

---

Using poetry to manage --user site-packages

Create a directory for your poetry project:

mkdir -p ~/.config/pypoetry/envs/default
cd ~/.config/pypoetry/envs/default
poetry init

Symlink the resulting virtualenv, prepend to PATH, and set up a convenient alias:

PYVER=3.11

rm -rf ~/.local/lib/python$PYVER
ln -s ~/.cache/pypoetry/virtualenvs/default-*-py$PYVER/lib/python$PYVER ~/.local/lib/python$PYVER
ln -s ~/.cache/pypoetry/virtualenvs/default-*-py$PYVER/bin ~/.local/bin_python

echo 'export PATH="$HOME/.local/bin_python:$PATH"' >> ~/.bashrc
echo "alias poetry-user='poetry --directory ~/.config/pypoetry/envs/default'" >> ~/.bashrc

And now, instead of poetry --user, just do poetry-user:

poetry-user add pyyaml

[EgorBlagov]

I think I've found a workaround:

just create a venv in same location as user's .local

RUN python3 -m venv $HOME/.local \
&& source $HOME/.local/bin/activate \
&& poetry install

[abn]

@sryabkov @DougPlumley @fjmacagno out of curiosity, is there a reason why https://python-poetry.org/docs/configuration/#virtualenvsoptionssystem-site-packages does not work for your scenario? In this case, poetry will only install the packages that it needs to override or if the version available in the system environment does not satisfy the locked versions. And that is the reason why this option was added in the first place. If you have a container example I can tinker with to understand the issue better - airflow or devcontainer scenario, I would maybe offer specific resolutions or better understand how we can improve the poetry features.

And as for when running as an unprivileged user, poetry should not (when using the new installer) try to install to the root site. I will detect the user site (or the venv site) and install it there. This is determined by the property below.

poetry/src/poetry/utils/env/site_packages.py

Lines 61 to 71 in cff4d7d

	def writable_candidates(self) -> list[Path]:
	if self._writable_candidates is not None:
	return self._writable_candidates
	self._writable_candidates = []
	for candidate in self._candidates:
	if not is_dir_writable(path=candidate, create=True):
	continue
	self._writable_candidates.append(candidate)
	return self._writable_candidates

[DougPlumley]

@sryabkov @DougPlumley @fjmacagno out of curiosity, is there a reason why https://python-poetry.org/docs/configuration/#virtualenvsoptionssystem-site-packages does not work for your scenario? In this case, poetry will only install the packages that it needs to override or if the version available in the system environment does not satisfy the locked versions. And that is the reason why this option was added in the first place. If you have a container example I can tinker with to understand the issue better - airflow or devcontainer scenario, I would maybe offer specific resolutions or better understand how we can improve the poetry features.

And as for when running as an unprivileged user, poetry should not (when using the new installer) try to install to the root site. I will detect the user site (or the venv site) and install it there. This is determined by the property below.

poetry/src/poetry/utils/env/site_packages.py

Lines 61 to 71 in cff4d7d

	def writable_candidates(self) -> list[Path]:
	if self._writable_candidates is not None:
	return self._writable_candidates
	self._writable_candidates = []
	for candidate in self._candidates:
	if not is_dir_writable(path=candidate, create=True):
	continue
	self._writable_candidates.append(candidate)
	return self._writable_candidates

Hi @abn, it might?

I took an example project where I ran into the permission issues and trimmed it down to this https://github.com/DougPlumley/poetry-devcontainer. Hopefully that provides some context.

If I recall correctly the reason I was using poetry config virtualenvs.create false was because VS Code/Pylance was not gracefully handling which Python virtualenv to use, so static code analysis wasn't happening as expected and running code selections wasn't happening as expected.

It could be there is an easy workaround that would have solved that, I used poetry config virtualenvs.create false and opted to just run as root for the development environment.

[gemmell]

I'm in a similar boat as other users: vscode devcontainer with the vscode user, and either I have to use sudo with poetry config virtualenvs.create false or I have to use the virtualenv and contend with vscode trying to deal with both a system python and a venv python. I am leaning towards the former, because for other users of the development environment who might not be as savvy (but are quantum physicists, so still very smart), I want the development environment to "just work" without them having to select python interpreters and mess about with venvs. That's sort of the point of the devcontainers IMO. A packaged/one click development environment. Admittedly, the only thing that's really missing for me is a feature in vscode to suppress the system python so it's not an option to use it as the interpreter.

[gemmell]

As a work around I have a script make_packages_available.sh that looks like this:

#!/bin/bash

set -e

# Check if poetry's virtualenvs.create is false, which means it'll be trying to install into system packages
if [[ $(poetry config virtualenvs.create) == "false" ]]; then
    command_prefix="sudo"
else
    command_prefix=""
fi

${command_prefix} poetry install

The other thing that confused me was that because you're doing sudo it's using the poetry config of the root user. So whilst the poetry config virtualenvs.create of the current user might be false, as soon as you do sudo you might find that root has virtualenvs.create set to true and it goes and creates a venv for the root user... So now in my Dockerfile I have to set virtualenvs.create false for both vscode and root.

In summary, whilst it's possible to use poetry in a devcontainer, I'm having to do a lot of fiddle work to get it to work nicely, and it would entirely be solved if I could just do poetry config virtualenvs.create false && poetry config install.user true.