Providing pip configuration in sys.base_prefix

[pelson]

pip version

main

Python version

all

OS

all

Additional information

When a pip config is installed in an installation prefix $PREFIX/pip.conf, $PREFIX/bin/python -m pip correctly picks up the config. When one makes a virtual environment with $PREFIX/bin/python -m venv ./my-venv then ./my-venv/bin/pip does not pick up the config.

It is questionable if this is a bug or a feature request, but essentially, I believe that pip should be looking in sys.base_prefix as well as sys.prefix for a config file.

Description

No response

Expected behavior

No response

How to Reproduce

Starting with a non-virtual environment (e.g. a conda environment):

$ touch ./env/pip.conf
$ pip config debug
env_var:
env:
global:
  /etc/xdg/xdg-ubuntu/pip/pip.conf, exists: False
  /etc/xdg/pip/pip.conf, exists: False
  /etc/pip.conf, exists: False
site:
  /media/important/github/pypa/pip/env/pip.conf, exists: True
user:
  /home/pelson/.pip/pip.conf, exists: False
  /home/pelson/.config/pip/pip.conf, exists: False

$ python -m venv ./venv

## BEWARE THAT YOUR VENV HAS THE BUNDLED PIP, SO INSTALL A NEWER PIP FOR DEBUGGING
$ ./venv/bin/pip install -e /path/to/pip/repo

$ ./venv/bin/pip config debug
env_var:
env:
global:
  /etc/xdg/xdg-ubuntu/pip/pip.conf, exists: False
  /etc/xdg/pip/pip.conf, exists: False
  /etc/pip.conf, exists: False
site:
  /media/important/github/pypa/pip/venv/pip.conf, exists: False
user:
  /home/pelson/.pip/pip.conf, exists: False
  /home/pelson/.config/pip/pip.conf, exists: False

What I want to see for a venv is that the base_prefix is searched as well as the prefix.

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct

[pradyunsg]

Hi! Thanks for filing this issue and PR!

Could you please elaborate on why you want this? What use case does this help with?

[uranusjr]

I think this is by design? A virtual environment is supposed to isolated things from the global environment, so configuration from sys.base_prefix should not be picked up.

[pelson]

Could you please elaborate on why you want this? What use case does this help with?

Sure, I'll try with my specific case, and then hopefully we can draw parallels elsewhere.

I an the responsible for a Python distribution to a controlled network. There are other Python distributions on the controlled network (both accessible from the same machine) for which I am not responsible, and which need specific pip configurations. I too have a specific pip configuration (e.g. index-url), I therefore configure them in the pip.conf of the distribution. It is designed that users (developers) take virtual environments from this distribution in order to extend it as needed - pip is a fundamental part of that work flow to allow them to extend as required. I therefore want those users to have the appropriate pip configuration within their virtual environments (out of the box).

To re-iterate, it is not acceptable to have the config be global (e.g. in /etc/pip.conf) nor user enabled (e.g. in ~/pip.conf) as they need to also be able to use another Python distribution with different pip configurations (i.e. the pip config needs to be isolated to the environment).

Environment variables defining the config aren't acceptable because the two Python distributions wouldn't be able to inter-operate (you can only have one PIP_INDEX_URL.

To me the idea that a virtual environment should inherit the pip configuration from the base environment is natural. After all, a virtual environment inherits the Python and its standard library from the base environment - if you change the config of how Python is built such config propagates/leaks to the virtual environment.

[pfmoore]

IMO, the only sort of environment that should inherit is a --use-system-site-packages one, as that's explicitly not isolated from the system environment. Certainly changing the default behaviour to inherit would be a significant backward incompatibility, and would likely cause problems for people who rely on global settings not leaking into a virtualenv (e.g., testing).

[pelson]

Thank you for your input so far.

would likely cause problems for people who rely on global settings not leaking into a virtualenv (e.g., testing)

We should try to be more precise with the terminology here. global settings do leak into a virtualenv:

$ cat /etc/pip.conf 
[global]
no-cache-dir = true

$ cat ./venv/pip.conf 
[global]
no-dependencies = yes

$ pip config list
global.no-cache-dir='true'
global.no-dependencies='yes'

What I'm suggesting is, just like Python config (e.g. compile flags) leaks into a virtual environment, so too should the pip config. i.e. site configuration should leak into a virtual env.

To turn this discussion around a little, perhaps I could ask you how you would control a virtual environment's pip configuration if global, user, and environment settings are not acceptable (because (a) a pip config doesn't apply to all Python installations on a machine, (b) it isn't a user configuration to define how a Python installation should interact with the distribution)?

I'm trying to not be hypothetical here, so my concrete scenario:

• 1 machine
• No internet, but internal repositories exist
• 2 completely different Python distributions
• Each python distribution has different pip configuration requirements (e.g. index urls)
• Users use virtual environments to give them extensibility

The problem occurs fairly quickly as users take a virtual environment from one of the distributions and they immediately have no pip configuration. Even worse, they try to use a PEP517 tool such as build to create a wheel python -m build ./ using the correctly (pip) configured Python distribution, but because the build isolation creates a new virtual environment the pip isn't configured and you get an error.

[piotr-dobrogost]

@pelson
I have to admit the scenario you brought up is interesting. :)

To turn this discussion around a little, perhaps I could ask you how you would control a virtual environment's pip configuration if global, user, and environment settings are not acceptable (because (a) a pip config doesn't apply to all Python installations on a machine, (b) it isn't a user configuration to define how a Python installation should interact with the distribution)?

After taking a quick look at docs of venv it looks like you could extend EnvBuilder class and define post_setup() method.

From https://docs.python.org/3/library/venv.html#venv.EnvBuilder.post_setup

post_setup(context)
A placeholder method which can be overridden in third party implementations to pre-install packages in the virtual environment or perform other post-creation steps. (emphasis mine)

The post-creation step in your case could be creation of appropriate pip configuration file in the newly created virtual environment.

[gaborbernat]

After taking a quick look at docs of venv it looks like you could extend EnvBuilder class and define post_setup() method.

That's my recommendation for using build too, to extend virtualenv via its plugins- pypa/build#270 (comment).

[pelson]

Thanks for the suggestion of extending EnvBuilder. Indeed this does work to automatically configure a venv with pip.conf, and I've been running with such a tweak for quite some time. Unfortunately the python -m venv and python -m build invocations aren't something that you can override in this way (without monkeypatching during Python startup 🙊).

Perhaps I could have been a bit clearer here: I'm not looking for a workaround per se (I have one, though I do appreciate ideas of other potential workaround solutions), I'm looking to address what I believe is a genuine shortfall (i.e. unexpected behaviour) in the way pip behaves with virtual environments.

That's my recommendation for using build too, to extend virtualenv via its plugins

I'm not using virtualenv, I'm using venv, the standard library way to make virtual environments. So too does build (https://github.com/pypa/build/blob/0.3.1/src/build/env.py#L188), no? There is no equivalent plugin system for venv.

the only sort of environment that should inherit is a --use-system-site-packages one

I disagree: I am not proposing that you inherit a single extra package with this change, merely that a locally (site) scoped pip.conf from a parent environment should be inherited in the child (virtual) one. I think considering configuration inheritance as the same as package inheritance (system site packages) would be a conflation of the --use-system-site-packages flag.

I'm trying to figure out real-world use cases where you wouldn't want to inherit the pip.conf from the parent environment. The one provided by @pfmoore "isolation testing" is a reasonable one, but I don't believe it should necessarily trump the one in which a user wants to create a virtual environment on a Python distribution which has been correctly pip configured and expect that their virtual env pip to "just work™".

In my PR (#9753) I implemented the easier approach of blending together the pip.conf in sys.base_prefix and sys.prefix (just like all of the other config search path items). It would totally solve my use case if, instead of doing a blend, it simply didn't look in sys.base_prefix if sys.prefix / pip.conf existed. This would solve the "isolated testing" usecase, as you would simply touch sys.prefix / pip.conf to avoid sys.base_prefix / pip.conf being considered. This may be best of both worlds - you can easily enough have isolated testing, but still get working behaviour out of the box.

[uranusjr]

build uses virtualenv if it’s available, and falls back to venv when it’s not.

[pfmoore]

"isolation testing" is a reasonable one, but I don't believe it should necessarily trump the one in which a user wants to create a virtual environment on a Python distribution which has been correctly pip configured and expect that their virtual env pip to "just work™".

Also, isolation is important from the point of view of bug triage - "can you reproduce your issue in an empyth virtualenv" is much easier than "can you reproduce your issue if you create a new virtualenv, hunt down and disable any config files you may have created in the past and forgotten about, ..."

But the real problem is that both are valid requirements. As a result, we're not discussing what the behaviour should be, but rather which behaviour should be default, and which should be opt-in.

Backward compatibility is a significant weight in favour of ignoring the site config by default.

IMO you've made some good arguments that your use case is worth considering, but your arguments are not strong enough to switch the default. If you want to continue arguing for your behaviour being the default, I suggest you focus on how to address the backward compatibility issue.

[layday]

I don't understand how #9753 is going to help with build inheriting a parent venv's configuration. When you create a venv from another venv, sys.base_prefix does not point to the parent venv's prefix; it always points to the installed Python's prefix. pip does not look inside sys.base_prefix I assume because (a) they would not want to encourage users to drop pip configuration directly in e.g. /usr and (b) because, in most cases, global configuration serves the same purpose.

[pelson]

Also, isolation is important from the point of view of bug triage - "can you reproduce your issue in an empyth virtualenv" is much easier than "can you reproduce your issue if you create a new virtualenv, hunt down and disable any config files you may have created in the past and forgotten about, ..."

I'm only semi-convinced about this one. You still have to deal with peoples environment variables, global config and user config. In reality you'd tell them to run pip config debug and, if it existed, perhaps tell users to PIP_DISABLE_CONFIG=1 for a truly clean config-free reproducer.

IMO you've made some good arguments that your use case is worth considering, but your arguments are not strong enough to switch the default.

Thank you for your insight and pointing out where I should focus the discussion.

But the real problem is that both are valid requirements. As a result, we're not discussing what the behaviour should be, but rather which behaviour should be default, and which should be opt-in.

If you want to continue arguing for your behaviour being the default, I suggest you focus on how to address the backward compatibility issue.

My proposal to read sys.base_prefix / pip.conf IFF sys.prefix doesn't exist is the easiest way to enable both behaviours conveniently, but it is indeed backwards incompatible.

The problem is that the two use cases are in conflict - so we need to be able to configure which behaviour we want. But in order to configure this using non-global config, we need to read the base_prefix's config...

Backward compatibility is a significant weight in favour of ignoring the site config by default.

The previous statement I made is the strongest I have in favour of changing the default. If we change the default, it is easy to then make an isolated virtual environment (touch sys.prefix / pip.conf) if you need it. The existing pip config debug is clear and continues to apply. If we don't change the default then it requires some more complexity in pip to decide if it should consider sys.base_prefix / pip.conf.

So let's take a look at what that might actually entail if we don't change the default:

We need some non-global, non-venv and non-envvar means to tell pip to blend sys.base_prefix / pip.conf with sys.prefix / pip.conf. So we could perhaps look at sys.base_prefix / pip.conf and look for a config item called, say use-base-pip-conf: true (default to false). If it is set, we include the rest of sys.base_prefix / pip.conf when building the pip configuration.

It isn't the nicest behaviour because we need to pre-read the config to figure if we want to read the rest of the config. pip config debug needs to become more nuanced ("the base config file exists, but we don't use it because it is not enabled"). This is the cost of keeping the "isolated by default" behaviour though. I can imagine this feature leading to a bit of confusion, if I'm honest.

Do you have different ideas about how we might be able to satisfy the two use cases without changing the default? Fundamentally I'm struggling because the "isolation" behaviour a subset of the "non-isolation" one (i.e. you can configure the "non-isolated by default" one to be isolated, but you can't configure the "isolated by default" one to be non-isolated).

[layday]

If we can step away from discussing the implementation for a minute, what's not been made clear through the course of this conversation is that this change would only be applicable to Conda environments, because Conda apparently mangles the value of sys.base_prefix of a Python installed in a Conda environment. This means that you can keep a development environment-local pip configuration in the "global" Python prefix, a peculiarity only found in Conda environments.

[piotr-dobrogost]

@layday
I'm not familiar with Conda but the problem seems perfectly valid in the realm of the standard Python. If I understand correctly the original issue was "Isolated venv does not inherit PIP index " (pypa/build#270). This has nothing to do with Conda. Actually I am surprised this issue has not come up way earlier. Just think of all these corporate environments when one is required to use specific pip index. It is natural to set this pip index as part of the base Python installation (often on a machine shared by many users) and hope it would propagate to virtual environments based on the base installation.
I am in favour of reevaluating what the isolation of virtual environment is supposed to mean. I guess the original intend was to isolate virtual environment from packages installed in the base installation. We should consider if isolation from the base installation's configuration was kind of an accident making more harm than good.

[layday]

I'd question whether this is a valid configuration vector outside of a Conda environment. The intention with loading configuration from {sys.prefix}/pip.conf was to allow configuring pip in a virtual environment (and extended to support Conda environments in #6268), i.e. from a development-local config file. Corporate entities can configure pip globally instead of on a Python prefix basis - sys.base_prefix can be shared by but it can also vary (!) for multiple Python installations on the same system. What this means in practice is that you can ensure neither that a pip config file located in sys.base_prefix applies to only one specific Python installation nor that it is global.

[pelson]

what's not been made clear through the course of this conversation is that this change would only be applicable to Conda environments

I don't think this conversation has anything to do with conda. Here is a clean build of Python:

$ wget https://www.python.org/ftp/python/3.9.4/Python-3.9.4.tgz
$ tar xzf Python-3.9.4.tgz 
$ ./Python-3.9.4/configure --prefix $(pwd)/py39
$ make
$ make install

With this:

$ ./py39/bin/python3 -m venv ./my-venv

$ ./py39/bin/python3
>>> import sys
>>> sys.base_prefix
'/home/pelson/Downloads/cpython_clean/py39'
>>> sys.prefix
'/home/pelson/Downloads/cpython_clean/py39'

$ ./my-venv/bin/python
>>> import sys
>>> sys.base_prefix
'/home/pelson/Downloads/cpython_clean/py39'
>>> sys.prefix
'/home/pelson/Downloads/cpython_clean/my-venv'

And the pip config:

$ touch ./py39/pip.conf
$ ./py39/bin/pip3 config debug
env_var:
env:
global:
  /etc/xdg/xdg-ubuntu/pip/pip.conf, exists: False
  /etc/xdg/pip/pip.conf, exists: False
  /etc/pip.conf, exists: True
    global.no-cache-dir: true
site:
  /home/pelson/Downloads/cpython_clean/py39/pip.conf, exists: True
user:
  /home/pelson/.pip/pip.conf, exists: False
  /home/pelson/.config/pip/pip.conf, exists: False

$ ./my-venv/bin/pip config debug
env_var:
env:
global:
  /etc/xdg/xdg-ubuntu/pip/pip.conf, exists: False
  /etc/xdg/pip/pip.conf, exists: False
  /etc/pip.conf, exists: True
    global.no-cache-dir: true
site:
  /home/pelson/Downloads/cpython_clean/my-venv/pip.conf, exists: False
user:
  /home/pelson/.pip/pip.conf, exists: False
  /home/pelson/.config/pip/pip.conf, exists: False

To "step away from the implementation", as you say, I want the pip config for the base environment to apply to the virtual environment one. sys.base_prefix is the documented way to identify "base environment" (https://docs.python.org/3/library/sys.html#sys.base_prefix).

Perhaps I missed something in what you were saying though.

---

To keep the conversation on track, the 3 proposals that have so far been discussed:

1. pip always looks at sys.base_prefix/pip.conf and blends it together with sys.prefix/pip.conf if they exist
2. pip looks at sys.base_prefix if-and-only-if sys.prefix/pip.conf doesn't exist
3. pip looks at sys.base_prefix/pip.conf to determine if it should consider the rest of sys.base_prefix/pip.conf to be blended with sys.prefix/pip.conf. By default this would be false so the current default behaviour would remain.

My preference is option 2 as it represents a far simpler implementation and is far less confusing to explain, and therefore less likely to generate support requests to pip. Unfortunately this is a breaking change as pointed out by @pfmoore. To get back to the old behaviour, one just needs to touch sys.prefix/pip.conf (or remove the sys.base_prefix/pip.conf!).

As far as I can see right now there is no other way for us to use non-global, non-user, non-environmental means to allow a venv to be configured out of the box - I have an open question to hear of other viable implementation options. To re-iterate, for the use case it is essential that a venv can be configured out of the box (e.g. isolated wheel building), it is completely acceptable that the base environment needs to be configured to do this (i.e. it isn't the default behaviour) but is not OK for it to be a post-venv creation step as this would preclude the use of venv using tools.

[layday]

It is relevant to Conda because only with Conda will you have a different base prefix for every development environment. Without this precondition there is no use case for base prefix dependent configuration as explained above. Sent with [ProtonMail](https://protonmail.com) Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

…

On Tuesday, 13 April 2021 15:06, Phil Elson ***@***.***> wrote: > what's not been made clear through the course of this conversation is that this change would only be applicable to Conda environments I don't think this conversation has anything to do with conda. Here is a clean build of Python: $ wget https://www.python.org/ftp/python/3.9.4/Python-3.9.4.tgz $ tar xzf Python-3.9.4.tgz $ ./Python-3.9.4/configure --prefix $(pwd)/py39 $ make $ make install With this: $ ./py39/bin/python3 -m venv ./my-venv $ ./py39/bin/python3 >>> import sys >>> sys.base_prefix '/home/pelson/Downloads/cpython_clean/py39' >>> sys.prefix '/home/pelson/Downloads/cpython_clean/py39' $ ./my-venv/bin/python >>> import sys >>> sys.base_prefix '/home/pelson/Downloads/cpython_clean/py39' >>> sys.prefix '/home/pelson/Downloads/cpython_clean/my-venv' And the pip config: $ touch ./py39/pip.conf $ ./py39/bin/pip3 config debug env_var: env: global: /etc/xdg/xdg-ubuntu/pip/pip.conf, exists: False /etc/xdg/pip/pip.conf, exists: False /etc/pip.conf, exists: True global.no-cache-dir: true site: /home/pelson/Downloads/cpython_clean/py39/pip.conf, exists: True user: /home/pelson/.pip/pip.conf, exists: False /home/pelson/.config/pip/pip.conf, exists: False $ ./my-venv/bin/pip config debug env_var: env: global: /etc/xdg/xdg-ubuntu/pip/pip.conf, exists: False /etc/xdg/pip/pip.conf, exists: False /etc/pip.conf, exists: True global.no-cache-dir: true site: /home/pelson/Downloads/cpython_clean/my-venv/pip.conf, exists: False user: /home/pelson/.pip/pip.conf, exists: False /home/pelson/.config/pip/pip.conf, exists: False To "step away from the implementation", as you say, I want the pip config for the base environment to apply to the virtual environment one. sys.base_prefix is the documented way to identify "base environment" (https://docs.python.org/3/library/sys.html#sys.base_prefix). Perhaps I missed something in what you were saying though. --------------------------------------------------------------- To keep the conversation on track, the 3 proposals that have so far been discussed: - pip always looks at sys.base_prefix/pip.conf and blends it together with sys.prefix/pip.conf if they exist - pip looks at sys.base_prefix if-and-only-if sys.prefix/pip.conf doesn't exist - pip looks at sys.base_prefix/pip.conf to determine if it should consider the rest of sys.base_prefix/pip.conf to be blended with sys.prefix/pip.conf. By default this would be false so the current default behaviour would remain. My preference is option 2 as it represents a far simpler implementation and is far less confusing to explain, and therefore less likely to generate support requests to pip. Unfortunately this is a breaking change as pointed out by ***@***.***(https://github.com/pfmoore). To get back to the old behaviour, one just needs to touch sys.prefix/pip.conf (or remove the sys.base_prefix/pip.conf!). As far as I can see right now there is no other way for us to use non-global, non-user, non-environmental means to allow a venv to be configured out of the box - I have an open question to hear of other viable implementation options. To re-iterate, for the use case it is essential that a venv can be configured out of the box (e.g. isolated wheel building), it is completely acceptable that the base environment needs to be configured to do this (i.e. it isn't the default behaviour) but is not OK for it to be a post-venv creation step as this would preclude the use of venv using tools. — You are receiving this because you were mentioned. Reply to this email directly, [view it on GitHub](#9752 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AHNRFWDIDUZZP3DB2HMTBGTTIQXVDANCNFSM42HKRNZA).

[piotr-dobrogost]

It is relevant to Conda because only with Conda will you have a different base prefix for every development environment.

No one is talking about different base prefix for every environment. The point is to have all virtual environments based on one, specific base Python installation to inherit pip configuration from this one, specific base Python installation. Also no one besides you mentioned Conda in any way.

Corporate entities can configure pip globally instead of on a Python prefix basis - sys.base_prefix can be shared by but it can also vary (!) for multiple Python installations on the same system.

That's exactly what OP stated:

There are other Python distributions on the controlled network (both accessible from the same machine) for which I am not responsible, and which need specific pip configurations.

As to:

What this means in practice is that you can ensure neither that a pip config file located in sys.base_prefix applies to only one specific Python installation nor that it is global.

Exactly the opposite is true. sys.base_prefix is unique per Python installation thus taking its pip's configuration into consideration in virtual environments allows all such environments to share common configuration tailored to the needs of this specific base Python installation.

[pfmoore]

To keep the conversation on track, the 3 proposals that have so far been discussed

... and to provide context, pip's current behaviour

• pip only looks at sys.prefix and never looks at sys.base_prefix.

This boils down to, pip looks at the currently active environment, the site configuration, and the user configuration. This is also simple to explain.

Do you have any examples of other software that looks up configuration from both sys.prefix and sys.base_prefix? I'm not aware of any.

🤷 I guess what it boils down to is that for me, the change is more difficult to describe than the current behaviour, it's not backward compatible, and it doesn't seem useful enough to justify the maintenance cost (without trying to be dismissive, your situation is clearly a fairly rare special case).

[layday]

Conda is mentioned in the bug report. I’ve explained why inheriting from a base prefix is problematic in other contexts. I’m going to drop out of this conversation now since we’re going round in circles and people are starting to forget their manners. Sent with [ProtonMail](https://protonmail.com) Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

…

On Tuesday, 13 April 2021 16:12, Piotr Dobrogost ***@***.***> wrote: > It is relevant to Conda because only with Conda will you have a different base prefix for every development environment. No one is talking about different base prefix for every environment. The point is to have all virtual environments based on one, specific base Python installation to inherit pip configuration from this one, specific base Python installation. Also no one besides you mentioned Conda in any way. — You are receiving this because you were mentioned. Reply to this email directly, [view it on GitHub](#9752 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AHNRFWD6ISAIFE5XWLYY6TLTIQ7NNANCNFSM42HKRNZA).

[pfmoore]

I currently represent 400+ scientific and engineering Python users, and have quite extensive experience of deploying Python to large scientific organisations who regularly have network isolated (internet-free) machines with internal package indexes (HPCs, critical op infrastructure, etc.). I'm not saying I'm correct or my opinion is worth more as a result, but I'm genuinely doing this in the interest of a significant user population who you would otherwise never see as they are behind an organisational firewall.

I really appreciate the fact that you've taken the time on this issue to represent that user base. As you say, it's extremely hard for us to find out about such use cases and the difficulties they face, and the input is really important to us. Please don't take the pushback you're receiving on this issue as a lack of interest in your use case. In particular, one reason I'm personally so involved in this issue is because I want to make sure we don't forget about these "hidden" use cases (I don't otherwise have much opinion on pip's config mechanisms 😉).

One question I asked above, which hasn't yet been answered:

Do you have any examples of other software that looks up configuration from both sys.prefix and sys.base_prefix? I'm not aware of any.

This is a key point - pip shouldn't be breaking new ground here, the problem our config system is addressing isn't unique to pip, so we shouldn't be implementing unique solutions. So I would expect you'd have similar problems with other software, and if you don't, I'd want pip to do things similarly to those tools, for consistency across the ecosystem if nothing else.

The PR I produced offered concrete evidence (#9753).

I apologise. I had somehow missed that you had provided this PR. It does indeed look OK to me, in terms of clearly explaining what you're proposing and demonstrating that the change is relatively straightforward at a technical level.

I personally think the complexity is low (please feel free to leave comments on the complexity in the PR if you disagree).

The amount of discussion and differing opinions here demonstrates (to me, at least) that this isn't the case. It may be simple in implementation terms, and maybe even in terms of explaining the technical implications ("you can now pick up config from sys.base_prefix as well") but in user terms, the fact that we're having this amount of discussion, suggests that calling the complexity "low" is missing a key point.

Regarding the chance of moving this forward, I still think it's mostly down to you. There have been a number of key questions/issues raised here. Maybe you feel that you have addressed them, but I think the lack of progress is essentially because the rest of the participants wouldn't agree...

You've said that the plugin mechanism for virtualenv solves the issue for that case, your problem is really only with the stdlib venv. That in itself limits the benefit of this change. And I'm not clear why, given that you control the Python installation, you don't patch the stdlib venv to fix the problem there. So what about this issue necessitates that it gets solved in pip? Is it just that you think the solution "fits better" in pip? If so, you haven't really made your point on that yet.

We currently have the following config locations in pip:

• global: system-wide configuration file, shared across users.
• user: per-user configuration file.
• site: per-environment configuration file; i.e. per-virtualenv.

You're proposing changing "site" to mean something very different (per-virtualenv plus inherited config from the base environment, I think). Could you explain how you'd re-word the description of "site" (without making it significantly longer than the other two!) to say what you intend it to now mean?

In my view, what you're after sounds more like you want to change "global". If so, that's something we inherit from platformdirs (where it's confusingly called "site" 🙁), so you should be proposing that change with them. I don't think we'd want to add a special case just for pip here.

If you want to add an extra level, then why? You've never (as far as I can see) said what precisely you want to put in this "shared base environment config". I can't think of anything that I'd want to set that would depend on which Python installation I used to create my virtual environment. If I have both your shared environment on my PC, plus a locally installed Python that's the same version, why would it matter which one I used to create a virtualenv? That's actually the biggest fundamental question here - what is the actual problem you're trying to solve here? Not the solution or approach you'd like to take, but what (in end-user terms) is the underlying problem?

I hope this helps. Sorry I can't give you a simple "yes, this seems like a good addition" or "no, sorry, we're not going to accept this", but I really don't think you've given us enough information about your requirement to let us make such a decision yet. I know you've tried, but hopefully the above helps you understand where there's still gaps in our understanding.

[pelson]

Maybe you feel that you have addressed them, but I think the lack of progress is essentially because the rest of the participants wouldn't agree...

Thank you for being explicit. This is also my feeling, and I appreciate having a concrete list of those things to focus on.

Unfortunately addressing those things is going to result in a long message - so apologies in advance. I will keep it as short as I can whilst addressing all that you have put to me.

You've said that the plugin mechanism for virtualenv solves the issue for that case, your problem is really only with the stdlib venv

In fact, right now I support venv (by patching it), but not virtualenv. Whilst virtualenv has hook support, I don't think it is any more easy to do by default than venv - the functionality to change the default seeder without explicit opt-in doesn't (yet) exist.

This isn't really about the tool though - I ideally want to support any tool that can produce virtual environments and which follows the standard sys.base_prefix and sys.prefix concepts. I'm advocating that pip knows how to handle virtual environments, not that the tools know how to make virtual environments with pip appropriately configured. The difference is subtle, but I would like to be able to create a virtual environment without pip, run python -m ensurepip in the venv and have a suitably configured pip as per the base, for example.

Is it just that you think the solution "fits better" in pip? If so, you haven't really made your point on that yet.

To be explicit: Yes exactly. That, and the fact that the tool used to create the virtual env isn't the important part - there are multiple valid ways to produce a virtual environment (a standard library concept), and it shouldn't be necessary to patch all tools which create them in order to get a working pip.

Do you have any examples of other software that looks up configuration from both sys.prefix and sys.base_prefix? I'm not aware of any.

pip shouldn't be breaking new ground here, the problem our config system is addressing isn't unique to pip, so we shouldn't be implementing unique solutions

I empathise with this point, but I disagree - IMO pip is special in this regard. pip is a fundamental part of an isolated virtual environment's subsequent population. There is no other library or tool which fundamentally needs to be appropriately configured immediately upon venv creation in order to allow tools to subsequently extend the newly created venv. To put this another way: for every other tool it is possible to manually configure a tool before running it (even if not ideal), but pip is so fundamental that tools (such as build, pre-commit) bake into their design the idea of "create venv + install packages" as a single step - there is no opportunity for users to configure pip manually in-between.

Finding something that does config like this is going to be tough as a result. Perhaps the most similar behaviour, whilst not using virtual environments precisely, is found in conda. Their config search path follows:

• CONDARC env var
• Currently active environment
• $HOME/
• Root environment (akin to base)
• System (/var/lib/, /etc/conda, C:/ProgramData/conda)

But finding other projects that read their config from sys.prefix, never mind sys.base_prefix, is challenging. Often it is based on $XDG_HOME, or /etc.

You're proposing changing "site" to mean something very different

I accept this directly. Both platformdirs and site.py (std-lib) have similar naming issues, IMO. The term "site" to mean virtual environment doesn't make much sense to me, since you tend to provide a "site-wide" (base) Python installation, rather than a site-wide virtual environment - but I am happy to fall into line on the terminology 👍.

Perhaps, given the conflation in many places, it is best to move away from the term "site" altogether. Python environment would be good enough to convey that it is coming from sys.prefix. I leave that to another discussion (on another issue) though. For now site is good enough to mean current Python environment.

In my view, what you're after sounds more like you want to change "global".

I'm explicitly not after this - I do not want to influence the pip.config of all Python invocations on a machine. I only want to influence those that use the Python installation I provide on a network drive - two reasons: the Python invocations aren't all happening on the same machine, and I don't want to touch what happens to other Python distributions running on that machine (platform managed Python, or other Python installations that may exist).

This latter point you've probed specifically about, so will talk about later on.

If you want to add an extra level, then why?

Given the naming confusion around site, I agree that I am fundamentally requesting a new level.

I think the motivation is coming through in my points above, but I believe that venv specific config should be relatively rare. I will try to highlight what I see as the advantages of each:

• Environment variable: Good for changing something quickly without writing a file
• Global: Multiple users on the same machine (in /etc) have pip configured 👍
• User: Same user on multiple machines (if homespace network mounted) have pip configured 👍
  Also good if you want to change a global setting, but don't have the permission to change global settings.
• Site (venv): Useful for testing pip.config and avoiding unwanted common config. Especially good if you have system python (owned by root) which isn't correctly pip configured, and want others to be able to be able to use pip from your venv (this sound fairly contrived).

The new level that I would put in between Global and Site:

• Base: Multiple users on multiple machines (if prefix network mounted) have pip configured 👍

(FWIW, I would have put Environment variable last, proceeded by User, as that would have been more consistent with PYTHONPATH, and would maximise the utility of the benefits I highlighted - that is a different discussion though 😉)

If I have both your shared environment on my PC, plus a locally installed Python that's the same version, why would it matter which one I used to create a virtualenv?

My shared environment doesn't support your arbitrary locally installed Python version. If you have problems with your Python version with my pip.conf, I am going to tell you to use my shared Python environment to see if you can reproduce it there. Concretely, my shared environment patches Python in certain ways, including overriding ensurepip - this ensures that when a new venv is created a reasonably up-to-date pip is installed out of the box. This was essential at a certain point in order to pick up the site config (which I think was added around pip 9), but now is a convenience and ensures that people are using the improved resolver and benefiting from all of the other improvements that come with modern pip.

I think your question is essentially asking: What is the use case for having multiple pip.config files on the same machine for the same Python version.

Hypothetically I could imaging wanting to differentiate the package index to use. For example, one could timestamp the index being used based on the Python distribution release date, in order to provide reproducible and snapshotted pip install calls - I don't do this, but it is something that my sys-admins and security team have requested, and a strategy that is employed for Linux distributions w. RPMs.

Perhaps a list of reasons that I can't use the existing config locations would be useful:

• Global: I don't control the machines upon which the Python is mounted. I can't touch /etc/. In most cases, nor can the user (it is managed by sys admins).
• User: It isn't my user running Python, it is others - not desirable to have each user configure their pip manually.
• Site: The virtual environments are created by the user. I use this location currently by patching venv creation.
• Env var: There is no activation step - the Python distribution works without activation, and not desirable to request each user set environment variables.

but what (in end-user terms) is the underlying problem?

Simply put, as a user:

• I don't have permission to change my global config.
• I use multiple Python distributions. Some in a restricted network zone.
• My homespace is mounted on all machines, no matter the zone.
• I don't want to deal with pip.config - I want to let the experts set it up for me so that when I am in the restricted zone, it uses a specific package index.
• When I'm not in a restricted zone, I can't use the restricted package config (e.g. no config, or one managed by another Python environment, is needed)
• There are multple Python distributions in the restricted zone - not all of them should be configured (mostly because they aren't supported). The security team insists that we only configure the Python distributions that are being maintained.

If you are partially convinced, I can revive the PR to see what it actually entails to implement the extra level being discussed. If not, I think I've probably over-spent on getting a return on investment already 😄. Either way, I appreciate that this is also very time consuming for you, and want to say thank you for taking the time to engage.

[pfmoore]

If you are partially convinced

I'm not convinced that your proposed solution is the right one, but I do see that you have a problem that isn't easily solved by existing pip functionality. But equally, I don't want to let you think that convincing me is all that's involved here. I'd like to see you get some feedback from other pip developers if we're to do anything with this. And assuming the other devs are receptive, I'd be wanting to explore options that don't have the problems that this approach has (which have been discussed at length above).

I hope that helps - I'll let you decide whether you want to pursue this further.

I might follow up with another response addressing some of your particular points. But please treat that as simply exploring the problem a little further - there's no need for you to respond to it.

[pradyunsg]

I've retitled this issue, to better reflect what is being asked for.

As useful context/background, I wrote pip config and wrote/refactored most of the configuration logic a few years ago (stating this to establish I'm reasonably familiar with it, and not to imply that I somehow "own" this piece within pip or have any sort of authority over this beyond what my fellow maintainers do).

I think the rationale provided here for adding a new level is reasonable, and I'm fine with adding an additional config file at base_prefix.

I will say this: this is a weird organisational situation you're in, where you can't/don't want to influence the global configuration (i.e. it's fine to have pip reach out to whatever is configured there, or the default PyPI) and you don't trust/want your users to configure their tools to work correctly (i.e. by documenting/educating about a standard config file for the use case). I'm not sure what pieces are specific choices by you vs being forced upon you. :)

Coming back: I don't think an additional config file layer is particularly that expensive for us in terms of the code + maintenance, and it feels like the right level of abstraction to solve this particular class of organisational/usage-pattern problems at -- the main concern I have is around (a) documenting this clearly+completely, and (b) a clean migration to this. Since it's effectively a new feature, (b) is less of a concern. (a) is something that will end up needing to be a part of the PR adding this. I'll guess that this is something that's mostly useful for organisational users, where there's some shared configuration for Python development that only applies to a subset of users (likely the package index?).

I'd be happy to review and (assuming no other maintainers are strongly opposed) merge a PR for this.

To err on the side of overcommunicating.... I'll explicitly state that I'm not promising that I'd take a look at such a PR quickly though -- trying to juggle quite a few things at the moment. I also haven't been looking at this thread closely due to limited bandwidth on my end. I've quickly skimmed through the initial bits of this thread and the most recent comments (sorry, there's way too much text here and I have limited mental energy right now) and that's the context I had when I wrote this. Sorry if I've missed some nuances/points that have been brought up already. 🙈

[pradyunsg]

I'll tentatively mark this as awaiting PR, to reflect that we'd review a PR for this and that's the next step forward here. I'll also note that I likely won't have the bandwidth to actually write this PR but likely will for reviewing one.

To my fellow maintainers, please feel free to remove that label if you think I applied that too eagerly. :)

[pelson]

We can close this out thanks to #11487 having been merged 🎉. Should be in pip 23.1 when it is released.

Thanks to all who contributed to the discussion. It was a long thread, over a long time span, but I believe we came out of it with a very worthwhile improvement, particularly for the case where a common base environment is supplied (e.g. by sys-admins) and where virtual environments are expected to be correctly configured from the base.

I personally will be happy to delete some ugly workarounds as a result of this 😄

[pfmoore]

This change caused issues in the 23.1 release (see #11982). As a result it is being reverted and a new PR will need to be submitted to re-implement the functionality, with a new design.

The design here was explicitly noted (above) as being backward incompatible, and indeed it was this incompatibility that caused the issues. Any replacement PR will need to either be fully backward compatible, or will need to have a suitable transition plan, including a deprecation period for the existing behaviour and a plan for what we will do if further compatibility issues occur.

Furthermore, this design breaks the isolation between virtual environments and their "base" environment - again, something that was noted in the discussion but which wasn't really addressed as a potential issue. Any replacement proposal needs to avoid doing this somehow. IMO, pip should not break the "virtual environments are isolated from their base environment" principle.

On a personal note, as a result of the issues, I'm now even less convinced that this is a feature that pip should accept. I understand there's a use case here, but there are other ways of addressing it. I'm very aware that "outsiders explaining how you can solve your problem" is generally more annoying than useful, but I would think that the following are possibilities to consider:

1. A wrapper for creating virtual environments that populates the environment's pip.ini appropriately. The venv module is explicitly designed for this sort of extension. Yes, I know that one constraint is that the site can't control how users create environmnts, but "do it this way or we can't support you" is often a reasonable compromise.
2. A global "Python startup" hook (implemented using one of the core Python mechanisms for such a thing) that checks the environment for a pip.ini and if there isn't one present, creates it. Again, this is using supported methods for customising your installation and therefore should not be a problem.

To be clear, my reason for preferring a solution using existing mechanisms is that if you take that route, you don't have to consider all of the other ways in which people can use (or abuse!) the existing mechanisms, and so you can implement a simpler, dedicated solution that doesn't have to take an "extreme defensiveness" position - which is what a new feature in pip has to take.

[pelson]

Firstly, thanks to @pfmoore for handling the fallout from this. I appreciate that it has caused grief and a lot of expended effort, so thank you.

my reason for preferring a solution using existing mechanisms is that if you take that route, you don't have to consider all of the other ways in which people can use (or abuse!) the existing mechanisms

To re-iterate what we already discussed (in admittedly very long thread above), I do have local solutions and am proposing this into pip because I believe it is a valuable requirement for providers of Python distributions who aren't shipping as a system Python.

The truth is: this situation is unfortunate and was not practically foreseeable. I believe the change that was introduced was reasonable given all that we knew, and to be entirely frank, I consider the fundamental problem here is that the Windows Store builds are configuring pip in such a way (I would be happy to elaborate if it is considered a contentious statement). That being said, we now know more than we did before, and that may influence our design going forwards.

In practice, the workaround to this issue is quite simple (you documented a number of them yourself - simply setting an environment variable, for example, addresses the problem). Would suitable and specific documentation of this breaking change, plus a transition guide not be considered sufficient in this case? ("if you wish to use Windows Store Python with pip and venv, you must configure pip in your virtual environment, else it has been pre-configured to install into --user").

Any replacement PR will need to either be fully backward compatible, or will need to have a suitable transition plan, including a deprecation period for the existing behaviour

The deprecation period would presumably warn if a base pip config file is detected, and proceed to ignore it until a future release of pip? As well as a future flag to acknowledge the ignore behaviour, and enable the new. This is something I could buy-in to, and would be willing to proceed with, if it is agreed that this is the desired approach.

[pfmoore]

The deprecation period would presumably warn if a base pip config file is detected, and proceed to ignore it until a future release of pip? As well as a future flag to acknowledge the ignore behaviour, and enable the new. This is something I could buy-in to, and would be willing to proceed with, if it is agreed that this is the desired approach.

No, that's not sufficient (IMO). While I agree that the Windows Store distribution's approach is problematic, it is a real-world situation we have to deal with. As such, the problem is that the end user has no way of addressing any such deprecation warning - they can't remove the "base" config file, nor can they be expected to switch Python distributions.

If you can persuade the Windows Store distribution to change their behaviour (see python/cpython#103646) then when they have done so and there's a reasonable expectation that the majority of their users have upgraded to a version that no longer has the config file, then we could try again with this approach. But that will be a long time happening, and even then I still object to the idea of base environment configuration "leaking" into derived virtual environments that don't have --use-system-site set.

I think what's needed here is a different approach that solves your use case without using an existing config location. And it should probably also block setting configuration values (like --user) that don't behave the same in "system" and virtual environments. Or something. Remember that I'm the person who has been consistently uncomfortable with this feature - I never approved it, I simply stopped objecting in the face of other pip maintainers being willing to support it. I'm definitely not the person you want to be advising on how it should be designed. I'm the person who will pick apart any design you propose 😉

[dalebrydon]

I think I tend to agree with @pfmoore, that this is a bit of a confusing change to me. I admit I never fully grasped why changes to other tools aren't the preferred method. Wouldn't something like a --copy-pip-config option in venv or virtualenv work too?

The big thing I don't like with the current design of pip config is that putting things in the Python install is treated as site config when not working in a virtualenv. To me things in the Python install are clearly global config but sort of get stuck as project-level config because we use sys.site for this. Note that even the comments in pip source refer to the site configuration as virtualenv config, so I think this is an accidental side effect more than intentional design. The base thing only extends this problem, which is why I would prefer some other solution.