Coordinate releases of pip and pipenv

[Froskekongen]

Environment

• pip version: 18.1
• pipenv version: 2018.7.1
• Python version: 3.6.6
• OS: Ubuntu 18.04

Description
On at least two occasions, pipenv and pip has had independent releases, breaking pipenv environments, automated builds, etc...

I suggest that pypa tries to coordinate releases of pip and pipenv to avoid such happenings. The easiest way to do this would perhaps involve a testing regime that entails pip always testing vs pipenv and vice-versa for any new prospective releases.

Expected behavior
pipenv install pip installs pip in the pipenv environment.

How to Reproduce

This instance of

1. pip install -U pipenv pip
2. pipenv install pip
3. An error occurs: TypeError: 'module' object is not callable

Reference
pypa/pipenv#2924

[benoit-pierre]

pip has not supported public API, pipenv import some internal modules from pip, clearly the onus is on pipenv's developers to keep up with pip's latest developments (running some tests with pip master?).

[Froskekongen]

For this to be practical, the tests have to be bilateral. Is there a rationale to keep pip entirely independent from other pypa projects? In the case of pipenv, there should be some communication, stopping such incidents from happening.

[jmcs]

@benoit-pierre notice that from an outside perspective what you said translates to

clearly the onus is on pypa to keep up with pypa

I know the projects are run independently, but it would be nice to not have this kind of breakage between pypa projects.

[benoit-pierre]

No, because, again, there's no supported public API! Importing from pip._internal is not supported (it really ever was), if you do it, you're on your own: it's on you to keep up with pip developments.

[Froskekongen]

Whether it is supported at this time or not is beside the point.

Maybe a subset of that API should be supported in order to have smooth transitions between versions of pip and pipenv?

[benoit-pierre]

It's not new: https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html

[jmcs]

Importing from pip._internal is not supported (it really ever was), if you do it, you're on your own: it's on you to keep up with pip developments.

This was nice and fair if both projects weren't hosted in PyPA 's organization. At some point pip's and pipenv's developers need to make sure they are not stepping on each other's shoes or the PyPA needs to make it clear that pipenv is not and will not be an "official" PyPA project and move it elsewhere.

Saying you don't have a public API is legit, what is not legit is having one project in a organization breaking because of a change in another project in the same organization and the anwser be "we never talked about APIs and the other former project is not my concern".

As the situation stands we have to assume that at any single moment pip might even make breaking changes that prevent pipenv from working altogether.

[cjerdonek]

Out of curiosity, why do issues like this happen even when pipenv vendors pip?
https://github.com/pypa/pipenv/tree/d65038ecc99945baa133467e6d80884268a82c34/pipenv/patched/notpip

[di]

@cjerdonek it's due to a third-party dependency that pipenv uses that doesn't use the vendored pip: sarugaku/pip-shims#12

[pradyunsg]

I do understand that this is a perception issue. Perhaps the folks at pipenv can keep an eye on the "release XX.X" issues (like #5782, #5516)?

/cc @techalchemy @uranusjr

[techalchemy]

@pradyunsg My basic understanding was that pip switched over to a 3 month release cadence, I just wasn't tracking that this was 3 months already. I don't feel this is on you guys at all, although continuously telling users that 'its their problem' is obviously creating a sense that we aren't communicating, even when we are, since we are having a bunch of discussions about how to break our reliance on the internal API going forward.

@di we actually maintain all of those libraries, that is a separate organization that we made to put all of our tooling/libraries in as we sort through some of these issues. pip-shims is one that I maintain

@cjerdonek there were two causes for this one --

• pip-shims has compatibility shims for pip 9 through current, because I got tired of rewriting the same import shims everywhere
• It also has an override environment variable for specifying the import path, PIP_SHIMS_BASE_MODULE, which was not being set properly somewhere along the way
• Additionally, I mistakenly didn't add pip's master branch to CI for that library as I did for pip-tools -- I now have CI set up to run daily against the master branch: see here

So typically I should have some advanced notice even if I don't hear in any actual communication from one of the maintainers over here.

Given the vendoring situation there really is no case where a pip release should immediately have any impact on pipenv, that's the whole point of vendoring -- pip-shims really just needed to be pointed at the right pip module to avoid all of this.

[di]

@di we actually maintain all of those libraries, that is a separate organization that we made to put all of our tooling/libraries in as we sort through some of these issues. pip-shims is one that I maintain

To be clear, by "we" you mean "the pipenv maintainers", not "the PyPA", which was who @jmcs was originally criticizing.

I was just trying to point out that the responsibility for fixing this lies with that non-PyPA group (even if there is some overlap).

[pradyunsg]

I don't feel this is on you guys at all, although continuously telling users that 'its their problem' is obviously creating a sense that we aren't communicating

Agreed.

I think both the cases that OP is referring were mostly results of oversight on someone's part rather than due to lack of communication (or hostility). OTOH, I'm not sure how exactly we could be cleanly communicating the not-so-straightforward relationship between pip and pipenv, to end users who stumble upon such bugs.

[techalchemy]

To be clear, by "we" you mean "the pipenv maintainers", not "the PyPA", which was who @jmcs was originally criticizing.

Considering that the group is run exclusively by pipenv maintainers the distinction likely seems like an attempt to just dodge the issue; we could have (and previously did) put the code directly into pipenv... would that change the conversation?

[pradyunsg]

pip's and pipenv's developers need to make sure they are not stepping on each other's shoes

pipenv is using pip's internals (for a variety of valid reasons) and pipenv has taken the responsibility of making sure they track pip's internal changes, hence the initial response you got.

(insert failed attempt at changing the "stepping on each other's shoes analogy here" to fit the actual situation here)

As you have pointed out, this isn't a good situation for us to be in and the maintainers of both projects agree; it can result in things breaking for users of pipenv when pip makes internal changes as it has. This isn't a good look when they both are under the same organization: PyPA.

This is why the maintainers of pip and pipenv are working together to create underlying tooling/libraries so that both projects (and any current/future packaging tooling) can use the same backing implementations for common operations that these projects need to do.

The effort to do this involves working together on reducing technical debt in pip's codebase, lots of use-case/implementation/API design discussions for the future underlying "shared" libraries and more (I'd rather not digress here), while considering current and potential future use cases. All this will take time, especially given that a lot of this is done on volunteered time, with AFAIK little/no financial incentive to do this work for anyone involved.

As the situation stands we have to assume that at any single moment pip might even make breaking changes that prevent pipenv from working altogether.

I can understand how you have reached this conclusion.

I think the past 2 pip releases brought out the need for a process improvement in the short term, to catch the kinds of issues that can show up in pipenv due to internal changes in pip. @techalchemy has since made such an improvement in pipenv's CI (as he outlined above) which should be able to catch them early.

Things should be better on this front moving forward.

[pradyunsg]

Closing this as nothing-immediately-actionable-for-now. The work on the shared libraries has been happening, slowly but surely, and there isn't much benefit to keeping this issue open for now IMO.

Thanks everyone who participated here! ^>^