Support Pipfile

[gsemet]

I prefer using Pipfile over requirements.txt. Please support this file format to set up the virtual environment.

[RichardLitt]

Thanks @stibbons. Contributions are always welcome, of course, if you'd like to submit a PR for this.

[agjohnson]

I started playing with support for pipenv, it looks like it might be a drop in replacement for calls to pip install. We'll probably want to offer the option through our readthedocs.yml file, so this will take:

• Conditional call to pipenv in doc_builder/python_environments.py
• readthedocs.yml support in our readthedocs-build repo
• readthedocs.yml support in this repo

[gsemet]

yes, it is a straigh replacement of pip (actually, a wrapper), you do pipenv install thispackage and it installs the package in the virtualenv (using pew) and modify Pipfile accordingly. For ReadTheDocs, I suggest you do pipenv install --dev and ensure the lib is installed in the environment. I usually use -e . to inject my lib under developement into the virtualenv, but I am not sure if everybody does it

[tuukkamustonen]

You should actually do pipenv install --dev --ignore-pipfile. Without --ignore-pipfile, pipenv will just seek to upgrade the packages in Pipfile and not really do deterministic build. See https://github.com/kennethreitz/pipenv/issues/954#issuecomment-338777638.

[gsemet]

indeed

[agjohnson]

I'm already using pipenv for other projects, so have some familiarity. I believe for our case, we actually want --system as well, to utilize our existing virtualenvs. We have special storage for this, so that venvs persist across builds.

@tuukkamustonen this confuses me a little, but I've only recently started with pipenv and pipfile. What does ignoring the pipfile mean for projects that don't check in their pipfile.lock? It doesn't seem checking in pipfile.lock is a requirement.

I suppose the most correct is check for pipfile.lock, if so, --ignore-pipfile, otherwise don't?

Some of these options will likely be added/allowed override through our readthedocs.yml, --dev is likely one of those.

In the case of a requirements.txt, we should be able to just use pipenv install instead of pip install

[gsemet]

Both side is ok. When you freeze the requirements.txt (ex using pip-tools), it act like when you use the lock file.
When you only describe the version by range or without version, it act like Pipfile.
The lock file is specific to the machine, it does not have the markers (ex « python_version < 3 »), but both should work.
I would use ‘pipenv install —dev —system’, it is like ´pip install -r requirements.txt -r requirements-dev.txt’

[tuukkamustonen]

@agjohnson I am new to pipenv as well, but as I understand it, if you run just pipenv install it will NOT follow your Pipfile.lock, but instead look into Pipfile, and upgrade any packages that you haven't locked into exact version, and then update Pipfile.lock. So it's like pip-compile and pip-sync (from pip-tools package) combined. I think it's a bit confusing.

I suppose the most correct is check for pipfile.lock, if so, --ignore-pipfile, otherwise don't?

Yeah, if we are after deterministic builds, then the lock-file needs to be followed and --ignore-pipfile given. I don't know how it would make sense otherwise. And then there is pipenv update, that uninstalls/installs packages like pip-sync but it doesn't have --ignore-pipfile option... hmm.

The lock file is specific to the machine, it does not have the markers (ex « python_version < 3 »), but both should work.

This is a good reminder. Neither pip-tools or pipenv (which uses pip-tools underneath) support "universal" lock files. So if you create the lockfile under python 3.6 on Windows and then run it on 3.4 on Linux, things may break, because lockfile expects 3.6/Windows environment. See https://github.com/kennethreitz/pipenv/issues/857 and jazzband/pip-tools#563.

What does ignoring the pipfile mean for projects that don't check in their pipfile.lock

Those projects won't get deterministic builds. It's ok, but should be documented. One of the main benefits with pipenv is to get deterministic builds (improvement over simple requirements.txt), but as you say, it's not a requirement. User should just acknowledge that.

In the case of a requirements.txt, we should be able to just use pipenv install instead of pip install

I don't think pipenv reads requirements.txt-style files. In case there is no Pipfile why not just resort to old behavior?

[agjohnson]

This is a good reminder. Neither pip-tools or pipenv (which uses pip-tools underneath) support "universal" lock files. So if you create the lockfile under python 3.6 on Windows and then run it on 3.4 on Linux, things may break, because lockfile expects 3.6/Windows environment. See kennethreitz/pipenv#857 and jazzband/pip-tools#563.

Ah yes, so it seems that ignoring the Pipfile and relying on Pipfile.lock should be an option, but is off by default. Without universal lock files, we can't guarantee that the environment is the same OS or even python version as our build images.

I don't think pipenv reads requirements.txt-style files. In case there is no Pipfile why not just resort to old behavior?

It looks like if a Pipfile is missing and a requirements.txt is available, pipenv install will automatically translate this to a Pipfile. Likewise, if the file is named foo.txt, pipenv install -r foo.txt will pick up the arbitrarily named file. These are the two modes we need to support on RTD.

[gsemet]

It does not change from requirements.txt, it may or may not be reproductible, depending on the work of the programmer... I would not bother, simply use the --dev, because most of the time ReadTheDoc is triggered at the same time than travis.

Later when universal lock file will be settled this can be adapted to Rtd, but this is not as "important" than in the travis build (because the artefact is the doc, not the actual wheel), and we dont have this ability in the travis.

Or, simpler, you let the developer write its install line (pip install -r requirements.txt, or pipenv install --dev), it is the more versatile solution.

[sobolevn]

What's the status of this issue? Maybe there is something I can help with?
Thanks!

[DmytroLitvinov]

Hey guys.
Are there any news?

[agjohnson]

No updates. Feel free to take this work on if you want.

[Tobotimus]

Hey guys, firstly, thanks for all the work you guys do on this project 😃

I've taken an interest in this issue, and I'd just like to note a few things down about how we expect people will use this feature, and which use cases we should accommodate for. People often get confused about how pipenv should be used, and thus I expect it will be difficult to please everyone with this feature.

I am going to use the information provided in this section of pipenv's documentation as the main point of reference. Accommodating for the use cases and guidelines listed there should satisfy most developers.

RTD's installation process

So currently, there are three stages to the installation process after the virtualenv is created:

1. Installing RTD's core requirements
2. Installing user requirements from a requirements.txt file with pip (optional stage)
3. Installing the package itself with pip install . or python setup.py install --force (also optional)

How pipenv fits into these stages

Installation of pipenv itself should be done in the first stage. I think this should be conditional based on the YAML config.

Now, some (or perhaps even "most") usages of pipenv would actually combine stages two and three, as many projects have -e . or some equivalent specification in their Pipfile (and subsequently their Pipfile.lock, which would also contain sub-dependencies of the package).

However, as specified in the documentation linked above, Pipfiles are also capable of managing dependencies by themselves without relying on the project's install_requires. In this case, pipenv would be used to complete stage 2, and pip install . or python setup.py install for stage 3, if required.

The simplest and most malleable approach I can think of is doing pipenv install {opts} as an additional stage between 2 and 3. Since stages 2 and 3 are already optional, this gives the user lots of freedom for how they'd like to set up the environment, without confusing the user with mutually exclusive configuration options. The documentation would obviously need to be quite clear on how these various configuration options can be used together.

Arguments to pipenv install

I think in the interest of giving the user freedom and also future-proofing the configuration for any new pipenv install flags, we should simply allow the user to specify the flags they'd like to use as a string in the YAML, however we should have --system set as a persistent flag to ensure pipenv doesn't create another virtualenv. Using pipenv to manage virtual environments is unnecessary as we already have a lovely robust design and API for setting up virtual environments with virtualenv, and pipenv's virtual environment management is more of an end-user QoL feature.

I think --ignore-pipfile is unnecessary as the default behaviour is to install from Pipfile.lock.

However, in any case, I think we ought not to specify packages or requirements files as arguments to pipenv install, since that is used for adding dependencies to a Pipfile and not deterministically recreating a development environment.

Config layout

So given the above points, I think my approach would be to add these options to the YAML:

pipenv_install: True|False
pipenv_install_opts: OPTIONS

Edit: On second though perhaps having these options as sub-keys below a pipenv key would be preferable.

The reason I'm not including it in the python key is because I think it should be on the same level as the requirements_file key, which this is most similar to.

Summary of installation with pipenv

1. Install RTD's core requirements with pip, including pipenv if pipenv_install is True
2. Optionally install requirements from a requirements.txt file with pip
3. Optionally do pipenv install --system {opts}
4. Optionally do either pip install .[extras] or python setup.py install --force, or neither

Please let me know if you have a different view and/or point out improvements which could be made 😃

[stsewd]

@Tobotimus thanks, that's a great summary of the use case of pipenv, currently, we are doing a v2 of the configuration file https://github.com/orgs/rtfd/projects/2, I think this should be there.

[Tobotimus]

@stsewd Ah I see! Great. Unfortunately that link is a 404 for me, and I can't see any projects on rtfd's org page, however I did find the YAML file completion milestone, which I'm happy to help out with when I get some time 😄 I am a new here so still somewhat stumbling around.

I've opened up #4254 as a patch to the old config, but I am more than happy to refactor it according to any new spec. Is there a feature branch of some sort for the new config? Thanks!

[stsewd]

We are moving the rtd-build repo to the readthedocs.org repo #4242, there are some ideas here https://docs.readthedocs.io/en/latest/design/yaml-file.html and this is kind of the base schema we have for the v2 https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/rtd_tests/fixtures/spec/v2/schema.yml. There are some design decisions to make.

[Tobotimus]

Thanks very much, I suppose my approach from #4254 could easily adapt to that schema by simply placing the pipenv section under the python key. For reference, here is the patch I prepared for the rtd-build repo. I suppose I will wait for #4242 to be merged and for more progress to be made on the V2 config parser before continuing with pipenv PRs 😃

[agjohnson]

There is more work to finish up the porting process, so it would probably be best to wait for that process to be completed first. I suppose we can consider readthedocs_build frozen for now.

Once things are ported, you can rework your PR to be housed here.

For now, a separate PR that updates our v2 schema spec file would be the best way to start. You'll find tests and the schema here:
https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/rtd_tests/fixtures/spec/v2/schema.yml
https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/rtd_tests/tests/test_build_config.py

This would be the best place to describe your changes and rationale for schema changes.

The immediate feedback I'd have is that I don't like exposing a pipenv.options, we don't do this for any other tools, we rather support specific options. I agree that this should all be under a python.pipenv key, but perhaps we also need to change python.install to support pipen instead of pip.

Edit: cc @Tobotimus, also 🎉 for the contribution! Sorry your PR landing right in the middle of a large project

[petersmithca]

Is this still in the plans? My work uses pipenv exclusively. And without its support, i need to build some hack to make things work

[honzajavorek]

@stsewd You previously mentioned we should comment with projects running on pipenv. I maintain https://github.com/pyvec/docs.pyvec.org and https://github.com/honzajavorek/cojeapi.

Both currently require developers of the docs to run pipenv lock --requirements > requirements.txt every time they mess with dependencies. Although we have CI to check that this file isn't out of sync, it is annoying to work with. I'm unsure whether there's a better workaround at this moment - I'd be glad if someone has a suggestion which would lead to a smoother workflow.

(If anyone is interested in the linter, see this script. Works correctly with dependabot.)

[stsewd]

Hi everyone, sorry for the silence here. We are trying to decide if support pipenv or not.

• One thing is that pipenv isn't (or going to) adopting PEP 517 [question] What is the recommended best practice for Pipenv and apps on Pypi? pypa/pipenv#2787 (comment) like other tools like poetry or flit did (this allows us to just call pip and don't implement more options in the config file).
• Last release of pipenv was one year ago https://pypi.org/project/pipenv/#history, we are not sure about what direction the project is going to take.
• Pipenv is slow when resolving deps, we already have the same problem with conda. But conda did an improvement recently, so pipenv could have improvements in that area over time too.
• We may try to support pipenv in an experimental way (we already have a PR kind of ready with pipenv support), so we can remove/change it without any guaranty.

Part of those decisions is that if we implement pipenv is something that we'll need to maintain for ever. Please, let us know your opinions.

[Peque]

I personally stopped using Pipenv (still using Conda and Poetry). I think going with PEP 517 may be the best choice.

[brianmay]

Pipenv is still the best supported option. Editors like pycharm and spacemacs supports pipenv. Dependabot supports pipenv. The only tool that I use that doesn't support pipenv is readthedocs.

I have no idea what will happen in the future. The fact pipenv hasn't been updated in over a year doesn't look good for pipenv. I see poetry[1] and flit[2] are actively maintained. Pip itself might be improved.

However right now, I think pipenv, like it or not is the best option, and even if it does get replaced in the future I suspect it will be used by legacy projects for a long time.

Personally, the sooner the Python community picks the preferred solution to solve the problems in requirements.txt based pip installs, the better. Otherwise tools such as readthedocs may all have to support a number of differing solutions as everything picks a different solution for their projects.

Notes:

[1] I tried poetry recently and found it didn't have a good way of migrating from an existing requirements.txt based system - however if poetry was the way of the future this could be easily fixed. poetry+dependencies is not particularly lightweight - which is important when installing on the fly for CI runs.

[2] Not familiar with flt - I guess I should have a look.

[petersmithca]

In that case, is there any plans to support poetry? Currently I cannot use the product with Pipenv or Poetry unless I hack the repo locally, and then of course its overwritten on every update.

…

On Tue, Nov 26, 2019 at 2:04 PM Santos Gallegos ***@***.***> wrote: Hi everyone, sorry for the silence here. We are trying to decide if support pipenv or not. - One thing is that pipenv isn't (or going to) adopting PEP 517 pypa/pipenv#2787 (comment) <pypa/pipenv#2787 (comment)> like other tools like poetry or flit did (this allows us to just call pip and don't implement more options in the config file). - Last release of pipenv was one year ago https://pypi.org/project/pipenv/#history, we are not sure about what direction the project is going to take. - Pipenv is slow when resolving deps, we already have the same problem with conda. But conda did an improvement recently, so pipenv could have improvements in that area over time too. - We may try to support pipenv in an experimental way (we already have a PR kind of ready with pipenv support), so we can remove/change it without any guaranty. Part of those decisions is that if we implement pipenv is something that we'll need to maintain for ever. Please, let us know your opinions. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3181?email_source=notifications&email_token=ABGUL7JZGO2MZLY7WMZJ6X3QVVQJJA5CNFSM4EAMFA62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFG5UDI#issuecomment-558750221>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGUL7JU6HHROEZP6IWAE3TQVVQJJANCNFSM4EAMFA6Q> .

[stsewd]

In that case, is there any plans to support poetry?

@petersmithca You should be able to use poetry now https://poetry.eustace.io/docs/pyproject/#poetry-and-pep-517

[petersmithca]

I meant will read the docs work with poetry to build my documentation.

…

On Wed, Nov 27, 2019 at 11:18 AM Santos Gallegos ***@***.***> wrote: In that case, is there any plans to support poetry? You should be able to use poetry now https://poetry.eustace.io/docs/pyproject/#poetry-and-pep-517 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3181?email_source=notifications&email_token=ABGUL7P7B5C5HSU4DDSMDNTQV2FTBA5CNFSM4EAMFA62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFJ2NGY#issuecomment-559130267>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGUL7KI4LUTHYUITCB7YYLQV2FTBANCNFSM4EAMFA6Q> .

[stsewd]

I meant will read the docs work with poetry to build my documentation.

Yes, we don't call poetry directly, but it's called with pip (that's PEP 517)

[petersmithca]

oh ok. cool thank you

…

On Wed, Nov 27, 2019 at 11:22 AM Santos Gallegos ***@***.***> wrote: I meant will read the docs work with poetry to build my documentation. Yes, we don't call poetry directly, but it's called with pip (that's PEP 517) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3181?email_source=notifications&email_token=ABGUL7M2WA2SR2JL54AIIG3QV2GEPA5CNFSM4EAMFA62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFJ25CA#issuecomment-559132296>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGUL7OXN4AK63UAIL7TO3LQV2GEPANCNFSM4EAMFA6Q> .

[zbeekman]

My understanding is that pip is going to start supporting the Pipfile and Pipfile.lock formats at some point in the future. Either way, I find dependency management with Pipenv to be more intuitive and reliable than with pip, and I like the built in venv.

[brianmay]

@zbeekman Do you have a source for that understanding?

[agjohnson]

Thanks for all the input so far, everyone. This is helpful to have.

Otherwise tools such as readthedocs may all have to support a number of differing solutions as everything picks a different solution for their projects

I feel the same way to a certain degree. RTD might be overly opinionated about process and tooling, but if we're looking to support the Python community at-large, we probably should have more flexibility in our workflow.

One option I've raised, and this option is not being actively considered at this moment, is to implement pipenv support as an "experimental" feature. That would mean that pipenv is not supported by core team (we can't help with support requests), and pipenv support could go away (though we'd deprecate with warnings).

Normally I'd probably vote as hard pass on pipenv support at this point, but this is a highly requested feature still -- even given some of the arguments against pipenv.

The question that I think core team needs more answers to, if you are a pipenv user, is "why not use poetry?". Our decisions on pipenv might be further skewed because we're not users of either tool in daily development, so poetry feels synonymous with pipenv -- only RTD actually supports poetry.

[honzajavorek]

One of the reasons I started to use pipenv (and not poetry) back then was wider support for Pipfile in integrations - e.g. dependabot, GitHub (security warnings), Snyk, Heroku, or now.sh support Pipfile for specifying dependencies alongside requirements.txt, while support for pyproject.toml and poetry is still rare. Last time I checked I noticed dependabot now supports it, but I doubt about the others. RTD was actually the only place where I had to figure out hacks for Pipfile, and where poetry is supported sooner.

I don’t know which tool wins the users in the end, but to me it feels like people desperately wanted to have something like pip + virtualenv and when pipenv appeared, they jumped on using it. Poetry wasn’t so well known and when I learned about it, I felt like it’s distraction from finally having one tool to rule the ecosystem. Later people realized pipenv’s flaws and started going back to pip or looking for alternatives, discovering poetry. I feel like this process is still happening. The tooling catches up with noticing poetry, but pipenv was mass-adopted earlier and is supported “everywhere” already.

Today, like many, I see pipenv’s limitations and I think of migrating to poetry, but I won’t do it until the tooling I use catches up.

[astrojuanlu]

The lack of activity on this issue + the recent (from ~2018 onwards) governance and maintenance problems of Pipenv + lack of standardization around Pipfiles + discussion in pypa/packaging.python.org#912, pypa/packaging.python.org#701, and related issues make me think that the interest for Pipfile support on Read the Docs has greatly decreased and that it might be a dead end. In particular, I copy-paste a fragment I wrote on pypa/packaging.python.org#912:

• Pipenv relies on pypa/pipfile, a format that advertises itself as "a replacement for the existing standard pip's requirements.txt file". However, the README has a big WARNING: This project is under active design and development. Nothing is set in stone at this point in time., and the experimental nature of the spec is not in line with the privileged position of Pipenv in the tool recommendations. (This might of course be a problem with the README instead.) On the other hand, the proposal to make pip adopt it has been frozen for almost two years, and in fact, it hasn't received much attention, so it is to be expected that, for the short and medium-term at least, Pipenv will be the only tool to support this format.

I think it is very unlikely that pip gains support of Pipfiles. In any case, even if I'm wrong about Pipfile long-term success, I think it's fair to say that we won't implement support for it in the short nor medium-term.

For "alternative" package managers, Poetry has much better support, is actively developed, it's engaging with the PyPA to bring forward new standards (like PEP 621 and others), and supports generating requirements.txt files when necessary. In fact, I am working on improving our documentation on how Read the Docs supports Poetry.

For "classical" approaches, pip has improved a lot, and pip-tools provides a lightweight way to freeze application requirements using existing formats (metadata files, requirements.txt files).

The community is working hard to achieve consensus on the next-generation lock files. Discussions started in February 2019, then PEP 665 was proposed, next we had PEP 665, take 2 a couple of weeks ago, and the discussion is still ongoing.

Whatever comes out of this, we will probably strive for supporting existing standards, rather than specific tools that do their own thing.

Therefore, I am going to go ahead and close this.