pip-compile: extras_require is ignored if it differs to a previously specified requirement

[jkp]

Steps to replicate

• In requirements.in specify that you want to install dependency X in editable mode.
• Run pip-compile to generate requirements.txt
• In requirements.dev.in include requirements.txt as the first requirement and then specify that you want to install dependency X in editable mode with extras (e.g dev).
• Run pip-compile requirements.dev.in to generate requirements.dev.txt

See the minimal test case here for a full example of the bug.

Expected result

requirements.dev.txt contains the extra dependencies specified in package's X extras_require - [dev] section.

Actual result

The packages are missing.

[atugushev]

Hello @jkp,

Thank you for the detailed report (especially for the repo so nicely) and good catch! There is a bug indeed with editable packages. I'll take a look shortly.

[atugushev]

Actually, it's not a bug. This is the way how the pip works. It might be related to the installation order, since pip-tools parses requirements via pip internal api.

For example:

$ cat requirements.in
-e git+https://github.com/jschneier/django-storages.git#egg=django-storages

$ cat requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile -v
#
-e git+https://github.com/jschneier/django-storages.git#egg=django-storages
django==2.1.7
pytz==2018.9              # via django

Look how the order is important to pip install. First case - django-storages will be installed:

$ cat requirements.dev.in
-r requirements.txt
-e git+https://github.com/jschneier/django-storages.git#egg=django-storages[boto3]

$ pip install -r requirements.dev.in
Obtaining django-storages from git+https://github.com/jschneier/django-storages.git#egg=django-storages (from -r requirements.txt (line 7))
  Cloning https://github.com/jschneier/django-storages.git to ./.venv/src/django-storages
Collecting django==2.1.7 (from -r requirements.txt (line 8))
  Using cached https://files.pythonhosted.org/packages/c7/87/fbd666c4f87591ae25b7bb374298e8629816e87193c4099d3608ef11fab9/Django-2.1.7-py3-none-any.whl
Collecting pytz==2018.9 (from -r requirements.txt (line 9))
  Using cached https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl
Installing collected packages: pytz, django, django-storages
  Running setup.py develop for django-storages
Successfully installed django-2.1.7 django-storages pytz-2018.9

Second case - django-storages[boto3] will be installed:

$ cat requirements.dev.in
-e git+https://github.com/jschneier/django-storages.git#egg=django-storages[boto3]
-r requirements.txt

$ pip install -r requirements.dev.in
Obtaining django-storages[boto3] from git+https://github.com/jschneier/django-storages.git#egg=django-storages[boto3] (from -r requirements.dev.in (line 1))
  Updating ./.venv/src/django-storages clone
Requirement already satisfied: django==2.1.7 in ./.venv/lib/python3.6/site-packages (from -r requirements.txt (line 8)) (2.1.7)
Requirement already satisfied: pytz==2018.9 in ./.venv/lib/python3.6/site-packages (from -r requirements.txt (line 9)) (2018.9)
Collecting boto3>=1.4.4 (from django-storages[boto3]->-r requirements.dev.in (line 1))
  Using cached https://files.pythonhosted.org/packages/22/7d/d558ed7f0d916f912cfab65104375e1d7e41972c97284afe43bb6a53dd3c/boto3-1.9.114-py2.py3-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from boto3>=1.4.4->django-storages[boto3]->-r requirements.dev.in (line 1))
  Using cached https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Collecting botocore<1.13.0,>=1.12.114 (from boto3>=1.4.4->django-storages[boto3]->-r requirements.dev.in (line 1))
  Using cached https://files.pythonhosted.org/packages/63/f6/a3e18fd0b15a57efad4c2f20de4fad0a3abbdc9a60d7033af71375d0b39c/botocore-1.12.114-py2.py3-none-any.whl
Collecting s3transfer<0.3.0,>=0.2.0 (from boto3>=1.4.4->django-storages[boto3]->-r requirements.dev.in (line 1))
  Using cached https://files.pythonhosted.org/packages/d7/de/5737f602e22073ecbded7a0c590707085e154e32b68d86545dcc31004c02/s3transfer-0.2.0-py2.py3-none-any.whl
Collecting docutils>=0.10 (from botocore<1.13.0,>=1.12.114->boto3>=1.4.4->django-storages[boto3]->-r requirements.dev.in (line 1))
  Using cached https://files.pythonhosted.org/packages/36/fa/08e9e6e0e3cbd1d362c3bbee8d01d0aedb2155c4ac112b19ef3cae8eed8d/docutils-0.14-py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore<1.13.0,>=1.12.114->boto3>=1.4.4->django-storages[boto3]->-r requirements.dev.in (line 1))
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Requirement already satisfied: urllib3<1.25,>=1.20; python_version >= "3.4" in ./.venv/lib/python3.6/site-packages (from botocore<1.13.0,>=1.12.114->boto3>=1.4.4->django-storages[boto3]->-r requirements.dev.in (line 1)) (1.24.1)
Requirement already satisfied: six>=1.5 in ./.venv/lib/python3.6/site-packages (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore<1.13.0,>=1.12.114->boto3>=1.4.4->django-storages[boto3]->-r requirements.dev.in (line 1)) (1.12.0)
Installing collected packages: jmespath, docutils, python-dateutil, botocore, s3transfer, boto3, django-storages
  Found existing installation: django-storages 1.7.1
    Uninstalling django-storages-1.7.1:
      Successfully uninstalled django-storages-1.7.1
  Running setup.py develop for django-storages
Successfully installed boto3-1.9.114 botocore-1.12.114 django-storages docutils-0.14 jmespath-0.9.4 python-dateutil-2.8.0 s3transfer-0.2.0

[jkp]

OK! So basically put the new requirement first and it should solve the issue...nice. I will try that!

[atugushev]

That’s right!

[atugushev]

I'll close the issue then. Feel free to reopen if you still have an issue with it.