Python 2 package is added to Python 3 dependency list using pip-compile #558

MHLut · 2017-09-05T13:22:52Z

Issue occurs in conjunction with django-auth-ldap.

django-auth-ldap requires different LDAP packages, depending on whether Python 2 or 3 is used. As defined in setup.py:

[...]

install_requires=[
    "django",
    "pyldap" if PY3 else "python-ldap >= 2.0",
],

[...]

When using pip-compile to compile a list of dependencies on a Python 3 environment, the Python 2-compatible package (python-ldap) is added to the list. This results in a broken application when all dependencies are installed through pip.

Environment Versions

Fedora 26 (4.12.9-300.fc26.x86_64 GNU/Linux)
Python version: 3.6.2
pip version: 9.0.1
pip-tools version: 1.9.0

Steps to replicate

Create a file called base.in and add at least the following dependencies (non-related packages have been omitted):

Django==1.10.6
django-auth-ldap==1.2.10
pyldap==2.4.28

[...]

Run the following command in a Python 3 environment to compile a list of dependencies:

pip-compile --output-file base.txt base.in

Note: The resulting file should now contain a line like python-ldap==2.4.42 # via django-auth-ldap.

Install packages through pip install -r /path/to/base.txt.

Expected result

I expect all packages to install without errors.

Actual result

The following error occurs:

Collecting python-ldap==2.4.42 (from -r /requirements/dev.txt (line 18))
  Downloading python-ldap-2.4.42.tar.gz (297kB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-32v49evs/python-ldap/setup.py", line 53
        print name + ': ' + cfg.get('_ldap', name)
                 ^
    SyntaxError: invalid syntax

Note here that my base.txt file is called dependencies.txt in the snippet above. This is because the file is renamed and used in a Docker environment.

When I remove the line python-ldap==2.4.42 # via django-auth-ldap from by dependency file manually, all packages are installed without any error.

The text was updated successfully, but these errors were encountered:

suutari-ai · 2017-09-05T14:19:00Z

Can't reproduce this (Ubuntu 16.04, Python 3.5.2, pip-tools 1.9.0). Are you sure that you compiled the requirement file with Python 3? Try removing the txt file and running pip-compile again in Python 3 venv.

vphilippon · 2017-09-05T14:40:14Z

Also, try with the --rebuild option to clear your pip-tools cache. If you happen to use both py2 and py3 on your system, you might get hit by the py2 dependencies being cached for django-auth-ldap.

In would actually make sense, as pip-tools caches the dependencies based on the package name and version, and django-auth-ldap provides 2 different wheel for py2 and py3, which each provides a different dependencies list.

@MHLut Please check with what @suutari-ai said, and then try clearing your cache, and tell us about the result. If you could save the verbose output of you different attempts (both failed and successful), that would be really helpful for us to pinpoint the exact cause for the future.

It's the 3rd time in a short while we get reports of this. I'd really like to know the exact cause to do something about it.

MHLut · 2017-09-05T14:54:25Z

@suutari-ai The file was compiled in a Python 3 virtualenv on a machine that never even has contained an installation of Python 2. I have removed the .txt file and recompiled as per your instructions and I still get the python-ldap package in the results.

@vphilippon I have run the command with a rebuild and verbose flag. The full results of my compilation can be read in the attached text file. Look at line 50 in the file to see the point where it's first (wrongly) asserted that there is a dependency on python-ldap.

issue558-pip-compile-verbose.txt

vphilippon · 2017-09-05T22:04:45Z

Can't reproduce either, using Python 3.6.2 in a Docker container.
For some reason I can't figure out, pip seems to download the Python 2 django-auth-ldap wheel.
This starts looking a lot like #552.

Could you try simply running pip install django-auth-ldap==1.2.10 to confirm it works?

MHLut · 2017-09-07T11:32:30Z

@vphilippon Installation of this package goes well on my local venv (a new P3.6) and inside my Docker container (P3.4).

$ pip install django-auth-ldap==1.2.10
Collecting django-auth-ldap==1.2.10
  Using cached django_auth_ldap-1.2.10-py3-none-any.whl
Requirement already satisfied: pyldap in ~/.virtualenvs/python3/lib/python3.6/site-packages (from django-auth-ldap==1.2.10)
Collecting django (from django-auth-ldap==1.2.10)
  Downloading Django-1.11.5-py2.py3-none-any.whl (6.9MB)
    100% |████████████████████████████████| 7.0MB 198kB/s 
Requirement already satisfied: setuptools in ~/.virtualenvs/python3/lib/python3.6/site-packages (from pyldap->django-auth-ldap==1.2.10)
Collecting pytz (from django->django-auth-ldap==1.2.10)
  Using cached pytz-2017.2-py2.py3-none-any.whl
Installing collected packages: pytz, django, django-auth-ldap
Successfully installed django-1.11.5 django-auth-ldap-1.2.10 pytz-2017.2

You are correct that this is very much like #552. I did try the solution presented in that issue, but this did not solve the problem for me. For now I have manually removed the dependency line from the .txt file. Will test this setup on another workstation later, to see whether the issue occurs there as well.

vphilippon · 2017-09-07T21:41:47Z

@MHLut I noticed your pip install ended up using a cached py3 wheel. Could you try doing the same with the --no-cache and --ignore-installed flags? I want to be 100% sure that pip won't go and grab the py2 wheel.

If pip-compile --rebuild with no .txt file, in the same venv as the succeeding pip install --no-cache --ignore-installed, grabs py2 dependencies, then when have some major issue.

MHLut · 2017-09-08T08:42:23Z

@vphilippon Both on my local Python 3.6 environment and Python 3.4 Docker container I get the same result when running the installation instructions. As you can see below, the Py3 wheel is used:

$ pip install django-auth-ldap==1.2.10 --no-cache --ignore-installed
Collecting django-auth-ldap==1.2.10
  Downloading django_auth_ldap-1.2.10-py3-none-any.whl
Collecting pyldap (from django-auth-ldap==1.2.10)
  Downloading pyldap-2.4.37.tar.gz (303kB)
    100% |████████████████████████████████| 307kB 10.8MB/s 
Collecting django (from django-auth-ldap==1.2.10)
  Downloading Django-1.11.5-py2.py3-none-any.whl (6.9MB)
    100% |████████████████████████████████| 7.0MB 21.8MB/s 
Collecting setuptools (from pyldap->django-auth-ldap==1.2.10)
  Downloading setuptools-36.4.0-py2.py3-none-any.whl (478kB)
    100% |████████████████████████████████| 481kB 13.2MB/s 
Collecting pytz (from django->django-auth-ldap==1.2.10)
  Downloading pytz-2017.2-py2.py3-none-any.whl (484kB)
    100% |████████████████████████████████| 491kB 18.3MB/s 
Installing collected packages: setuptools, pyldap, pytz, django, django-auth-ldap
  Running setup.py install for pyldap ... done

However, the pip-compile still creates a dependency file with the Py2 dependency added.

Note: I've just realised I neglected to mention that the dependency list also contains the Py3 package. So both packages mentioned in line "pyldap" if PY3 else "python-ldap >= 2.0", (see first post for full snippet) are included.

Attached to this comment is the output of $ pip-compile --output-file base.txt base.in --rebuild in a text file. This command was run on my local Py3.6 venv.

issue558-pip-compile-output-file.txt

vphilippon · 2017-09-08T15:46:48Z

pyldap comes directly from your base.in file (pyldap==2.4.28). It does not appear as required by django_auth_ldap in the computation. So pip-compile really only detects the Py2 dependency here (unless it reappears if you remove it from your base.in)

I used a Fedora 26 Docker image and updated to Python 3.6.2 to try and reproduce this, and I can't.
~~Only difference, I removed the mysqlclient==1.3.10 line as it failed (probably don't have the dependencies to build it).~~ (Edit: installed the dependencies and included it, still works)

I just realised that when using pip install, I told you to use --no-cache, which is nice, but didn't actually clear pip's cache (or at least, I'm not a 100% sure it did). You could try doing that.
It's likely to be around ~/.cache/pip.

suutari-ai · 2017-09-08T16:30:18Z

I'm also wondering if there could still be some part of Python 2 tool chain installed, e.g. pip or virtualenv. Another thing that could affect the results is if there is some environment variable affecting pip configuration or a pip configuration file present.

Can you copy-paste the output of these commands for us?

# activate the problematic virtualenv first
ls -l $VIRTUAL_ENV/bin
ls -l /usr/bin/{python,pip,virtualenv}*
head -n1 $VIRTUAL_ENV/bin/pip
head -n1 /usr/bin/pip
ls -l ~/.config/pip/
ls -l ~/.pip/
ls -l /etc/pip.conf
env | grep PIP_

Also it seems that if a new virtualenv is created with a -v option, then there is very good debugging info in the output. So it would also be very helpful, if you could create a new virtualenv with the -v (and every other option that you used previously) and copy-paste its output too.

suutari-ai · 2017-09-13T21:13:16Z

I just fixed a very similar problem with Prequ. See Prequ PR #12. It was caused by commit 4900c7c, which isn't part of pip-tools 1.9.0 though. (First pip-tools version which has it is 1.10.0.rc1.) Are you sure you're using pip-tools 1.9.0?

vphilippon · 2017-09-14T13:41:39Z

@suutari-ai I looked at the PR you linked and your analysis makes a lot of sense. But here's a thing: 1.10.0rc1 and 1.10.0rc2 are not really out yet: they were attempts for a release that failed as the Pypi upload was (and still is) disabled for all of Jazzband's projects.

So, unless someone is pointing to master or something, this shouldn't be out in the wild.

Maybe the issue is with a something related to hashes or platforms, but included in 1.9.0? IIRC, there was quite a bit of moving around these parts when releasing 1.9.0. Still, I think you've put the finger on something : pip-tools seems to be overriding some of pip's behaviour, which breaks my "we're not messing up with pip" assumption. I'm now sad.

suutari-ai · 2017-09-15T08:50:27Z

OK. I don't really know anything in the 1.9.0 code base that would cause similar behaviour and that's why I was guessing that this could be a newer code. Or maybe the cache was built with newer code? (If a broken pip-tools is used to compile incorrect dependencies, they will be cached and won't be recalculated until the cache is purged.)

Still, I think you've put the finger on something : pip-tools seems to be overriding some of pip's behaviour, which breaks my "we're not messing up with pip" assumption. I'm now sad.

That's one of the reasons why I maintain my own fork of pip-tools, Prequ. (Another reason is that there's no new releases.)

azaghal · 2017-09-28T10:17:42Z

Funnily enough, I am facing opposite problem after upgrading to pip-tools 1.10.x with the django-auth-ldap and Python 2. On my side pip-compile keeps pulling in pyldap insetad of python-ldap. My environment is:

Distribution: Gentoo x86_64
Python version: 2.7.12
pip version: 9.0.1
pip-tools version: 1.10.1
requirements.in:
```
 django-auth-ldap
```

For issue to be reproduced reliably I had to clear caches (via --rebuild option). As per what @suutari-ai said, after doing git bisect, the culprit is commit 4900c7c. However, when utilising Python 3 (3.4.5), I do get pyldap as requirement.

vphilippon · 2017-09-28T13:56:06Z

And I can reproduce too with python 2.7 on Windows. We need to check if the feature of generating hashes on all platform can be fixed an kept, or removed.

@jdufresne maybe you can give some input here.

azaghal · 2017-09-28T14:07:35Z

I wonder if it would be possible to somehow fake what the platform is instead? Doing a quick search (did not actually go through script in detail), there's some script that does something similar.

jdufresne · 2017-09-28T15:48:47Z

Can someone provide an easy to follow bash script to reproduce?

Here is what I'm running, but it works fine. I'm first clearing all caches. The requirements.txt contains only the platform dependencies. pip install -r requirements.txt executes successfully.

Here is my test script:

#!/bin/bash

set -x

cat requirements.in

rm -rf ~/.cache/pip
rm -rf ~/.cache/pip-tools
rm -f requirements.txt

rm -rf venv
python3 -m venv venv
venv/bin/pip install --upgrade pip setuptools
venv/bin/pip install pip-tools
venv/bin/pip-compile -o requirements.txt requirements.in

rm -rf venv
python3 -m venv venv
venv/bin/pip install --upgrade pip setuptools
venv/bin/pip install -r requirements.txt

Here is the output running this. Commands are prefixed with a "+":

+ cat requirements.in
django-auth-ldap
+ rm -rf /home/jon/.cache/pip
+ rm -rf /home/jon/.cache/pip-tools
+ rm -f requirements.txt
+ rm -rf venv
+ python3 -m venv venv
+ venv/bin/pip install --upgrade pip setuptools
Requirement already up-to-date: pip in ./venv/lib/python3.6/site-packages
Collecting setuptools
  Using cached setuptools-36.5.0-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 36.2.0
    Uninstalling setuptools-36.2.0:
      Successfully uninstalled setuptools-36.2.0
Successfully installed setuptools-36.5.0
+ venv/bin/pip install pip-tools
Collecting pip-tools
  Using cached pip_tools-1.10.1-py2.py3-none-any.whl
Collecting six (from pip-tools)
  Using cached six-1.11.0-py2.py3-none-any.whl
Collecting click>=6 (from pip-tools)
  Using cached click-6.7-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./venv/lib/python3.6/site-packages (from pip-tools)
Collecting first (from pip-tools)
  Using cached first-2.0.1-py2.py3-none-any.whl
Installing collected packages: six, click, first, pip-tools
Successfully installed click-6.7 first-2.0.1 pip-tools-1.10.1 six-1.11.0
+ venv/bin/pip-compile -o requirements.txt requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
django-auth-ldap==1.2.15
django==1.11.5            # via django-auth-ldap
pyldap==2.4.37            # via django-auth-ldap
pytz==2017.2              # via django
+ rm -rf venv
+ python3 -m venv venv
+ venv/bin/pip install --upgrade pip setuptools
Requirement already up-to-date: pip in ./venv/lib/python3.6/site-packages
Collecting setuptools
  Using cached setuptools-36.5.0-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 36.2.0
    Uninstalling setuptools-36.2.0:
      Successfully uninstalled setuptools-36.2.0
Successfully installed setuptools-36.5.0
+ venv/bin/pip install -r requirements.txt
Collecting django-auth-ldap==1.2.15 (from -r requirements.txt (line 7))
  Using cached django_auth_ldap-1.2.15-py3-none-any.whl
Collecting django==1.11.5 (from -r requirements.txt (line 8))
  Using cached Django-1.11.5-py2.py3-none-any.whl
Collecting pyldap==2.4.37 (from -r requirements.txt (line 9))
Collecting pytz==2017.2 (from -r requirements.txt (line 10))
  Using cached pytz-2017.2-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./venv/lib/python3.6/site-packages (from pyldap==2.4.37->-r requirements.txt (line 9))
Installing collected packages: pytz, django, pyldap, django-auth-ldap
Successfully installed django-1.11.5 django-auth-ldap-1.2.15 pyldap-2.4.37 pytz-2017.2

vphilippon · 2017-09-28T17:17:00Z

@jdufresne I took you script and adjusted it for python2, and I reproduced (in a Docker container: python:2.7) :

(Edit: First execution did not clear cache, fixed it)

#!/bin/bash

set -x
cat requirements.in

rm -rf ~/.cache/.pip
rm -rf ~/.cache/.pip-tools
rm -f requirements.txt

rm -rf venv
python2 -m virtualenv venv
venv/bin/pip install --upgrade pip setuptools
venv/bin/pip install pip-tools
venv/bin/pip-compile -o requirements.txt requirements.in

rm -rf venv
python2 -m virtualenv venv
venv/bin/pip install --upgrade pip setuptools
venv/bin/pip install -r requirements.txt

Output

+ cat requirements.in
django-auth-ldap
+ rm -rf /root/.cache/.pip
+ rm -rf /root/.cache/.pip-tools
+ rm -f requirements.txt
+ rm -rf venv
+ python2 -m virtualenv venv
New python executable in /venv/bin/python2
Also creating executable in /venv/bin/python
Installing setuptools, pip, wheel...done.
+ venv/bin/pip install --upgrade pip setuptools
Requirement already up-to-date: pip in /venv/lib/python2.7/site-packages
Requirement already up-to-date: setuptools in /venv/lib/python2.7/site-packages
+ venv/bin/pip install pip-tools
Collecting pip-tools
  Using cached pip_tools-1.10.1-py2.py3-none-any.whl
Collecting six (from pip-tools)
  Using cached six-1.11.0-py2.py3-none-any.whl
Collecting click>=6 (from pip-tools)
  Using cached click-6.7-py2.py3-none-any.whl
Collecting first (from pip-tools)
  Using cached first-2.0.1-py2.py3-none-any.whl
Requirement already satisfied: setuptools in /venv/lib/python2.7/site-packages (from pip-tools)
Installing collected packages: six, click, first, pip-tools
Successfully installed click-6.7 first-2.0.1 pip-tools-1.10.1 six-1.11.0
+ venv/bin/pip-compile -o requirements.txt requirements.in
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
django-auth-ldap==1.2.15
django==1.11.5            # via django-auth-ldap
pyldap==2.4.37            # via django-auth-ldap
pytz==2017.2              # via django
+ rm -rf venv
+ python2 -m virtualenv venv
New python executable in /venv/bin/python2
Also creating executable in /venv/bin/python
Installing setuptools, pip, wheel...done.
+ venv/bin/pip install --upgrade pip setuptools
Requirement already up-to-date: pip in /venv/lib/python2.7/site-packages
Requirement already up-to-date: setuptools in /venv/lib/python2.7/site-packages
+ venv/bin/pip install -r requirements.txt
Collecting django-auth-ldap==1.2.15 (from -r requirements.txt (line 7))
  Using cached django_auth_ldap-1.2.15-py2-none-any.whl
Collecting django==1.11.5 (from -r requirements.txt (line 8))
  Using cached Django-1.11.5-py2.py3-none-any.whl
Collecting pyldap==2.4.37 (from -r requirements.txt (line 9))
  Using cached pyldap-2.4.37.tar.gz
Collecting pytz==2017.2 (from -r requirements.txt (line 10))
  Using cached pytz-2017.2-py2.py3-none-any.whl
Collecting python-ldap>=2.0 (from django-auth-ldap==1.2.15->-r requirements.txt (line 7))
  Using cached python-ldap-2.4.44.tar.gz
Requirement already satisfied: setuptools in /venv/lib/python2.7/site-packages (from pyldap==2.4.37->-r requirements.txt (line 9))
Building wheels for collected packages: pyldap, python-ldap
  Running setup.py bdist_wheel for pyldap ... error
  Complete output from command /venv/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-8oDn1L/pyldap/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpDpPPeopip-wheel- --python-tag cp27:
  defines: HAVE_SASL HAVE_TLS HAVE_LIBLDAP_R
  extra_compile_args:
  extra_objects:
  include_dirs: /usr/include /usr/include/sasl /usr/local/include /usr/local/include/sasl
  library_dirs: /usr/lib /usr/lib64 /usr/local/lib /usr/local/lib64
  libs: ldap_r
  running bdist_wheel
  running build
  running build_py
  file Lib/ldap.py (for module ldap) not found
  file Lib/ldap/controls.py (for module ldap.controls) not found
  file Lib/ldap/extop.py (for module ldap.extop) not found
  file Lib/ldap/schema.py (for module ldap.schema) not found
  creating build
  creating build/lib.linux-x86_64-2.7
  copying Lib/ldapurl.py -> build/lib.linux-x86_64-2.7
  copying Lib/ldif.py -> build/lib.linux-x86_64-2.7
  copying Lib/dsml.py -> build/lib.linux-x86_64-2.7
  copying Lib/slapdtest.py -> build/lib.linux-x86_64-2.7
  creating build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/__init__.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/async.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/compat.py -> build/lib.linux-x86_64-2.7/ldap
  creating build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/__init__.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/deref.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/libldap.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/openldap.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/ppolicy.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/psearch.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/pwdpolicy.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/readentry.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/sessiontrack.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/simple.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/sss.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/cidict.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/dn.py -> build/lib.linux-x86_64-2.7/ldap
  creating build/lib.linux-x86_64-2.7/ldap/extop
  copying Lib/ldap/extop/__init__.py -> build/lib.linux-x86_64-2.7/ldap/extop
  copying Lib/ldap/extop/dds.py -> build/lib.linux-x86_64-2.7/ldap/extop
  copying Lib/ldap/filter.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/functions.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/ldapobject.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/logger.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/modlist.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/resiter.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/sasl.py -> build/lib.linux-x86_64-2.7/ldap
  creating build/lib.linux-x86_64-2.7/ldap/schema
  copying Lib/ldap/schema/__init__.py -> build/lib.linux-x86_64-2.7/ldap/schema
  copying Lib/ldap/schema/models.py -> build/lib.linux-x86_64-2.7/ldap/schema
  copying Lib/ldap/schema/subentry.py -> build/lib.linux-x86_64-2.7/ldap/schema
  copying Lib/ldap/schema/tokenizer.py -> build/lib.linux-x86_64-2.7/ldap/schema
  copying Lib/ldap/syncrepl.py -> build/lib.linux-x86_64-2.7/ldap
  file Lib/ldap.py (for module ldap) not found
  file Lib/ldap/controls.py (for module ldap.controls) not found
  file Lib/ldap/extop.py (for module ldap.extop) not found
  file Lib/ldap/schema.py (for module ldap.schema) not found
  running egg_info
  writing requirements to Lib/pyldap.egg-info/requires.txt
  writing Lib/pyldap.egg-info/PKG-INFO
  writing top-level names to Lib/pyldap.egg-info/top_level.txt
  writing dependency_links to Lib/pyldap.egg-info/dependency_links.txt
  file Lib/ldap.py (for module ldap) not found
  file Lib/ldap/controls.py (for module ldap.controls) not found
  file Lib/ldap/extop.py (for module ldap.extop) not found
  file Lib/ldap/schema.py (for module ldap.schema) not found
  reading manifest file 'Lib/pyldap.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'Makefile'
  warning: no files found matching 'Modules/LICENSE'
  writing manifest file 'Lib/pyldap.egg-info/SOURCES.txt'
  running build_ext
  building '_ldap' extension
  creating build/temp.linux-x86_64-2.7
  creating build/temp.linux-x86_64-2.7/Modules
  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DHAVE_SASL -DHAVE_TLS -DHAVE_LIBLDAP_R -DHAVE_LIBLDAP_R -DLDAPMODULE_VERSION=2.4.37 -IModules -I/usr/include -I/usr/include/sasl -I/usr/local/include -I/usr/local/include/sasl -I/usr/local/include/python2.7 -c Modules/LDAPObject.c -o build/temp.linux-x86_64-2.7/Modules/LDAPObject.o
  In file included from Modules/LDAPObject.c:9:0:
  Modules/errors.h:8:18: fatal error: lber.h: No such file or directory
   #include "lber.h"
                    ^
  compilation terminated.
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for pyldap
  Running setup.py clean for pyldap
  Running setup.py bdist_wheel for python-ldap ... error
  Complete output from command /venv/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-8oDn1L/python-ldap/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpkWNqOVpip-wheel- --python-tag cp27:
  defines: HAVE_SASL HAVE_TLS HAVE_LIBLDAP_R
  extra_compile_args:
  extra_objects:
  include_dirs: /usr/include /usr/include/sasl /usr/local/include /usr/local/include/sasl
  library_dirs: /usr/lib /usr/lib64 /usr/local/lib /usr/local/lib64
  libs: ldap_r
  running bdist_wheel
  running build
  running build_py
  file Lib/ldap.py (for module ldap) not found
  file Lib/ldap/controls.py (for module ldap.controls) not found
  file Lib/ldap/extop.py (for module ldap.extop) not found
  file Lib/ldap/schema.py (for module ldap.schema) not found
  creating build
  creating build/lib.linux-x86_64-2.7
  copying Lib/ldapurl.py -> build/lib.linux-x86_64-2.7
  copying Lib/ldif.py -> build/lib.linux-x86_64-2.7
  copying Lib/dsml.py -> build/lib.linux-x86_64-2.7
  copying Lib/slapdtest.py -> build/lib.linux-x86_64-2.7
  creating build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/__init__.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/async.py -> build/lib.linux-x86_64-2.7/ldap
  creating build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/__init__.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/deref.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/libldap.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/openldap.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/ppolicy.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/psearch.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/pwdpolicy.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/readentry.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/sessiontrack.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/simple.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/controls/sss.py -> build/lib.linux-x86_64-2.7/ldap/controls
  copying Lib/ldap/cidict.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/dn.py -> build/lib.linux-x86_64-2.7/ldap
  creating build/lib.linux-x86_64-2.7/ldap/extop
  copying Lib/ldap/extop/__init__.py -> build/lib.linux-x86_64-2.7/ldap/extop
  copying Lib/ldap/extop/dds.py -> build/lib.linux-x86_64-2.7/ldap/extop
  copying Lib/ldap/filter.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/functions.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/ldapobject.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/logger.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/modlist.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/resiter.py -> build/lib.linux-x86_64-2.7/ldap
  copying Lib/ldap/sasl.py -> build/lib.linux-x86_64-2.7/ldap
  creating build/lib.linux-x86_64-2.7/ldap/schema
  copying Lib/ldap/schema/__init__.py -> build/lib.linux-x86_64-2.7/ldap/schema
  copying Lib/ldap/schema/models.py -> build/lib.linux-x86_64-2.7/ldap/schema
  copying Lib/ldap/schema/subentry.py -> build/lib.linux-x86_64-2.7/ldap/schema
  copying Lib/ldap/schema/tokenizer.py -> build/lib.linux-x86_64-2.7/ldap/schema
  copying Lib/ldap/syncrepl.py -> build/lib.linux-x86_64-2.7/ldap
  file Lib/ldap.py (for module ldap) not found
  file Lib/ldap/controls.py (for module ldap.controls) not found
  file Lib/ldap/extop.py (for module ldap.extop) not found
  file Lib/ldap/schema.py (for module ldap.schema) not found
  running egg_info
  writing requirements to Lib/python_ldap.egg-info/requires.txt
  writing Lib/python_ldap.egg-info/PKG-INFO
  writing top-level names to Lib/python_ldap.egg-info/top_level.txt
  writing dependency_links to Lib/python_ldap.egg-info/dependency_links.txt
  file Lib/ldap.py (for module ldap) not found
  file Lib/ldap/controls.py (for module ldap.controls) not found
  file Lib/ldap/extop.py (for module ldap.extop) not found
  file Lib/ldap/schema.py (for module ldap.schema) not found
  reading manifest file 'Lib/python_ldap.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching 'Makefile'
  warning: no files found matching 'Modules/LICENSE'
  writing manifest file 'Lib/python_ldap.egg-info/SOURCES.txt'
  running build_ext
  building '_ldap' extension
  creating build/temp.linux-x86_64-2.7
  creating build/temp.linux-x86_64-2.7/Modules
  gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DHAVE_SASL -DHAVE_TLS -DHAVE_LIBLDAP_R -DHAVE_LIBLDAP_R -DLDAPMODULE_VERSION=2.4.44 -IModules -I/usr/include -I/usr/include/sasl -I/usr/local/include -I/usr/local/include/sasl -I/usr/local/include/python2.7 -c Modules/LDAPObject.c -o build/temp.linux-x86_64-2.7/Modules/LDAPObject.o
  In file included from Modules/LDAPObject.c:9:0:
  Modules/errors.h:8:18: fatal error: lber.h: No such file or directory
   #include "lber.h"
                    ^
  compilation terminated.
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for python-ldap
  Running setup.py clean for python-ldap
Failed to build pyldap python-ldap
Installing collected packages: pytz, django, python-ldap, django-auth-ldap, pyldap
  Running setup.py install for python-ldap ... error
    Complete output from command /venv/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-8oDn1L/python-ldap/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-CsVUiX-record/install-record.txt --single-version-externally-managed --compile --install-headers /venv/include/site/python2.7/python-ldap:
    defines: HAVE_SASL HAVE_TLS HAVE_LIBLDAP_R
    extra_compile_args:
    extra_objects:
    include_dirs: /usr/include /usr/include/sasl /usr/local/include /usr/local/include/sasl
    library_dirs: /usr/lib /usr/lib64 /usr/local/lib /usr/local/lib64
    libs: ldap_r
    running install
    running build
    running build_py
    file Lib/ldap.py (for module ldap) not found
    file Lib/ldap/controls.py (for module ldap.controls) not found
    file Lib/ldap/extop.py (for module ldap.extop) not found
    file Lib/ldap/schema.py (for module ldap.schema) not found
    creating build
    creating build/lib.linux-x86_64-2.7
    copying Lib/ldapurl.py -> build/lib.linux-x86_64-2.7
    copying Lib/ldif.py -> build/lib.linux-x86_64-2.7
    copying Lib/dsml.py -> build/lib.linux-x86_64-2.7
    copying Lib/slapdtest.py -> build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/ldap
    copying Lib/ldap/__init__.py -> build/lib.linux-x86_64-2.7/ldap
    copying Lib/ldap/async.py -> build/lib.linux-x86_64-2.7/ldap
    creating build/lib.linux-x86_64-2.7/ldap/controls
    copying Lib/ldap/controls/__init__.py -> build/lib.linux-x86_64-2.7/ldap/controls
    copying Lib/ldap/controls/deref.py -> build/lib.linux-x86_64-2.7/ldap/controls
    copying Lib/ldap/controls/libldap.py -> build/lib.linux-x86_64-2.7/ldap/controls
    copying Lib/ldap/controls/openldap.py -> build/lib.linux-x86_64-2.7/ldap/controls
    copying Lib/ldap/controls/ppolicy.py -> build/lib.linux-x86_64-2.7/ldap/controls
    copying Lib/ldap/controls/psearch.py -> build/lib.linux-x86_64-2.7/ldap/controls
    copying Lib/ldap/controls/pwdpolicy.py -> build/lib.linux-x86_64-2.7/ldap/controls
    copying Lib/ldap/controls/readentry.py -> build/lib.linux-x86_64-2.7/ldap/controls
    copying Lib/ldap/controls/sessiontrack.py -> build/lib.linux-x86_64-2.7/ldap/controls
    copying Lib/ldap/controls/simple.py -> build/lib.linux-x86_64-2.7/ldap/controls
    copying Lib/ldap/controls/sss.py -> build/lib.linux-x86_64-2.7/ldap/controls
    copying Lib/ldap/cidict.py -> build/lib.linux-x86_64-2.7/ldap
    copying Lib/ldap/dn.py -> build/lib.linux-x86_64-2.7/ldap
    creating build/lib.linux-x86_64-2.7/ldap/extop
    copying Lib/ldap/extop/__init__.py -> build/lib.linux-x86_64-2.7/ldap/extop
    copying Lib/ldap/extop/dds.py -> build/lib.linux-x86_64-2.7/ldap/extop
    copying Lib/ldap/filter.py -> build/lib.linux-x86_64-2.7/ldap
    copying Lib/ldap/functions.py -> build/lib.linux-x86_64-2.7/ldap
    copying Lib/ldap/ldapobject.py -> build/lib.linux-x86_64-2.7/ldap
    copying Lib/ldap/logger.py -> build/lib.linux-x86_64-2.7/ldap
    copying Lib/ldap/modlist.py -> build/lib.linux-x86_64-2.7/ldap
    copying Lib/ldap/resiter.py -> build/lib.linux-x86_64-2.7/ldap
    copying Lib/ldap/sasl.py -> build/lib.linux-x86_64-2.7/ldap
    creating build/lib.linux-x86_64-2.7/ldap/schema
    copying Lib/ldap/schema/__init__.py -> build/lib.linux-x86_64-2.7/ldap/schema
    copying Lib/ldap/schema/models.py -> build/lib.linux-x86_64-2.7/ldap/schema
    copying Lib/ldap/schema/subentry.py -> build/lib.linux-x86_64-2.7/ldap/schema
    copying Lib/ldap/schema/tokenizer.py -> build/lib.linux-x86_64-2.7/ldap/schema
    copying Lib/ldap/syncrepl.py -> build/lib.linux-x86_64-2.7/ldap
    file Lib/ldap.py (for module ldap) not found
    file Lib/ldap/controls.py (for module ldap.controls) not found
    file Lib/ldap/extop.py (for module ldap.extop) not found
    file Lib/ldap/schema.py (for module ldap.schema) not found
    running egg_info
    writing requirements to Lib/python_ldap.egg-info/requires.txt
    writing Lib/python_ldap.egg-info/PKG-INFO
    writing top-level names to Lib/python_ldap.egg-info/top_level.txt
    writing dependency_links to Lib/python_ldap.egg-info/dependency_links.txt
    file Lib/ldap.py (for module ldap) not found
    file Lib/ldap/controls.py (for module ldap.controls) not found
    file Lib/ldap/extop.py (for module ldap.extop) not found
    file Lib/ldap/schema.py (for module ldap.schema) not found
    reading manifest file 'Lib/python_ldap.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching 'Makefile'
    warning: no files found matching 'Modules/LICENSE'
    writing manifest file 'Lib/python_ldap.egg-info/SOURCES.txt'
    running build_ext
    building '_ldap' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/Modules
    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DHAVE_SASL -DHAVE_TLS -DHAVE_LIBLDAP_R -DHAVE_LIBLDAP_R -DLDAPMODULE_VERSION=2.4.44 -IModules -I/usr/include -I/usr/include/sasl -I/usr/local/include -I/usr/local/include/sasl -I/usr/local/include/python2.7 -c Modules/LDAPObject.c -o build/temp.linux-x86_64-2.7/Modules/LDAPObject.o
    In file included from Modules/LDAPObject.c:9:0:
    Modules/errors.h:8:18: fatal error: lber.h: No such file or directory
     #include "lber.h"
                      ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/venv/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-8oDn1L/python-ldap/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-CsVUiX-record/install-record.txt --single-version-externally-managed --compile --install-headers /venv/include/site/python2.7/python-ldap" failed with error code 1 in /tmp/pip-build-8oDn1L/python-ldap/

pyldap was included as a dependency on Python 2, while it should not (the error on the pip install is with trying to build pyldap)

It's hard to make 100% reproducible case, as I'm not sure of the pattern used to select the wheel when multiple are detected as ok.

suutari-ai · 2017-09-28T18:11:32Z

I think the non-deterministic part of this is the following line in Resolver.find_best_match:

        candidates_by_version = lookup_table(all_candidates, key=lambda c: c.version, unique=True)

This builds a lookup table of the candidates by version. An unordered set of InstallationCandidate objects is assigned for each version. The py2 wheel, py3 wheel and tar.gz packages are in this set. The best candidate is then selected from this set using finder._candidate_sort_key as sorter key, but since this is monkey-patched to always return the same value it effectively selects a random candidate with the best version.

You can try to set environment variable PYTHONHASHSEED to an integer value to create deterministic results. (I.e. try with export PYTHONHASHSEED=1, export PYTHONHASHSEED=2, ...)

vphilippon · 2017-09-28T19:43:00Z

@suutari-ai In my case, the best_match still point to Python2 wheel, and I get pyldap.

But what you explained happens almost exactly in get_dependencies, within this call:

self._dependencies_cache[ireq] = reqset._prepare_file(self.finder, ireq)

I'll try and use a context manager to keep the monkeypatch of "Wheel" only in the context of get_hashes. I think that should do it.

azaghal · 2017-09-28T19:54:56Z

Another resolution difference between 1.9.0 and 1.10.x that I've seen is for package kallithea - to be more precise for one of this dependencies - mercurial. This is probably the same issue?

With 1.9.0, pip-compile would end-up using version 4.2.2 of the package, while with 1.10.x, it would use 4.2.3. Looking at the PyPI download page, it looks like 4.2.3 is available only for 32 and 64-bit Windows platform?

The issue should be reproducible with requirements.in being set to mercurial~=4.2.0 (you may need to remove .txt before each run of pip-compile).

vphilippon · 2017-09-28T20:01:25Z

@azaghal Seems like the same issue yes, assuming you're on Linux.

azaghal · 2017-09-28T20:32:51Z

Yep, Gentoo 64-bit.

vphilippon · 2017-09-28T21:22:15Z

I've got something working that fixes this while keeping the feature from #520.
It'll be a fix, not a piece of art.
I should be submitting a PR soon-ish after cleaning my stuff up.

Fixes jazzband#558

…endencies Respect platform's wheels for dependencies (Fixes #558)

azaghal · 2017-10-23T09:35:47Z

First of all, thanks for looking into this and fixing it :) Now, what's the chance of getting a patch release out that includes this fix, since it does make it troublesome to use 1.10.x?

vphilippon · 2017-10-23T13:41:29Z

@azaghal We're currently unable to make releases, as the PyPi upload has been deactivated for all project part of Jazzband at the moment (see #531 and jazzband/help#64). That's in the hands of the admin/roadies right now.

As soon as we're able to make new releases, it will be my pleasure to make one (believe me, I'm waiting for it too).

azaghal · 2017-10-24T13:11:23Z

Ho-hum, that sounds even slightly scary. Thanks for letting me know what's up, much appreciated :)

suutari-ai mentioned this issue Sep 20, 2017

Environment markers do not carry out to transitive dependencies #563

Closed

vphilippon added the bug Something is not working label Sep 28, 2017

vphilippon mentioned this issue Sep 28, 2017

Respect platform's wheels for dependencies (Fixes #558) #571

Merged

3 tasks

vphilippon added a commit to vphilippon/pip-tools that referenced this issue Sep 28, 2017

Keep pip.Wheel monkey patch only in the get_hashes context.

a984fdf

Fixes jazzband#558

vphilippon closed this as completed in #571 Oct 2, 2017

vphilippon added a commit that referenced this issue Oct 2, 2017

Merge pull request #571 from vphilippon/respect-plaform-wheel-for-dep…

ef0aa2d

…endencies Respect platform's wheels for dependencies (Fixes #558)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 2 package is added to Python 3 dependency list using pip-compile #558

Python 2 package is added to Python 3 dependency list using pip-compile #558

MHLut commented Sep 5, 2017 •

edited

Loading

suutari-ai commented Sep 5, 2017

vphilippon commented Sep 5, 2017

MHLut commented Sep 5, 2017 •

edited

Loading

vphilippon commented Sep 5, 2017 •

edited

Loading

MHLut commented Sep 7, 2017 •

edited

Loading

vphilippon commented Sep 7, 2017

MHLut commented Sep 8, 2017

vphilippon commented Sep 8, 2017 •

edited

Loading

suutari-ai commented Sep 8, 2017

suutari-ai commented Sep 13, 2017

vphilippon commented Sep 14, 2017

suutari-ai commented Sep 15, 2017

azaghal commented Sep 28, 2017

vphilippon commented Sep 28, 2017

azaghal commented Sep 28, 2017

jdufresne commented Sep 28, 2017 •

edited

Loading

vphilippon commented Sep 28, 2017 •

edited

Loading

suutari-ai commented Sep 28, 2017 •

edited

Loading

vphilippon commented Sep 28, 2017

azaghal commented Sep 28, 2017

vphilippon commented Sep 28, 2017

azaghal commented Sep 28, 2017

vphilippon commented Sep 28, 2017

azaghal commented Oct 23, 2017

vphilippon commented Oct 23, 2017

azaghal commented Oct 24, 2017

Python 2 package is added to Python 3 dependency list using pip-compile #558

Python 2 package is added to Python 3 dependency list using pip-compile #558

Comments

MHLut commented Sep 5, 2017 • edited Loading

Environment Versions

Steps to replicate

Expected result

Actual result

suutari-ai commented Sep 5, 2017

vphilippon commented Sep 5, 2017

MHLut commented Sep 5, 2017 • edited Loading

vphilippon commented Sep 5, 2017 • edited Loading

MHLut commented Sep 7, 2017 • edited Loading

vphilippon commented Sep 7, 2017

MHLut commented Sep 8, 2017

vphilippon commented Sep 8, 2017 • edited Loading

suutari-ai commented Sep 8, 2017

suutari-ai commented Sep 13, 2017

vphilippon commented Sep 14, 2017

suutari-ai commented Sep 15, 2017

azaghal commented Sep 28, 2017

vphilippon commented Sep 28, 2017

azaghal commented Sep 28, 2017

jdufresne commented Sep 28, 2017 • edited Loading

vphilippon commented Sep 28, 2017 • edited Loading

suutari-ai commented Sep 28, 2017 • edited Loading

vphilippon commented Sep 28, 2017

azaghal commented Sep 28, 2017

vphilippon commented Sep 28, 2017

azaghal commented Sep 28, 2017

vphilippon commented Sep 28, 2017

azaghal commented Oct 23, 2017

vphilippon commented Oct 23, 2017

azaghal commented Oct 24, 2017

MHLut commented Sep 5, 2017 •

edited

Loading

MHLut commented Sep 5, 2017 •

edited

Loading

vphilippon commented Sep 5, 2017 •

edited

Loading

MHLut commented Sep 7, 2017 •

edited

Loading

vphilippon commented Sep 8, 2017 •

edited

Loading

jdufresne commented Sep 28, 2017 •

edited

Loading

vphilippon commented Sep 28, 2017 •

edited

Loading

suutari-ai commented Sep 28, 2017 •

edited

Loading