Difficulty using pip in a venv

[mawis]

Describe the bug
I'm using pip inside a default.nix on NixOS 19.09 as well as on the docker container nixos/nix:latest. I had a working setup that used pip to install the package nrfutils. In the last build on my CI server 11 days ago everything worked as expected. When I trigger a rebuild it's now failing with the following message:

Installing pip, wheel...                                                         
done.                                                                            
Traceback (most recent call last):                                               
File "/drone/src/.venv/bin/pip", line 6, in <module>                             
from pip._internal.main import main                                              
ImportError: No module named main

To Reproduce
Save the following file as default.nix and start nix-shell:

with import <nixpkgs> {};
with pkgs.python27Packages;

stdenv.mkDerivation {
  name = "impurePythonEnv";
  buildInputs = [
    automake
    autoconf
    gcc-arm-embedded-7
    # these packages are required for virtualenv and pip to work:
    #
    python27
    python27Packages.virtualenv
    python27Packages.pip
    # the following packages are related to the dependencies of your python
    # project.
    # In this particular example the python modules listed in the
    # requirements.txt require the following packages to be installed locally
    # in order to compile any binary extensions they may require.
    #
    taglib
    openssl
    git
    stdenv
    zlib ];
  src = null;
  shellHook = ''
    # set SOURCE_DATE_EPOCH so that we can use python wheels
    SOURCE_DATE_EPOCH=$(date +%s)
    virtualenv --no-setuptools .venv
    export PATH=$PWD/.venv/bin:$PATH
    #python2.7 -m pip install nrfutil
    pip install nrfutil

    # the following is required to build micro_ecc_lib_nrf52.a in the SDK
    export GNU_INSTALL_ROOT="${gcc-arm-embedded-7}/bin/"

    unset CC
  '';
}

Expected behavior
I would expect a shell in which pip installed the program nrfutil.

Additional context
I found out that I can fix the problem with either one of these two:

• Instead of calling pip install nrfutil I can call python2.7 -m pip install nrfutil.
• I can edit the file .venv/bin/pip and change the module path:

After starting the shell .venv/bin/pip contains:

#!/home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
# -*- coding: utf-8 -*-
import re
import sys

from pip._internal.main import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

It works if I change this to:

#!/home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
# -*- coding: utf-8 -*-
import re
import sys

from pip._internal import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

(The change is from pip._internal.main to pip._internal.)

Metadata

• system: "x86_64-linux"
• host os: Linux 4.19.79, NixOS, 19.09.809.5000b1478a1 (Loris)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3
• channels(root): "nixos-19.09.809.5000b1478a1, unstable-19.09pre192418.e19054ab3cd"
• channels(matthias): "home-manager-19.03"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

attribute: [python27Packages.pip, python37Packages.pip]
module: []

[scoates]

At risk of adding noise to this issue, I can confirm that s/pip._internal.main/pip._internal/ fixed this for me in a new nix-shell + pipenv.

[mawis]

@FRidh Sure that this is a “question” instead of a “bug”?

[mawis]

@scoates Another way to work around the bug, that doesn't require to patch anything is to call python -m pip instead of pip. That works as well an can be easily placed inside the shellHook if required.

[FRidh]

Yes, this is not a Nixpkgs bug. Python packaging in Nixpkgs covers building expressions using Nix, and not generating impure environments.

[scoates]

pipenv doesn't exactly like to do python -m pip instead of pip, unfortunately (there might be a way to trick it, but this breaks the workflow for not-particularly-techy team members).

Is this a problem with pip itself? I thought it was with nix packaging, but I could be wrong, I suppose. It's not like pip's internal stuff hasn't burned me before.

[tsusanka]

I'm also experiencing this and I can confirm that @mawis' hotfix works.

[mawis]

To add more context to this: it seems to me, that this is related to the following change in PIP: pypa/pip#7061

What I still don't get is why this change is present in pythonPackage*.pip on NixOS 19.09 which is still on version 19.2.3 – while this change was introduced in upstream not before version 19.3.

[FRidh]

because the pip of the venv is used

[FRidh]

or is that the same one as the nixpkgs one?

[mawis]

@FRidh That might be an explanation. I must admit, that I have no idea how all that gets resolved. I'm no Python developer.

[silviogutierrez]

@mawis : your fixes worked for me too. But I'm just as curious how my pip/virtualenv setup wasn't "pinned" and suddenly started failing? I even pinned the nix channel.

So, before, this worked:

nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/nixpkgs-19.03-darwin.tar.gz -p python37 python37Packages.virtualenv python37Packages.pip

virtualenv --no-setuptools test
test/bin/pip

and now, it doesn't. Same error as you. And same workaround.

I thought one of the main points of NixOS, particularly with pinned channel, was reproducibility? I'm dying to know what changed here overnight.

[silviogutierrez]

@mawis: By the way, for me, simply removing python37Packages.pip but leaving python37Packages.virtualenv worked. I can still create a virtualenv with pip inside of it. And it works!

[avanov]

Same issue with pipenv when used within nix-shell (19.09 snapshot), sed'ing helped as well.

[Kaali]

I think it's a problem with nix, that breaks virtualenv and thus also pipenv. As far as I understand the problem seems to be that Python wrapper will always push the nix installed pip as the first item in PYTHONPATH even though virtualenv tries to override it.

It can be repeated easily by running:

$ nix-shell -p python37Packages.pip python37Packages.virtualenv
$ virtualenv --no-site-packages venv
$ venv/bin/python -c "import pprint; import sys; pprint.pprint(sys.path)" 

You can see that nix pip is still in the path even though the virtualenv installed its own pip. Now if the environment is activated, the pip command in PATH is venv/bin/pip which expects the new import that people in this issue is sedding. But as the nix pip is forced, the import doesn't work as that version of pip is incompatible with the binary/script.

[FRidh]

Do not use nix-shell -p python37Packages.pip python37Packages.virtualenv. That will cause Python's setup hook to run, exporting PYTHONPATH. If you need an application or environment, use nix run. For an environment, use python.withPackages.

There is an issue with pipenv that a certain commit wraps it with PYTHONPATH which is bad.

[silviogutierrez]

@FRidh : how does this explain something that was previously working (and pinned with a channel) stopped working? Isn't that the whole point of nix?

Or is there some special code inside those nix packages that auto-downloads a newer version of pip or the like?

[scoates]

In case it's immediately useful to anyone, I have this in one of my bootstrap scripts (on MacOS at least), to fix this problem:

sed -i -e's/pip._internal.main/pip._internal/' `nix-shell --run 'pipenv run which pip'`

[scoates]

This appears to be where this is discussed upstream: pypa/pip#7209

[Kaali]

There is an issue with pipenv that a certain commit wraps it with PYTHONPATH which is bad.

What commit and is there a fix coming?

Me using python37Packages.pip like that was to replicate what happens with pipenv at the moment. Is there more info somewhere how python37Packages should be used, and what it means for it to run Python's setup hook?

[FRidh]

pipenv is fixed in 7f63ecf.
The Nixpkgs manual has a huge section on how to use Python.

[Kaali]

pipenv is fixed in 7f63ecf.

That commit seems to break pipenv for me completely: #71771

The Nixpkgs manual has a huge section on how to use Python.

Thanks, it's been a long time since I read that. Will reread it 👍

[FRidh]

New pipenv fix #71800.

I do not see anything more actionable in this ticket. As I explained, by using nix-shell $PYTHONPATH will be set and this will cause issues as it will be prepended to whatever would be in sys.path already.

[jonringer]

for my personal venv use, i need to do:

export PYTHONPATH=$PWD/.venv/lib/${python.sitePackages}:$PYTHONPATH

so that the install packages appear on the PYTHONPATH

[novoxd]

same issue additionally can say:

env/bin/python
env/bin/python: error while loading shared libraries: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory

previously I could execute python without nix-shell just from virtualenv location by path (And did so from ide (maybe need to swith to nix-build))

[FRidh]

I have yet to see a reproducible example here that is not using nix-shell to set $PYTHONPATH.

[scoates]

Current status: seems to be fixed for me after a bunch of updates, and changing my shell.nix files to use python37.withPackages instead of python37Packages, and wiping out my virtualenvs (to rebuild)… (and waiting for lorri to catch up).

Example shell.nix in case it's helpful to others:

with import <nixpkgs> {};

stdenv.mkDerivation rec {
  name = "sample-dev";
  env = buildEnv { name = name; paths = buildInputs; };
  buildInputs = [
    (python37.withPackages (pypkgs: [ pypkgs.pip pypkgs.virtualenv ]))
    pipenv
    (buildEnv { name = "python-packages-bin-only"; paths = [
      awscli
    ]; })
  ];
  shellHook = ''
    # set SOURCE_DATE_EPOCH so that we can use python wheels
    SOURCE_DATE_EPOCH=$(date +%s)
  '';
}

Another vote to close this issue, here, from me.

[silviogutierrez]

Can anyone who knows Nix explain, why, even with a pinned channel[1], this broke? Isn't the whole point of nix to make things deterministic? How could nix be downloading a new version of a package that is broken? I would imagine the pinning of the channel locks packages as they were at the time.

Appreciate any help.

[1] #71178 (comment)

[mawis]

@silviogutierrez If you're using PIP your installing code that you don't get it from Nix but from PyPI. PyPI does not provide a reproducible build. At any moment you will just install the latest version of the PythonPackage available at PyPI.

[FRidh]

also pointing to a channel is not pinning, because the channel is a moving target. To pin you need to point to a specific revision.

[silviogutierrez]

@mawis , that helps, and is sort of what I suspect. But I would imagine pip itself is bundled in the python37Packages.virtualenv or python37Packages.virtualenv? Isn't that the point of asking for these packages? Or do those not "come" with pip? Where does it download pip? Is it the nix formula, or virtualenv itself?

[scoates]

@silviogutierrez virtualenv itself goes out to pypi to install packages:

(this is Little Snitch showing what's happening)

[silviogutierrez]

@scoates : interesting! I never would have imagined that. Of course pip itself goes out to Pypi to install packages, but I never thought virtualenv itself needed to hit the internet to start a new virtualenv to get python/pip. That seems very odd.

But! I just turned off the internet entirely, and virtualenv foo still worked fine, and I see a pip and everything. Just took a lot longer. Maybe it gives up and uses a local one?

(I could just go through the virtualenv code, but this process of deduction is just as interesting)

silviogutierrez@Silvios-MacBook-Pro (testing): time virtualenv with-internet
Using base prefix '/nix/store/jd8200285gwy8p538hrd2wpwfnm5b27r-python3-3.7.4'
New python executable in /Users/silviogutierrez/Sites/silviogutierrez/joyapp.com/testing/spam/bin/python3.7
Also creating executable in /Users/silviogutierrez/Sites/silviogutierrez/joyapp.com/testing/spam/bin/python
Installing setuptools, pip, wheel...
done.

real    0m4.238s
user    0m2.904s
sys     0m0.738s

silviogutierrez@Silvios-MacBook-Pro (testing): time virtualenv without-internet
Using base prefix '/nix/store/jd8200285gwy8p538hrd2wpwfnm5b27r-python3-3.7.4'
New python executable in /Users/silviogutierrez/Sites/silviogutierrez/joyapp.com/testing/bar/bin/python3.7
Also creating executable in /Users/silviogutierrez/Sites/silviogutierrez/joyapp.com/testing/bar/bin/python
Installing setuptools, pip, wheel...
done.

real    0m25.969s
user    0m2.547s
sys     0m0.708s

[scoates]

@silviogutierrez We're getting way off topic, here, so I suggest this be the last of these comments, but you can supply --no-download to virtualenv to get the same (but faster) results you're seeing without a connection above.

[silviogutierrez]

@scoates @FRidh : incredibly helpful. Thank you.

[jraygauthier]

In case someone comes here confused, some notes I took on how to use virtualenv on nix/nixos:

Python virtualenv

Using plain virtualenv / pip on nix / nixos

From:

• Difficulty using pip in a venv - Issue #71178 - NixOS/nixpkgs
  • silviogutierrez commented on Oct 18

Installing pip in the nix environment prevent proper virtualenv creation.

Also, one should not create the virtual env using the
python -m venv myvenvname way but rather go with
virtualenv myvenvname. This is because the latter installs
a proper pip in the venv which allow one to proceed.

The following shell.nix file should be used:

{ nixpkgs ? import <nixpkgs> {}
}:

with nixpkgs;

stdenv.mkDerivation rec {
  name = "python-virtualenv-shell";
  env = buildEnv { name = name; paths = buildInputs; };
  buildInputs = [
    python3
    python3Packages.virtualenv
    # It is essential **not** to add `pip` here as
    # it would prevent proper virtualenv creation.
  ];
  shellHook = ''
    # set SOURCE_DATE_EPOCH so that we can use python wheels
    SOURCE_DATE_EPOCH=$(date +%s)
  '';
}

An example of using this nix shell and virtualenv / pip:

$ touch shell.nix
# ..

# Write paste the above nix code into the 'shell.nix' file
# using an editor of your choice.
$ editor shell.nix

# Enter the nix shell which should bring the python interpreter and
# the virtualenv executable.
$ nix-shell

$ virtualenv venv
Using base prefix '/nix/store/c2n0xp0j3nacr1l17lmrdzwp16ljsvll-python3-3.7.5'
New python executable in /home/myuser/dev/mypiptest/venv/bin/python3.7
Also creating executable in /home/myuser/dev/mypiptest/venv/bin/python
Installing setuptools, pip, wheel...
done.

$ source ./venv/bin/activate
(venv)

$ which python
/home/myuser/dev/mypiptest/venv/bin/python
(venv)

$ which pip
/home/myuser/dev/mypiptest/venv/bin/pip
(venv)

$ pip install ipython
# ..
(venv)

$ which ipython
/home/myuser/dev/mypiptest/venv/bin/ipython
(venv)

$ echo "request" > requirements.txt
(venv)

$ pip install -r requirements.txt
# ..
(venv)
$ ipython
Python 3.7.5 (default, Oct 14 2019, 23:08:55)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.10.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import request

In [2]:
exit
(venv)

$

[silviogutierrez]

@jraygauthier : years later, I took another stab at getting rid of virtualenv and using the built in python -m venv method. You'd think this would have taken over by now.

And yet inside nix, python39 and python310 would create venvs (not virtualenvs) without pip. Couldn't figure out why.

Turns out, it was because I had python39Packages.pip or python310Packages.pip as well. It's very counterintuitive, and I don't quite know why. But removing that and leaving just python39 (no python39Packages.virtualenv either) made it work.