19.3 doesn't work after upgrading from 19.2.3

[j0sete]

Environment

• pip version: 19.3
• Python version: 3.7.3
• OS: Windows 10

Description
I used pip 19.2.3 and today I upgrade it to 19.3 in my system and it doesn't work using pip command (image attached)

Expected behavior

How to Reproduce

1. Using pip with version 19.2.3 upgrade to 19.3 with command "pip install --upgrade pip --user"
2. Then run any command in pip.
3. It's going to throw an exception (same as image attached)

Output

C:\Users\<user>>pip --version
Traceback (most recent call last):
  File "c:\users\<user>\appdata\local\programs\python\python37-32\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\<user>\appdata\local\programs\python\python37-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\<user>\AppData\Local\Programs\Python\Python37-32\Scripts\pip.exe\__main__.py", line 9, in <module>
TypeError: 'module' object is not callable

Workaround

1. Copying the files from %AppData%\Roaming\Python\Python37-32\Scripts\pip* to %AppData%\Local\Programs\Python\Python37-32\Scripts\

[vwvw]

See #7205

[pfmoore]

Looks like you have mixed --user and non---user installs of pip (the previous install appears to have been done without --user, presumably as part of your Python install). This is causing clashes.

You should either not use --user to upgrade your pip, or you should uninstall the old version of pip and then install pip afresh (using get-pip.py) with --user.

[jayvdb]

Also see #5599 (comment)

That it is trying to do anything with ..\Scripts\pip.exe\__main__.py means there is some string manipulation problem somewhere. If fixing it only makes the error slightly better, it still seems worth doing, as that stands out as "wrong".

[pfmoore]

Honestly, my preference would be to bite the bullet and deprecate all use of wrapper scripts for pip, making python -m pip the only supported way of running pip. This obviously wouldn't remove the existing issues (people will still have old versions of wrappers lying around from previous installs, distributions may still supply wrappers, etc) but it would send a clear message to users of the most reliable fix to use, while starting the (necessarily very slow) process of removing wrappers altogether.

[pradyunsg]

Hmm... This looks like another case of redistributor executables need to be updated because pip changed -- similar to how pip broke for folks when we did the _internal move: #5599.

[pradyunsg]

Honestly, my preference would be to bite the bullet and deprecate all use of wrapper scripts for pip, making python -m pip the only supported way of running pip.

That'd be #3164. I have a proposal for how we can go about this in a manner that minimizes the disruption -- #3164 (comment).

[pfmoore]

This is Windows, so no redistributor involved as such. The issue here seems to be that the original install was to site-packages, the upgrade went to user-site, but PATH still had the old wrapper first which assumed old pip, whereas sys.path gives user-site priority over site-packages.

pip install --upgrade --user when the original install was a system install is the problem here. Maybe we should warn if someone does that? But we can't fix the issue because to do that we'd either have to modify the user's PATH (impossible) or try to uninstall the old site-packages installed version (may not work, and is likely not what the user wants).

[pradyunsg]

Ah right I missed that.

Yea, this is an issue with an "out-of-date" executable, since pip._internal:main moved to pip._internal.main:main and whatever executable is getting called is running pip._internal.main.

I think we can detect that (--user overiden by non---user executable) pretty easily and incorporate that into our script warning setup. Could someone file a dedicated issue for that (labeled "awaiting PR")?

[Vel-San]

I'm having same issue with pip 19.3 (upgrading from 19.2.2 ) Running Ubuntu 18.04.3 - Python 2.7.15+

It doesn't matter if i use --user or not, i'm running python and pip in venv (16.7.5) always same error as in OP.

I tried uninstalling then installing solely with get-pip.py, same error. Downgrading to 19.2.2 works fine. Any suggestions?

[chrahunt]

Hi @Vel-San, do you mind creating a separate issue with clear steps to reproduce (and output from running those steps)? Those details seem different enough that it would be easier to address the issue with that approach.

[cloventt]

I am experiencing the same issue. Running python -m pip install instead of pip install works.

This issue is fairly significant for me as it has broken a number of Docker builds. We typically run pip install --upgrade pip as part of the Docker build process to make sure we are always up-to-date (some base images ship with out-of-date pip), but due to some breaking change in pip 19.3 the Docker builds now fail.

We will be removing the automatic pip upgrade to fix this, which is a shame because I like using up-to-date software. But I can no longer trust that pip versions will remain fully backwards compatible, so we will have to stick with a fixed out-of-date version.

[chrahunt]

Hi @cloventt. As mentioned in #5599, this isn't anyone's "fault", but the intersection of several misunderstandings and mis-features involved with resolving a plain pip invocation to a concrete executable thing and the import precedence in Python itself.

path/to/python -m pip is unambiguous and should have 0 problems regardless of pip version. If you want your CI setup or build process to be robust then you should take this approach.

[chrahunt]

I created #7228 based on @pradyunsg's comment above, since that seems like the best action we can take away from this. I will close this issue now, but please let me know if that's not agreeable!

[j0sete]

Moving the files of directoy now gets the following error:

C:\Users\jmrm>python -m pip install --upgrade pip --user
Traceback (most recent call last):
  File "C:\Users\jmrm\AppData\Local\Programs\Python\Python37-32\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\jmrm\AppData\Local\Programs\Python\Python37-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\jmrm\AppData\Roaming\Python\Python37\site-packages\pip\__main__.py", line 19, in <module>
    sys.exit(_main())
TypeError: 'module' object is not callable

I'm uneable of run pip using \path\to\python.exe -m pip [option]

And when I try to upgrade my pip version (19.3) to 19.3.1 I get this error:

C:\Users\jmrm>pip install --upgrade pip --user
Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
ERROR: To modify pip, please run the following command:
c:\users\jmrm\appdata\local\programs\python\python37-32\python.exe -m pip install --upgrade pip --user

[chrahunt]

Hi @j0sete. Can you please describe in detail the changes that were made to get the above error?

[j0sete]

Hi @j0sete. Can you please describe in detail the changes that were made to get the above error?

I did the workarround that I explained in the first message:

1. Copying the files from %AppData%\Roaming\Python\Python37-32\Scripts\pip* to %AppData%\Local\Programs\Python\Python37-32\Scripts\

That fix the invocation of pip but I'm unable to call python -m pip and, in consequence, unable to upgrade pip

[chrahunt]

Please delete all traces of pip from

1. %AppData%\Python\Python37-32\Scripts
2. %AppData%\Python\Python37-32\lib\site-packages
3. %LocalAppData%\Programs\Python\Python37-32\Scripts
4. %LocalAppData%\Programs\Python\Python37-32\lib\site-packages

Once finished, install pip using get-pip.py as described here. I don't think there's any specific reason to use --user on Windows with a user-mode Python installation, so I would just leave it off. That will make things simpler and get you back to a working installation.

[thijstriemstra]

Seeing same issue on raspbian buster with python 3.7:

$ pip3 --version
Traceback (most recent call last):
  File "/usr/bin/pip3", line 11, in <module>
    sys.exit(main())
TypeError: 'module' object is not callable

What happened:

pi@flowmeter:~ $ pip3 --version
pip 18.1 from /usr/lib/python3/dist-packages/pip (python 3.7)
pi@flowmeter:~ $ sudo pip3 install -U pip
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pip
  Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 68kB/s 
Installing collected packages: pip
  Found existing installation: pip 18.1
    Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'pip'. No files were found to uninstall.
Successfully installed pip-19.3.1
pi@flowmeter:~ $ pip3 --version
Traceback (most recent call last):
  File "/usr/bin/pip3", line 11, in <module>
    sys.exit(main())
TypeError: 'module' object is not callable
pi@flowmeter:~ $ pip3 --version
Traceback (most recent call last):
  File "/usr/bin/pip3", line 11, in <module>
    sys.exit(main())
TypeError: 'module' object is not callable
pi@flowmeter:~ $ sudo pip3 install --ignore-installed -U pip 
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Installing collected packages: pip
Successfully installed pip-19.3.1
pi@flowmeter:~ $ pip3 --version
Traceback (most recent call last):
  File "/usr/bin/pip3", line 11, in <module>
    sys.exit(main())
TypeError: 'module' object is not callable
pi@flowmeter:~ $

Eventually:

pi@flowmeter:~ $ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1734k  100 1734k    0     0   411k      0  0:00:04  0:00:04 --:--:--  411k
pi@flowmeter:~ $ python3 get-pip.py 
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pip
  Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB)
     |████████████████████████████████| 1.4MB 708kB/s 
Installing collected packages: pip
  Found existing installation: pip 19.3.1
    Uninstalling pip-19.3.1:
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/bin/pip3'
Consider using the `--user` option or check the permissions.

pi@flowmeter:~ $ sudo python3 get-pip.py 
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.3.1
    Uninstalling pip-19.3.1:
      Successfully uninstalled pip-19.3.1
Successfully installed pip-19.3.1
pi@flowmeter:~ $ pip3 --versionn
-bash: /usr/bin/pip3: No such file or directory
pi@flowmeter:~ $ pip3 --version
-bash: /usr/bin/pip3: No such file or directory
pi@flowmeter:~ $ /usr/local/bin/pip
pip     pip3    pip3.7  
pi@flowmeter:~ $ /usr/local/bin/pip3 --version
pip 19.3.1 from /usr/local/lib/python3.7/dist-packages/pip (python 3.7)

[chrahunt]

Hi @thijstriemstra. That seems like a separate issue, probably what happened is:

1. In the first case the pip installation created the script in /usr/local/bin/pip3 and left the original in /usr/bin/pip3. The original is what is getting invoked and it has the old entrypoint location. The solution to this is to not use sudo pip install -U pip but pip install -U --user pip, otherwise it impacts your system at large in unexpected ways because distributions will have specific configuration for the system-wide pip.
2. In the second case, bash has "hashed" (remembered) the location of the last pip3 that it previously invoked. You need to run hash pip3 after installation in order for it to pick up the new script. We have several issues open to add a warning for this case, notably Consider warning users about shell path hashing on upgrade #6863.

You may also check out #5599 for more information, it's our go-to reference for these kinds of situations, but we could probably take this as a sign that it can be made more approachable/easily found.

If you still have problems, please create a separate issue. This one hasn't had action in awhile so I will close it.

[pfmoore]

@chrahunt is our advice to use pip install -U --user pip sufficient? The system /usr/bin/pip3 will still use the old API, and will fail because the newly installed pip in user-site takes precedence. The user also needs to ensure that ~/.local/bin is ahead of /usr/bin on PATH (as well as never using /usr/bin/pip3 via the full pathname).

IMO, user installs are still good advice (certainly better than sudo!!!), but they will still have issues if the user follows the advice blindly without fully understanding the underlying (OS and Python) mechanisms.

[sujoy108]

use pip install --upgrade pip --user command
its working

[chrahunt]

@pfmoore, I think it will be once we have implemented some of the script-related warnings like #6863 mentioned above. Also we would switch to pip install -U pip after #7002 gets into a release.