tellurium pip package not working on python 3.6 with pip 9.0.2: KeyError: 'pip._vendor.urllib3.contrib'

[matthiaskoenig]

Hi all,
I just tried to run tellurium in a clean virtualenv and get the following error when trying to do:

import tellurium

or

import tellurium as te

Python 3.6.3 (default, Oct  6 2017, 08:44:35) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.2.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import tellurium
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-1-c43a5795b314> in <module>()
----> 1 import tellurium

~/envs/test/lib/python3.6/site-packages/tellurium/__init__.py in <module>()
    129 
    130 # Package utilities
--> 131 from tellurium.utils.package import (
    132     searchPackage,
    133     installPackage,

~/envs/test/lib/python3.6/site-packages/tellurium/utils/package.py in <module>()
      5 
      6 from __future__ import print_function, absolute_import
----> 7 import pip
      8 
      9 

~/envs/test/lib/python3.6/site-packages/pip/__init__.py in <module>()
     43 from pip.utils import get_installed_distributions, get_prog
     44 from pip.utils import deprecation, dist_is_editable
---> 45 from pip.vcs import git, mercurial, subversion, bazaar  # noqa
     46 from pip.baseparser import ConfigOptionParser, UpdatingDefaultsHelpFormatter
     47 from pip.commands import get_summaries, get_similar_commands

~/envs/test/lib/python3.6/site-packages/pip/vcs/mercurial.py in <module>()
      7 from pip.utils import display_path, rmtree
      8 from pip.vcs import vcs, VersionControl
----> 9 from pip.download import path_to_url
     10 from pip._vendor.six.moves import configparser
     11 

~/envs/test/lib/python3.6/site-packages/pip/download.py in <module>()
     38 from pip.locations import write_delete_marker_file
     39 from pip.vcs import vcs
---> 40 from pip._vendor import requests, six
     41 from pip._vendor.requests.adapters import BaseAdapter, HTTPAdapter
     42 from pip._vendor.requests.auth import AuthBase, HTTPBasicAuth

~/envs/test/lib/python3.6/site-packages/pip/_vendor/requests/__init__.py in <module>()
     96 
     97 from . import utils
---> 98 from . import packages
     99 from .models import Request, Response, PreparedRequest
    100 from .api import request, get, head, post, patch, put, delete, options

~/envs/test/lib/python3.6/site-packages/pip/_vendor/requests/packages.py in <module>()
     10     for mod in list(sys.modules):
     11         if mod == package or mod.startswith(package + '.'):
---> 12             sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
     13 
     14 # Kinda cool, though, right?

KeyError: 'pip._vendor.urllib3.contrib'

This seems to be a bug/issue in the lates pip. Just posting this here so people will find a solution.
Workaround is to down-grade pip to 9.0.1 via

pip install pip==9.0.1

Best Matthias

[matthiaskoenig]

Found the pip issue. This seems to be a major issue affecting many projects, so hopefully fixed soon
pypa/pip#5081
But also affecting us right now.

[kirichoi]

That's a nasty one. Let's keep this issue until the issue is resolved.

[matthiaskoenig]

Yes, but seems this will not be fixed. This is basically the code in tellurium to install packages. But not officially supported :/ and probably only will get worse.

As I understand it, the issue is with code that uses import pip to access internal functionality of pip. We've never supported such usage officially, and we have explicitly documented that lack of support for some time now (albeit only in the "latest" version of the docs at https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program, because we haven't had a new stable release since that doc section was added). We also announced a reorganisation of the internals to make it clear that use of internal APIs is unsupported, last October (see https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html). That change, which is in pip 10, will break any such usage regardless of what pip a possible pip 9.0.3 release would do. So it's hard to see this as a sudden, unexpected breakage.

[hsauro]

*I've uninstalled pip 9.0.2* *H*

…

On Mon, Mar 19, 2018 at 2:43 PM, Matthias König ***@***.***> wrote: Yes, but seems this will not be fixed. This is basically the code in tellurium to install packages. But not officially supported :/ and probably only will get worse. As I understand it, the issue is with code that uses import pip to access internal functionality of pip. We've never supported such usage officially, and we have explicitly documented that lack of support for some time now (albeit only in the "latest" version of the docs at https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program, because we haven't had a new stable release since that doc section was added). We also announced a reorganisation of the internals to make it clear that use of internal APIs is unsupported, last October (see https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html). That change, which is in pip 10, will break any such usage regardless of what pip a possible pip 9.0.3 release would do. So it's hard to see this as a sudden, unexpected breakage. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#354 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABAZDkveEr7H_ojBpQ4wM69vpBBiddoJks5tgCZlgaJpZM4Sw6O_> .

[matthiaskoenig]

Basically we have to rewrite the few lines of code used for pip imports via the following

Having said all of the above, it is worth covering the options available if you decide that you do want to run pip from within your program. The most reliable approach, and the one that is fully supported, is to run pip in a subprocess. This is easily done using the standard subprocess module:

subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'my_package'])

If you want to process the output further, use one of the other APIs in the module:

reqs = subprocess.check_output([sys.executable, '-m', 'pip', 'freeze'])

[0u812]

Thanks Matthias, I'll fix that line in Tellurium.

[luciansmith]

@matthiaskoenig : This seems as good a place to ask as any: you aren't using these tellurium wrappers for pip any more, are you? I assume that nowadays everyone just uses pip directly. I noticed them because they use distutils which is being removed in 3.12, and figured I'd just remove the functionality entirely.