💫 Improve "spacy download" and support for different installation prefixes

[honnibal]

Currently spacy download seems to struggle when Python is installed into a different prefix. See #1220 .

Possible resolutions could include an argument to spacy download. Ideal resolution would be detecting the correct prefix, and installing there.

If you work on this, please try to pay attention to cross-platform and cross-version support. Try to use the built-in pkg_resources module. The tools in spacy.compat might assist with this. Using pathlib.Path for file-system manipulation is also helpful.

[ines]

Related issues: #924, #1284

Also related and could be included in the same enhancement to spacy download:

#1373: python -m spacy download should not overwrite an existing model of the same name

I think the solution should be as simple as appending the model name and version to the file URL in cli.download, e.g. #egg=en_core_web_sm-2.0.0a0 (see here). This should tell pip which package and version you're trying to download and make it behave like it does when you're downloading packages from PyPi.

Edit: Okay, looks like this wasn't so easy – specifying the version with #egg= doesn't seem to work. However, with only the package name, it does skip the download – but this doesn't take the version into account. But maybe I was doing something wrong...

One option could be to make this the default behaviour, and introduce a -U flag for upgrading only. But this would be a big change and not exactly intuitive (since all the user gets to see is the default pip response, which doesn't always make it clear whether a package was installed or whether all requirements were already satisfied.) The alternative would be to have spaCy perform the check – but I'm not sure I like that option, since it'd again require getting the installed distributions and performing hacky checks on them (which always introduces error potential).

But maybe that's what we have to do. If spaCy thinks you already have the model and version installed, spacy download exits and prints a warning, telling you to run spacy download -U to download and install regardless.

[jnothman]

FWIW, for me pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-1.2.0/en_core_web_sm-1.2.0.tar.gz#egg=en_core_web_sm-1.2.0 successfully avoids downloading when run a second time.

I'm a bit confused about @honnibal's description of #1220 as being an issue about Python being installed into a different prefix. download_model currently correctly ensures that it is the current Python version, and its PYTHONPATH being used (although an alternative is to avoid subprocess and just use import pip; pip.commands.InstallCommand().main(args_list) assuming that API is stable). The problem in #1220 is that the user does not want to install into the default site-packages, particularly because it is not writable to that user. Forwarding args to pip is one option, but so is asking the user to create a pip.conf containing [install]\nuser=true because that might be useful to them in any case... Changed my mind on pip.conf which may override virtualenvs...

[parajain]

Facing same issue:

  error: could not create '/usr/local/lib/python2.7/dist-packages/en_vectors_web_lg': Permission denied

    ----------------------------------------
Command "/usr/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-qn3PBj-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-g4hcUl-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-qn3PBj-build/

I tried

pip2 install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-1.2.0/en_core_web_sm-1.2.0.tar.gz#egg=en_core_web_sm-1.2.0

 Running setup.py clean for en-core-web-sm
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-vXeKyY/en-core-web-sm/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-build-vXeKyY/en-core-web-sm/setup.py'

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.