Pip should interpret relative paths in requirements.txt relative to requirements.txt

[OptiverTimAll]

• Pip version: 8.1.2
• Python version: 2.7.9
• Operating System: CentOS 5

Background:

I'm coming up with a plan for repeatable, automated deployment of production applications. The problem-space looks something like this:

1. A commit triggers the build system.
2. The build system checks out the new version, installs all the required dependencies, runs all the tests.
3. If the tests pass, the build system uses pip wheel to archive the exact version of the code (and the exact versions of all the dependencies) that passed the tests at that time.
4. All the wheels get posted on a dumb HTTP server.
5. I want pip, running on the target host, to download and install all those wheels.

Steps to reproduce:

# Make a directory to serve wheels from
SOURCE=$(mktemp -d)

# Put some wheels in it.
cd $SOURCE
pip download --only-binary :all: unittest2

# Make a requirements.txt that references all the wheels.
echo "--no-index" > requirements.txt
ls *.whl >> requirements.txt

# Start a webserver serving up this directory.
python -m SimpleHTTPServer

# In another session, make a virtualenv to install things into
virtualenv tmp-venv
./tmp-venv/bin/pip install --upgrade pip setuptools wheel

# Deploy!
./tmp-venv/bin/pip install -r http://localhost:8000/requirements.txt

Expected results:

I expected pip to download requirements.txt file, see the paths inside it, and resolve them relative to the requirements.txt location.

Actual results:

$ ./tmp-venv/bin/pip install -r http://localhost:8000/requirements.txt
Requirement u'argparse-1.4.0-py2.py3-none-any.whl' looks like a filename, but the file does not exist
Requirement u'linecache2-1.0.0-py2.py3-none-any.whl' looks like a filename, but the file does not exist
Requirement u'six-1.10.0-py2.py3-none-any.whl' looks like a filename, but the file does not exist
Requirement u'traceback2-1.4.0-py2.py3-none-any.whl' looks like a filename, but the file does not exist
Requirement u'unittest2-1.1.0-py2.py3-none-any.whl' looks like a filename, but the file does not exist
Processing ./argparse-1.4.0-py2.py3-none-any.whl
Exception:
Traceback (most recent call last):
  File "/tmp/tmp-venv/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/tmp/tmp-venv/lib/python2.7/site-packages/pip/commands/install.py", line 310, in run
    wb.build(autobuilding=True)
  File "/tmp/tmp-venv/lib/python2.7/site-packages/pip/wheel.py", line 750, in build
    self.requirement_set.prepare_files(self.finder)
  File "/tmp/tmp-venv/lib/python2.7/site-packages/pip/req/req_set.py", line 370, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/tmp/tmp-venv/lib/python2.7/site-packages/pip/req/req_set.py", line 587, in _prepare_file
    session=self.session, hashes=hashes)
  File "/tmp/tmp-venv/lib/python2.7/site-packages/pip/download.py", line 798, in unpack_url
    unpack_file_url(link, location, download_dir, hashes=hashes)
  File "/tmp/tmp-venv/lib/python2.7/site-packages/pip/download.py", line 705, in unpack_file_url
    unpack_file(from_path, location, content_type, link)
  File "/tmp/tmp-venv/lib/python2.7/site-packages/pip/utils/__init__.py", line 599, in unpack_file
    flatten=not filename.endswith('.whl')
  File "/tmp/tmp-venv/lib/python2.7/site-packages/pip/utils/__init__.py", line 482, in unzip_file
    zipfp = open(filename, 'rb')
IOError: [Errno 2] No such file or directory: u'/tmp/argparse-1.4.0-py2.py3-none-any.whl'

Notes:

Before filing this issue, I came across #328 which seems similar, but weirdly specific to file: URLs. Maybe the same solution could fix both tickets.

As a workaround, I can use pip --no-index --find-links but it requires the client to have a bit more context than just an opaque URL.

I guess this would technically be a breaking change, although I can't really imagine why somebody would keep a complete package archive locally and download requirements.txt over the network.

[dstufft]

While I agree this is confusing, because it is a breaking change I think this would be better handled during the migration to requirements 2.0 format instead (see #1795).

[tony]

This one kinda hurts because I wanted to use an editable package pulled in as a git submodule (not that I had to do it, just wanted the open source community to get the improvements while not stymying my momentum on the larger task at hand).

Going to have to find another way to do it.

[mitar]

Yes, my use case is also that I have packages through git submodules and then I would like to locally reference them.