Requirements file which contains -e with a relative local path, is only installable from a specific directory

[Thiel]

Environment

• pip version: 18.1
• Python version: 2.7.14
• OS: Windows (but I don't think it's restricted to a specific platform)

Description
I would like to be able to call pip install -r path/to/requirements.txt from any directory. However usage of --editable/-e with a relative path inside of a requirements.txt assume that the path is relative the current directory I'm currently in, rather than the directory where the file is located.

My specific use case, involve usage of multiple independent requirements.txt in a directory hierarchy where they can reference each other. And seems to exhibit the same issue.

Expected behavior
Expected behavior would be to not require being in a specific directory in order to call pip install -r path/to/requirements.txt when --editable/-e is being used with relative local path.

How to Reproduce

#! /bin/bash

mkdir dir/
echo '-e .' > dir/requirements.txt
cat - > dir/setup.py <<EOF
from distutils.core import setup, Extension
setup(name='foo', version='1.0')
EOF

pip install -r dir/requirements.txt # FAIL

cd dir
pip install -r requirements.txt # PASS

Output

Directory '.' is not installable. File 'setup.py' not found.
Obtaining file:///C:/project/repro_github/pip_change_dir_r/dir (from -r requirements.txt (line 1))
Installing collected packages: foo
  Found existing installation: foo 1.0
    Uninstalling foo-1.0:
      Successfully uninstalled foo-1.0
  Running setup.py develop for foo
Successfully installed foo

Potential solution
I assume that pip is missing a chdir when the requirements is being interpreted. And doing so would fix all mentioned issue.
Would there be a reason that prevent us to do that?

[cjerdonek]

I wonder how this could be rolled out if people are relying on the cwd behavior for relative paths. Would we need a mechanism to warn people of the change in advance (e.g. triggered by the appearance of relative paths inside a requirements file)?

[piotr-dobrogost]

Duplicate of issue #3772
Some history around relative paths in requirements files can be found in issue #328.

[pradyunsg]

Thanks (again!) for looking up the duplicate issue that would have been difficult to find otherwise @piotr-dobrogost! :)

@cjerdonek I agree with the closing comment on #3772 that this should be handled in Requirements 2.0.

[Thiel]

@cjerdonek Good point. In order keep backward compatibly there would have to be a hint to distinguish the 2 cases. The only ideas that come to mind are having a distinguishable syntax in the requirements.txt or in the command line flag. I have to admit I'm not a big fan of either solution.

@pradyunsg By requirements 2.0 do you mean Pipfile ?

[pradyunsg]

@Thiel Yes. That's what it has become now. :)

[kevincantu]

We could use a / like .gitignore and .gitattributes do to indicate a path relative to the repository root, or some other symbol like that.

So ../other/requirements.txt would still go wherever it goes today. But /my/other/requirements.txt would go to the same file no matter which directory I attempted to install it from.

[DavidAntliff]

Since requirements.txt supports environment variables, could there be a special variable set by pip that is the absolute canonical location of the requirements.txt file? Then the requirements.txt file could reference, for example:

-e ${PIP_REQUIREMENTS_PATH}[test]

[uranusjr]

Yes I think that’s a good idea and would approve a PR adding it. (The variable name needs some discussion since PIP_REQUIREMENTS_PATH sounds like it points the the requirements.txt itself, not the directory.) We can also special-case PWD (already available on Linux and macOS but would provide consistency across all platforms), and maybe in the long run deprecate relative paths entirely to avoid ambiguity.

[sbidoul]

Alternatively we could transition to making all relative paths in requirement files to be relative to the requirement file itself instead of the current working directory - which is a surprising behaviour, especially when you have nested requirement files. That is a breaking change, which would need to be weighted carefully, of course.

[DavidAntliff]

Any suggestions for a better variable name? PIP_REQUIREMENTS_ROOT? PIP_REQUIREMENTS_DIRNAME?

[kevincantu]

PIP_REQUIREMENTS_EDITABLE_PREFIX

[uranusjr]

The variable should make sense for non-editable requirements as well so having EDITABLE in there is wrong IMO.

[kevincantu]

That's what -e means, though. I'm a little rusty, though. Are requirements like this even valid without that flag?

path/to/mylibrary

[uranusjr]

Yes

[TBBle]

Since the requirements file doesn't need to be named requirements.txt, perhaps something like PIP_THIS_FILE_DIR to indicate that the env-var is only set while processing "this file", i.e. the one it appears in, and will have a different value for each such file, i.e. a -r ${PIP_THIS_FILE_DIR}/subdir/site-local.txt line in a requirements.txt would see a different value of the env-var inside site-local.txt.

As a comparison, CMake uses CMAKE_CURRENT_LIST_DIR for the same use-case, where the equivalent filename for requirements.txt is canonically CMakeLists.txt.

[DavidAntliff]

@TBBle for what it's worth, I like this idea. Sometimes I install multiple requirements files (that aren't even called requirements.txt in some cases), from different directories, so the path should be relative to the file currently being processed.

[lunalucadou]

I second the suggestion to create a git/VCS-root environment variable. My use case is as follows:

• Project repo is organized as several mini-modules
• There are several database model classes that are basically identical across each module
• Because of several bugs arising from forgetting to update all the model classes across all modules together, we are moving the database models to their own module
  • Due to some quirks of our testing setup, it is not feasible to publish it to a package registry; local project installs are the best way to go
• Each module's requirements file (located at <module>/src/requirements.txt) received a new line, ../../models/src, to make the new module available
• However, our CI is setup to cd <module>; pip install -r src/requirements.txt, which causes errors because ../../models/src does not exist from the module root where pip is executed from
• Having a VCS-root variable to use in the requirements file (e.g. $REPO_ROOT/models/src) would allow us to easily run pip install -r requirements.txt or pip install -r src/requirements.txt

Of course, there is a simple workaround in my case (since CI always does pip install from the module root, just change the requirements file to ../models/src and let my coworkers know they need to change their IDE pip configurations), but it would be nice to have the flexibility of a VCS-root environment variable.