Shebang length exceeded in pip executable

[ajgillis]

When a new virtualenv is created, there are a number of things installed into that virtualenv. One of these things is the python package installer, pip.

On Linux, the pip executable is a shell script. At the top of this pip shell script is a shebang (#!) line that specifies the python interpreter in the virtualenv. This line contains the absolute path to the python interpreter.

If the absolute path to the python interpreter in the virtualenv is very long (deeply nested and/or large path names), it may exceed the maximum length allowed for a shebang line.

The maximum length for a shebang line is limited in the kernel by BINPRM_BUF_SIZE, set in /usr/include/linux/binfmts.h. On the Linux machines I looked at, this limit is set to 128.

So, when the path, to where a python virtualenv is created, becomes too long, using that virtualenv's pip will not work. It will fail with the error: "bad interpreter: No such file or directory"

A work-around for this problem is to not run pip directly, but by run the virtualenv's python interpreter and pass the pip script to python as the source to execute.

[d6e]

I am also experiencing this issue.

[sholsapp]

Me too.

[sblask]

Thanks for the workaround!

[pfmoore]

Note that the wrapper scripts are actually generated by setuptools (if you install from sdist) or distlib (if you install from wheel). So any fix for this issue should really be requested from those projects.

But if it's an OS limitation, maybe there simply isn't a viable fix? I seem to remember that at one time, Perl used a magic incantation to execute scripts (googles a bit...) Yes, something like this (translated for Python)

#!/bin/sh
eval 'exec /the/long/interpreter/path/to/python $0 ${1+"$@"}'

You'd need some extra stuff so that Python didn't try to execute the eval-exec line, but that might work.

If anyone wanted to propose something like that as a feature request for setuptools and distlib, that'd be great.

[b-long]

So, when the path, to where a python virtualenv is created, becomes too long, using that virtualenv's pip will not work.

It would be really nice if running virtualenv on a system in which this problem occurs, the command would output some warning (or even an error) that being in a directory with such a long path name is essentially not supported by the defaults at the moment.

[jantman]

I'm hitting this also. It's specifically a problem for me as I'm running pip from within Jenkins builds, which have very long paths to the pwd. Interestingly, I have a number of identically-built build slaves (all have BINPRM_BUF_SIZE set to 128, and the same pip and python and virtualenv versions) and I'm experiencing this on some of them but not others, even with varying path lengths.

I like @b-long's suggestion; if virtualenv is going to create a wrapper that doesn't work on the current system, it should at least give the user a warning, if not flat-out fail.

[pfmoore]

It's not clear how virtualenv could detect this situation. We can't exactly use a value from a C header, and I don't know how we'd detect we're on a system with this limitation (Windows doesn't have it, does OSX?)

I'm inclined to close this issue as an OS limitation, not a virtualenv problem.

[jantman]

For what it's worth, some thoughts...

1. can ctypes do this?
2. How about something as simple as:

# on Linux, BINPRM_BUF_SIZE == 128, giving us a maximum shebang length of 127
# 2 bytes for #!, 11 bytes for '/bin/python' leaves us with 114
if sys.platform() == 'Linux' and len(home_dir) > 114:
    log.warn("bin/activate may not work your system, as the length of the shebang line will exceed 128 characters")

[pfmoore]

1. No idea, sorry.
2. I'm a Windows user so I have no opinion. If you think this is reasonable, I suggest raising a PR and seeing what the Linux people think...

[b-long]

I'm sure other users would also appreciate the change I suggested. Can we leave this open for comment and to inspire a PR? Thanks @jantman and @pfmoore 😄

[jantman]

BINPRM_BUF_SIZE is defined in a kernel header, and seems to only be used by a small handful of kernel functions. I don't believe there's any way to programmatically detect this (short of reading in the headers, which isn't anywhere near feasible).

That being said, I think it's reasonable to assume that this won't be changing any time soon (there's a quick overview of the maximum shebang length here). The current length is also clearly given in the "Notes" section of the execve(2) man page, where it states, "A maximum line length of 127 characters is allowed for the first line in a #! executable shell script."

I would assume that all POSIX-compliant operating systems have a limit of some sort, though it seems that some of them, specifically the BSD variants, may have limits upwards of 8,000 characters.

While it seems like it would be a reasonable solution to simply hard-code something as naive as my above suggestion (if the platform is Linux and the line is longer than 127 characters), it feels overly crude and specific and inflexible to me.

I'm still thinking about how I'd programmatically test this in a way that wouldn't add too much overhead to virtualenv (granted, we'd only need to test this at venv creation time, so I'd assume some overhead might be acceptable).

I have some simple proof-of-concept code to test whether or not a given path is an acceptable she-bang length, but it's rather ugly as it writes out a file to disk and then executes the file to capture the output. The example and output is here: https://gist.github.com/jantman/ba39f98936643bc948bd

I'm going to have to call it a night, but I'll try to come back to this. I'd certainly appreciate the input of any other Linux users, or users of other OSes who can confirm whether or not they have a similar limit.

[johntyree]

Instead of trying to do something tricky like make assumptions about the limit, why not just make a functional test bash script? If python correctly gets invoked then everything is peachy, otherwise it isn't.

[pfmoore]

To reiterate:

1. Please submit PRs to setuptools and distlib if you want this fixed - the code is not in virtualenv or pip.
2. To be acceptable, code would need to work on all supported platforms, including OSX, BSD, ... and not just Linux (platform-specific conditionals are acceptable, of course, if there's no other way)

Closing this issue, as it isn't a virtualenv problem (you can get the same effect by installing a full Python interpreter to a long directory name)..

[dotchev]

We just hit this issue.

Why not use the python from the PATH? virtualenv already prepends it with the bin dir.

#!/usr/bin/env python

[pfmoore]

virtualenv already prepends it with the bin dir

Only if you activate the virtualenv. Virtualenv supports being used without activation.

[crccheck]

If anyone find this confusing:

A work-around for this problem is to not run pip directly, but by run the virtualenv's python interpreter and pass the pip script to python as the source to execute.

That means instead of pip install -r requirements.txt do python -m pip install -r requirements.txt