Allow customizing Python stub script's shebang in the Python toolchain

[brandjon]

Currently the stub script requires (via its shebang) that the target system have a python command available on PATH. If the target is PY3, the user might reasonable expect that it work if python is not available but python3 is. We can modify the shebang in this case to be #!/usr/bin/env python3. However doing so might break if the system has a (Python 3) python command but not a python3 command.

See #8665 for a case where the shebang has been a problem in practice.

[brandjon]

More generally, we can support customizing the shebang as part of the Python toolchain. I'm not clear on how useful this feature would be in the short to medium term. Long term we'd like to eliminate the Python stub altogether and replace it with a C++ one.

[brandjon]

Repurposing this bug to track the FR to customize the shebang string in the toolchain. That supersedes requests to make it work with a particular interpreter version / system environment xyz. See also #11269.

[ghost]

@brandjon I'm not sure if the Bazel maintainers prefer to resolve this independent of issues like #8446 or #137, but I'm pursuing this internally and would be happy to generate a PR within the next day or two.

• Any naming suggestions for the new py_runtime parameter?
• Are PyRuntimeInfoTest test cases sufficient, or do I need to get down and dirty with python_stub_test.sh?

[chrislovecnm]

@brandjon / @beasleyr-vmw what is the status on this issue? I just ran smack into it with trying to do a hermetic python build. When was this issue introduced?

[fahhem]

Potentially fixed by: #6632

[meteorcloudy]

This issue shouldn't be P4 anymore since Python 2 has already been deprecated.

[meteorcloudy]

@brandjon Is there any plan to fix this?

[stdbug]

Seems like the problem holds if --build_python_zip flag is used. At least when running bazel test (still puts #!/usr/bin/env python as shebang for the binary, everything is OK in the zip archive)

[meteorcloudy]

@codeschlosser Are you testing this on Windows?
Probably the problem is in the python launcher for Windows:

bazel/src/tools/launcher/python_launcher.cc

Line 45 in ba4435c

if (GetBinaryPathWithoutExtension(python_binary) != L"python") {

[stdbug]

@meteorcloudy Just the opposite, it works without an issue on Windows, I faced the problem on Debian 10. This change seems to resolve it (checked within a Docker container with hermetic python and without system-provided interpreter)

[meteorcloudy]

@codeschlosser Thanks! Can you send this change as a PR 😃

[EdSchouten]

One downside of the current approach is that in the case of remote execution, workers need to have a copy of Python present, even if you've got a hermetic Python toolchain declared. This leads to situations where people accidentally invoke the wrong copy of Python. To work around this issue, I have written a small Go tool that can be installed as "python" in $PATH. It parses the Python stub script, relaunching it with the copy of Python stored in the input root.

Source code: https://github.com/buildbarn/bb-remote-execution/blob/master/cmd/fake_python/main.go
Precompiled versions can be obtained by looking at the GitHub Actions page for that project.

[jheaff1]

One downside of the current approach is that in the case of remote execution, workers need to have a copy of Python present, even if you've got a hermetic Python toolchain declared. This leads to situations where people accidentally invoke the wrong copy of Python. To work around this issue, I have written a small Go tool that can be installed as "python" in $PATH. It parses the Python stub script, relaunching it with the copy of Python stored in the input root.

Source code: https://github.com/buildbarn/bb-remote-execution/blob/master/cmd/fake_python/main.go Precompiled versions can be obtained by looking at the GitHub Actions page for that project.

Is there any way for Bazel to invoke your fake python without having to manually add it to the PATH? I’m trying to make my builds as hermetic as I can