Enable custom py_binary stub_template attribute

[fahhem]

Fixes #137

Wasn't too hard after all!

One note: the fact that the TemplateExpansionAction constructors switch the order of the template and the output depending on the type of the template (an Artifact vs a Template) took the longest amount of time in debugging for this PR.

Lastly, I'm aware I didn't document the attribute very well. I'm open to suggestions!

[brandjon]

Thanks for the PR! Please see my comments in #137. I'd consider this PR blocked pending a design doc. I'll see if I can write that doc tomorrow.

[laurentlb]

What's the current state, @brandjon?

[aiuto]

ping everyone.

• tests need to be fixed
• to we still want this or is it obsolete?

[meisterT]

ping @brandjon

[danmx]

Any updates on this PR?

[hrfuller]

I'm running into some issues debugging an os error in the stub template and having a custom script would be useful.

[sgowroji]

Hello @fahhem, The above PR has been stale from long time. Could you please check buildkite failures and address the above comments. Thanks

[fahhem]

I have this working on the bazel 4.0.0 tag internally. However, I don't want to spend time on this PR until I have a signal that someone from the bazel team will actually consider it. Last I checked, the issue was 'under design review' a couple years ago and no updates since. It's marked a P4 (though this PR is a P1??).

If there's an actual chance of getting this in I'd be happy to update the branch and get it in

[sventiffe]

@brandjon can you have a look/comment, please?

[meteorcloudy]

@rickeylev is currently working on Python rules, he can probably take a look ;)

[fahhem]

I'm actively working on the CI failures, will post again when it's ready

[fahhem]

Unfortunately I can't seem to figure out why @bazel_tools//tools/python:python_stub_template.txt doesn't exist in CI when it's there, checked in, and mentioned in BUILD.tools like the other files. Anyone have an idea on what I'm missing in adding a file to @bazel_tools?

[rickeylev]

I somewhat echo Jon's thinking in #137 from back in Nov 2018.

This is more appropriate as an attribute of py_runtime rather than py_binary. Most of what the stub template does is find the interpreter, runfiles, and setup the runtime environment for a binary. The majority of this doesn't vary on a per-binary bases. It does vary on a per-platform basis, though, which is better handled by the toolchain lookup process. A good example of this is windows vs mac vs linux differences; e.g. windows might use a batch file (or powershell or etc), while mac might use an old version of bash, and linux uses a newer version. There's other stuff, too, like coverage setup and execution. Handling those sort of differences on a per-binary basis is a hassle.

I would accept adding an optional {something}_template[1] attribute to py_runtime, which is treated as a file. If py_binary finds this, it uses it as it would the builtin template. The API for this (i.e the set of keywords substituted and their values) is considered experimental and unstable (at least until the py rule implementation is moved out of Bazel itself -- then we can revisit).

[1] not "stub_template"; the startup code is fairly complicated and does real things; "executable_template", "binary_template", etc are all more descriptive.

If you want to experiment with something to allow a greater degree of customization, then we can try adding a callback somewhere on the py_runtime_pair or PyRuntimeInfo values to give the toolchain greater control over how the executable is created (basically allowing the toolchain to directly do what PythonSemantics.createExecutable is able to do).

All that said, there's definitely still some binary-level customizations to propagate into the startup process. e.g., a binary might want to have certain interpreter flags set at startup that are difficult/impossible to customize post-interpreter startup (-b, -u, and other flags come to mind). So I'm +1 on exposing some attribute to pass along that information.

[fahhem]

@rickeylev Thank you for the comment! py_runtime came out years after I opened this PR, and now that it exists I'd also love to move this there.

However, seeing as I can't make progress due to the @bazel_tools files not being exposed/updated the way they used to be, I can't make any useful changes here. I'm currently leaning to close this PR out, if anyone is interested in fixing this PR or getting a py_runtime version over the line, please do so. In the meantime, we'll keep updating our fork every few months, as we have for 4 years

[fahhem]

Somehow I got it to work in #16806, so going to close this in favor of that PR getting in.