Add support for all stacks

[jpena-r7]

Describe the Enhancement

I would like to propose making the python buildpack support all stacks similar to the java buildpack.

The groundwork for this change has already been laid by the PR to the cpython buildpack implemented here. This allows python to be installed from source on non-paketo stacks.

With the cpython buildpack updated, the remaining buildpacks are: pip, pipenv, poetry, miniconda, and conda-env-update

Possible Solution

For all buildpacks the primary change needed is to update stack definitions in the buildpack.toml file as show below. Other changes are noted below. Of course unit and integration tests will be updated accordingly as needed.

cat buildpack.toml | yj -tj | jq '. + {stacks: [{id: "*"}]} | . |= (.metadata.dependencies[] += (.metadata.dependencies[] | .stacks = ["*"]))' | yj -jt

See below docker hub images and READMEs on how all of these proposed changes were tested before this issue was proposed.

• https://hub.docker.com/r/jericop/pip
• https://hub.docker.com/r/jericop/pipenv
• https://hub.docker.com/r/jericop/poetry
• https://hub.docker.com/r/jericop/miniconda
• https://hub.docker.com/r/jericop/conda-env-update

The pip buildpack needs to be updated first before pipenv and poetry because they depend on pip.
The miniconda buildpack needs to be updated first before conda-env-update because it depends on miniconda.

pip

I found the following issue in the pip buildpack. This is outside of the scope of this issue. I plan to create a separate issue in the pip buildpack repo for this but noting it here for reference because it is related to the poetry buildpack change below.

• The pip3 and pip3.x shebang lines point to invalid paths for python
  • e.g, #!/tmp/d20220606-38-1nlqzzv/bin/python3.10
  • This only occurs when using the bionic or jammy builders because python is pre-commpiled on another system
  • This makes is so that you can't run pip3 directly but have to run python -m pip instead
  • See: https://hub.docker.com/r/jericop/pip

poetry

Because of the above mentioned issue the pip process needs to be updated to use python -m pip instead. This change solves the problem regardless of whether the pip buildpack is updated to fix this issue.

# Create a patch file to fix above noted issue
cat <<PATCH_EOF > pip-executable-changes.patch 
diff --git a/poetry_install_process.go b/poetry_install_process.go
index 77325fe..7eb1a0c 100644
--- a/poetry_install_process.go
+++ b/poetry_install_process.go
@@ -32,7 +32,7 @@ func (p PoetryInstallProcess) Execute(version, targetLayerPath string) error {
 	buffer := bytes.NewBuffer(nil)
 
 	err := p.executable.Execute(pexec.Execution{
-		Args: []string{"install", fmt.Sprintf("poetry==%s", version), "--user"},
+		Args: []string{"-m", "pip", "install", fmt.Sprintf("poetry==%s", version), "--user"},
 		// Set the PYTHONUSERBASE to ensure that pip is installed to the newly created target layer.
 		Env:    append(os.Environ(), fmt.Sprintf("PYTHONUSERBASE=%s", targetLayerPath)),
 		Stdout: buffer,
diff --git a/run/main.go b/run/main.go
index df7b32c..654c013 100644
--- a/run/main.go
+++ b/run/main.go
@@ -26,7 +26,7 @@ func main() {
 		poetry.Detect(poetry.NewPyProjectParser()),
 		poetry.Build(
 			postal.NewService(cargo.NewTransport()),
-			poetry.NewPoetryInstallProcess(pexec.NewExecutable("pip")),
+			poetry.NewPoetryInstallProcess(pexec.NewExecutable("python")),
 			poetry.NewSiteProcess(pexec.NewExecutable("python")),
 			Generator{},
 			chronos.DefaultClock,
PATCH_EOF

Motivation

While most users are well served by paketo stacks and builders, there are instances where stacks based on different operating systems such as RHEL are required. Adding this support will help users with such requirements.

[robdimsdale]

Thanks @jpena-r7 - I'm in favor of supporting all stacks across the python buildpack.

I am slightly confused as to why we need to switch from pip install to python -m pip install if we fix the underlying issue you mentioned in pip/cpython where the shebang lines are pointing to the wrong location.

It sounds like the proposal to switch to python -m pip is a workaround a bug in an upstream buildpack. Is that a fair characterization? Am I missing something?

[jpena-r7]

You are correct that fixing pip should also resolve the issue for poetry. While the proposed switch to python -m pip install is a workaround, I think it actually insulates the poetry buildpack from such potential issues in the future.

I expect that you would need to update the shebang after the dependency is created or ensure that the dependency is built inside a similar layer directory structure to what will be created in the end. I was trying to avoid changes to the dependency creation process and find a universal solution to the problem, which is why I proposed switching to python -m pip install.

Let me know which direction you would like to go and I can create an issue for the pip buildpack.

[robdimsdale]

Sure that makes sense. I'm happy with python -m pip install if you think that is more idiomatic for python users.

I see some examples around (e.g. here and here) suggesting that python -m pip install is preferred for pretty much exactly this reason.

So I'm fairly convinced that this is a reasonable change to make. If you prefer to use python -m pip install then please do that!

[jpena-r7]

Excellent! python -m pip install it is. 😄

One thing that will need to happen before I can create PRs is that a new release of cpython is needed because pip, pipenv, and poetry all rely on it for integration tests.

Once that is in place should I create PRs and post them here for review?

[robdimsdale]

I just released cpython v1.4.0 so you should be unblocked on that front.

Yeah, I think creating PRs in each repo and referencing this tracking issue is probably the best way to proceed. That way they are all easily discoverable and reference the same conversation here.

[jpena-r7]

The following PRs were created.

• Add support for all stacks pip#306
• Add support for all stacks miniconda#265

Both will need to be merged and released before the remaining buildpacks can be updated.

[jpena-r7]

The remaining PRs have been created.

• Add support for all stacks pipenv#279
• Add support for all stacks poetry#103
• Add support for all stacks conda-env-update#174

Once these have been merged and released this issue can be closed out.

[robdimsdale]

All buildpacks above have been released with support for * stacks.

[robdimsdale]

With the release of the Python Buildpack v.2.2.0 this issue can be closed!

Thanks for all the PRs, @jpena-r7 🎉