Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python: add patchelf option #840

Closed
wants to merge 3 commits into from
Closed

python: add patchelf option #840

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f712021
examples/python-poetry: test for pandas, scipy, opencv and scikit
bobvanderlinden Oct 10, 2023
58a55d7
python: add venv.patchelf option
bobvanderlinden Oct 10, 2023
d2fcec5
examples/python-poetry: enable patchelf option
bobvanderlinden Oct 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions examples/python-poetry/.test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,9 @@ POETRY_VENV="$PWD/.venv"
python --version
poetry --version
poetry run python -c 'import numpy'
poetry run python -c 'import pandas'
poetry run python -c 'import scipy'
poetry run python -c 'import opencv'
poetry run python -c 'import sklearn'
python -c 'import numpy'
python -c 'import pjsua2'
1 change: 1 addition & 0 deletions examples/python-poetry/devenv.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,6 @@
languages.python = {
enable = true;
poetry.enable = true;
venv.patchelf = true;
};
}
2 changes: 2 additions & 0 deletions examples/python-poetry/devenv.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
inputs:
nixpkgs:
url: github:NixOS/nixpkgs/nixpkgs-unstable
autopatchelf:
url: github:bobvanderlinden/autopatchelf/7fe81db2d94989e41e52fcab65c9d0a88d46e25c
326 changes: 290 additions & 36 deletions examples/python-poetry/poetry.lock
View file
Open in desktop

Large diffs are not rendered by default.

8 changes: 6 additions & 2 deletions examples/python-poetry/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,12 @@ authors = ["Bob van der Linden <bobvanderlinden@gmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"
numpy = "^1.24.1"
python = ">=3.9,<3.13"
pandas = "^2.1.1"
numpy = "^1.26.0"
scipy = "^1.11.3"
opencv-python = "^4.8.1.78"
scikit-learn = "^1.3.1"


[build-system]
Expand Down
1 change: 1 addition & 0 deletions examples/python/devenv.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,6 @@

venv.enable = true;
venv.requirements = ./requirements.txt;
venv.patchelf = true;
};
}
21 changes: 21 additions & 0 deletions src/modules/languages/python.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,16 @@
let
cfg = config.languages.python;

autopatchelf-flake = inputs.autopatchelf or (throw ''
To use patchelf, you need to add the following to your devenv.yaml:

inputs:
autopatchelf:
url: github:bobvanderlinden/autopatchelf/7fe81db2d94989e41e52fcab65c9d0a88d46e25c
'');

autopatchelf = autopatchelf-flake.packages.${pkgs.system}.default;

requirements = pkgs.writeText "requirements.txt" (
if lib.isPath cfg.venv.requirements
then builtins.readFile cfg.venv.requirements
Expand Down Expand Up @@ -135,6 +145,12 @@ in
description = "Whether `pip install` should avoid outputting messages during devenv initialisation.";
};

venv.patchelf = lib.mkOption {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this should be a no-op on non-elf architectures to not break macOS by accident.

type = lib.types.bool;
default = false;
description = "Whether to patch ELF files in the Python virtual environment, so that they refer to libraries/packages defined in devenv.";
};

poetry = {
enable = lib.mkEnableOption "poetry";
install = {
Expand Down Expand Up @@ -219,6 +235,11 @@ in
source ${initVenvScript}
'') ++ (lib.optional cfg.poetry.install.enable ''
source ${initPoetryScript}
'') ++ (lib.optional cfg.venv.patchelf ''
if [ -d "$VIRTUAL_ENV" ]
then
${autopatchelf}/bin/autopatchelf --libs ${config.devenv.profile}/lib --path $VIRTUAL_ENV
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know the devenv code well enough but find this approach interesting.
Here a few hints: 1. Avoid running patchelf repeatably as might change the binary on every run (sometimes with weird side effects). 2. Make sure that ${config.devenv.profile}/lib cannot be garbage collected.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback!

  1. Avoid running patchelf repeatably as might change the binary on every run (sometimes with weird side effects).

Indeed. Maybe storing the profile inside VIRTUAL_ENV and comparing it is a good enough approach.

  1. Make sure that ${config.devenv.profile}/lib cannot be garbage collected.

I presumed the profile is not GCed, as it has a GC root that in .devenv. Not entirely sure whether this is a good presumption though.

fi
'')
);
};
Expand Down