Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add (experimental) option to use current active python to create venv ("pyenv way") #4852

Merged
merged 5 commits into from
Jan 17, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ virtualenvs.in-project = null
virtualenvs.options.always-copy = true
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs" # /path/to/cache/directory/virtualenvs
virtualenvs.prefer-active-python = false
```

## Displaying a single configuration setting
Expand Down Expand Up @@ -188,6 +189,13 @@ Give the virtual environment access to the system site-packages directory.
Applies on virtualenv creation.
Defaults to `false`.

### `virtualenvs.prefer-active-python` (experimental)

**Type**: boolean

Use currently activated Python version to create a new venv.
Defaults to `false`, which means Python version used during Poetry installation is used.

### `repositories.<name>`

**Type**: string
Expand Down
17 changes: 16 additions & 1 deletion docs/managing-environments.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,29 @@ To achieve this, it will first check if it's currently running inside a virtual
If it is, it will use it directly without creating a new one. But if it's not, it will use
one that it has already created or create a brand new one for you.

By default, Poetry will try to use the currently activated Python version
By default, Poetry will try to use the Python version used during Poetry's installation
to create the virtual environment for the current project.

However, for various reasons, this Python version might not be compatible
with the `python` requirement of the project. In this case, Poetry will try
to find one that is and use it. If it's unable to do so then you will be prompted
to activate one explicitly, see [Switching environments](#switching-between-environments).

{{% note %}}
If you use a tool like [pyenv](https://github.com/pyenv/pyenv) to manage different Python versions,
you can set the experimental `virtualenvs.prefer-active-python` option to `true`. Poetry
than will try to find the current `python` of your shell.

For instance, if your project requires a newer Python than is available with
your system, a standard workflow would be:

```bash
pyenv install 3.9.8
pyenv local 3.9.8 # Activate Python 3.9 for the current project
poetry install
```
{{% /note %}}

## Switching between environments

Sometimes this might not be feasible for your system, especially Windows where `pyenv`
Expand Down
2 changes: 2 additions & 0 deletions src/poetry/config/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ class Config:
"in-project": None,
"path": os.path.join("{cache-dir}", "virtualenvs"),
"options": {"always-copy": False, "system-site-packages": False},
"prefer-active-python": False,
},
"experimental": {"new-installer": True},
"installer": {"parallel": True, "max-workers": None},
Expand Down Expand Up @@ -138,6 +139,7 @@ def _get_normalizer(name: str) -> Callable:
"virtualenvs.in-project",
"virtualenvs.options.always-copy",
"virtualenvs.options.system-site-packages",
"virtualenvs.options.prefer-active-python",
"experimental.new-installer",
"installer.parallel",
}:
Expand Down
5 changes: 5 additions & 0 deletions src/poetry/console/commands/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,11 @@ def unique_config_values(self) -> Dict[str, Tuple[Any, Any, Any]]:
lambda val: str(Path(val)),
str(Path(CACHE_DIR) / "virtualenvs"),
),
"virtualenvs.prefer-active-python": (
boolean_validator,
boolean_normalizer,
False,
),
"experimental.new-installer": (
boolean_validator,
boolean_normalizer,
Expand Down
33 changes: 32 additions & 1 deletion src/poetry/utils/env.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
import tomlkit
import virtualenv

from cleo.io.outputs.output import Verbosity
from packaging.tags import Tag
from packaging.tags import interpreter_name
from packaging.tags import interpreter_version
Expand Down Expand Up @@ -456,6 +457,30 @@ class EnvManager:
def __init__(self, poetry: "Poetry") -> None:
self._poetry = poetry

def _detect_active_python(self, io: "IO") -> str:
executable = None

try:
io.write_line(
"Trying to detect current active python executable as specified in the config.",
verbosity=Verbosity.VERBOSE,
)
executable = decode(
subprocess.check_output(
list_to_shell_command(
["python", "-c", '"import sys; print(sys.executable)"']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not necessarily universal.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean python? In theory yes.

But if you have e.g. python2 and python3 in your shell but no python it is impossible to decide what python the user expects.

The main use case for the implementation here is for users who switch there python to different versions (e.g. with pyenv). I'm not aware of other tools, but I guess they all will provide a python. As a fallback if no python is found and the option is activated, the default behavior will be used.

The only generic solution for all edge cases I can imagine right now, is to let the user set a "default python executable" that should be used. But in this case I have some security concerns. Running a user provided command listed in a config, doesn't sound good.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not just about python2 vs python3. There also are other issues like for .exe extensions on windows etc depending on where poetry is running. Additionally, quite a lot of tools rely on environment magic to wrangle which python version to use etc., this can also be impacted by how the subprocess shell is spawned (eg: is it a login shell?, how does the tool configure the "current" python?, are profile scripts required? is the PATH discoery set correctly? etc.). There are too many variables here. Not sure if this is a show stopper, but we should really consider if this makes things more obscure.

An additional point might be that we should fall back gracefully to compatible version detection than crashing as the option is "prefer" and not "use".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An additional point might be that we should fall back gracefully to compatible version detection than crashing as the option is "prefer" and not "use".

This is already the case :)

),
shell=True,
).strip()
)
io.write_line(f"Found: {executable}", verbosity=Verbosity.VERBOSE)
except CalledProcessError:
io.write_line(
"Unable to detect the current active python executable. Falling back to default.",
verbosity=Verbosity.VERBOSE,
)
return executable

def activate(self, python: str, io: "IO") -> "Env":
venv_path = self._poetry.config.get("virtualenvs.path")
if venv_path is None:
Expand Down Expand Up @@ -784,6 +809,12 @@ def create_venv(
create_venv = self._poetry.config.get("virtualenvs.create")
root_venv = self._poetry.config.get("virtualenvs.in-project")
venv_path = self._poetry.config.get("virtualenvs.path")
prefer_active_python = self._poetry.config.get(
"virtualenvs.prefer-active-python"
)

if not executable and prefer_active_python:
executable = self._detect_active_python(io)

if root_venv:
venv_path = cwd / ".venv"
Expand Down Expand Up @@ -820,7 +851,7 @@ def create_venv(
# If an executable has been specified, we stop there
# and notify the user of the incompatibility.
# Otherwise, we try to find a compatible Python version.
if executable:
if executable and not prefer_active_python:
raise NoCompatiblePythonVersionFound(
self._poetry.package.python_versions, python_patch
)
Expand Down
3 changes: 3 additions & 0 deletions tests/console/commands/test_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ def test_list_displays_default_value_if_not_set(
virtualenvs.options.always-copy = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = {venv_path} # {config_cache_dir / 'virtualenvs'}
virtualenvs.prefer-active-python = false
"""

assert expected == tester.io.fetch_output()
Expand All @@ -79,6 +80,7 @@ def test_list_displays_set_get_setting(
virtualenvs.options.always-copy = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = {venv_path} # {config_cache_dir / 'virtualenvs'}
virtualenvs.prefer-active-python = false
"""

assert config.set_config_source.call_count == 0
Expand Down Expand Up @@ -126,6 +128,7 @@ def test_list_displays_set_get_local_setting(
virtualenvs.options.always-copy = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = {venv_path} # {config_cache_dir / 'virtualenvs'}
virtualenvs.prefer-active-python = false
"""

assert config.set_config_source.call_count == 1
Expand Down