pdm run python then import package return system packages if not installed with pdm

[mde-pach]

• I have searched the issue tracker and believe that this is not a duplicate.

Steps to reproduce

pip install black
pdm add typer

Actual behavior

$> pdm run pip freeze
black==21.9b0
typer==0.4.0

Expected behavior

$> pdm run pip freeze
typer==0.4.0

The pdm run command should launch the command in a pdm isolated environment but it's more like a "system overloaded" environment. Is this a voluntary behavior ?
It makes more difficult to be sure than pdm.lock file is complete if system had some package than the project need.

In this case, if a script did a import black and I run it with pdm run my_script.py it will not raise a ModuleNotFoundError, so I can't know I'm missing the black package in my lock file and the install will fail in a clean environment without the system black package already installed.

Environment Information

$> pdm info
PDM version:        1.10.0
Python Interpreter: /usr/bin/python3.9 (3.9)
Project Root:       $HOME/test
Project Packages:   $HOME/test/__pypackages__/3.9
$> pdm info --env
{
  "implementation_name": "cpython",
  "implementation_version": "3.9.2",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "5.10.0-9-amd64",
  "platform_system": "Linux",
  "platform_version": "#1 SMP Debian 5.10.70-1 (2021-09-30)",
  "python_full_version": "3.9.2",
  "platform_python_implementation": "CPython",
  "python_version": "3.9",
  "sys_platform": "linux"
}

[frostming]

The correct way to do this is pdm list --freeze

[pawamoy]

Oh, I get this error:

$ pdm list --freeze -v
Traceback (most recent call last):
  File "/home/user/.local/bin/pdm", line 8, in <module>
    sys.exit(main())
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/core.py", line 200, in main
    return Core().main(args)
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/core.py", line 153, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/core.py", line 148, in main
    f(options.project, options)
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/cli/commands/list.py", line 31, in handle
    actions.do_list(
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/cli/actions.py", line 404, in do_list
    reqs = [
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/cli/actions.py", line 405, in <listcomp>
    frozen_requirement_from_dist(dist)
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/cli/utils.py", line 608, in frozen_requirement_from_dist
    return str(FrozenRequirement.from_dist(pkg_dist))
  File "/home/user/.local/pipx/shared/lib/python3.6/site-packages/pip/_internal/operations/freeze.py", line 235, in from_dist
    editable = dist.editable
  File "/home/user/.local/pipx/shared/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2816, in __getattr__
    return getattr(self._provider, attr)
AttributeError: 'PathMetadata' object has no attribute 'editable'

pdm list works fine.

[mde-pach]

pdm run pip freeze was just a (bad) example for what I want to describe because if I do that:

$> pdm run python
>>> import black
>>>

This will import my system black and not raise a ModuleNotFoundError

Maybe it's the desired behavior but I wanted to log it here

[pawamoy]

I believe that is the normal behavior. PDM adds the __pypackages__ lib dirs to the Python path rather than isolate them from the rest of the interpreter libs paths.

Note that you could run python directly instead of pdm run python: it will have access to your installed packages as well.

pdm run python is useful when you want to make sure that the Python version specified with pdm use is used, though (it could differ from the first python found in your path).

Another note: I usually install tools like Black with pipx instead of pip, so as to not pollute my interpreter site-packages directory.

[frostming]

let's make an enhancement item: allow disabling the system site-packages with an env var and turn it on for pdm run.

[mde-pach]

Thanks @pawamoy for explanations !

@frostming Maybe it's my "old way usage" of pip with venv that make me feel that weird, if there is no normal usage of that, we don't need to make a new feature of this.

Maybe we can make this issue well referenced and wait for more community inputs before make you wasting your time ?

[frostming]

No, I think we can disable system site-packages in pdm run mode but keep them in PEP 582 mode, sounds good? @pawamoy

Because by including system site-packages, global tools like ipython and other Python utilities won't suddenly break when the current directory has a __pypackages__ directory if PEP 582 mode is on. But when running pdm run, people know they are running within the project's context and don't expect external libraries to contaminate the project environment.

[pawamoy]

I wouldn't mind actual isolation for the pdm run commands, yes. Being able to import packages from the interpreter or user site-packages is usually confusing, and not very well tooled to deal with.

Yup. Totally agree with your last two sentences.