System site-packages still not reused upon install

[sanmai-NL]

• I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

See below.

pyproject.toml snippet:

dependencies = [
  "cryptography>=41.0.3",
  "kubernetes>=27.2.0",
]

Actual behavior

Installing PDM (2.9.3): Creating virtual environment
Installing PDM (2.9.3): Installing PDM and dependencies
Installing PDM (2.9.3): Making binary at /home/resources/.local/bin

Successfully installed: PDM (2.9.3) at /home/resources/.local/bin/pdm
+ pdm config --local install.cache false
+ pdm config --verbose install.cache
False
+ pdm config --verbose python.use_venv
True
+ pdm venv create --with-pip 3.11 --system-site-packages
STATUS: Creating virtualenv using virtualenv...
Virtualenv /opt/resources/.venv is created successfully
+ pdm info
Virtualenv /opt/resources/.venv is reused.
PDM version:
  2.9.3
Python Interpreter:
  /opt/resources/.venv/bin/python (3.11)
Project Root:
  /opt/resources
Local Packages:
  
+ pdm info --env
{
  "implementation_name": "cpython",
  "implementation_version": "3.11.5",
  "os_name": "posix",
  "platform_machine": "aarch64",
  "platform_release": "6.4.15-200.fc38.aarch64",
  "platform_system": "Linux",
  "platform_version": "#1 SMP PREEMPT_DYNAMIC Thu Sep  7 00:51:10 UTC 2023",
  "python_full_version": "3.11.5",
  "platform_python_implementation": "CPython",
  "python_version": "3.11",
  "sys_platform": "linux"
}
+ pdm list --graph
attrs 23.1.0 [ Not required ]
cython 0.29.36 [ Not required ]
html5lib 1.1 [ Not required ]
├── six 1.16.0 [ required: >=1.9 ]
└── webencodings 0.5.1 [ required: Any ]
kubernetes 28.1.0 [ required: >=27.2.0 ]
├── certifi 2023.7.22 [ required: >=14.05.14 ]
├── google-auth 2.22.0 [ required: >=1.0.1 ]
│   ├── cachetools 5.3.1 [ required: <6.0,>=2.0.0 ]
│   ├── pyasn1-modules 0.3.0 [ required: >=0.2.1 ]
│   │   └── pyasn1 0.5.0 [ required: <0.6.0,>=0.4.6 ]
│   ├── rsa 4.9 [ required: <5,>=3.1.4 ]
│   │   └── pyasn1 0.5.0 [ required: >=0.1.3 ]
│   ├── six 1.16.0 [ required: >=1.9.0 ]
│   └── urllib3 1.26.16 [ required: <2.0 ]
├── oauthlib 3.2.2 [ required: >=3.2.2 ]
├── python-dateutil 2.8.2 [ required: >=2.5.3 ]
│   └── six 1.16.0 [ required: >=1.5 ]
├── pyyaml 6.0.1 [ required: >=5.4.1 ]
├── requests 2.31.0 [ required: Any ]
│   ├── certifi 2023.7.22 [ required: >=2017.4.17 ]
│   ├── charset-normalizer 3.2.0 [ required: <4,>=2 ]
│   ├── idna 3.4 [ required: <4,>=2.5 ]
│   └── urllib3 1.26.16 [ required: <3,>=1.21.1 ]
├── requests-oauthlib 1.3.1 [ required: Any ]
│   ├── oauthlib 3.2.2 [ required: >=3.0.0 ]
│   └── requests 2.31.0 [ required: >=2.0.0 ]
│       ├── certifi 2023.7.22 [ required: >=2017.4.17 ]
│       ├── charset-normalizer 3.2.0 [ required: <4,>=2 ]
│       ├── idna 3.4 [ required: <4,>=2.5 ]
│       └── urllib3 1.26.16 [ required: <3,>=1.21.1 ]
├── six 1.16.0 [ required: >=1.9.0 ]
├── urllib3 1.26.16 [ required: <2.0,>=1.24.2 ]
└── websocket-client 1.6.3 [ required: !=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 ]
lit 16.0.6.dev0 [ Not required ]
matplotlib 3.7.2 [ Not required ]
├── contourpy 1.1.1 [ required: >=1.0.1 ]
│   └── numpy 1.25.2 [ required: <2.0,>=1.16 ]
├── cycler 0.11.0 [ required: >=0.10 ]
├── fonttools 4.42.1 [ required: >=4.22.0 ]
├── kiwisolver 1.4.5 [ required: >=1.0.1 ]
├── numpy 1.25.2 [ required: >=1.20 ]
├── packaging 23.1 [ required: >=20.0 ]
├── pillow 10.0.1 [ required: >=6.2.0 ]
├── pyparsing 3.1.1 [ required: <3.1,>=2.3.1 ]
└── python-dateutil 2.8.2 [ required: >=2.7 ]
    └── six 1.16.0 [ required: >=1.5 ]
maturin 1.2.3 [ Not required ]
opencv 4.8.0 [ Not required ]
└── numpy 1.25.2 [ required: Any ]
opencv-python 4.8.0 [ Not required ]
└── numpy 1.25.2 [ required: Any ]
pandas 2.0.3 [ Not required ]
├── numpy 1.25.2 [ required: >=1.23.2 ]
├── python-dateutil 2.8.2 [ required: >=2.8.2 ]
│   └── six 1.16.0 [ required: >=1.5 ]
├── pytz 2023.3 [ required: >=2020.1 ]
└── tzdata [ not installed ] [ required: >=2022.1 ]
pip 23.2.1 [ Not required ]
pycairo 1.24.0 [ Not required ]
python-dotenv 1.0.0 [ required: >=1.0.0 ]
ruamel-yaml-clib 0.2.7 [ Not required ]
setuptools 68.2.0 [ Not required ]
twine 4.0.2 [ Not required ]
├── importlib-metadata 5.2.0 [ required: >=3.6 ]
│   └── zipp 3.16.2 [ required: >=0.5 ]
├── keyring 24.2.0 [ required: >=15.1 ]
│   ├── importlib-metadata 5.2.0 [ required: >=4.11.4 ]
│   │   └── zipp 3.16.2 [ required: >=0.5 ]
│   ├── jaraco-classes 3.3.0 [ required: Any ]
│   │   └── more-itertools 10.1.0 [ required: Any ]
│   ├── jeepney 0.8.0 [ required: >=0.4.2 ]
│   └── secretstorage 3.3.3 [ required: >=3.2 ]
│       ├── cryptography 41.0.4 [ required: >=2.0 ]
│       │   └── cffi 1.16.0 [ required: >=1.12 ]
│       │       └── pycparser 2.21 [ required: Any ]
│       └── jeepney 0.8.0 [ required: >=0.6 ]
├── pkginfo 1.9.6 [ required: >=1.8.1 ]
├── readme-renderer 40.0 [ required: >=35.0 ]
│   ├── bleach 5.0.1 [ required: >=2.1.0 ]
│   │   ├── six 1.16.0 [ required: >=1.9.0 ]
│   │   └── webencodings 0.5.1 [ required: Any ]
│   ├── docutils 0.20.1 [ required: >=0.13.1 ]
│   └── pygments 2.16.1 [ required: >=2.5.1 ]
├── requests 2.31.0 [ required: >=2.20 ]
│   ├── certifi 2023.7.22 [ required: >=2017.4.17 ]
│   ├── charset-normalizer 3.2.0 [ required: <4,>=2 ]
│   ├── idna 3.4 [ required: <4,>=2.5 ]
│   └── urllib3 1.26.16 [ required: <3,>=1.21.1 ]
├── requests-toolbelt 1.0.0 [ required: !=0.9.0,>=0.8.0 ]
│   └── requests 2.31.0 [ required: <3.0.0,>=2.0.1 ]
│       ├── certifi 2023.7.22 [ required: >=2017.4.17 ]
│       ├── charset-normalizer 3.2.0 [ required: <4,>=2 ]
│       ├── idna 3.4 [ required: <4,>=2.5 ]
│       └── urllib3 1.26.16 [ required: <3,>=1.21.1 ]
├── rfc3986 2.0.0 [ required: >=1.4.0 ]
├── rich 13.5.3 [ required: >=12.0.0 ]
│   ├── markdown-it-py 3.0.0 [ required: >=2.2.0 ]
│   │   └── mdurl 0.1.2 [ required: ~=0.1 ]
│   └── pygments 2.16.1 [ required: <3.0.0,>=2.13.0 ]
└── urllib3 1.26.16 [ required: >=1.26.0 ]
wheel 0.41.2 [ Not required ]
+ python -c 'import kubernetes'
+ pdm install --check --fail-fast --no-editable --no-isolation --production --verbose
STATUS: Resolving packages from lockfile...
STATUS: Fetching hashes for resolved packages...
Synchronizing working set with resolved packages: 3 to add, 0 to update, 0 to remove

  ✖ Install pyyaml 6.0 failed
pdm.termui: Error occurs: 
Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/installers/synchronizers.py", line 283, in install_candidate
    self.manager.install(can)
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/installers/manager.py", line 33, in install
    installer(str(prepared.build()), self.environment, prepared.direct_url())
                  ^^^^^^^^^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 381, in build
    self.obtain(allow_all=False)
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 422, in obtain
    raise CandidateNotFound(
pdm.exceptions.CandidateNotFound: No candidate is found for `pyyaml` that matches the environment or hashes
  ✖ Install cffi 1.15.1 failed
pdm.termui: Error occurs: 
Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/installers/synchronizers.py", line 283, in install_candidate
    self.manager.install(can)
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/installers/manager.py", line 33, in install
    installer(str(prepared.build()), self.environment, prepared.direct_url())
                  ^^^^^^^^^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 381, in build
    self.obtain(allow_all=False)
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 422, in obtain
    raise CandidateNotFound(
pdm.exceptions.CandidateNotFound: No candidate is found for `cffi` that matches the environment or hashes
unearth.preparer: Downloading <Link https://files.pythonhosted.org/packages/9d/44/5a992cb9d7bf8aaae73bc5adaf721ad08731c9d00c1c17999a8691404b0c/google_auth-2.23.0-py2.py3-none-any.whl (from https://pypi.org/simple/google-auth/)> (181 kB)
  ✔ Install google-auth 2.23.0 successful
Traceback (most recent call last):
  File "/home/resources/.local/bin/pdm", line 8, in <module>
    sys.exit(main())

             ^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 290, in main
    return Core().main(args or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 208, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 203, in main
    self.handle(project, options)
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 157, in handle
    command.handle(project, options)
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/cli/commands/install.py", line 100, in handle
    actions.do_sync(
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/cli/actions.py", line 214, in do_sync
    synchronizer.synchronize()
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/installers/synchronizers.py", line 451, in synchronize
    raise InstallationError("Some package operations are not complete yet")
pdm.exceptions.InstallationError: Some package operations are not complete yet

Expected behavior

PDM doesn't try to install packages that are already available in the venv through system site-packages (#2216).

Environment Information

See previous log.

[sanmai-NL]

Additionally, locking doesn't work.

+ pdm lock --group :all --refresh
STATUS: Re-calculating hashes...
Changes are written to pdm.lock.
+ pdm install --fail-fast --no-editable --no-isolation --production --verbose
STATUS: Resolving packages from lockfile...
Traceback (most recent call last):
  File "/home/resources/.local/bin/pdm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 290, in main
    return Core().main(args or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 208, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 203, in main
    self.handle(project, options)
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 157, in handle
    command.handle(project, options)
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/cli/commands/install.py", line 100, in handle
    actions.do_sync(
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/cli/actions.py", line 198, in do_sync
    candidates = resolve_candidates_from_lockfile(project, requirements)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/cli/actions.py", line 141, in resolve_candidates_from_lockfile
    mapping, *_ = resolve(
                  ^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/resolver/core.py", line 36, in resolve
    result = resolver.resolve(requirements, max_rounds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/resources/.local/share/pdm/venv/lib/python3.11/site-packages/resolvelib/resolvers.py", line 399, in resolve
    raise ResolutionImpossible(e.criterion.information)
resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=NamedRequirement(name='cryptography', marker=None, extras=set(), specifier=<SpecifierSet('>=41.0.3')>, editable=False, prerelease=False), parent=None)]

Yet if I let pdm install ... do the locking, then a subsequent pdm install --dev ... can't find the dev group.

[sanmai-NL]

As for a solution requirement: I would want the lockfile to only cover dependencies that aren't sourced from the system site-packages. The system site-packages are effectively locked through the OS package manager, whereas the other dependencies aren't if I forego locking during build.

[sanmai-NL]

Another interesting case that can't be worked around by using --no-lockfile:

...
ruamel-yaml-clib 0.2.7 [ Not required ]
...
pdm.termui: ======== Starting round 61 ========
pdm.termui:   Adding requirement ruamel-yaml-clib>=0.2.7; platform_python_implementation == "CPython" and python_version < "3.13"(from ruamel-yaml 0.17.34)
STATUS: Resolving: new pin ruamel-yaml 0.17.34
pdm.termui: Pinning: ruamel-yaml 0.17.34
...
 ✖ Install ruamel-yaml-clib 0.2.8 failed
pdm.termui: Error occurs: 
Traceback (most recent call last):
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/datapipeline_ff/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/installers/synchronizers.py", line 283, in install_candidate
    self.manager.install(can)
  File "/home/datapipeline_ff/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/installers/manager.py", line 33, in install
    installer(str(prepared.build()), self.environment, prepared.direct_url())
                  ^^^^^^^^^^^^^^^^
  File "/home/datapipeline_ff/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 381, in build
    self.obtain(allow_all=False)
  File "/home/datapipeline_ff/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/models/candidates.py", line 422, in obtain
    raise CandidateNotFound(
pdm.exceptions.CandidateNotFound: No candidate is found for `ruamel-yaml-clib` that matches the environment or hashes
...

It seems platform_python_implementation somehow restricts package resolution to exclude system site-packages.

[sanmai-NL]

I don't know if you understand the trickery in https://sourceforge.net/p/ruamel-yaml/code/ci/default/tree/setup.py, @frostming ... I'm out of my depth.

[frostming]

The dependencies will always be resolved whether they are included by system-site-package or not. Even when writing lockfile is not required(--no-lock). This step is mandatory to make sure the dependencies are compatible with each other. The system site packages Can only skip the installation of packages if its version matches exactly with the one in the lockfile.

[sanmai-NL]

For context, does Poetry have the same behavior?

[sanmai-NL]

@frostming I fail to wrap my head around how to reuse system site-packages with PDM. If I remove/disregard the lockfile and my dependency has a dependency with a lower bound version range, then the latest version will be collected from the package index, whereas my system site-package version is lower but still compatible with the lower bound version range. How to avoid this issue?

[frostming]

This was resolved by the introduction of --update-reuse-installed strategy.