`ResolutionImpossible` when using `pdm add` but `install` and `lock` work fine #2358

reversefold · 2023-11-01T22:28:26Z

I have searched the issue tracker and believe that this is not a duplicate.

I'm running into an issue with pdm add succeeding in adding dependencies to pyproject.toml and pdm.lock but then failing on install with ResolutionImpossible when resolving packages from the lockfile during the install step. The error output without -v looks like:

Adding packages to default dependencies: jupyter
🔒 Lock successful
Changes are written to pyproject.toml.
See /var/folders/wc/1tq03hqj0m781dy88gksnqyr0000gq/T/pdm-install-resolve-q3fjo1vp.log for detailed debug log.
[ResolutionImpossible]: [RequirementInformation(requirement=NamedRequirement(name='jsonschema', marker=None, extras={'format-nongpl'}, specifier=<SpecifierSet('>=4.18.0')>, editable=False, prerelease=False), parent=<Candidate jupyter-events@0.8.0>)]

Detailed output below. Apologies for the massive amount of transitive dependencies for these 2 packages but this was the smallest I could come up with in a reason able amount of time.

Steps to reproduce

pyproject.toml

[project]
name = ""
version = ""
description = ""
authors = [
    {name = "", email = ""},
]
requires-python = ">=3.11,<4.0.0"
readme = "README.md"
license = {text = "MIT"}

Run:

pdm add --group test 'moto[all]'
pdm add -v jupyter

Actual behavior

jupyter is added to pyproject.toml and all expected packages are added to pdm.lock is updated but when resolving packages from the lockfile to install pdm fails with (only end of output included here):

🔒 Lock successful
Changes are written to pyproject.toml.
STATUS: Resolving packages from lockfile...
Traceback (most recent call last):
  File "/Users/XXX/.local/bin/pdm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/XXX/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/core.py", line 295, in main
    return Core().main(args or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/XXX/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/core.py", line 208, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "/Users/XXX/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/core.py", line 203, in main
    self.handle(project, options)
  File "/Users/XXX/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/core.py", line 157, in handle
    command.handle(project, options)
  File "/Users/XXX/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/cli/commands/add.py", line 69, in handle
    self.do_add(
  File "/Users/XXX/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/cli/commands/add.py", line 181, in do_add
    do_sync(
  File "/Users/XXX/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/cli/actions.py", line 199, in do_sync
    candidates = resolve_candidates_from_lockfile(project, requirements)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/XXX/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/cli/actions.py", line 142, in resolve_candidates_from_lockfile
    mapping, *_ = resolve(
                  ^^^^^^^^
  File "/Users/XXX/.local/pipx/venvs/pdm/lib/python3.11/site-packages/pdm/resolver/core.py", line 36, in resolve
    result = resolver.resolve(requirements, max_rounds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/XXX/.local/pipx/venvs/pdm/lib/python3.11/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/XXX/.local/pipx/venvs/pdm/lib/python3.11/site-packages/resolvelib/resolvers.py", line 434, in resolve
    success = self._backjump(causes)
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/XXX/.local/pipx/venvs/pdm/lib/python3.11/site-packages/resolvelib/resolvers.py", line 318, in _backjump
    raise ResolutionImpossible(causes)
resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=NamedRequirement(name='jsonschema', marker=None, extras={'format-nongpl'}, specifier=<SpecifierSet('>=4.18.0')>, editable=False, prerelease=False), parent=<Candidate jupyter-events@0.8.0>)]

Subsequently running pdm lock or pdm install executes without error.

What is especially odd is that if you remove the --group test from the commands above everything works fine.

Expected behavior

Installation happens without error.

Environment Information

PDM version:
  2.10.0
Python Interpreter:
  /Users/XXX/src/pdm-test/.venv/bin/python (3.11)
Project Root:
  /Users/XXX/src/pdm-test
Local Packages:

{
  "implementation_name": "cpython",
  "implementation_version": "3.11.4",
  "os_name": "posix",
  "platform_machine": "arm64",
  "platform_release": "23.0.0",
  "platform_system": "Darwin",
  "platform_version": "Darwin Kernel Version 23.0.0: Fri Sep 15 14:43:05 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T6020",
  "python_full_version": "3.11.4",
  "platform_python_implementation": "CPython",
  "python_version": "3.11",
  "sys_platform": "darwin"
}

The text was updated successfully, but these errors were encountered:

g-ananth · 2023-11-05T22:58:25Z

Here is another scenario which is happening on OSX (Sonoma)

`

pdm install
See /var/folders/1x/x11tnzgd1tz2qctq7mryh2080000gn/T/pdm-install-resolve-8vx66p_0.log for detailed debug log.
[ResolutionImpossible]: [RequirementInformation(requirement=PythonRequirement(name='python', marker=None, extras=None, specifier==3.10>, editable=False, prerelease=False), parent=None), RequirementInformation(requirement=PythonRequirement(name='python', marker=None, extras=None, specifier=<PySpecSet
=3.9,<3.13>, editable=False, prerelease=False), parent=<Candidate numpy@1.26.1>)]
`

The stanza for numpy (for which the above error was raised) looks like the following in pdm.lock

`
[[package]]
name = "numpy"
version = "1.26.1"
requires_python = "<3.13,>=3.9"
summary = "Fundamental package for array computing in Python"
files = [
{file = "numpy-1.26.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:82e871307a6331b5f09efda3c22e03c095d957f04bf6bc1804f30048d0e5e7af"},
{file = "numpy-1.26.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:cdd9ec98f0063d93baeb01aad472a1a0840dee302842a2746a7a8e92968f9575"},
{file = "numpy-1.26.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d78f269e0c4fd365fc2992c00353e4530d274ba68f15e968d8bc3c69ce5f5244"},
{file = "numpy-1.26.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8ab9163ca8aeb7fd32fe93866490654d2f7dda4e61bc6297bf72ce07fdc02f67"},
{file = "numpy-1.26.1-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:78ca54b2f9daffa5f323f34cdf21e1d9779a54073f0018a3094ab907938331a2"},
{file = "numpy-1.26.1-cp310-cp310-win32.whl", hash = "sha256:d1cfc92db6af1fd37a7bb58e55c8383b4aa1ba23d012bdbba26b4bcca45ac297"},
{file = "numpy-1.26.1-cp310-cp310-win_amd64.whl", hash = "sha256:d2984cb6caaf05294b8466966627e80bf6c7afd273279077679cb010acb0e5ab"},
{file = "numpy-1.26.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:cd7837b2b734ca72959a1caf3309457a318c934abef7a43a14bb984e574bbb9a"},
{file = "numpy-1.26.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:1c59c046c31a43310ad0199d6299e59f57a289e22f0f36951ced1c9eac3665b9"},
{file = "numpy-1.26.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d58e8c51a7cf43090d124d5073bc29ab2755822181fcad978b12e144e5e5a4b3"},
{file = "numpy-1.26.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:6081aed64714a18c72b168a9276095ef9155dd7888b9e74b5987808f0dd0a974"},
{file = "numpy-1.26.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:97e5d6a9f0702c2863aaabf19f0d1b6c2628fbe476438ce0b5ce06e83085064c"},
{file = "numpy-1.26.1-cp311-cp311-win32.whl", hash = "sha256:b9d45d1dbb9de84894cc50efece5b09939752a2d75aab3a8b0cef6f3a35ecd6b"},
{file = "numpy-1.26.1-cp311-cp311-win_amd64.whl", hash = "sha256:3649d566e2fc067597125428db15d60eb42a4e0897fc48d28cb75dc2e0454e53"},
{file = "numpy-1.26.1-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:1d1bd82d539607951cac963388534da3b7ea0e18b149a53cf883d8f699178c0f"},
{file = "numpy-1.26.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:afd5ced4e5a96dac6725daeb5242a35494243f2239244fad10a90ce58b071d24"},
{file = "numpy-1.26.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a03fb25610ef560a6201ff06df4f8105292ba56e7cdd196ea350d123fc32e24e"},
{file = "numpy-1.26.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:dcfaf015b79d1f9f9c9fd0731a907407dc3e45769262d657d754c3a028586124"},
{file = "numpy-1.26.1-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:e509cbc488c735b43b5ffea175235cec24bbc57b227ef1acc691725beb230d1c"},
{file = "numpy-1.26.1-cp312-cp312-win32.whl", hash = "sha256:af22f3d8e228d84d1c0c44c1fbdeb80f97a15a0abe4f080960393a00db733b66"},
{file = "numpy-1.26.1-cp312-cp312-win_amd64.whl", hash = "sha256:9f42284ebf91bdf32fafac29d29d4c07e5e9d1af862ea73686581773ef9e73a7"},
{file = "numpy-1.26.1-pp39-pypy39_pp73-macosx_10_9_x86_64.whl", hash = "sha256:06934e1a22c54636a059215d6da99e23286424f316fddd979f5071093b648668"},
{file = "numpy-1.26.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:76ff661a867d9272cd2a99eed002470f46dbe0943a5ffd140f49be84f68ffc42"},
{file = "numpy-1.26.1-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:6965888d65d2848e8768824ca8288db0a81263c1efccec881cb35a0d805fcd2f"},
{file = "numpy-1.26.1.tar.gz", hash = "sha256:c8c6c72d4a9f831f328efb1312642a1cafafaa88981d9ab76368d50d07d93cbe"},
]

`

Rerunning the command did not help me in 2.10

This was reproducible in 2.9.3 version of pdm as well.

One way the pdm install command worked for me was when I manually edited the python version specifier for numpy. Changing the requires_python = "<3.13,>=3.9" TO requires_python = ">=3.9" and rerunning pdm install worked fine for pdm 2.9.3. I believe this manual hack had got nothing to do with my virtual env python version which is 3.10

reversefold · 2023-11-09T20:34:46Z

I've also gotten this error for the project with this issue when running in CI. I'll try to get more detailed output for you.
[KeyError]: ('jsonschema', '4.19.2', None, False)

reversefold · 2023-11-09T20:45:53Z

verbose output:

pdm install -v
STATUS: Resolving packages from lockfile...
python.use_venv is on, creating a virtualenv for this project...
Run command: ['/root/.local/share/pdm/venv/bin/python', '-m', 'virtualenv', '/project/.venv', '-p', '/usr/local/bin/python', '--prompt=project', '--no-pip', '--no-setuptools', '--no-wheel']
created virtual environment CPython3.11.4.final.0-64 in 61ms
  creator CPython3Posix(dest=/project/.venv, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Virtualenv is created successfully at /project/.venv
Traceback (most recent call last):
  File "/root/.local/bin/pdm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 2[95](<REDACTED>}), in main
    return Core().main(args or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 208, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 203, in main
    self.handle(project, options)
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/core.py", line 157, in handle
    command.handle(project, options)
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/cli/commands/install.py", line 100, in handle
    actions.do_sync(
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/cli/actions.py", line 1[99](<REDACTED>), in do_sync
    candidates = resolve_candidates_from_lockfile(project, requirements)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/cli/actions.py", line 142, in resolve_candidates_from_lockfile
    mapping, *_ = resolve(
                  ^^^^^^^^
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/resolver/core.py", line 36, in resolve
    result = resolver.resolve(requirements, max_rounds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/resolvelib/resolvers.py", line 426, in resolve
    name = min(unsatisfied_names, key=self._get_preference)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/resolvelib/resolvers.py", line 203, in _get_preference
    return self._p.get_preference(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/resolver/providers.py", line 91, in get_preference
    deps = self.get_dependencies(candidate)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/resolver/providers.py", line 214, in get_dependencies
    deps, requires_python, _ = self.repository.get_dependencies(candidate)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/models/repositories.py", line [103](<REDACTED>), in get_dependencies
    requirements, requires_python, summary = getter(candidate)
                                             ^^^^^^^^^^^^^^^^^
  File "/root/.local/share/pdm/venv/lib/python3.11/site-packages/pdm/models/repositories.py", line 555, in _get_dependencies_from_lockfile
    return self.candidate_info[self._identify_candidate(candidate)]
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: ('jsonschema[format-nongpl]', '4.19.2', None, False)

frostming · 2023-11-12T13:15:22Z

It apparently is an issue, but I failed to reproduce it following the steps given by @reversefold (Python 3.11, M1)
The issue is caused by missing package in the lockfile, pdm install -v just uncovers it and is not the first scene. Therefore, the problem is brought by some previous pdm add step that fails to record a package with extra in the lockfile. In the concrete example, jsonschema[format-nongpl] should have been pinned in the lockfile but it isn't. Showing the traceback of pdm install isn't very helpful.

It's also possible that this is a bug in older versions and uncovered by the latest version.

If someone could give a reproducible example to create the broken lockfile I would be appreciated. Or if @reversefold can reproduce constantly can you provide a MRE for it. If that doesn't work out well try to make a Dockerfile to freeze all the environment.

JacobCoffee · 2023-11-12T19:46:26Z

I'm sorry I do not have a useful or smaller example as most of my encounters with this have been in private work projects, but we did have litestar-org/litestar#2639 which resolved it for us, and the previous lock file should be available in the diff (at https://github.com/litestar-org/litestar/tree/0c6a57f8454838572aa6eaffd89123c778d638df).

frostming · 2023-11-13T01:25:58Z

It's also possible that this is a bug in older versions and uncovered by the latest version.

This looks like the case. I looked into the git history and found the lockfile is problemtic at litestar-org/litestar@b0dd1ee, the same lockfile fails on PDM 2.10.1 but succeeded on PDM 2.10.0.

So it is not a bug with the latest PDM but a legacy bug with older versions that was not surfaced.

Fixes #2358 Signed-off-by: Frost Ming <me@frostming.com>

reversefold · 2023-11-13T18:52:57Z

The KeyError is simply another case of jsonschema failing, @frostming are your comments the same for the ResolutionImpossible error originally reported? That one has been 100% reproducible for me.

reversefold added the 🐛 bug Something isn't working label Nov 1, 2023

frostming added this to the 2.11.0 milestone Nov 12, 2023

frostming added the 🧩 dependency resolution Resolution failures label Nov 12, 2023

frostming self-assigned this Nov 13, 2023

frostming added a commit that referenced this issue Nov 13, 2023

fix: improve the error message to tell user how to fix it

447fad5

Fixes #2358 Signed-off-by: Frost Ming <me@frostming.com>

frostming mentioned this issue Nov 13, 2023

fix: improve the error message to tell user how to fix it #2396

Merged

2 tasks

frostming closed this as completed in #2396 Nov 13, 2023

frostming mentioned this issue Nov 15, 2023

pdm export raises KeyError #2407

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`ResolutionImpossible` when using `pdm add` but `install` and `lock` work fine #2358

`ResolutionImpossible` when using `pdm add` but `install` and `lock` work fine #2358

reversefold commented Nov 1, 2023 •

edited

Loading

g-ananth commented Nov 5, 2023

reversefold commented Nov 9, 2023

reversefold commented Nov 9, 2023

frostming commented Nov 12, 2023 •

edited

Loading

JacobCoffee commented Nov 12, 2023

frostming commented Nov 13, 2023

reversefold commented Nov 13, 2023

ResolutionImpossible when using pdm add but install and lock work fine #2358

ResolutionImpossible when using pdm add but install and lock work fine #2358

Comments

reversefold commented Nov 1, 2023 • edited Loading

Steps to reproduce

Actual behavior

Expected behavior

Environment Information

g-ananth commented Nov 5, 2023

reversefold commented Nov 9, 2023

reversefold commented Nov 9, 2023

frostming commented Nov 12, 2023 • edited Loading

JacobCoffee commented Nov 12, 2023

frostming commented Nov 13, 2023

reversefold commented Nov 13, 2023

`ResolutionImpossible` when using `pdm add` but `install` and `lock` work fine #2358

`ResolutionImpossible` when using `pdm add` but `install` and `lock` work fine #2358

reversefold commented Nov 1, 2023 •

edited

Loading

frostming commented Nov 12, 2023 •

edited

Loading