PDM update not picking up the latest version of a dev. dependency (Sphinx)

[sr-murthy]

• [Y ] I have searched the issue tracker and believe that this is not a duplicate.

Steps to reproduce

My project documentation depends on Sphinx (latest @ v8.0.2). This is defined as a dev. dependency in the project TOML, without any range pinning:

...
[tool.pdm.dev-dependencies]
docs = [
    "myst-parser",
    "pydata-sphinx-theme",
    "Sphinx",
    ...
]
...

Note that almost all of the non-Sphinx docs dependencies depend on Sphinx.

But when I recently tried to update all of the dev. dependencies (excluding the project itself) in the PDM lockfile via:

pdm update -v --dev --no-editable --no-self --update-all

I do not see the latest version of Sphinx in the lockfile. Instead, I see that Sphinx has been pinned to an older version v7.4.7:

...
[[package]]
name = "sphinx"
version = "7.4.7"
...

Why does this happen?

When I tried to force the issue by requiring Sphinx>=8.0.2 in the TOML and re-running the PDM update I got a circular dependency resolution issue, where this line appears repeatedly in the resolution pinning stages for various dependencies related to Sphinx, for example myst-parser:

...
pdm.termui:   Adding requirement sphinx<8,>=6(from myst-parser 3.0.1)
pdm.termui: Candidate rejected: myst-parser@3.0.1 because it introduces a new requirement sphinx<8,>=6 that conflicts with other requirements:
...
pdm.termui:   Adding requirement sphinx<8,>=6(from myst-parser 3.0.0)
pdm.termui: Candidate rejected: myst-parser@3.0.0 because it introduces a new requirement sphinx<8,>=6 that conflicts with other requirements:
...
pdm.termui:   Adding requirement sphinx<8,>=6(from myst-parser 2.0.0)
pdm.termui: Candidate rejected: myst-parser@2.0.0 because it introduces a new requirement sphinx<8,>=6 that conflicts with other requirements:
...

Actually, the latest myst-parser (v3.0.1) is compatible with Sphinx>=7,<9. So where is sphinx<8,>=6 coming from with the Sphinx>=8.0.2 option?

So, in summary, without any range pinning of Sphinx in the project TOML the latest version is not being found and pinned in the lockfile with the PDM update command for --dev dependencies, and, for the same command, the lower range pinning of Sphinx to the latest version (v8.0.2) is causing the resolution of non-Sphinx dependencies that depend on Sphinx to loop because there is a mistaken conclusion that those dependencies are not compatible with the latest version of Sphinx.

Actual behavior

See above.

Expected behavior

An updated lockfile with the latest version of Sphinx (v8.0.2).

Environment Information

PDM version:
  2.17.3
Python Interpreter:
  /Users/srm/Documents/dev/cfenv/bin/python (3.11)
Project Root:
  /Users/srm/Documents/dev/continuedfractions
Local Packages:
  
{
  "implementation_name": "cpython",
  "implementation_version": "3.11.2",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "23.6.0",
  "platform_system": "Darwin",
  "platform_version": "Darwin Kernel Version 23.6.0: Fri Jul  5 17:54:20 PDT 2024; root:xnu-10063.141.1~2/RELEASE_X86_64",
  "python_full_version": "3.11.2",
  "platform_python_implementation": "CPython",
  "python_version": "3.11",
  "sys_platform": "darwin"
}

[sr-murthy]

Possibly related to #3027.

[frostming]

Actually, the latest myst-parser (v3.0.1) is compatible with Sphinx>=7,<9

False alarm. You should not look at the main branch which is not released yet.
The 3.0.1 metadata shows it is using >=6,<8, so PDM is doing right.

[sr-murthy]

OK, my mistake, thanks.