Fork version selection based on `requires-python` requirements #9827

charliermarsh · 2024-12-11T23:15:23Z

Summary

This PR addresses a significant limitation in the resolver whereby we avoid choosing the latest versions of packages when the user supports a wider range.

For example, with NumPy, the latest versions only support Python 3.10 and later. If you lock a project with requires-python = ">=3.8", we pick the last NumPy version that supported Python 3.8, and use that for all Python versions. So you get 1.24.4 for all versions, rather than 2.2.0. And we'll never upgrade you unless you bump your requires-python. (Even worse, those versions don't have wheels for Python 3.12, etc., so you end up building from source.)

(As-is, this is intentional. We optimize for minimizing the number of selected versions, and the current logic does that well!)

Instead, we know recognize when a version has an elevated requires-python specifier and fork. This is a new fork point, since we need to fork once we have the package metadata, as opposed to when we see the dependencies.

In this iteration, I've made this behavior the default. I'm sort of undecided on whether I want to push on that... Previously, I'd suggested making it opt-in via a setting (#8686).

Closes #8492.

charliermarsh · 2024-12-11T23:17:00Z

I suggest taking a look at the changes in the lock.rs tests.

zanieb · 2024-12-12T03:50:30Z

Is this breaking as a default behavior? Or will it not invalidate existing lockfiles?

charliermarsh · 2024-12-12T03:55:31Z

No, it wouldn't invalidate existing lockfiles.

charliermarsh · 2024-12-12T04:05:49Z

From the test changes, I think this behavior is actually a bit more intuitive.

konstin

This is elegant!

I do like this as a default. I'm unsure if it's the right default when there is a universal wheel, but it's definitely better for packages with platform specific wheels such as numpy, where it solves a class of problems that is otherwise hard to understand and fix for the user with convenient defaults.

konstin · 2024-12-12T10:25:28Z

crates/uv-distribution-types/src/prioritized_distribution.rs

+    /// Return the `requires-python` specifier for the distribution, if any.
+    pub fn requires_python(&self) -> Option<&VersionSpecifiers> {
+        match self {
+            CompatibleDist::InstalledDist(_) => None,


Can't we read the requires python from disk for it, or is that irrelevant here?

I believe it's irrelevant here.

crates/uv-resolver/src/resolver/mod.rs

konstin · 2024-12-12T10:46:15Z

crates/uv/tests/it/lock.rs

+        version = "1.2.1"
+        source = { registry = "https://pypi.org/simple" }
+        resolution-markers = [
+            "python_full_version >= '3.7.9' and python_full_version < '3.8'",


imo we should only split at major.minor and ignore patch levels (see also https://discuss.python.org/t/requires-python-and-pre-release-python-versions/62959/24).

konstin · 2024-12-12T10:58:25Z

crates/uv/tests/it/lock.rs

+            { url = "https://files.pythonhosted.org/packages/4c/0c/9c603826b6465e82591e05ca230dfc13376da512b25ccd0894709b054ed0/numpy-1.26.4-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:ab47dbe5cc8210f55aa58e4805fe224dac469cde56b9f731a4c098b91917159a", size = 13572172 },
+            { url = "https://files.pythonhosted.org/packages/76/8c/2ba3902e1a0fc1c74962ea9bb33a534bb05984ad7ff9515bf8d07527cadd/numpy-1.26.4-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:1dda2e7b4ec9dd512f84935c5f126c8bd8b9f2fc001e9f54af255e8c5f16b0e0", size = 17786643 },
+            { url = "https://files.pythonhosted.org/packages/28/4a/46d9e65106879492374999e76eb85f87b15328e06bd1550668f79f7b18c6/numpy-1.26.4-cp312-cp312-win32.whl", hash = "sha256:50193e430acfc1346175fcbdaa28ffec49947a06918b7b92130744e81e640110", size = 5677803 },
+            { url = "https://files.pythonhosted.org/packages/16/2e/86f24451c2d530c88daf997cb8d6ac622c1d40d19f5a031ed68a4b73a374/numpy-1.26.4-cp312-cp312-win_amd64.whl", hash = "sha256:08beddf13648eb95f8d867350f6a018a4be2e5ad54c8d8caed89ebca558b2818", size = 15517754 },


Note to self: the cp313 wheels aren't missing, this is just our exclude newer

konstin · 2024-12-12T11:05:30Z

crates/uv-resolver/src/resolver/mod.rs

+        {
+            if env.marker_environment().is_none() {
+                let forks = VersionForker::fork(requires_python, python_requirement, env);
+                if !forks.is_empty() {


maybe something that we handle implicitly, but wouldn't we always return at least one fork (because union(left, right) is the entire requires-python of the fork we're in, so partitioning it at least one partition must overlap), so when we get one fork we should also continue implicitly?

I had a similar thought here. But I find it somewhat difficult to reason about invariants that are always true. For example, if the fork has a marker that always evaluates to false, then VersionForker::fork could return an empty set of forks. But... I think it's impossible at this point to have such a marker?

Other than that, I think I agree with you. We could add an assert! here.

Both of the forks need to be non-empty when splitting the Python requirement. We then filter out forks that are disjoint with the markers, because it's subtly different... We could have a project with requires-python = ">=3.9", and then be in a fork with python_version < '3.8' or python_version > '3.11.

If we then split at Python 3.10, we'd split the requires-python into >=3.9, <3.10 and >=3.10.

If we intersect >=3.9, <3.10 with python_version < '3.8' or python_version > '3.11, that's empty, so we need to drop that piece.

BurntSushi

I'm a little uneasy about making this the default (I see that the markers get a fair bit more complicated), but I do agree that this does seem to work well in the Python ecosystem. I trust your judgment. :)

crates/uv-resolver/src/requires_python.rs

crates/uv-resolver/src/resolver/environment.rs

BurntSushi · 2024-12-12T14:16:52Z

crates/uv-resolver/src/resolver/mod.rs

+        {
+            if env.marker_environment().is_none() {
+                let forks = VersionForker::fork(requires_python, python_requirement, env);
+                if !forks.is_empty() {


I had a similar thought here. But I find it somewhat difficult to reason about invariants that are always true. For example, if the fork has a marker that always evaluates to false, then VersionForker::fork could return an empty set of forks. But... I think it's impossible at this point to have such a marker?

Other than that, I think I agree with you. We could add an assert! here.

konstin · 2024-12-12T15:07:35Z

We should add a sentence or two about this new behavior in resolver-internals.md.

charliermarsh · 2024-12-13T04:14:31Z

Current thinking is to make this the default, but add a setting such that users can opt-out if necessary.

## Summary This PR makes the behavior in #9827 the default: we try to select the latest supported package version for each supported Python version, but we still optimize for choosing fewer versions when stratifying by platform. However, you can opt out with `--fork-strategy fewest`. Closes #7190.

henryiii · 2024-12-13T21:05:51Z

Minor comment on the PR summary: NumPy now only supports 3.11+, no longer 3.10+ (following SPEC 0). ;)

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.5.8` -> `0.5.11` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.5.11`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0511) [Compare Source](astral-sh/uv@0.5.10...0.5.11) ##### Enhancements - Normalize `platform_system` to `sys_platform` ([#9949](astral-sh/uv#9949)) - Improve retry mechanisms on Windows for `copy_atomic` and `write_atomic` ([#10026](astral-sh/uv#10026)) - Add nuance to prefetch logging ([#9984](astral-sh/uv#9984)) - Update to [`python-build-standalone 20241219`](https://github.com/astral-sh/python-build-standalone/releases/tag/20241219) ##### Preview features - Build backend: Preserve executable bits for scripts in distributions ([#10027](astral-sh/uv#10027)) - Build backend: Handle case where `metadata_directory` already contains `dist-info` directory ([#10005](astral-sh/uv#10005)) ##### Performance - Batch resolver pre-fetches per fork ([#10029](astral-sh/uv#10029)) ##### Bug fixes - Allow `--script` to be provided with `uv run -` ([#10035](astral-sh/uv#10035)) - Allow `uv run` arguments when reading from `stdin` ([#10034](astral-sh/uv#10034)) - Prefer higher Python lower-bounds when forking ([#10007](astral-sh/uv#10007)) - Remove references to deprecated `first-match` ([#10036](astral-sh/uv#10036)) ##### Documentation - Add `uv python install --preview` to the documentation ([#10010](astral-sh/uv#10010)) - Fix `uv python install --default` note about multiple requests ([#10011](astral-sh/uv#10011)) - Fix typo in Caching docs ([#10032](astral-sh/uv#10032)) - Remove remaining references to deprecated `first-match` ([#10038](astral-sh/uv#10038)) - Supplement missing separators for `UV_INSTALL_DIR` directions on Windows ([#9507](astral-sh/uv#9507)) ### [`v0.5.10`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0510) [Compare Source](astral-sh/uv@0.5.9...0.5.10) ##### Enhancements - Improve backtracking behavior when packages conflict repeatedly ([#9843](astral-sh/uv#9843)) - Patch Python `sysconfig` values such as `AR` at `ar` install time ([#9905](astral-sh/uv#9905)) - Patch Python `sysconfig` values such as `clang` to `cc` at install time ([#9916](astral-sh/uv#9916)) - Skip `--native-tls` in `pip compile` header ([#9913](astral-sh/uv#9913)) - Add resolver error hint for no-binary and no-build failures ([#9948](astral-sh/uv#9948)) - Improve build error messages ([#9660](astral-sh/uv#9660)) - Reduce redundant Python version incompatibilities in resolver error message ([#9957](astral-sh/uv#9957)) - Reduce redundant enumeration of all package versions in some resolver errors ([#9885](astral-sh/uv#9885)) - Improve display of ranges when pre-releases are not allowed ([#9944](astral-sh/uv#9944)) - Improve error messages for `uv remove` ([#9959](astral-sh/uv#9959)) - Improve phrasing for single term incompatibilities ([#9953](astral-sh/uv#9953)) - Improve styling of `uv remove` dependency hints ([#9960](astral-sh/uv#9960)) - Omit trailing zeros on Python requirements inferred from versions ([#9952](astral-sh/uv#9952)) - Show a concise error message for missing `version` field ([#9912](astral-sh/uv#9912)) - Use the build options value to improve hints for no wheel / source distribution errors ([#9950](astral-sh/uv#9950)) ##### Bug fixes - Allow multiple disjoint URLs in overrides ([#9893](astral-sh/uv#9893)) - Include explicit indexes in publish index choice ([#9932](astral-sh/uv#9932)) - Fix Python interpreter detection for 32-bit operating systems on 64-bit hosts ([#9970](astral-sh/uv#9970)) ##### Documentation - Fix typo "operation system" ([#9971](astral-sh/uv#9971)) - Clarify uninstallation docs ([#9938](astral-sh/uv#9938)) - Add a note to say that dependencies between workspace members are editable ([#9363](astral-sh/uv#9363)) - Correctly document default value of `fork-strategy` setting ([#9931](astral-sh/uv#9931)) - Use double quotes for Windows support in examples ([#9946](astral-sh/uv#9946)) - Remove `pypy` from top-level pin example ([#9896](astral-sh/uv#9896)) - Update references to `python-build-standalone` to reflect the transferred project ([#9977](astral-sh/uv#9977)) - Use a different Ruff version in documentation ([#9943](astral-sh/uv#9943)) - Change example so it works as-is on `powershell` and `cmd.exe` ([#9903](astral-sh/uv#9903)) - Clarify best practice for Python matrix strategy in GitHub Actions ([#9454](astral-sh/uv#9454)) - Add documentation for `uv-lock` and `uv-export` pre-commit hooks ([#9872](astral-sh/uv#9872)) ##### Preview features - Build backend: Fix pre-PEP 639 license files ([#9965](astral-sh/uv#9965)) ### [`v0.5.9`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#059) [Compare Source](astral-sh/uv@0.5.8...0.5.9) ##### Enhancements - Fork version selection based on `requires-python` requirements ([#9827](astral-sh/uv#9827)) - Patch `sysconfig` data at install time ([#9857](astral-sh/uv#9857)) - Remove `-isysroot` when patching sysconfig ([#9860](astral-sh/uv#9860)) ##### Configuration - Introduce a `--fork-strategy` preference mode ([#9868](astral-sh/uv#9868)) - Add support for `UV_OFFLINE` ([#9795](astral-sh/uv#9795)) ##### Bug fixes - Avoid `panic!()` when current directory does not exist ([#9876](astral-sh/uv#9876)) - Avoid reusing interpreter metadata when running under Rosetta ([#9846](astral-sh/uv#9846)) - Avoid trailing slash when deserializing from lockfile ([#9848](astral-sh/uv#9848)) - Fix bug in terms when collapsing unavailable versions in resolver errors ([#9877](astral-sh/uv#9877)) - Fix suggestion to use `uv help python` on invalid install requests ([#9820](astral-sh/uv#9820)) - Skip root when assessing prefix viability ([#9823](astral-sh/uv#9823)) - Avoid spurious 'Upgraded tool environment' in `uv tool upgrade` ([#9870](astral-sh/uv#9870)) ##### Rust API - Upgrade minimum Rust version to 1.83 ([#9815](astral-sh/uv#9815)) ##### Documentation - Document the `--fork-strategy` setting ([#9887](astral-sh/uv#9887)) ##### Preview features - Build backend: Allow underscores in entrypoints ([#9825](astral-sh/uv#9825)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

With a past version of uv, `uv add onnxruntime` would fail. To address that, we overly constrained the allowed onnxruntime version (#801). More recent versions of uv have now fixed this (astral-sh/uv#9827), so we remove the constraint. Fixes #835.

charliermarsh requested review from konstin, BurntSushi and zanieb and removed request for konstin and BurntSushi December 11, 2024 23:15

charliermarsh added the enhancement New feature or improvement to existing functionality label Dec 11, 2024

charliermarsh force-pushed the charlie/max-version-split branch from d899fa2 to 6607e99 Compare December 11, 2024 23:16

charliermarsh changed the title ~~Fork version selection based on requires-python requirements~~ Fork version selection based on requires-python requirements Dec 11, 2024

charliermarsh force-pushed the charlie/max-version-split branch from 6607e99 to fc13a4d Compare December 11, 2024 23:18

charliermarsh mentioned this pull request Dec 11, 2024

Fork when wheels only support a specific range #8492

Closed

charliermarsh force-pushed the charlie/max-version-split branch 2 times, most recently from 95139fd to 3491470 Compare December 12, 2024 03:44

charliermarsh force-pushed the charlie/max-version-split branch from 3491470 to 71839e0 Compare December 12, 2024 03:53

konstin approved these changes Dec 12, 2024

View reviewed changes

BurntSushi approved these changes Dec 12, 2024

View reviewed changes

charliermarsh force-pushed the charlie/max-version-split branch 4 times, most recently from 86621aa to 67de436 Compare December 13, 2024 02:52

Fork version selection based on requires-python requirements

a6ab42f

charliermarsh force-pushed the charlie/max-version-split branch from 67de436 to a6ab42f Compare December 13, 2024 02:54

charliermarsh mentioned this pull request Dec 13, 2024

Introduce a --fork-strategy preference mode #9868

Merged

charliermarsh merged commit 0ee2114 into main Dec 13, 2024
64 checks passed

charliermarsh deleted the charlie/max-version-split branch December 13, 2024 20:33

BrewTestBot mentioned this pull request Dec 13, 2024

uv 0.5.9 Homebrew/homebrew-core#201113

Merged

This was referenced Dec 14, 2024

MNT: planning for maintenance mode volodia99/nonos#387

Open

MNT: upgrade uv.lock (pyqt5-qt5 5.15.5, pyqt5-sip 12.16.1) 1313e/CMasher#188

Closed

resolver: requiring PyQt5 produces non-portable locks #9907

Closed

zanieb mentioned this pull request Dec 17, 2024

Build from source packages #9961

Closed

reyammer mentioned this pull request Dec 17, 2024

Revisit python's onnxruntime constraint (in light of uv's latest release) google/magika#835

Closed

neutrinoceros mentioned this pull request Jan 6, 2025

TST: decoupling stable CI and incompatibilities discovery (let's use lock files) astropy/astropy#17596

Open

reyammer mentioned this pull request Jan 23, 2025

python: relax constraint on onnxruntime google/magika#917

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fork version selection based on `requires-python` requirements #9827

Fork version selection based on `requires-python` requirements #9827

charliermarsh commented Dec 11, 2024

charliermarsh commented Dec 11, 2024

zanieb commented Dec 12, 2024

charliermarsh commented Dec 12, 2024

charliermarsh commented Dec 12, 2024

konstin left a comment •

edited

Loading

konstin Dec 12, 2024

charliermarsh Dec 13, 2024

konstin Dec 12, 2024

konstin Dec 12, 2024

konstin Dec 12, 2024

BurntSushi Dec 12, 2024

charliermarsh Dec 12, 2024

BurntSushi left a comment

BurntSushi Dec 12, 2024

konstin commented Dec 12, 2024

charliermarsh commented Dec 13, 2024

henryiii commented Dec 13, 2024 •

edited

Loading

Fork version selection based on requires-python requirements #9827

Fork version selection based on requires-python requirements #9827

Conversation

charliermarsh commented Dec 11, 2024

Summary

charliermarsh commented Dec 11, 2024

zanieb commented Dec 12, 2024

charliermarsh commented Dec 12, 2024

charliermarsh commented Dec 12, 2024

konstin left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BurntSushi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

konstin commented Dec 12, 2024

charliermarsh commented Dec 13, 2024

henryiii commented Dec 13, 2024 • edited Loading

Fork version selection based on `requires-python` requirements #9827

Fork version selection based on `requires-python` requirements #9827

konstin left a comment •

edited

Loading

henryiii commented Dec 13, 2024 •

edited

Loading