Allow users to mark platforms as "required" for wheel coverage #10067

charliermarsh · 2024-12-20T22:00:55Z

Summary

This PR revives #10017, which might be viable now that we don't enforce any platforms by default.

The basic idea here is that users can mark certain platforms as required (empty, by default). When resolving, we ensure that the specified platforms have wheel coverage, backtracking if not.

For example, to require that we include a version of PyTorch that supports Intel macOS:

[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = ["torch>1.13"]

[tool.uv]
required-platforms = [
    "sys_platform == 'darwin' and platform_machine == 'x86_64'"
]

Other than that, the forking is identical to past iterations of this PR.

This would give users a way to resolve the tail of issues in #9711, but with manual opt-in to supporting specific platforms.

BurntSushi

I like this!

BurntSushi · 2024-12-21T13:04:59Z

crates/uv-distribution-types/src/prioritized_distribution.rs

@@ -196,6 +199,15 @@ impl Display for IncompatibleDist {
                IncompatibleWheel::RequiresPython(python, _) => {
                    write!(f, "Python {python}")
                }
+                IncompatibleWheel::MissingPlatform(marker) => {
+                    if let Some(platform) = KnownPlatform::from_marker(*marker) {
+                        write!(f, "no {platform}-compatible wheels")


BurntSushi · 2024-12-21T13:22:17Z

docs/reference/settings.md

@@ -273,6 +273,36 @@ package = false

 ---

+### [`required-platforms`](#required-platforms) {: #required-platforms }


I think it might help if the docs specifically "compared and contrasted" required-platforms with environments. They do different things of course, but they take the same inputs and I could see end users being confused by them.

(What I tend to do in docs is to mention the other in the docs and contrast them. e.g., mention environments in this section and required-platforms in the environments section. That way, users see the contrasting doc regardless of where they look.)

I think doing this would be helpful for naming. (as well as very helpful for users)

charliermarsh · 2024-12-21T23:59:14Z

Another case that this could solve (but would require user intervention): #10085. Ideally, I think we should be able to give users a hint when the sync fails -- like "There are no available wheels for this platform, add required-platforms".

konstin

The feature is definitely needed and i like the approach, though the find_environments if i haven't missed something fails to uphold the required platforms in certain conditions.

konstin · 2025-01-10T13:17:18Z

crates/uv-workspace/src/pyproject.rs

+            ]
+        "#
+    )]
+    pub required_platforms: Option<SupportedEnvironments>,


Should this be required_environments or required_markers? The other markers list is environments, platforms lacks symmetry

It was somewhat intentional but I can change it.

We might want to change the other one? I feel like environments is a bit overloaded in the longterm.

konstin · 2025-01-10T13:25:15Z

crates/uv-resolver/src/resolver/mod.rs

+
+    // Find all dependencies on the current package.
+    let mut marker = MarkerTree::FALSE;
+    for index in incompatibilities {


Don't we add additional incompatibilities that target id when we find a new package foo with foo --[marker]--> id? Could that lead to packages missing platform support because we could have a case like the following:

Required platform linux

Choose A 2, with A -> B; windows, A -> C
Choose B 2. B 2 does have only a mac wheel but that's fine because it's only required on windows, so our required linux is coverd.
Choose C 2, with C -> B

Try to install on linux. We try to install A 2, C 2 and B 2, but B 2 fails since it has no wheel.

@konstin -- I think this would actually work as-is, though for "bad" reasons. Basically, when we visit B, we won't know that it's "only required for Windows", because we don't do "aggressive forking" today, so we'd still try to solve for Linux. It might get filtered out later? I can't quite remember.

Could you add a test case covering this scenario? So it doesn't fail with future resolver changes.

zanieb · 2025-01-10T17:50:07Z

Since this is specific to wheels should we name this differently? Like, if I encounter a build failure why would it be obvious that setting a required_platform would solve that? Is this sort of no_build_platforms?

charliermarsh · 2025-01-11T14:13:17Z

@zanieb -- That's true, although I'd read that setting as "platforms on which building should be disabled", which isn't quite the same. We're not enforcing that all packages have wheels on these platforms. We're enforcing that all packages have wheels or source distributions on these platforms (and so, building is considered ok).

charliermarsh · 2025-01-11T14:13:42Z

I sort of prefer something that doesn't get into the source vs. binary distinction, and instead gets at the intent of the flag: to ensure that the lockfile supports the platforms you care about.

zanieb · 2025-01-11T19:53:21Z

Oh I guess I'm getting confused by the pull request title and body which say...

When resolving, we ensure that the specified platforms have wheel coverage

This sounds like we require wheels for the platforms?

charliermarsh · 2025-01-11T19:54:39Z

We require that there are compatible distributions for the platform. That could be a source distribution, or (for packages that lack source distributions) a wheel.

zanieb · 2025-01-11T20:26:04Z

That makes sense, it'd help to change the way this is presented in the title / body then. With this context, no_build_platforms definitely does not make sense.

charliermarsh · 2025-02-11T02:47:57Z

crates/uv-workspace/src/pyproject.rs

+    /// For example, `environments = ["sys_platform == 'darwin'"]` would limit uv to solving for
+    /// macOS (and ignoring Linux and Windows). On the other hand, `required-environments = ["sys_platform == 'darwin'"]`
+    /// would _require_ that any package without a source distribution include a wheel for macOS in
+    /// order to be installable.


I re-did the docs here. Is it any better, @zanieb @BurntSushi?

Beautiful. Thank you!

konstin

I like the updated documentation, i left some nits.

konstin · 2025-02-11T11:06:05Z

docs/reference/settings.md

+the resulting resolution contains wheels for specific platforms, or fails if no such wheels
+are available.


I would add something about using older versions here, like "If a version of a package does not contain the required source distribution or wheels, uv will try older versions until it finds a version with sufficient support".

konstin · 2025-02-11T11:06:34Z

docs/reference/settings.md

+For example, `environments = ["sys_platform == 'darwin'"]` would limit uv to solving for
+macOS (and ignoring Linux and Windows). On the other hand, `required-environments = ["sys_platform == 'darwin'"]`
+would _require_ that any package without a source distribution include a wheel for macOS in
+order to be installable.


Suggested change

order to be installable.

order to be installable, while still including Linux and Windows wheels in the lockfile.

BurntSushi

Docs are much improved!

zanieb

The docs are great. Not sure if we want to consider another name. This seems fine for now.

Uhh trying Fork on... Add heuristic Gate to local versions Remove KnownMarkers Add required platform support

johnthagen · 2025-02-17T19:43:11Z

I've wanted this for so long I almost can't believe my eyes that someone has implemented it. ❤️ ❤️ ❤️

charliermarsh · 2025-02-17T19:43:51Z

Please let me know what you use it for and if it works for you (or not)!

johnthagen · 2025-02-17T19:50:06Z

Please let me know what you use it for and if it works for you (or not)!

My use case is that I work with dependencies that will sometimes (suddenly) drop publishing wheels for certain platforms (often something like ARM64 Linux) and it can become very challenging to detect this. What happens is that we'll poetry update or similar on, say, macOS ARM and that will resolve to the latest version that is missing wheels for other platforms that we need.

A couple real-life examples;

(Missing Linux ARM wheels): https://pypi.org/project/ai-edge-litert-nightly/1.1.2.dev20250211/#files
(Now they're back!): https://pypi.org/project/ai-edge-litert-nightly/1.1.2.dev20250216/#files

We wouldn't want to accidentally lock to the version that is missing Linux ARM wheels while locking from macOS.

Another example:

0.18 has Linux ARM wheels: https://pypi.org/project/open3d/0.18.0/
0.19 does not have Linux ARM wheels: https://pypi.org/project/open3d/0.19.0/#files

Our current workaround is to pin versions in pyproject.toml after we either hit issues later (for trying to poetry sync in a Linux ARM64 container on an ARM64 macOS host).

Being able to express our platform needs so that the resolver can make this work properly would be great.

I hope this feedback helps!

uwu-420 · 2025-02-18T09:18:35Z

@charliermarsh Love to see this making it into the latest release, thank you ❤️
This also helps me massively at work because I can at least ensure that my team only adds packages that have wheels for {x86_64, aarch64} x {linux, macos}. The only thing that's missing for us is to ensure that there are wheels for macos 10.15 (for x86_64) and 11.0 (for aarch64) because these are the minimum macos versions we have to support. Related to my issue #9837

trim21 · 2025-02-18T13:44:02Z

the option in PR description is not updated, it should be required-environments instead of required-platforms ...

charliermarsh force-pushed the charlie/required-platforms branch from 59b6690 to eeb249d Compare December 21, 2024 00:21

charliermarsh marked this pull request as ready for review December 21, 2024 04:14

charliermarsh requested review from BurntSushi, zanieb and konstin December 21, 2024 04:14

BurntSushi approved these changes Dec 21, 2024

View reviewed changes

konstin mentioned this pull request Jan 10, 2025

error: Distribution kaleido==0.2.1.post1 @ registry+https://pypi.org/simple can't be installed because it doesn't have a source distribution or wheel for the current platform #10464

Open

konstin reviewed Jan 10, 2025

View reviewed changes

konstin mentioned this pull request Jan 22, 2025

inconsistency between uv sync and uv pip install #10849

Closed

charliermarsh force-pushed the charlie/required-platforms branch 4 times, most recently from 97b4965 to d567801 Compare February 11, 2025 02:47

charliermarsh commented Feb 11, 2025

View reviewed changes

konstin approved these changes Feb 11, 2025

View reviewed changes

BurntSushi approved these changes Feb 11, 2025

View reviewed changes

zanieb approved these changes Feb 11, 2025

View reviewed changes

charliermarsh force-pushed the charlie/required-platforms branch from d567801 to 7385e09 Compare February 14, 2025 19:54

Respect all platforms

71e7ecb

Uhh trying Fork on... Add heuristic Gate to local versions Remove KnownMarkers Add required platform support

charliermarsh force-pushed the charlie/required-platforms branch from 7385e09 to 71e7ecb Compare February 14, 2025 20:00

charliermarsh merged commit 172305a into main Feb 14, 2025
73 checks passed

charliermarsh deleted the charlie/required-platforms branch February 14, 2025 20:11

johnthagen mentioned this pull request Feb 14, 2025

Port from Poetry to uv johnthagen/python-blueprint#238

Open

11 tasks

BrewTestBot mentioned this pull request Feb 17, 2025

uv 0.6.1 Homebrew/homebrew-core#208046

Merged

uwu-420 mentioned this pull request Feb 18, 2025

Feature request: Have MACOSX_DEPLOYMENT_TARGET as uv config or have a default place from where env vars are loaded for uv #9837

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow users to mark platforms as "required" for wheel coverage #10067

Allow users to mark platforms as "required" for wheel coverage #10067

charliermarsh commented Dec 20, 2024 •

edited

Loading

BurntSushi left a comment

BurntSushi Dec 21, 2024

BurntSushi Dec 21, 2024

zanieb Jan 11, 2025

charliermarsh commented Dec 21, 2024

konstin left a comment

konstin Jan 10, 2025

charliermarsh Jan 10, 2025

zanieb Jan 10, 2025

konstin Jan 10, 2025

charliermarsh Jan 10, 2025

konstin Jan 13, 2025

zanieb commented Jan 10, 2025

charliermarsh commented Jan 11, 2025

charliermarsh commented Jan 11, 2025

zanieb commented Jan 11, 2025

charliermarsh commented Jan 11, 2025

zanieb commented Jan 11, 2025

charliermarsh Feb 11, 2025

BurntSushi Feb 11, 2025

konstin left a comment

konstin Feb 11, 2025

konstin Feb 11, 2025

BurntSushi left a comment

zanieb left a comment

johnthagen commented Feb 17, 2025

charliermarsh commented Feb 17, 2025

johnthagen commented Feb 17, 2025 •

edited

Loading

uwu-420 commented Feb 18, 2025

trim21 commented Feb 18, 2025

		@@ -273,6 +273,36 @@ package = false

		---

		### [`required-platforms`](#required-platforms) {: #required-platforms }

		the resulting resolution contains wheels for specific platforms, or fails if no such wheels
		are available.

	order to be installable.
	order to be installable, while still including Linux and Windows wheels in the lockfile.

Allow users to mark platforms as "required" for wheel coverage #10067

Allow users to mark platforms as "required" for wheel coverage #10067

Conversation

charliermarsh commented Dec 20, 2024 • edited Loading

Summary

BurntSushi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

charliermarsh commented Dec 21, 2024

konstin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zanieb commented Jan 10, 2025

charliermarsh commented Jan 11, 2025

charliermarsh commented Jan 11, 2025

zanieb commented Jan 11, 2025

charliermarsh commented Jan 11, 2025

zanieb commented Jan 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

konstin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BurntSushi left a comment

Choose a reason for hiding this comment

zanieb left a comment

Choose a reason for hiding this comment

johnthagen commented Feb 17, 2025

charliermarsh commented Feb 17, 2025

johnthagen commented Feb 17, 2025 • edited Loading

uwu-420 commented Feb 18, 2025

trim21 commented Feb 18, 2025

charliermarsh commented Dec 20, 2024 •

edited

Loading

johnthagen commented Feb 17, 2025 •

edited

Loading