Add a uv build backend #3957
Comments
chrisrodrigue
changed the title
uv should provide a build backend
I personally believe that this is pretty much against the whole idea of modern approach and splitting the backend vs. frontend responsibilities. The idea (philosophically) behind the backend/frontend split is that the maintainers of the project (via pyproject.toml) choose the backend that should be used to build their tool, while the user installing Python project is free to choose whatever fronted they prefer. This is IMHO a gamechanger in the python packaging and we should rather see a stronger push in that direction than weakening it. And I think it's not going to go back - because more and more projects will revert to use pyproject.toml and backend specification of build environment. In case of Airflow for example - as of December there is no way to install airlfow "natively" without actually installing hatchling and build environment. But we do not give the option to the frontend. You MUST use hatchling in specified version and few other dependencies specified in specific versions to build airflow. Full stop. UV won't be able to make their own choices (It will be able to choose a way how to create such an environment but not to choose what should be used to build airlfow). But also maybe my understanding of it is wrong and maybe you are proposing something different than I understand. BTW. I do not think access to PyPI is needed to build a project with build backend. The frontend my still choose any mechanism (including private repos if needed) to install build environment, no PyPI is needed for it, the only requirement is that pyproject.toml specifies the environment. |
|
I agree and think the backend and frontend specifications should be separate, I am merely suggesting that You do need access to PyPI or some repo hosting the build backend to build a project. Python does not include IMO, Without the capability to build from source out of the box, Python is hamstringed and can only run the most rudimentary scripts, leaving users with multi-module projects to resort to ugly |
But this is where the whole packaging for Python is heading. The PEPs of packaging precisely specify this is the direction and if project maintainers choose so, unless you manually (like conda) maintain your build recipes for all the packages our there, you won't be able to build project "natively". Just to give you example of Airflow. Without The only way to find out what dependencies Airflow needs for editable build, or to build a wheel package is to get the right version of hatchling and get the frontend execute the build_hook - the build hook returrns such dependencies dynamically. You can see it yourself here https://github.com/apache/airflow/blob/main/pyproject.toml -> there is no way to build airflow from sources in current main without actually installing those packages: And letting hatchling invoke And I think - personally (though I was not part of it) - that the decisions made by the packaging team were pretty sound and smart, and they deliberately left the decision for maintainers of a project to choose the right backend packages needed (and set of 3rd-party tools) and all frontends have no choice but to follow it. I understand you might have different opinion, but here - the process of Python Software Foundation and Packaging team is not an opinion - they have authoritative power to decide it by voting and PEP approval. And the only way to change it is to get another PEP approved. Here is the list of those PEPs (and I am actually quite happy Airflow after 10 years finally migrated out of setuptools and setup.py by following those standards as finally the tooling - including the modern backends you mentioned support it for sufficently long time):
|
|
And BTW. if uv provides a build backend, you will still be able to choose it when you maintain your project - but it will also be a 3rd-party dependency :) Someone installing your project with any frontend will have to download and install according to your specification in pyproject.toml. Similarly as hatch/hatchling pair that are separate. Even if you use hatch to install airlfow, it has to anyhow download hatchling in the version specified by the maintainer of the project you are installing and use it to build the package. |
|
It's interesting that One of the nice things about Astral tools like If you use I think [build-system]
requires = ["uv"]
build-backend = "uv.api" |
Well. There are always trade-offs and what you see from your position might be important for you, might not be important for others and the other way. For example - If (like we did with airflow being popular package) you had gone through some of the pains where new releases of setuptools suddently started breaking packages being built - because deliberately or accidentally breaking compatibilities and suddenly being flooded by 100s of your users having problem with installing your package without you doing anything you'd understand that bundling the way how things are run with specific version of frontend is a good idea when you have even moderately big package. That's what I love that we as maintainers can choose and "lock" the backend to the tools and version of our choice - rather than relying that the version of the tool that our users choose will behave consistently over the years. That was a very smart choice of packaging team based on actual learnings from their millions of users over many years, and while I often criticized their choices in the past, I came to understanding that it's my vision that is short-sighted and limited - I learned a bit of empathy. I'd strongly recommend a bit more reading and understanding what they were (and still do) cooking there. Python actually deliberately removeed setuptools in order to drive more the adoption of what's being developed as packaging standards (and that's really smart plan that was laid out years ago and is meticulously and consistently, step-by-step put in motion. And I admire the packaging team for that to be honest. What you really think about is not following and bringing back old "setuptools" behaviour is something else that packaging team has already accepted and a number of tools are implementing https://peps.python.org/pep-0723/ - which allows you to define a small subset of pyproject.toml metadata (specifically dependencies) in the single-file scripts. And this is really where yes - any front-end implementing PEP-723 should indeed prepare a venv, install dependencies and run the script that specifies such dependencies. Anything more complex that really has a bit more complex packaging need - putting more files together, should really define their backend, in order to maintain "build consistency", otherwise you start to be at mercy of tool developers who might change their behaviours at any time and suddenly not only you but anyone else who want to build your package will suddenly have problems with it. But yes. If BTW. Piece of advise - for that very reason, as a maintainer you should do: [build-system]
requires = ["uv==x.y.z"]
build-backend = "uv.build"Otherwise you never know which version of uv people have and whether they have a version that is capable of building your package at all (i.e. old version that has no backend). |
|
Also - again if you have proposal how to improve packaging, there are discourse threads there and PEP could be written, so I also recommend you, if you are strongly convinced that you can come up with a complete and better solution - please start discussion there about new PEP, and propose it, lead to approval and likely help to implement in a number of tools - this is the way how standard in packaging are being developed :) |
FYI, I beleive this is because pip maintainers are very conservative when it comes to breaking changes, not because it is the intended future of Python packaging. For example, the old resolver, which can easily install a broken environment, is still available to use even though the new resolver has been available for over 5 years and turned on by default for over 4 years. |
|
A uv-aware build backend would enable private git packages to depend on other private git packages, and still be installed with |
|
Just some more thoughts, notes, and ramblings on this. Full stack uvThe build backend capability could be rolled into the This would give developers the option to utilize uv as either a frontend, a backend, or both, while still maintaining the single static dependency. Developers electing to use uv as both frontend and backend could have access to some optimizations that might not be possible individually. We could call this use case “full stack uv” since that’s usually what we call frontend + backend, right? 🤪 Full stack uv without a specified versionrequires = ["uv"]
build-backend = "uv"Backend Or,
Full stack uv with pinned or maximum versionrequires = ["uv==0.5"]
build-backend = "uv"PEP 517 build isolation can guarantee that frontend
However, a decision could be made to maintain backward compatibility for the backend, such that newer versions of uv could satisfy whichever version is declared. On PEP 517 complianceThe backend feature could be PEP 517 compliant so that other frontends (like poetry, pdm, pip, hatch, etc.) can use uv as a backend, but this could be distinct from full stack uv.
uv would need to expose some hooks in the build environment via Python API. The mandatory and optional hooks at the time of this writing are: # Mandatory hooks
def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
...
def build_sdist(sdist_directory, config_settings=None):
...
# Optional hooks
def get_requires_for_build_wheel(config_settings=None):
...
def prepare_metadata_for_build_wheel(metadata_directory, config_settings=None):
...
def get_requires_for_build_sdist(config_settings=None):
...Perhaps a highly optimized, importable module or package named
|
|
If it was up to me, and assuming the build backend isn't too complicated, I'd consider doing both:
|
|
Hatch does 2. with Hatchling. |
charliermarsh
added
the
enhancement
|
To make it both sides of this thread happy I want to point to @hauntsaninja's comment above. In addition, Definitely, that would have some maintainability costs:
Current vs Future implementationI think that current path of implementation is correct (which is still in preview) that it provides an option for build backend (which is correctly not set as default) to force every build frontend to use Building projects with
|
|
Interesting. So, like what Flit is doing actually, without the mistake (IMO) of extending the backend features in the frontend. Also what Hatch is doing based on a comment from @ofek earlier. I wouldn't recommend having two build-backends though, but only one that is pure-python. Then uv, as a frontend, can detect it and hijack it with a Rust implementation. Other frontends would work as usual with the pure Python backend. |
|
The Astral team requested that we not keep going back and forth with pros/cons etc until they have time to review the current discussion and talk it out internally As Charlie said, everyone seems to have said their piece and has been heard. Have a Merry Christmas everyone 🎄 |
|
@charliermarsh Are there any news on this? Is it discussed only internally or is there some other location where the discussion takes place? |
|
We're discussing this internally. We'll chime in here eventually. |
|
We've been thinking about all the concerns noted here and have some thoughts to share. As a TL;DR:
I’ll talk through our decision, starting with some background on our motivations. When we first launched uv's project management capabilities, we tried using a third-party backend by default. The feedback from users about this experience was very negative: there was confusion that errors were coming from another tool, that they needed to read its documentation to understand what was happening, and that they often had to add configuration for it to work with their project. In response to this feedback, we decided not to use a build system in We cannot deliver the user experience we want with existing build backends — we need better error messages, better defaults, and more robust handling of edge cases. Due to the intentionally isolated nature of a build backend, there is context about the user's intent that cannot be captured without integration with the build frontend. For example, an integrated backend lets us understand why the frontend is building the package or why the backend failed so we can provide error messages that guide people to a successful build. We have devoted a lot of resources to improving build error messages already by parsing the output of a build backend to provide hints to users, but that’s brittle. While it’s a major benefit that we can special case our own backend (as done in Poetry and Hatch), the build backend will be usable with a different frontend than uv and all of the other build backends will continue to be supported by uv’s frontend. We’ve always focused heavily on standards compliance and will continue to do so. Shifting gears, there are a few alternative suggestions in the discussion:
I'll respond to these next. With uv, we've invested heavily in a solid foundation for Python packaging in Rust. We have a lot of code and abstractions, with a focus on correctness and performance. It's not feasible for us to maintain duplicate implementations in Python, as in (1). We think this is infeasible both due to time constraints and, as mentioned by some, foundational differences in behavior between the Rust and Python standard libraries — subtly different behavior here would be a disaster. We want to keep our rate of development high and deliver features to users — we can't do that while maintaining multiple versions of our core abstractions. For similar reasons, we think that (2) is infeasible. Even if we accept the premise that it is feasible, it's against our product principles to shadow another tool implicitly and re-use their configuration. Additionally, our goal is not just to "make a more performant build backend". As some mentioned, the overhead of a build backend is generally minimal (though noticeable when using uv). We want to innovate on the experience of building Python packages because it is a core part of working on Python projects. The feedback from our users (in the form of questions and bug reports in our issue tracker) has made it clear that improving integration with a build backend will be very high impact. At this point, we’re at “build a backend in Rust” or “don’t do it at all” (3) — there are a few recurring concerns that support the latter conclusion. We’re taking these concerns seriously, but want to balance them with the value we see for users. I'll do my best to address them here, but I'll admit that packaging is a very hard problem and I do not have good answers to all of the concerns raised.
With that, I want to take a second to thank everyone for their feedback here. It’s helped us prioritize some changes (like a separate package) that we may not have otherwise. Going forward, I'd like to focus this issue on the behavior of the build backend within this framing instead of debating if it should be done or not. |
(2) should help a lot, I think. In particular it seems reasonable to assume that the MSRV of a dedicated build-backend crate can be a lot lower than the MSRV of uv as a whole. In particular, if you can avoid entirely the use of problem crates such as ring. That being said, the arguments with regard to (1) aren't as compelling as they could be. There are CPU architectures where Rust isn't available at all, and therefore cryptography isn't available either. I would say pydantic isn't available either, except that really, pydantic can be commonly downloaded all it wants but it is hardly core infrastructure. Cryptography isn't universally needed, there's absolutely loads of stuff Gentoo users commonly install that are written in Python, without ever ending up hitting a dependency tree that leads back to cryptography... but there are definitely some packages that are uninstallable for Gentoo users on less common architectures, solely because of cryptography. Build backends are a much bigger influence here than something like pydantic or cryptography, because they could potentially influence "core infrastructure" packages that are written in pure python. For something verging on a worst-case example, what if "trove-classifiers" ported to the uv build backend because they heard it was so popular nowadays, and also it's basically a data package so what's the big deal and who really needs it except for other package developers. Except, setuptools and flit don't care but hatchling depends on it so now a massive number of packages can't be built on non-rust platforms. There are other packages that are common runtime packages needed by other software which could be a huge problem as well, e.g. attrs, certifi, appdirs, xdg... so we end up in a situation where people are told about a new integrated backend that Having a pure python version would at least prevent this invisible line across which package maintainers have to magically know that it's socially bad to use a specific backend.
I'm not sure it's conceptually solvable. Some projects install their testsuite inside the wheel, that's about all you can do. But most projects consider it bad to install tests as part of the wheel, as it penalizes pip install users. The handful that install their testsuite tend to argue that they are the exceptional case and that end users creating a virtualenv should then go ahead and import and run the testsuite to make sure everything actually works... because they're pretty confident that there are cases where it doesn't work. It's a common concern in the science ecosystem, for example (delicate interactions between various moving parts can do that for you 🤷). We may also have to first solve the problem where large subsets of the python packaging ecosystem object to including tests even in the sdist, as they regard sdists as purely for pip to build wheels and see tests as dead weight. That includes going around to other projects and advocating to not include tests in the sdist! :( One possibility is to download both the wheel and the sdist and then install the wheel and test it using the sdist copy of the test suite. But it's not immediately clear what the advantages are over just building the sdist using a build backend. It would add great complexity to the process, doesn't solve the occasional need to backport a patch that fixes crashes, can't be used for anything that has compiled extensions, doubles or triples the download size... It's something I've thought about before, but with the association "worst case scenario, we might be able to do this to get out of a very sticky situation". |
I should take this back. Sigstore appears to rely on it, which means it is extremely problematic to validate cryptographic signatures for cpython starting with 3.14. Oh well, we can just assume it was verified on an amd64 developer machine and disable signature verification -- and the Gentoo PGP signature upon the checksum manifest for cpython will prove that that was checked by someone else. Not great, but not the end of the world. Still, I do admit I was wrong. Pydantic is used for more than I thought... |
@zanieb Thanks for this detailed write-up, tradeoff of each approach is now more obvious to me.
When I compare these tradeoffs to the rust-only backend tradeoffs (those five mentioned here), and see how impactful each side is, I'd ended-up to the |
|
If you're looking for tips on how to minimize the binary size of the Rust build backend, I maintain a repository with some information on that: I hope it can help a bit. |
will |
|
I think it's a good decision. It addresses most concerns of mine. One suggestion for consideration though: I think it would be great if uv tooliing ( If package creator wants to still use Of course, that's also a bit problematic - which other backends would be supported in this case? not sure how to solve it easily, but maybe simply arbitrary choosing (and maintaining?) some of the popular backends would be a good choice, or maybe a way to plugin different other backends as well ? not so sure. |
We haven't worked out the details of this yet. If we do version the build backend separately, then presumably if there have been no changes to it then a later uv version could use its bundled version of the build backend. We'll need to design this carefully. It's probably easiest to start with 1:1 versioning though.
I think this makes sense. We already do support quite a few: |
|
I'm sorry but I'm a little confused now — will this backend be providing a PEP 517 wrapper, or will packages using it be installable from source only using dedicated tooling (i.e. |
As I read what @zanieb wrote - it will be fully compliant with PEP 517, but it will also have some custom ways to hook into At least that's how I interpret it - and having experience with a number of fronted + backend combinations, this is often what happens in other cases. Just to give an example - hatch can interact with some specific "hatchling" behaviours that allow to do some specific things, for example hatch allows to run a custom build hook when it is implemented in Another example is enabling reproducible builds - while many backends have a way to produce reproducible builds, one of the ways how to do it is to pass an environment variable to set the "date" with which files will be stored in .whl, but you also should do some other stuff before you run the build - for example make sure that group bits are cleared on all source files, because by default git uses system umask and it might impact reproducibility. Those behaviours (and likely a number of others) are not entirely covered by PEP 517 - possibly some of those will be in the future by follow-up PEPs - but in order to make a good user experience, some custom ways of interaction between frontend and backend have to be implemented. I think it will not impact the basic "build wheel" or "build sdist" or "install package in editable mode" features that PEP 517 standardised, it's more driven be improving user experience for more advanced cases. But that's my interpretation, so I might be wrong. |
|
👍 We'll distribute the build backend as a dedicated package on PyPI. And it will be usable as a standalone build backend entirely independently of uv, e.g.: requires = ["uv-build>=0.5.15,<0.6"]
build-backend = "uv_build"(We might change the package or module name before releasing this; consider it an illustrative example.) But when you're using uv, and we see a package that's using the uv build backend, we can (e.g.) execute the build hooks directly or have other integrations. |
and others
uv is a fantastic tool that is ahead of its time. In the same vein as ruff, it is bundling many capabilities that Python developers need into a single tool. It currently provides the capabilities of
pip,pip-tools, andvirtualenvin one convenient binary.Python can't build out of the box
As of Python 3.12, which has removed
setuptoolsandwheelfrom standard Python installations, a user is unable to performpip install -e .orpip install .in a localpyproject.tomlproject without pulling in external third-party package dependencies.This means that in an offline environment without access to PyPI, a developer is dead in the water and cannot even install their own project from source. This is a glaring flaw with Python, which is supposed to be "batteries included."
uv can fix this
I propose that the
uvbinary expand its capabilities to also function as a build backend.If uv could natively build projects from source, it would be a game changer!
The text was updated successfully, but these errors were encountered: