Extract wheels directly to install location

[zooba]

The current sequence of events for installing a wheel involves extracting it to a temporary location, then copying it to the install location. This requires additional work to copy metadata and permissions, which would not be necessary if it started in the target location (or within one directory rename of the target location).

Avoiding the copy step would be a significant performance increase for installing wheels, particularly on Windows where the current process triggers anti-malware scanners multiple times.

(Forked from #3055)

[zooba]

Okay, so I have two choices for the shorter install step. I'm happy to do either, but the latter is a bigger change and I'll only go that way if you were wanting to do it anyway:

1. Pass enough options (prefix, root, etc.) to RequirementPreparer that I can calculate the location to extract wheels to
2. Defer wheel extraction (other than metadata) until the install step

Right now there's consistency between all the different source types as handled by the pieces of the install command, so taking the second approach will complicate that. Then again, if you were hoping to streamline the install command to something more like "acquire files, resolve dependencies, repeat until ready, build non-wheels, install wheels" then I can start moving this way?

[pradyunsg]

Let's do the later.

[zooba]

I'll try and map things out and come up with a proper design proposal before moving too far forward. I think I want a new requirement type/subclass for wheels (to separate "install requirement" from "build/install requirement"), but I'm not totally sure yet.

[zooba]

Okay, so here's my high level plan:

• in the install command, ensure all reqs are editable or wheels before starting the actual install (this should just be a bit more filtering)
• in InstallRequirement.install, only support editable and wheel installs
• make building non-wheel requirements always produce a wheel file or directory (no point unzipping when we can copy, but the wheel file will be produced for the cache)
• when an install requirement links to a wheel, treat source_dir as pointing directly at the wheel file
• (maybe this means just changing the value; maybe it means adding more if checks; not 100% sure yet)
• use ZipImporter (or a manually implemented equivalent) for extracting metadata from wheel files
• make move_wheel_files able to extract from a file when source_dir points at a wheel, and in this case use the AdjacentTempDirectory I'm adding in Fixes #3055 Uninstall causes paths to exceed MAX_PATH limit #6029 to avoid overwriting files before they're fully extracted

I think because of the way that pkg_resources is being used for metadata, it'll turn out to be simplest to change source_dir to point directly at the wheel file. Flushing out the code paths that don't like that assumption is the tricky part.

[pradyunsg]

There is a no-binary option - which basically means that a certain package should be installed from an sdist directly. We can't remove the ability to setup.py install just yet. ;)

#5051 has some notes on a potential design for changing InstallRequirement. It's basically making a Distribution class which will be used for preparation / installation that differs for legacy sdist vs PEP 517 sdist vs wheel. It's a multi stage process to get there and I've been working on actually implementing it, for a day now. One of the effects of that would be of untangling the code to make things like identifying code that's only run for sdist vs wheel vs editable become easier to spot.

[pfmoore]

@pradyunsg My understanding is that --no-binary simply prohibits the download of wheels. It doesn't require that a sdist has to be direct-installed - and indeed if PEP 517 is being used, a sdist is installed by building a wheel, and then installing the wheel, even when --no-binary is present.

Having said that, switching to always installing by building a wheel first, even on the "legacy" path, is a definite change in behaviour, and one that we need to explicitly and clearly document. It'll probably be controversial, but honestly I think we'll have to bite the bullet at some point, and doing it as part of a change that adds other value is better than doing it as a standalone piece of work that only has internal benefits.

[zooba]

I agree with Paul that biting the bullet is going to have to happen, and this seems like a good time.

I'm following #5051 now, so let me know how I can help. It's certainly got a lot of overlap with what I'm thinking here - once it's done, the change to avoid eager-extraction of wheels should be much simpler.

[pradyunsg]

--no-binary says "Do not use binary packages" in it's docstring. Currently, doing a pip install --no-binary six six results in a setup.py install (without going through a wheel).

My understanding stems from the fact that it is that a commonly suggested way on Stackoverflow etc for cases where wheel based installation is to be avoided.

I think I've suggested it to people offline as well as a workaround for when wheels don't do what they want.

That said, I have no issues removing that code path. In fact, the idea behind #5051 came from an attempt to understand what be done to separate the build paths that use setup.py install to be able to treat them differently and removing them in a post PEP 517 world, is something I'm on board for.

[pradyunsg]

@zooba I've started working on #5051 already -- I'd hate for us to duplicate effort here.

If you want to take up some parts/all of it, I have no issues deferring to you - just let's not do the exact same thing independently in slightly different ways. ;)

[pfmoore]

Currently, doing a pip install --no-binary six six results in a setup.py install (without going through a wheel).

If six had a pyproject.toml, we'd go via a wheel (using the PEP 517 backend). But you're correct that the "legacy" setuptools install path doesn't build a wheel and then install it. I don't consider that to be something that --no-binary guarantees, though. (If it were, then we would have needed to remove that guarantee when PEP 517 support landed).

[pradyunsg]

Right. I've been thinking about it in terms of the most common use case I've seen for that functionality - the way you put it makes sense to me. The no-binary direction is probably a false alarm due to my (mis)interpretation of the option's effect.

[pradyunsg]

FTR, IIUC, the setup.py install removal discussion has to happen in #5204.