speed up x install by skipping archiving and compression

[onur-ozkan]

Performing archiving and compression on x install is nothing more than a waste of time and resources. Additionally, for systems like gentoo(which uses x install) this should be highly beneficial.

benchmark report

Resolves #109308

r? Mark-Simulacrum (I think you want to review this, feel free to change it if otherwise.)

[onur-ozkan]

Right now I am on a low-resource machine and can't perform benchmarks between current-new way of x install. Most likely, I'll be able to do that sometime next week.

[Mark-Simulacrum]

Propagating prepare_only throughout the code feels like a higher cost than is worth paying. I'd be interested in seeing a comparison of cost between doing nothing (i.e., this PR) vs a new compression format that is a no-op (i.e., we produce foo.tar not foo.tar.gz or foo.tar.xz).

My suspicion is that producing the tarball and unpacking it is basically free, at least on most systems, it's the compression & decompression that hurts (particularly with gzip or xz, neither of which are fast on either operation).

Adding a no-op compression format should be much cheaper in lines of code and added maintenance complexity, so it's my preferred approach even if the tarballing is slightly more expensive than nothing.

[onur-ozkan]

Most likely, I'll be able to do that sometime next week.

x install took 0:10:45 on this branch, and 0:12:12 on f536185 (immediate previous commit).

System specs:

CPU: AMD Ryzen 9 5900X 12-Core @ 24x 4.2GHz
RAM: 32G 3600 MHz DDR4
DISK: 500GB Nvme 3470MB-2600MB/S M.2 SSD

OS: Void Linux
Kernel: x86_64 Linux 6.5.13_1

Configuration:

change-id = 116881
profile = "dist"

[llvm]
download-ci-llvm = true

[build]
submodules = true
extended = true

[rust]
download-rustc = false
deny-warnings = false

[onur-ozkan]

Propagating prepare_only throughout the code feels like a higher cost than is worth paying. I'd be interested in seeing a comparison of cost between doing nothing (i.e., this PR) vs a new compression format that is a no-op (i.e., we produce foo.tar not foo.tar.gz or foo.tar.xz).

What about the current approach (91b56e4) ? This shouldn't add any complexity on bootstrap I believe.

[Mark-Simulacrum]

The current in-PR approach is significantly better, but it still introduces more modality than I'd like. It also means if we do want plain tarballs (e.g., to compress out of band, or not compress at all), we'd need an extra mode.

It also still adds complexity in the form of the unwrap()s added in various places that want a tarball.

[onur-ozkan]

The current in-PR approach is significantly better, but it still introduces more modality than I'd like. It also means if we do want plain tarballs (e.g., to compress out of band, or not compress at all), we'd need an extra mode.

In that case yes, we need another mode. I don't have strong feelings, but I don't understand how worth it is to generate tarballs under "build/dist" when we can simply avoid doing that altogether.

Currently, bootstrap handles 'install' with the following logic:

1) prepare --> 2)generate the tarball --> 3)compress --> 4)use the prepared directory (which is generated at the first step)

With this PR:
1)prepare -> 2)use the prepared directory

So, anything between 'prepare' and 'use the prepared directory' is a redundant operation for the actual functionality of x install.

From a performance perspective, I think there won't be much difference (if the disk is not too slow), but I'm not sure about the disk usage (for plain tarballs), maybe I should check that too.

[Mark-Simulacrum]

Yeah, I understand that anything beyond the current state makes the operation slower. The amount of usage of install should be very small though, only users building from source explicitly to install go through that path. Almost everyone is using rustup (or other package managers) that handle this behind the scenes.

But we need to thread the tradeoff between adding complexity and extra state to consider as we evolve, possible improved speed with more granular options, and impact of that speed on users. IMO, the tradeoff in this case makes me want to do minimum addition - i.e., supporting a no-op compression format - since that's somewhere we already have complexity (gz vs xz, and I think various profiles within those), so an extra variant is a cheap price to pay, and nothing except the compression code should need to care.

[Dylan-DPC]

@onur-ozkan any updates on this? thanks

[onur-ozkan]

@onur-ozkan any updates on this? thanks

Soon I will work on it again.

[onur-ozkan]

Functionality remains identical, but now there is no added complexity on bootstrap and it is solely controlled with the new compression format "no-op" added for x install. Let me know if you prefer to create the tarball sources rather than skipping them (with making the compression profiles as fast as possible).

@rustbot ready

[Mark-Simulacrum]

Thanks, this looks good to me.

@bors r+

[bors]

📌 Commit a13ec8d has been approved by Mark-Simulacrum

It is now in the queue for this repository.

[bors]

🌲 The tree is currently closed for pull requests below priority 50. This pull request will be tested once the tree is reopened.

[bors]

⌛ Testing commit a13ec8d with merge 26cd5d8...

[bors]

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 26cd5d8 to master...

[rust-timer]

Finished benchmarking commit (26cd5d8): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.7%	[-0.8%, -0.6%]	4
Improvements ✅ (secondary)	-0.4%	[-0.7%, -0.2%]	24
All ❌✅ (primary)	-0.7%	[-0.8%, -0.6%]	4

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.4%	[2.4%, 2.4%]	1
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	2.4%	[2.4%, 2.4%]	1

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-6.2%	[-7.0%, -5.4%]	6
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 649.797s -> 651.624s (0.28%)
Artifact size: 311.14 MiB -> 311.12 MiB (-0.01%)