Enable xz archives #1284
Enable xz archives #1284
Conversation
wt
force-pushed
the
enable_xz_archives
branch
8 times, most recently
from
760f1d4 to
65b64b5
Compare
|
The static rust environment variable is interfering with this since the url it adds ends up on the top of the list. I have removed it. It's not clear to me that the functionality is super useful since the default location for urls matches it. If it's needed, I would love some suggestions for how to deal with it. |
|
Is it possible to use Arch Linux switched from xz to zst because it decompresses significantly faster (~1300%), at the cost of slightly larger archives (~0.8%). https://archlinux.org/news/now-using-zstandard-instead-of-xz-for-package-compression/ Though, I don't know if rust is actually archived with zst. Shame. |
|
Are zstd archives supported by bazel? If so, the diff from here to there would be pretty small. There doesn't appear to be a programatic list of the archive types supported, so each much be added manually. Assuming that is done, at the very least you can supply the sha256s manually to the register_rust_toolchains function. Having said that, tar.zstd are not available from the rust distribution that I could find. Nothing would stop you from downloading an xz file and converting it to use zstd compression and distributing them from a web site. |
Yep! :) The list of supported archives can be found with the http_archive documentation.
That is a shame, though I thought this might be the case. Thank you for considering my suggestion. |
|
Find the files for download and it would be easy to add them to the fold. BTW, those archive formats are buried in many comments around bazel. It would be truly awesome if they were available programatically for extensions. If you wanted to create an issue to ask for that on the main bazel repo, feel free to tag me. Also, if you can find the location of the .tar.zst files. The additional diff on top of this one would be pretty small. |
I have explicitly added keys for the files with the extension explicitly on the file key. The point of this is to (in a future change) allow tar.xz archives to be used instead of tar.gz archives. This change keeps compatiblity with the old keys that assume the tar.gz archives. I don't know if this is something that we should drop in the future or just not worry about.
XZ archives are about half the size of the gz equivalents. This improves the cold repo bootstrap case a fair amount.
|
Cross posting from 841fc6f#commitcomment-72310891 to try and consolidate info on the PR. cc @krasimirgg @wt |
The commit 841fc6f for #1284 broke the ability to use rust_register_toolchains with a nightly channel, see 841fc6f#commitcomment-72310891. I tracked the issue down to this line which was updated there. While using a dated nightly, these variables look like: * tool_path: `rust-nightly-x86_64-unknown-linux-gnu` * tool_suburl: `2022-04-06/rust-nightly-x86_64-unknown-linux-gnu` Using tool_suburl causes the extract to fail like in the comment above: `Prefix "2022-04-06/rust-nightly-x86_64-unknown-linux-gnu/rustc" was given, but not found in the archive. Here are possible prefixes for this archive: "rust-nightly-x86_64-unknown-linux-gnu".` Switching back to the old behaviour fixes the glitch. I'm not sure if this breaks novel use cases introduced by 841fc6f; @wt, could you please take a look. On a sidenote, this is an instance where having some some build bot CI that uses nightly would have noticed this.
XZ archives are about half the size of the gz equivalents. This
improves the cold repo bootstrap case a fair amount.
Fixes: #871