Download components in parallel while installing a toolchain

Problem you are trying to solve

Not exactly a problem, rather a feature. It could make downloading a toolchain much faster, really helpful while bisecting Rustc.

Solution you'd like

I would want Rustup to download components in parallel while downloading a toolchain.

Notes

I'd love to contribute!

[kinnison]

I'd love to see a solution for this implemented, we might need to switch to a better progress bar implementation to support multiple parallel downloads though.

[fisherdarling]

Hi! I'd like to work on this :) It's been about 5 years 3 years since I last looked at the rustup repo, but I think I can implement this.

I'll definitely be asking many questions :D

[fisherdarling]

Ok, so this is rather complex. The primary problem is the &dyn Fn(Notification<'_>) callback threaded through the codebase and the fact that Notification has a lifetime.

I'm going to see if I can find a good place to split the dist::Notifications into something that's more Send/Sync friendly.

[fisherdarling]

@gimbles Could you give me a command that you run that you hope parallel downloads would speed up? I have a fork that downloads in parallel and I'd like to get some example commands to benchmark.

Could you give me a command that you run that you hope parallel downloads would speed up? I have a fork that downloads in parallel and I'd like to get some example commands to benchmark.

@fisherdarling Downloading components in parallel would really speed up bisecting Rustc, take a peek at cargo-bisect-rustc and their examples. cargo-msrv will also benefit from this. Really, just install a bunch of toolchains. =)

[fisherdarling]

Sounds good! I'll check that out :) I'm actually having problems showing that it's actually changing anything since I'm already saturating my download bandwidth, adding parallelism to the mix doesn't change anything for me.

However, parallelizing installs would be a huge speed up.

[rbtcollins]

We're already highly parallel in install on everything except decompression