[WIP] Rewrite the rust-installer in Rust

[cuviper]

For rust-lang/rust#41569, this branch rewrites almost all of rust-installer in Rust. The existing shell scripts are cut down to shims invoking cargo run, but otherwise they still accept the exact same options as before. Only the actual install script and the testing script are still written in shell, and all of those tests do pass as-is.

It's not clear to me if this will be wanted as an evolution of rust-installer, or dropped in directly to the Rust sources, but having a PR here should let people start reviewing it either way. My next step is to see if this will serve as a drop-in replacement in Rust, Cargo, etc., and then we can see about porting rustbuild dist.rs to use this directly as a library.

[alexcrichton]

Awesome, thanks so much for this @cuviper! Everything looks quite solid to me. I really like how we've still got a test suite in this repository, and I like the strategy here as well. I think we'll want to keep the test suite no matter what, and it's not necessarily critical to rewrite it just yet, I was mostly interested in the tarball-creation logic which just tends to be easier if it's cross-platform.

Note that we probably don't have to worry about updating the Cargo repository's rust-installer submodule, I've been meaning to phase that out now that it's all included in rust-lang/rust. In that sense I think the next steps would be:

• Land this
• Update the submodule in rust-lang/rust
• Add a src/tools/foo which depends on the library support in the rust-installer submodule
• Update src/bootstrap/dist.rs to invoke the src/tools/foo command instead of the shell scripts in rust-intaller.sh

Does that sounds reasonable? Or did you have different plans in mind perhaps?

[cuviper]

Note that we probably don't have to worry about updating the Cargo repository's rust-installer submodule, I've been meaning to phase that out now that it's all included in rust-lang/rust.

On the distro side, we still build cargo from its own source rpm, so I would at least hope that "make install" continues to work -- but that definitely doesn't need to use a full rust-installer. This is a tangent though, so if there's more to discuss here, let's spin off a cargo issue.

• Add a src/tools/foo which depends on the library support in the rust-installer submodule
• Update src/bootstrap/dist.rs to invoke the src/tools/foo command instead of the shell scripts in rust-intaller.sh

I had thought to just make dist.rs call the library directly. Is there a benefit to having it go through an external tool?

And FWIW, a basic tool already exists in the new src/main.rs here, which is what the script shims are using. How would you want a new tool to look?

[alexcrichton]

I had thought to just make dist.rs call the library directly. Is there a benefit to having it go through an external tool?

Ah my only thinking here is that this'd mean the libs would be a dependency of rustbuild itself, and rustbuild is the gate into compiling everything in the rust-lang/rust repo. If we add all the deps (semi-weighty here) it may make rust compilations more difficult to get working on all platforms all the time.

Basically I'd imagine that less-than-100% (maybe not even half?) of builds in rust-lang/rust don't actually end up using dist logic at all, so requiring those libs to build, for example, flate2/xz2/etc may just be a bit too onerous. By deferring the logic to a separate build we can at least cut out those deps from builds that don't end up invoking dist logic.

I think, however, it's always possible to add more deps to rustbuild, it'll just be a long road uphill unfortunately.

And FWIW, a basic tool already exists in the new src/main.rs here, which is what the script shims are using. How would you want a new tool to look?

Nah it seems fine to me! Either a custom tool or the one here should work great.

[cuviper]

Ah my only thinking here is that this'd mean the libs would be a dependency of rustbuild itself, and rustbuild is the gate into compiling everything in the rust-lang/rust repo. If we add all the deps (semi-weighty here) it may make rust compilations more difficult to get working on all platforms all the time.

Ah, that makes total sense. External tool it is!

[alexcrichton]

r? @brson

(would like to review before merging)

[brson]

lgtm

The remove_dir_all module should probably be replaced with the remove_dir_all crate.

[cuviper]

I saw the remove_dir_all crate exists, but with so few downloads I didn't feel comfortable using it. It really seems like something libstd should solve itself...