Figure out combined Rust+Cargo installation

[brson]

On Unix, the rustup.sh script will install both Rust+Cargo (though that may not end up the official installation mechanism). On Windows, acquiring Cargo is not so convenient.

The obvious thing to do on Windows is to put them both in the same installer, but since Cargo and Rust live in different repos (and should continue to do so for the foreseeable future imo) creating that installer will require some work.

Nominating.

[pnkfelix]

to properly categorize this bug, we need to decide whether cargo is a 1.0-blocking feature.

deferring for a later triage with @alexcrichton present (and hopefully then we can decide about categorizing cargo)

[steveklabnik]

Given that it's the foundation of the new Guide, I would argue that it is. (Or that the guide then needs changed)

[wycats]

I don't see any reason not to consider it a blocker. Cargo largely works already, and the remaining work item, the registry, is well under way.

[pnkfelix]

P-high, 1.0 milestone.

[vadimcn]

So... can we add it as a submodule?

[brson]

@vadimcn My inclination right now is to create an entirely new repository for the purposes of packaging Rust, moving out much of existing dist logic. @alexcrichton likes the idea of having both the Rust and Cargo repos just produce a simple tarball, then having another project that takes them and combines them into platform-specific installers.

[brson]

Here's my current plan:

general setup

• Rename the Rust package from 'rust' to 'rustc' to indicate that
  it is only the compiler, and not the entire 'platform', make room
  for calling the combined installer 'rust'.
• Create a rust-combined-installer repo to house a bunch of scripts
  for building the combined installers.
• The build.py script does platform detection, then drives a bunch
  of other scripts.
• get-component-tarballs.py syncs the Rust git repo, does a
  dist-build of the appropriate branch, downloads the Cargo nightly,
  to get the Rust and Cargo binary tarballs.
• combine-from-tarballs.py extracts both installers and overlays
  them on top of each other in a single directory, using the root
  files (README, LICENSE, etc.) from Rust's installer, not Cargo's.
  We might do more here in the future to concatenate licenses.

windows installer

• build-windows-installer.py create a new Windows .exe installer
  with Inno Setup.
• Drop the .exe installer from the Rust repo. Unneeded maintenance
  burden.

unix installer

• Create a rust-installer repo to house install.sh so that Rust,
  Cargo and the combined installer all use the same base script.
  Create another script, make-installer.sh that generates the
  customized install.sh script and builds the tarball from an
  on-disk template. Update Rust and Cargo to use this.
• Change install.sh to support multiple manifests so that the combined
  installer can use both. This allows the Rust, Cargo, and
  combined installers to interoperate (e.g. the Cargo uninstaller
  will work on a combined install).
• Have build-unix-installer.py do the dirty work of driving
  combine-from-tarballs.py and make-installer.sh.

potential improvements

• Change the Rust manifest to lib/rustlib/rust-manifest.in and the
  cargo to lib/rustlib/cargo-manifest.in for consistency? Paves the
  way for more Rust 'components' (which there may never be since we
  have a package manager - once cargo is installed you use it). I
  would rather have cargo's manifest in the rustlib dir though. This
  would require adding compatibility code to the uninstaller to handle
  upgrades.

OS X package

• Similar to the windows installer

automation

• To start we'll add completely new nightly/beta/stable dist builders
  that run in parallel to the old, since we're going to have to roll
  this out in stages.

@alexcrichton What you think?

[alexcrichton]

Sounds great!

• Is there any particular reason get-component-tarballs.py builds rust from source, but not cargo?
• Can you remind me where the source tarballs play into all this? Are they just generated by the nightly builders? I don't think we should ship a combined source tarball, but we should make sure the source tarballs are next to each other. Note that cargo does not currently have a source tarball (not that it would be terribly useful anyway).
• What are the prerequisites to build-windows-installer.py and the OSX pkg builder?
• The exe/pkg installers would be only combined, right? (that's what I would expect).
• Cargo has a hefty amount of licenses, so we should make sure that they're displayed prominently (especially the GPLv2 one). They're installed into $prefix/share/doc/cargo, so they're somewhere, but they should probably be at the top-level of the tarball as well?

[brson]

Let me clarify that at the end of this the current Rust dist builders will not exist and the combined dist builder will create and upload both the Rust and combined artifacts (and also Cargo - see first point).

• The reason it builds rust from source is so we don't have to worry about build dependencies in the automation (e.g. buildbot doesn't have to do the rust build then trigger the combined build). The reason it doesn't do the cargo build itself is because cargo only has nightlies currently, so the combined builders for all release channels are going to be building the same cargo nightlies. This reason is pretty weak though and we could make the beta/stable channels just not upload the cargo artifacts. I'll make that change, but that means that we're going to have to also download the current Rust nightly to build Cargo (we can't count on Cargo building with all Rust release channels).
• The source tarballs will be produced and uploaded by the combined builder as part of the Rust dist build on Linux, but aren't used beyond that (there's no combined source tarball).
• The prereqs for the windows and pkg combined build are the same as for Unix - Run the Rust/Cargo build to produce the Unix installer tarball, which is then re-extracted to turn into the Windows installer. The on-disk layout is the same for all installers (with the slight exception that on Windows we dump everything in bin, even in the Unix-style installer).
• exe/pkg are only combined and support is dropped from the Rust repo
• I'll put some more thought into how to combine the licenses at the top level.

[alexcrichton]

It feels somewhat odd to me that the combined builder would build rust from source as opposed to downloading the artifact as produced by someone else. I thought we were going to handle the scheduling with #14431 (comment) and just run this script every hour or so, and 23/24 runs a day would do nothing.

[alexcrichton]

That being said, I'm also fine if it builds rust from source

[brson]

@alexcrichton my intention was to run a periodic cron job to promote the artifacts from the staging directory to the distribution directory on s3. My concern with doing the same thing with buildbot was that something has to trigger buildbot to do the build based on the status of the dist directory.

We could have that cron job do multiple things, both promote from staging to dist, and kick off the combined build, though I kind of would prefer not to do that, keeping the bits of automation separate. If we do have that script start the combined build then we need some way to trigger buildbot, which i don't know how to do offhand, but is surely possible.

If we really wanted to we could probably do this and #14431 (comment) not as a cron job but as part of the buildbot script, running in response to successful dist builds, though of course I'm loath to pack more functionality into our script.

I'm somewhat worried about the possibility of race conditions causing commit mismatches across the variety of platforms we support, particularly with cargo. I've read mention of locks in buildbot, so we may be able to just throw in a big lock that protects the combined builds from the individual builds.

[brson]

@alexcrichton we don't particularly need to do the staging->dist commit-id tracking on s3, but can do it locally on the build master. we don't even need to upload to a staging directory, everything can be retained locally until the final build is complete. Not sure if that's better or worse: with the uploaded staging artifacts at least you have the option of downloading them before the full set is built.

[alexcrichton]

Tracking things locally instead of on S3 seems fine, certainly easier for the time being. We may have more freedom of directory structure there in order to prevent things from going awry as well.

I'm basically fine with whatever's simplest right now, we could always time things to weed out most race conditions with the possibility of making it "rock solid" in the future.

[brson]

OK, here's the plan as I see it currently, though details are still vague to me.

I am still maintaining the premise that the cargo and rust builds should remain entirely separate. The reasons for this are pretty squishy, I admit: the current independent development model is working well; both builds are very complex and self-hosting, so combining them doubly-so; cargo and rust won't necessarily always support the same platforms, so it may continue to be desirable to produce rustc installers independently.

• First, factor out install.sh, used by both cargo and rust, into a new project rust-installer, which generates installers in the form of tarballs. This is almost done.
• Upgrade rust-installer to support some new features:
  • Installer versioning: the installer drops a file rustlib/rust-installer-version that just contains a number, for versioning the installer for future backwards-incompatible changes.
  • Components: the rustlib/components file just contains a list of component names. Each of these has a corresponding manifest called rustlib/manifest-${component}.
  • Installing / removing entire directories at once: to fix Consider installing HTML documentation #14916. Under the current model, we cp or rm every file individually, but that is very slow with our massive documentation set. Under this scheme /lib/share/doc/rust would be copied and deleted with a single command. Requires a new manifest format, so I want to just do this at the same time as upgrading the installer version.
  • Upgrading from the old, non-versioned format.
• Add a new script to rust-installer than unpacks two installers for different components and recombines them into a single installer.

At that point we can create a rust-combined-installer project that actually merges the two installers, and builds the various other installers, like .msi and .pkg.

• Rename the rust artifacts to rustc-${version}, to make room for a new rust package.
• Do everything else.

[brson]

Need to start thinking about a future in which we support multiple simultaneous installs.

[brson]

I've made a lot of progress on this and rust-installer can merge two installers now. I still need to use it to create the combined installer project and then upgrade rust and cargo to use it, adjust automation, move the os x and win installers.

[brson]

Done.