rustbuild: Add support for compiling Cargo #39917

alexcrichton · 2017-02-17T21:56:29Z

This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.

The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.

The process for release Cargo will now look like:

The rust-lang/rust repository has a Cargo submodule which is used to build a
Cargo to pair with the rust-lang/rust release
Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
master branch
When branching beta we'll create a new branch of Cargo (as we do today), and
the first commit to the beta branch will be to update the Cargo submodule to
this exact revision.
When branching stable, we'll ensure that the Cargo submodule is updated and
then make a stable release.

Backports to Cargo will look like:

Send a PR to cargo's master branch
Send a PR to cargo's release branch (e.g. rust-1.16.0)
Send a PR to rust-lang/rust's beta branch updating the submodule
Eventually send a PR to rust-lang/rust's master branch updating the submodule

For reference, the process to add a new component to the rust-lang/rust release
would look like:

Add $foo as a submodule in src/tools
Add a tool-$foo step which compiles $foo with the specified compiler,
likely mirroring what Cargo does.
Add a dist-$foo step which uses src/tools/$foo and the tool-$foo output
to create a rust-installer package for $foo likely mirroring what Cargo
does.
Update the dist-extended step with a new dependency on dist-$foo
Update src/tools/build-manifest for the new component.

rust-highfive · 2017-02-17T21:56:41Z

r? @brson

(rust_highfive has picked a reviewer for you, use r? to override)

bors · 2017-02-20T05:57:26Z

☔ The latest upstream changes (presumably #39966) made this pull request unmergeable. Please resolve the merge conflicts.

brson · 2017-02-22T17:52:29Z

src/tools/cargotest/Cargo.toml

@@ -1,5 +1,5 @@
 [package]
-name = "cargotest"
+name = "cargotest2"


Why is this renamed?

Oh this conflicted with the cargotest package in Cargo itself, so I figured it'd be easiest to rename this here

brson · 2017-02-22T17:54:04Z

@bors r+

All the openssl logic in bootstrap is 😢

bors · 2017-02-22T17:54:05Z

📌 Commit 3120181 has been approved by brson

alexcrichton · 2017-02-22T18:12:50Z

@bors: r=brson

bors · 2017-02-22T18:12:51Z

📌 Commit ed2f0d6 has been approved by brson

alexcrichton · 2017-02-22T18:14:36Z

FWIW rust-openssl may get support soon for building OpenSSL given a source tarball, so we could at least do away with all the grungy build logic.

sfackler · 2017-02-22T18:31:36Z

src/bootstrap/native.rs

+    };
+
+    configure.arg(format!("--prefix={}", dst.display()));
+    configure.arg("no-dso");


Might be worth adding no-ssl2 no-ssl3 no-comp additionally just to be safe.

alexcrichton · 2017-02-22T18:37:31Z

@bors: r=brson

bors · 2017-02-22T18:37:32Z

📌 Commit b8fc75f has been approved by brson

alexcrichton · 2017-02-23T17:30:12Z

@bors: r=brson

bors · 2017-02-23T17:30:14Z

📌 Commit 28ea763 has been approved by brson

bors · 2017-02-24T05:09:52Z

☔ The latest upstream changes (presumably #39851) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2017-02-25T15:10:17Z

🔒 Merge conflict

alexcrichton · 2017-02-25T18:24:40Z

@bors: r=brson

bors · 2017-02-25T18:24:40Z

📌 Commit be5cb13 has been approved by brson

bors · 2017-02-25T20:40:16Z

⌛ Testing commit be5cb13 with merge 9d6ed79...

bors · 2017-02-25T21:32:07Z

💔 Test failed - status-travis

alexcrichton · 2017-02-25T21:45:51Z

@bors: r=brson

This commit adds support to rustbuild for compiling Cargo as part of the release process. Previously rustbuild would simply download a Cargo snapshot and repackage it. With this change we should be able to turn off artifacts from the rust-lang/cargo repository and purely rely on the artifacts Cargo produces here. The infrastructure added here is intended to be extensible to other components, such as the RLS. It won't exactly be a one-line addition, but the addition of Cargo didn't require too much hooplah anyway. The process for release Cargo will now look like: * The rust-lang/rust repository has a Cargo submodule which is used to build a Cargo to pair with the rust-lang/rust release * Periodically we'll update the cargo submodule as necessary on rust-lang/rust's master branch * When branching beta we'll create a new branch of Cargo (as we do today), and the first commit to the beta branch will be to update the Cargo submodule to this exact revision. * When branching stable, we'll ensure that the Cargo submodule is updated and then make a stable release. Backports to Cargo will look like: * Send a PR to cargo's master branch * Send a PR to cargo's release branch (e.g. rust-1.16.0) * Send a PR to rust-lang/rust's beta branch updating the submodule * Eventually send a PR to rust-lang/rust's master branch updating the submodule For reference, the process to add a new component to the rust-lang/rust release would look like: * Add `$foo` as a submodule in `src/tools` * Add a `tool-$foo` step which compiles `$foo` with the specified compiler, likely mirroring what Cargo does. * Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output to create a rust-installer package for `$foo` likely mirroring what Cargo does. * Update the `dist-extended` step with a new dependency on `dist-$foo` * Update `src/tools/build-manifest` for the new component.

alexcrichton · 2017-03-03T15:30:04Z

@bors: r=brson

Why thank you @bors, I did indeed not intend to check in 100MB build trees into the repo

bors · 2017-03-03T15:30:05Z

📌 Commit 44a01b8 has been approved by brson

bors · 2017-03-03T16:42:14Z

⌛ Testing commit 44a01b8 with merge 7d2bdb0...

bors · 2017-03-03T17:26:07Z

💔 Test failed - status-travis

alexcrichton · 2017-03-03T17:32:25Z

@bors: retry

sccache failure Spurious failures in sccache #40240

bors · 2017-03-03T19:48:43Z

⌛ Testing commit 44a01b8 with merge 4e0cfbd...

bors · 2017-03-03T20:26:09Z

💔 Test failed - status-travis

alexcrichton · 2017-03-03T20:34:52Z

@bors: retry * sccache failure #40240

…

On Fri, Mar 3, 2017 at 2:26 PM, bors ***@***.***> wrote: 💔 Test failed - status-travis <https://travis-ci.org/rust-lang/rust/builds/207483714> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#39917 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD95H4aP8toucGp1Kc1sDhv-Q26LriKks5riHdjgaJpZM4MEzbk> .

bors · 2017-03-03T23:26:30Z

⌛ Testing commit 44a01b8 with merge 5208090...

rustbuild: Add support for compiling Cargo This commit adds support to rustbuild for compiling Cargo as part of the release process. Previously rustbuild would simply download a Cargo snapshot and repackage it. With this change we should be able to turn off artifacts from the rust-lang/cargo repository and purely rely on the artifacts Cargo produces here. The infrastructure added here is intended to be extensible to other components, such as the RLS. It won't exactly be a one-line addition, but the addition of Cargo didn't require too much hooplah anyway. The process for release Cargo will now look like: * The rust-lang/rust repository has a Cargo submodule which is used to build a Cargo to pair with the rust-lang/rust release * Periodically we'll update the cargo submodule as necessary on rust-lang/rust's master branch * When branching beta we'll create a new branch of Cargo (as we do today), and the first commit to the beta branch will be to update the Cargo submodule to this exact revision. * When branching stable, we'll ensure that the Cargo submodule is updated and then make a stable release. Backports to Cargo will look like: * Send a PR to cargo's master branch * Send a PR to cargo's release branch (e.g. rust-1.16.0) * Send a PR to rust-lang/rust's beta branch updating the submodule * Eventually send a PR to rust-lang/rust's master branch updating the submodule For reference, the process to add a new component to the rust-lang/rust release would look like: * Add `$foo` as a submodule in `src/tools` * Add a `tool-$foo` step which compiles `$foo` with the specified compiler, likely mirroring what Cargo does. * Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output to create a rust-installer package for `$foo` likely mirroring what Cargo does. * Update the `dist-extended` step with a new dependency on `dist-$foo` * Update `src/tools/build-manifest` for the new component.

bors · 2017-03-04T01:47:40Z

☀️ Test successful - status-appveyor, status-travis
Approved by: brson
Pushing 5208090 to master...

shepmaster · 2017-03-04T03:53:21Z

@alexcrichton after pulling this change, I can no longer build:

$ rm -rf build-release/
$ mkdir build-release
$ cd build-release/
$ ../configure
$ make
downloading https://static.rust-lang.org/dist/2017-02-01/rust-std-beta-x86_64-apple-darwin.tar.gz
######################################################################## 100.0%
extracting /Users/shep/Projects/rust/build-release/build/cache/2017-02-01/rust-std-beta-x86_64-apple-darwin.tar.gz
downloading https://static.rust-lang.org/dist/2017-02-01/rustc-beta-x86_64-apple-darwin.tar.gz
######################################################################## 100.0%
extracting /Users/shep/Projects/rust/build-release/build/cache/2017-02-01/rustc-beta-x86_64-apple-darwin.tar.gz
downloading https://s3.amazonaws.com/rust-lang-ci/cargo-builds/bfee18f73287687c543bda8c35e4e33808792715/cargo-nightly-x86_64-apple-darwin.tar.gz
######################################################################## 100.0%
extracting /Users/shep/Projects/rust/build-release/build/cache/bfee18f73287687c543bda8c35e4e33808792715/cargo-nightly-x86_64-apple-darwin.tar.gz
error: failed to read `/Users/shep/Projects/rust/src/tools/cargo/Cargo.toml`

Caused by:
  No such file or directory (os error 2)
make: *** [all] Error 101

Is there another step I'm unaware of?

alexcrichton · 2017-03-04T04:38:07Z

Ah yup that'd do it.

Cargo's a submodule. We'll have to move submodule updates sooner in the process or warn about it. You can fix this with:

git submodule update --init

Mark-Simulacrum · 2017-03-04T15:30:23Z

Should that perhaps be posted in a more visible location (e.g., internals)? I suspect pretty much all developers working on rustc will run into it.

shepmaster · 2017-03-04T15:36:58Z

all developers working on rustc

I think everyone is playing that new Zelda game today 😉

More seriously, I just assumed that it would be fixed soon enough that not many people would run into it. I'm also a bit surprised that the build machines didn't have any issues. I guess there's differences between what a human does and the autobuilds?

alexcrichton · 2017-03-06T15:50:07Z

I've sent a PR to fix the issue @shepmaster ran into.

Sorry for the breakage!

Also yeah the auto builds explicitly update submodules for... reasons. So that's why they didn't run into this.

rust-highfive assigned brson Feb 17, 2017

alexcrichton force-pushed the build-cargo branch from 684305c to becb091 Compare February 17, 2017 22:05

alexcrichton force-pushed the build-cargo branch from becb091 to 3120181 Compare February 21, 2017 16:44

brson reviewed Feb 22, 2017

View reviewed changes

alexcrichton force-pushed the build-cargo branch from 3120181 to ed2f0d6 Compare February 22, 2017 18:12

sfackler reviewed Feb 22, 2017

View reviewed changes

alexcrichton force-pushed the build-cargo branch from ed2f0d6 to b8fc75f Compare February 22, 2017 18:37

steveklabnik mentioned this pull request Feb 22, 2017

Tracking issue for RFC 1828: Rust Bookshelf #39588

Closed

10 tasks

alexcrichton force-pushed the build-cargo branch from b8fc75f to 28ea763 Compare February 23, 2017 17:30

alexcrichton force-pushed the build-cargo branch from 28ea763 to be5cb13 Compare February 25, 2017 18:24

alexcrichton force-pushed the build-cargo branch from be5cb13 to 985be4f Compare February 25, 2017 21:45

alexcrichton force-pushed the build-cargo branch from 77fb172 to 44a01b8 Compare March 3, 2017 15:29

bors merged commit 44a01b8 into rust-lang:master Mar 4, 2017

This was referenced Mar 4, 2017

Move the reference into a submodule #40213

Merged

Extract nomicon to its own repo #40222

Merged

rustbuild: A few tweaks #40236

Merged

Rollup of 6 pull requests #40249

Closed

Restore creating the channel-rust-$channel-date.txt files #40225

Merged

alexcrichton deleted the build-cargo branch March 4, 2017 04:38

This was referenced Mar 5, 2017

error: failed to read src/tools/cargo/Cargo.toml #40284

Closed

Building Rust from source fails #40291

Closed

alexcrichton mentioned this pull request Mar 24, 2017

Overall cycle time has regressed ~25% from 2hrs to 2.5hrs #40804

Closed

rustbuild: Add support for compiling Cargo #39917

rustbuild: Add support for compiling Cargo #39917

Conversation

alexcrichton commented Feb 17, 2017

rust-highfive commented Feb 17, 2017

bors commented Feb 20, 2017

brson Feb 22, 2017

Choose a reason for hiding this comment

alexcrichton Feb 22, 2017

Choose a reason for hiding this comment

brson commented Feb 22, 2017

bors commented Feb 22, 2017

alexcrichton commented Feb 22, 2017

bors commented Feb 22, 2017

alexcrichton commented Feb 22, 2017

sfackler Feb 22, 2017

Choose a reason for hiding this comment

alexcrichton Feb 22, 2017

Choose a reason for hiding this comment

alexcrichton commented Feb 22, 2017

bors commented Feb 22, 2017

alexcrichton commented Feb 23, 2017

bors commented Feb 23, 2017

bors commented Feb 24, 2017

bors commented Feb 25, 2017

alexcrichton commented Feb 25, 2017

bors commented Feb 25, 2017

bors commented Feb 25, 2017

bors commented Feb 25, 2017

alexcrichton commented Feb 25, 2017

alexcrichton commented Mar 3, 2017

bors commented Mar 3, 2017

bors commented Mar 3, 2017

bors commented Mar 3, 2017

alexcrichton commented Mar 3, 2017

bors commented Mar 3, 2017

bors commented Mar 3, 2017

alexcrichton commented Mar 3, 2017 via email

bors commented Mar 3, 2017

bors commented Mar 4, 2017

shepmaster commented Mar 4, 2017

alexcrichton commented Mar 4, 2017

Mark-Simulacrum commented Mar 4, 2017

shepmaster commented Mar 4, 2017

alexcrichton commented Mar 6, 2017