bootstrap: when we detect travis-ci or appveyor, use 16 codegen-units for stage0 #48843

matthiaskrgr · 2018-03-08T15:26:22Z

No description provided.

rust-highfive · 2018-03-08T15:26:33Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @alexcrichton (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

ishitatsuyuki · 2018-03-08T15:28:11Z

3 is too low, use 8 or 16.

matthiaskrgr · 2018-03-08T15:31:43Z

Are you sure about that?
It seems travis and appveyor VMs only have 2 cores.
https://docs.travis-ci.com/user/reference/overview/#Virtualisation-Environment-vs-Operating-System
https://www.appveyor.com/docs/build-environment/#build-vm-configurations

ishitatsuyuki · 2018-03-08T15:34:41Z

We have 4 cores in Travis, and also that having too few codegen units may caused unbalanced load.

The parallel codegen follows jobserver, which means the number of thread will not exceed the core count.

matthiaskrgr · 2018-03-08T15:37:02Z

Alright, bumped to 16.

alexcrichton · 2018-03-08T15:53:31Z

Thanks for the PR! I think we already enable 16 cgus + thinlto on appveyor/travis, though, so are we sure that this has an effect?

matthiaskrgr · 2018-03-08T15:57:08Z

@alexcrichton see the logs @ https://api.travis-ci.org/v3/job/350827435/log.txt
it looks like CI runs in single-cgu mode
// cc #48841

[00:06:14] �[m�[m�[32m�[1m   Compiling�[m rls-data v0.15.0
[00:06:15] �[m�[m�[32m�[1m   Compiling�[m rustc_data_structures v0.0.0 (file:///checkout/src/librustc_data_structures)
[00:06:15] �[m�[m�[32m�[1m   Compiling�[m flate2 v1.0.1
[00:06:18] �[m�[m�[32m�[1m   Compiling�[m syntax_pos v0.0.0 (file:///checkout/src/libsyntax_pos)
[00:06:18] �[m�[m�[32m�[1m   Compiling�[m backtrace v0.3.5
[00:06:23] �[m�[m�[32m�[1m   Compiling�[m rustc_errors v0.0.0 (file:///checkout/src/librustc_errors)
top - 13:52:30 up 8 min,  1 user,  load average: 2.31, 1.49, 0.71 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 34.6 us,  2.8 sy,  0.0 ni, 59.9 id,  2.7 wa,  0.0 hi,  0.1 si,  0.0 st KiB Mem:  15400500 total,  5213396 used, 10187104 free,   110116 buffers 
top - 13:53:00 up 8 min,  1 user,  load average: 1.79, 1.44, 0.72 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 34.0 us,  2.6 sy,  0.0 ni, 60.7 id,  2.6 wa,  0.0 hi,  0.1 si,  0.0 st KiB Mem:  15400500 total,  5694704 used,  9705796 free,   110172 buffers 
top - 13:53:30 up 9 min,  1 user,  load average: 1.48, 1.40, 0.73 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 33.5 us,  2.5 sy,  0.0 ni, 61.5 id,  2.4 wa,  0.0 hi,  0.1 si,  0.0 st KiB Mem:  15400500 total,  5560748 used,  9839752 free,   110184 buffers 
top - 13:54:00 up 9 min,  1 user,  load average: 1.29, 1.36, 0.74 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 33.1 us,  2.4 sy,  0.0 ni, 62.2 id,  2.3 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  5627892 used,  9772608 free,   110192 buffers 
top - 13:54:31 up 10 min,  1 user,  load average: 1.17, 1.32, 0.75 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 32.7 us,  2.3 sy,  0.0 ni, 62.8 id,  2.2 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  5651796 used,  9748704 free,   110208 buffers 
[00:08:49] �[m�[m�[32m�[1m   Compiling�[m rustc_const_math v0.0.0 (file:///checkout/src/librustc_const_math)
[00:08:49] �[m�[m�[32m�[1m   Compiling�[m proc_macro v0.0.0 (file:///checkout/src/libproc_macro)
[00:08:56] �[m�[m�[32m�[1m   Compiling�[m syntax_ext v0.0.0 (file:///checkout/src/libsyntax_ext)
top - 13:55:01 up 10 min,  1 user,  load average: 1.33, 1.34, 0.77 Tasks: 123 total,   1 running, 122 sleeping,   0 stopped,   0 zombie %Cpu(s): 32.9 us,  2.2 sy,  0.0 ni, 62.7 id,  2.1 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  5425852 used,  9974648 free,   110232 buffers 
top - 13:55:31 up 11 min,  1 user,  load average: 1.60, 1.41, 0.81 Tasks: 123 total,   1 running, 122 sleeping,   0 stopped,   0 zombie %Cpu(s): 33.7 us,  2.1 sy,  0.0 ni, 62.2 id,  2.0 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  5948860 used,  9451640 free,   110288 buffers 
top - 13:56:01 up 11 min,  1 user,  load average: 1.36, 1.37, 0.82 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 33.3 us,  2.0 sy,  0.0 ni, 62.7 id,  1.9 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6184412 used,  9216088 free,   110308 buffers 
top - 13:56:31 up 12 min,  1 user,  load average: 1.22, 1.33, 0.83 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 33.0 us,  2.0 sy,  0.0 ni, 63.2 id,  1.8 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6825952 used,  8574548 free,   110324 buffers 
top - 13:57:01 up 12 min,  1 user,  load average: 1.13, 1.30, 0.83 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 32.7 us,  1.9 sy,  0.0 ni, 63.6 id,  1.8 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6435712 used,  8964788 free,   110336 buffers 
top - 13:57:31 up 13 min,  1 user,  load average: 1.08, 1.27, 0.84 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 32.4 us,  1.8 sy,  0.0 ni, 64.0 id,  1.7 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6561764 used,  8838736 free,   110336 buffers 
top - 13:58:02 up 13 min,  1 user,  load average: 1.05, 1.24, 0.84 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 32.1 us,  1.8 sy,  0.0 ni, 64.4 id,  1.6 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6641564 used,  8758936 free,   110348 buffers 
top - 13:58:32 up 14 min,  1 user,  load average: 1.03, 1.22, 0.85 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 31.9 us,  1.7 sy,  0.0 ni, 64.8 id,  1.6 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6674308 used,  8726192 free,   110356 buffers 
top - 13:59:02 up 14 min,  1 user,  load average: 1.02, 1.19, 0.86 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 31.6 us,  1.7 sy,  0.0 ni, 65.1 id,  1.5 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6728572 used,  8671928 free,   110372 buffers 
top - 13:59:32 up 15 min,  1 user,  load average: 1.01, 1.17, 0.86 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 31.4 us,  1.6 sy,  0.0 ni, 65.5 id,  1.5 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6787644 used,  8612856 free,   110380 buffers 
top - 14:00:02 up 15 min,  1 user,  load average: 1.00, 1.16, 0.87 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 31.2 us,  1.6 sy,  0.0 ni, 65.8 id,  1.4 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6753376 used,  8647124 free,   110388 buffers 
top - 14:00:32 up 16 min,  1 user,  load average: 1.00, 1.14, 0.87 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 31.0 us,  1.5 sy,  0.0 ni, 66.0 id,  1.4 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6819500 used,  8581000 free,   110400 buffers 
top - 14:01:03 up 16 min,  1 user,  load average: 1.00, 1.12, 0.88 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 30.8 us,  1.5 sy,  0.0 ni, 66.3 id,  1.4 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6953028 used,  8447472 free,   110408 buffers 
top - 14:01:33 up 17 min,  1 user,  load average: 1.00, 1.11, 0.88 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 30.7 us,  1.4 sy,  0.0 ni, 66.6 id,  1.3 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6982384 used,  8418116 free,   110416 buffers 
top - 14:02:03 up 17 min,  1 user,  load average: 1.00, 1.10, 0.89 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 30.5 us,  1.4 sy,  0.0 ni, 66.8 id,  1.3 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6965660 used,  8434840 free,   110432 buffers 
top - 14:02:33 up 18 min,  1 user,  load average: 1.00, 1.09, 0.90 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 30.4 us,  1.3 sy,  0.0 ni, 67.0 id,  1.2 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6952240 used,  8448260 free,   110440 buffers 
top - 14:03:03 up 18 min,  1 user,  load average: 1.00, 1.08, 0.90 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 30.2 us,  1.3 sy,  0.0 ni, 67.2 id,  1.2 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6888564 used,  8511936 free,   110464 buffers 
top - 14:03:33 up 19 min,  1 user,  load average: 1.00, 1.07, 0.91 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 30.1 us,  1.3 sy,  0.0 ni, 67.4 id,  1.2 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  6966332 used,  8434168 free,   110472 buffers 
top - 14:04:04 up 19 min,  1 user,  load average: 1.00, 1.06, 0.91 Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie %Cpu(s): 30.0 us,  1.2 sy,  0.0 ni, 67.6 id,  1.1 wa,  0.0 hi,  0.0 si,  0.0 st KiB Mem:  15400500 total,  7084940 used,  8315560 free,   110480 buffers 
[00:18:15] �[m�[m�[32m�[1m   Compiling�[m rustc_resolve v0.0.0 (file:///checkout/src/librustc_resolve)
[00:18:15] �[m�[m�[32m�[1m   Compiling�[m rustc_mir v0.0.0 (file:///checkout/src/librustc_mir)

alexcrichton · 2018-03-08T16:05:12Z

Oh I think that's specifically related to the llvm-3.9 builder there because it's not the "Rust LLVM" which means that ThinLTO isn't available so we build with one cgu. I believe other builders should already be using multiple cgus by default (with ThinLTO).

Given that the llvm-3.9 builder is already relatively speedy (~1.5hrs) are we sure that this speeds up the builder as well? In theory more cgus means less optimizations, so this could run the risk of slowing down the run-pass test suite, for example.

matthiaskrgr · 2018-03-08T16:22:16Z

I think we should be able to see how much faster stage0 build got and now much slower stage1 build got once the CI job is done building.

alexcrichton · 2018-03-08T18:43:27Z

The build clocked in at 1h25m which is about the same for the other PR builds I believe

matthiaskrgr · 2018-03-08T19:04:23Z

I'm taking the build of petrochenkov@e5b86ea // https://travis-ci.org/rust-lang/rust/builds/350827835?utm_source=github_status&utm_medium=notification
as time reference, both PRs are based on c90f682
(could no longer find the PR travis build of c90f682 unfortunately)

logs of this pr: https://travis-ci.org/rust-lang/rust/builds/350884124?utm_source=github_status&utm_medium=notification

	this PR (stg0 cgus)	underscore PR
building bootstrap	46.18	49.73
?? downloads	0:01:34	0:01:29
Building stage0 std artifacts	47.78	56.42
Building stage0 tool tidy	49.75	57.38
Building stage0 test artifacts	10.32	16.20
Building stage0 compiler artifacts	778.0	1067.11
Building stage0 codegen artifacts	54.79	80.13
Building stage1 std artifacts	79.3	78.57
Building stage1 test artifacts	13.49	14.66
Building stage1 compiler artifacts	1016.41	1065.49
Building stage1 codegen artifacts	73.63	83.84
Building rustdoc for stage2	121.57	137.12
Building stage0 tool unstable-book-gen	52.1	61.67
Building stage0 tool rustbook	132.43	152.62
Documenting stage2 std	64.79	68.29
Documenting stage2 compiler	33.82	35.25
Building stage0 tool compiletest	17.15	24.26
Check compiletest suite=ui mode=ui	82.157	80.388
Check compiletest suite=run-pass mode=run-pass	390.778	403.240
Check compiletest suite=compile-fail mode=compile-fail	115.920	119.536
...

It's a bit strange that stage1 improved as well though

kennytm · 2018-03-08T19:09:17Z

(cc #45444.)

alexcrichton · 2018-03-08T21:22:48Z

Note that builds on Travis have very variable performance so any two runs of the exact same code can have pretty large variations in compile times. The difference here between stage0 compiler artifacts is likely significant (as expected), but all other timings are probably "lost in the noise" in terms of the differences.

That means that there's probably a small gain to get on the stage0 compiler here, but it at least doesn't significantly regress the stage1 compiler!

matthiaskrgr · 2018-03-08T21:32:36Z

Yeah, travis' performance is not very reliable :/

I could push merge of this PR (#48843) and the top debug statistics (#48841) into a third branch though and have a look to make sure that 1) stage0 uses several cgus and 2) stage1 doesn't

alexcrichton · 2018-03-08T22:04:24Z

Ah that's ok the major difference between stage0/stage1 with this pr (700 -> 100s) seems pretty clear to me that it'd be caused by switching from 16 cgus to one cgu (way less usage of the 4 cores we've got on Travis)

If you'd like I'd be ok landing this, but given that this only affects the one Travis builder with LLVM 3.9 I'd prefer to have a configuration option for rustbuild which is set by that one builder.

matthiaskrgr · 2018-03-08T22:17:33Z

Hmm, well, my original plans were to speed up the auto branch as well....

Mark-Simulacrum · 2018-03-08T22:34:46Z

Thanks for the PR! I think we already enable 16 cgus + thinlto on appveyor/travis, though, so are we sure that this has an effect?

Not on dist builders: https://github.com/rust-lang/rust/blob/master/src/ci/run.sh#L49

So this could have a fairly major effect there, though probably not too much.

alexcrichton · 2018-03-09T21:16:25Z

@Mark-Simulacrum ah yes indeed! I forgot we hadn't gotten around to turning that back off yet, I'll send a PR.

alexcrichton · 2018-03-09T21:23:34Z

@matthiaskrgr would you like to land this PR for the llvm-3.9 builder on Travis?

matthiaskrgr · 2018-03-09T21:27:21Z

Well, what are the options?
The llvm-3.9 is the builder only used for pull requests, right?
Whats the opinion on enabling several stage0 CGUs for all jobs on the auto branch?

Of course, speeding up the pull request builder a bit would already be a win so I'm not against that.

alexcrichton · 2018-03-09T21:30:51Z

Yeah llvm-3.9 is only used for PRs (but is also on full auto builds). I actually thought we already enabled multiple cgus for all jobs on the auto branch but turns out this was only true for the builders that run tests, not the dist builders. I sent a PR to clean up some handling in this area and switch the default for the dist builders back to multiple cgus.

bors · 2018-03-16T02:49:11Z

☔ The latest upstream changes (presumably #49051) made this pull request unmergeable. Please resolve the merge conflicts.

Mark-Simulacrum · 2018-03-16T13:19:24Z

Please don't format the code since that should be done all at once and in a separate PR.

… when building stage0.

pietroalbini · 2018-03-20T08:30:15Z

Ping from triage @alexcrichton! The author pushed new commits, could you review them?

alexcrichton · 2018-03-20T14:13:32Z

Ah yeah as mentioned previously now that the PR has landed I think that we're actually using multiple CGUs on all builds, including the llvm-3.9 build (where at runtime we detect ThinLTO isn't available).

In light of that I'm going to close this as I don't think it's necessary any more

rust-highfive assigned alexcrichton Mar 8, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 8, 2018

matthiaskrgr mentioned this pull request Mar 8, 2018

travis: print some statistics from "top". #48841

Closed

matthiaskrgr force-pushed the CI_stage0_cgus branch from c71c8b1 to 696ec5c Compare March 8, 2018 15:36

matthiaskrgr force-pushed the CI_stage0_cgus branch from 696ec5c to 55446e5 Compare March 8, 2018 15:53

matthiaskrgr changed the title ~~bootstrap: when we detect travis-ci or appveyor, use 3 codegen-units for stage0~~ bootstrap: when we detect travis-ci or appveyor, use 16 codegen-units for stage0 Mar 8, 2018

matthiaskrgr force-pushed the CI_stage0_cgus branch 2 times, most recently from c4d9081 to a595c40 Compare March 16, 2018 11:55

bootstrap: when we detect travis-ci or appveyor, use 16 codegen-units…

e87abdb

… when building stage0.

matthiaskrgr force-pushed the CI_stage0_cgus branch from ecb4dfb to e87abdb Compare March 16, 2018 13:36

alexcrichton closed this Mar 20, 2018

matthiaskrgr deleted the CI_stage0_cgus branch August 28, 2018 17:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bootstrap: when we detect travis-ci or appveyor, use 16 codegen-units for stage0 #48843

bootstrap: when we detect travis-ci or appveyor, use 16 codegen-units for stage0 #48843

matthiaskrgr commented Mar 8, 2018

rust-highfive commented Mar 8, 2018

ishitatsuyuki commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018 •

edited

Loading

ishitatsuyuki commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018

alexcrichton commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018

alexcrichton commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018

alexcrichton commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018 •

edited

Loading

kennytm commented Mar 8, 2018

alexcrichton commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018

alexcrichton commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018

Mark-Simulacrum commented Mar 8, 2018

alexcrichton commented Mar 9, 2018

alexcrichton commented Mar 9, 2018

matthiaskrgr commented Mar 9, 2018

alexcrichton commented Mar 9, 2018

bors commented Mar 16, 2018

Mark-Simulacrum commented Mar 16, 2018

pietroalbini commented Mar 20, 2018

alexcrichton commented Mar 20, 2018

bootstrap: when we detect travis-ci or appveyor, use 16 codegen-units for stage0 #48843

bootstrap: when we detect travis-ci or appveyor, use 16 codegen-units for stage0 #48843

Conversation

matthiaskrgr commented Mar 8, 2018

rust-highfive commented Mar 8, 2018

ishitatsuyuki commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018 • edited Loading

ishitatsuyuki commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018

alexcrichton commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018

alexcrichton commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018

alexcrichton commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018 • edited Loading

kennytm commented Mar 8, 2018

alexcrichton commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018

alexcrichton commented Mar 8, 2018

matthiaskrgr commented Mar 8, 2018

Mark-Simulacrum commented Mar 8, 2018

alexcrichton commented Mar 9, 2018

alexcrichton commented Mar 9, 2018

matthiaskrgr commented Mar 9, 2018

alexcrichton commented Mar 9, 2018

bors commented Mar 16, 2018

Mark-Simulacrum commented Mar 16, 2018

pietroalbini commented Mar 20, 2018

alexcrichton commented Mar 20, 2018

matthiaskrgr commented Mar 8, 2018 •

edited

Loading

matthiaskrgr commented Mar 8, 2018 •

edited

Loading