Avoid one `rustc` rebuild in the optimized build pipeline #112012

Kobzol · 2023-05-26T22:16:36Z

This PR changes the optimized build pipeline to avoid one rustc rebuild, inspired by this comment. This speeds up the pipeline by 5-10 minutes. After this change, we no longer gather LLVM PGO profiles from compiling stage 2 of rustc.

Now we build rustc two times (1x PGO instrumented, 1x PGO optimized) and LLVM three times (1x normal, 1x PGO instrumented, 1x PGO optimized). It should be possible to cache the normal LLVM build, but I'll leave that for another PR.

Kobzol · 2023-05-26T22:16:41Z

@bors try

bors · 2023-05-26T22:16:49Z

⌛ Trying commit a5f4a6a33ea97eb31d87537e1cf85799897ef000 with merge 0b1d1a7586536966f299fc7f0e863abc2bd87c26...

bors · 2023-05-26T22:47:47Z

💔 Test failed - checks-actions

Kobzol · 2023-05-27T06:40:38Z

@bors try

bors · 2023-05-27T06:40:47Z

⌛ Trying commit f1cef095460ec7c82a30b82a02f32d5957f8bd0b with merge a7030bb5524ea580d7262ce64ae4976e22a69cc1...

bors · 2023-05-27T07:20:16Z

💔 Test failed - checks-actions

Kobzol · 2023-05-27T07:25:55Z

@bors try

bors · 2023-05-27T07:26:03Z

⌛ Trying commit 79ac5adcac9671b59218206b1d7ae9b7e7a1ac40 with merge 6dac01618ad5cdeb0b7ddc219ac23cc9cbb06049...

bors · 2023-05-27T08:14:02Z

💔 Test failed - checks-actions

Kobzol · 2023-05-27T11:20:51Z

@bors try

bors · 2023-05-27T11:21:00Z

⌛ Trying commit 47455a6f11865bebf4acd4d063bf83838affc56a with merge 5e2bb1b90772525393f2fc4640fc7b196c460783...

bors · 2023-05-27T12:07:43Z

💔 Test failed - checks-actions

Kobzol · 2023-05-27T12:45:46Z

@bors try

bors · 2023-05-27T12:45:55Z

⌛ Trying commit d6981af07da053d2f70f00d58bffcf2565099c0c with merge 0e7e1014c50ab9680e23008e30859f5dbc7966e7...

bors · 2023-05-27T13:50:09Z

💔 Test failed - checks-actions

Kobzol · 2023-06-07T10:38:08Z

The table was produced incorrectly, I fixed it in the meantime. I'll do another try run, so that we can see hopefully a correct version of the table. I'll have to look a bit closer into the final rebuild.

@bors try

bors · 2023-06-07T10:38:17Z

⌛ Trying commit 14ed9fa with merge 1bcffba3b86d52ce5726ff61d98719cf178bc073...

bors · 2023-06-07T12:00:26Z

☀️ Try build successful - checks-actions
Build commit: 1bcffba3b86d52ce5726ff61d98719cf178bc073 (1bcffba3b86d52ce5726ff61d98719cf178bc073)

nikic · 2023-06-07T12:04:43Z

The stats looks sensible now:


2023-06-07T12:00:07.2904061Z Stage 1 (rustc PGO):                            1632.52s (34.72%)
2023-06-07T12:00:07.2904450Z   Build PGO instrumented rustc and LLVM:         820.07s (17.44%)
2023-06-07T12:00:07.2904808Z     LLVM:                                        353.63s ( 7.52%)
2023-06-07T12:00:07.2905145Z     Rustc:                                       452.21s ( 9.62%)
2023-06-07T12:00:07.2905513Z   Gather profiles:                               520.31s (11.06%)
2023-06-07T12:00:07.2905884Z   Build PGO optimized rustc:                     292.14s ( 6.21%)
2023-06-07T12:00:07.2906237Z     LLVM:                                          0.00s ( 0.00%)
2023-06-07T12:00:07.2906573Z     Rustc:                                       291.57s ( 6.20%)
2023-06-07T12:00:07.2906913Z Stage 2 (LLVM PGO):                              713.83s (15.18%)
2023-06-07T12:00:07.2907287Z   Build PGO instrumented LLVM:                   420.48s ( 8.94%)
2023-06-07T12:00:07.2907762Z     LLVM:                                        406.73s ( 8.65%)
2023-06-07T12:00:07.2908124Z   Gather profiles:                               292.14s ( 6.21%)
2023-06-07T12:00:07.2908500Z Stage 3 (LLVM BOLT):                            1611.41s (34.27%)
2023-06-07T12:00:07.2908889Z   Build BOLT instrumented LLVM:                 1212.72s (25.79%)
2023-06-07T12:00:07.2909346Z     LLVM:                                       1128.15s (23.99%)
2023-06-07T12:00:07.2909742Z   Gather profiles:                               398.69s ( 8.48%)
2023-06-07T12:00:07.2910159Z Final stage (dist build):                        744.58s (15.83%)
2023-06-07T12:00:07.2910618Z   LLVM:                                            0.00s ( 0.00%)
2023-06-07T12:00:07.2910994Z   Rustc:                                         337.73s ( 7.18%)
2023-06-07T12:00:07.2911365Z                                                                  
2023-06-07T12:00:07.2911792Z Total duration:                                        1h 18m 22s

Kobzol · 2023-06-07T12:15:52Z

Good. In the final dist build, stage1 rustc is recompiled in 12 seconds starting from rustc_llvm, because LLVM has changed in the meantime. Then stage2 is built in 4 minutes, starting from scratch. We should be able to get rid of the final rebuild, but I'm not sure if --keep-stage N is safe to use in a dist build - I'll try it. I'm not actually sure what causes the rebuild, I was already trying to find it out for other reasons. Maybe the build script of rustc_llvm notices that the LLVM build files/hashes/mtimes have been changed?

I want to remove the BOLT step from boostrap and apply it in the PGO script instead, which will allow us to optimize libLLVM.so with BOLT without running bootstrap at all, which would resolve the rebuild issue, if there was only BOLT. However, it won't actually help here, because the LLVM that rustc sees will already be modified from the previous LLVM PGO build.

Kobzol · 2023-06-07T12:16:43Z

@bors try

bors · 2023-06-07T12:16:55Z

⌛ Trying commit 41f9f63 with merge bae6b1ef7a7dfd9842d250a3b64da45e71269346...

bors · 2023-06-07T13:50:24Z

☀️ Try build successful - checks-actions
Build commit: bae6b1ef7a7dfd9842d250a3b64da45e71269346 (bae6b1ef7a7dfd9842d250a3b64da45e71269346)

Kobzol · 2023-06-07T14:04:10Z

@rust-timer build bae6b1ef7a7dfd9842d250a3b64da45e71269346

rust-timer · 2023-06-07T16:07:53Z

Finished benchmarking commit (bae6b1ef7a7dfd9842d250a3b64da45e71269346): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.5%	[0.5%, 0.5%]	1
Regressions ❌ (secondary)	0.3%	[0.3%, 0.3%]	2
Improvements ✅ (primary)	-0.9%	[-1.5%, -0.3%]	24
Improvements ✅ (secondary)	-1.1%	[-1.3%, -0.9%]	10
All ❌✅ (primary)	-0.8%	[-1.5%, 0.5%]	25

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.2%	[2.2%, 2.2%]	1
Improvements ✅ (primary)	-2.1%	[-2.1%, -2.1%]	1
Improvements ✅ (secondary)	-2.0%	[-2.0%, -2.0%]	1
All ❌✅ (primary)	-2.1%	[-2.1%, -2.1%]	1

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-1.4%	[-1.8%, -0.9%]	14
Improvements ✅ (secondary)	-1.8%	[-2.0%, -1.4%]	3
All ❌✅ (primary)	-1.4%	[-1.8%, -0.9%]	14

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 647.329s -> 646.031s (-0.20%)

Kobzol · 2023-06-07T16:35:20Z

Ok, looks like it works :) This should be ready for review now.

nikic · 2023-06-07T20:12:30Z

@bors r+ rollup=never

bors · 2023-06-07T20:12:32Z

📌 Commit 41f9f63 has been approved by nikic

It is now in the queue for this repository.

bors · 2023-06-08T00:47:01Z

⌛ Testing commit 41f9f63 with merge 50f2176...

bors · 2023-06-08T03:54:47Z

☀️ Test successful - checks-actions
Approved by: nikic
Pushing 50f2176 to master...

rust-timer · 2023-06-08T05:09:19Z

Finished benchmarking commit (50f2176): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.4%	[0.4%, 0.5%]	2
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.9%	[-1.6%, -0.3%]	26
Improvements ✅ (secondary)	-1.1%	[-1.2%, -0.9%]	11
All ❌✅ (primary)	-0.8%	[-1.6%, 0.5%]	28

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.6%	[1.3%, 5.1%]	3
Improvements ✅ (primary)	-4.3%	[-4.8%, -3.8%]	2
Improvements ✅ (secondary)	-1.8%	[-1.8%, -1.7%]	2
All ❌✅ (primary)	-4.3%	[-4.8%, -3.8%]	2

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	3.5%	[1.0%, 5.3%]	10
Regressions ❌ (secondary)	3.7%	[3.4%, 4.4%]	3
Improvements ✅ (primary)	-1.1%	[-1.4%, -0.7%]	15
Improvements ✅ (secondary)	-1.7%	[-1.7%, -1.7%]	1
All ❌✅ (primary)	0.8%	[-1.4%, 5.3%]	25

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 649.083s -> 647.317s (-0.27%)

Kobzol · 2023-06-25T21:45:33Z

Looks like this has reduced the duration of x64 Linux dist (and try) and x64 Windows dist builds quite nicely:

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels May 26, 2023

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 26, 2023

This comment has been minimized.

Sign in to view

Use `--keep-stage also for the final build

41f9f63

This comment has been minimized.

Sign in to view

rustbot added the perf-regression Performance regression. label Jun 7, 2023

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 7, 2023

bors added the merged-by-bors This PR was explicitly merged by bors. label Jun 8, 2023

bors merged commit 50f2176 into rust-lang:master Jun 8, 2023

rustbot added this to the 1.72.0 milestone Jun 8, 2023

rustbot removed the perf-regression Performance regression. label Jun 8, 2023

Kobzol deleted the try-build-llvm-rebuild branch June 8, 2023 06:07

lqd mentioned this pull request Jun 8, 2023

Build rustc with 1CGU on x86_64-pc-windows-msvc #112267

Merged

Avoid one rustc rebuild in the optimized build pipeline #112012

Avoid one rustc rebuild in the optimized build pipeline #112012

Conversation

Kobzol commented May 26, 2023 • edited Loading

Kobzol commented May 26, 2023

bors commented May 26, 2023

bors commented May 26, 2023

This comment has been minimized.

Kobzol commented May 27, 2023

bors commented May 27, 2023

This comment has been minimized.

bors commented May 27, 2023

Kobzol commented May 27, 2023

bors commented May 27, 2023

This comment has been minimized.

bors commented May 27, 2023

Kobzol commented May 27, 2023

bors commented May 27, 2023

This comment has been minimized.

bors commented May 27, 2023

Kobzol commented May 27, 2023

bors commented May 27, 2023

This comment has been minimized.

bors commented May 27, 2023

Kobzol commented Jun 7, 2023

bors commented Jun 7, 2023

bors commented Jun 7, 2023

nikic commented Jun 7, 2023

Kobzol commented Jun 7, 2023 • edited Loading

Kobzol commented Jun 7, 2023

bors commented Jun 7, 2023

bors commented Jun 7, 2023

Kobzol commented Jun 7, 2023

This comment has been minimized.

rust-timer commented Jun 7, 2023

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Kobzol commented Jun 7, 2023

nikic commented Jun 7, 2023

bors commented Jun 7, 2023

bors commented Jun 8, 2023

bors commented Jun 8, 2023

rust-timer commented Jun 8, 2023

Overall result: ✅ improvements - no action needed

Kobzol commented Jun 25, 2023 • edited Loading

Avoid one `rustc` rebuild in the optimized build pipeline #112012

Avoid one `rustc` rebuild in the optimized build pipeline #112012

Kobzol commented May 26, 2023 •

edited

Loading

Kobzol commented Jun 7, 2023 •

edited

Loading

Kobzol commented Jun 25, 2023 •

edited

Loading