Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new rustc_panic_abort_runtime attribute for libpanic_abort #66489

Closed
wants to merge 2 commits into from

Conversation

Aaron1011
Copy link
Member

Supersedes #66311

This replaces the hack in bootstrap, allowing the special handling for
libpanic_abort to be encoded into the crate source itself, rather than
existing as special knowledge in the build system. This will allow Miri
(and any other users of Xargo) to correctly build libpanic_abort
without relying on bootstrap or custom wrapepr hacks.

The trickeist part of this PR is how we handle LLVM. The emscripten
target family requires the "-enable-emscripten-cxx-exceptions" flag
to be passed to LLVM when the panic strategy is set to "unwind".

Unfortunately, the location of this emscripten-specific check ends up
creating a circular dependency between LLVM and attribute resoltion.
When we check the panic strategy, we need to have already parsed crate
attributes, so that we determine if rustc_panic_abort_runtime was set.
However, attribute parsing requires LLVM to be initialized, so that we
can proerly handle cfg-gating of target-specific features. However, the
whole point of checking the panic strategy is to determinne which flags
to use during LLVM initialization!

To break this circular dependency, we explicitly set the
"-enable-emscripten-cxx-exceptions" in LLVM's argument-parsing framework
(using public but poorly-documented APIs). While this approach is
unfortunate, it only affects emscripten, and only modifies a
command-line flag which is never used until much later (when we create a
PassManager).

@rust-highfive
Copy link
Collaborator

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Nov 17, 2019
Copy link
Member

@bjorn3 bjorn3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to move codegen_backend.init(sess) itself after parsing? Having two places to initialize a codegen backend can lead to confusion about when to use which place when adding some new thing to init.

src/librustc_codegen_utils/codegen_backend.rs Outdated Show resolved Hide resolved
@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-11-17T14:04:13.6857426Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-11-17T14:04:13.7036998Z ##[command]git config gc.auto 0
2019-11-17T14:04:13.7121906Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-11-17T14:04:13.7186162Z ##[command]git config --get-all http.proxy
2019-11-17T14:04:13.7348365Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/66489/merge:refs/remotes/pull/66489/merge
---
2019-11-17T15:04:34.3328578Z .................................................................................................... 1500/9251
2019-11-17T15:04:40.6673614Z .................................................................................................... 1600/9251
2019-11-17T15:04:49.5773822Z .................................................................................................... 1700/9251
2019-11-17T15:04:59.0786962Z ........i........................................................................................... 1800/9251
2019-11-17T15:05:05.7814690Z .............................................................................................iiiii.. 1900/9251
2019-11-17T15:05:28.4643153Z .................................................................................................... 2100/9251
2019-11-17T15:05:30.9648256Z .................................................................................................... 2200/9251
2019-11-17T15:05:33.5618155Z .................................................................................................... 2300/9251
2019-11-17T15:05:40.1255306Z .................................................................................................... 2400/9251
---
2019-11-17T15:09:35.5241871Z .................................................................................................... 5400/9251
2019-11-17T15:09:46.8205099Z ...............................................................................i.................... 5500/9251
2019-11-17T15:09:55.1728338Z .................................................................................................... 5600/9251
2019-11-17T15:10:01.9597623Z .................................................................................................... 5700/9251
2019-11-17T15:10:13.0506365Z .................................................................ii...i..ii...........i............. 5800/9251
2019-11-17T15:10:36.5261971Z .................................................................................................... 6000/9251
2019-11-17T15:10:45.4794807Z .................................................................................................... 6100/9251
2019-11-17T15:10:45.4794807Z .................................................................................................... 6100/9251
2019-11-17T15:10:53.0329308Z ....................................................................................i..ii........... 6200/9251
2019-11-17T15:11:08.0181549Z .................................................................................................... 6300/9251
2019-11-17T15:11:18.2405832Z .........................................................................................FF.F.FF.... 6400/9251
2019-11-17T15:11:27.1166581Z .................................................................................................... 6600/9251
2019-11-17T15:11:29.6672780Z .........................................i.......................................................... 6700/9251
2019-11-17T15:11:32.9580530Z .................................................................................................... 6800/9251
2019-11-17T15:11:39.6187744Z .................................................................................................... 6900/9251
---
2019-11-17T15:16:32.3868619Z 
2019-11-17T15:16:32.3869449Z ---- [ui] ui/panic-runtime/abort-link-to-unwind-dylib.rs stdout ----
2019-11-17T15:16:32.3869684Z diff of stderr:
2019-11-17T15:16:32.3869817Z 
2019-11-17T15:16:32.3870893Z - error: the linked panic runtime `panic_unwind` is not compiled with this crate's panic strategy `abort`
2019-11-17T15:16:32.3871180Z + error: the crate `panic_abort` does not have the panic strategy `abort`
2019-11-17T15:16:32.3871490Z 3 error: aborting due to previous error
2019-11-17T15:16:32.3871649Z 4 
2019-11-17T15:16:32.3871794Z 
2019-11-17T15:16:32.3871915Z 
2019-11-17T15:16:32.3871915Z 
2019-11-17T15:16:32.3872078Z The actual stderr differed from the expected stderr.
2019-11-17T15:16:32.3872616Z Actual stderr saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panic-runtime/abort-link-to-unwind-dylib/abort-link-to-unwind-dylib.stderr
2019-11-17T15:16:32.3873085Z To update references, rerun the tests and pass the `--bless` flag
2019-11-17T15:16:32.3873608Z To only update this specific test, also pass `--test-args panic-runtime/abort-link-to-unwind-dylib.rs`
2019-11-17T15:16:32.3874101Z error: 1 errors occurred comparing output.
2019-11-17T15:16:32.3874425Z status: exit code: 1
2019-11-17T15:16:32.3874425Z status: exit code: 1
2019-11-17T15:16:32.3875353Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/panic-runtime/abort-link-to-unwind-dylib.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panic-runtime/abort-link-to-unwind-dylib" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-C" "panic=abort" "-C" "prefer-dynamic" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panic-runtime/abort-link-to-unwind-dylib/auxiliary" "-A" "unused"
2019-11-17T15:16:32.3875948Z ------------------------------------------
2019-11-17T15:16:32.3876493Z 
2019-11-17T15:16:32.3876921Z ------------------------------------------
2019-11-17T15:16:32.3877101Z stderr:
2019-11-17T15:16:32.3877101Z stderr:
2019-11-17T15:16:32.3877480Z ------------------------------------------
2019-11-17T15:16:32.3877660Z error: the crate `panic_abort` does not have the panic strategy `abort`
2019-11-17T15:16:32.3877961Z error: aborting due to previous error
2019-11-17T15:16:32.3878082Z 
2019-11-17T15:16:32.3878201Z 
2019-11-17T15:16:32.3878569Z ------------------------------------------
2019-11-17T15:16:32.3878569Z ------------------------------------------
2019-11-17T15:16:32.3878724Z 
2019-11-17T15:16:32.3878856Z 
2019-11-17T15:16:32.3879210Z ---- [ui] ui/panic-runtime/abort.rs stdout ----
2019-11-17T15:16:32.3879386Z 
2019-11-17T15:16:32.3879902Z error: test compilation failed although it shouldn't!
2019-11-17T15:16:32.3880068Z status: exit code: 1
2019-11-17T15:16:32.3880904Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/panic-runtime/abort.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panic-runtime/abort/a" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-C" "panic=abort" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panic-runtime/abort/auxiliary"
2019-11-17T15:16:32.3881472Z ------------------------------------------
2019-11-17T15:16:32.3881617Z 
2019-11-17T15:16:32.3881957Z ------------------------------------------
2019-11-17T15:16:32.3882487Z stderr:
2019-11-17T15:16:32.3882487Z stderr:
2019-11-17T15:16:32.3882849Z ------------------------------------------
2019-11-17T15:16:32.3883045Z error: the crate `panic_abort` does not have the panic strategy `abort`
2019-11-17T15:16:32.3883858Z error: aborting due to previous error
2019-11-17T15:16:32.3883994Z 
2019-11-17T15:16:32.3884131Z 
2019-11-17T15:16:32.3884640Z ------------------------------------------
2019-11-17T15:16:32.3884640Z ------------------------------------------
2019-11-17T15:16:32.3884819Z 
2019-11-17T15:16:32.3884934Z 
2019-11-17T15:16:32.3885342Z ---- [ui] ui/panic-runtime/abort-link-to-unwinding-crates.rs stdout ----
2019-11-17T15:16:32.3885490Z 
2019-11-17T15:16:32.3885850Z error: test compilation failed although it shouldn't!
2019-11-17T15:16:32.3886035Z status: exit code: 1
2019-11-17T15:16:32.3888174Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/panic-runtime/abort-link-to-unwinding-crates.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panic-runtime/abort-link-to-unwinding-crates/a" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-C" "panic=abort" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panic-runtime/abort-link-to-unwinding-crates/auxiliary"
2019-11-17T15:16:32.3888856Z ------------------------------------------
2019-11-17T15:16:32.3889036Z 
2019-11-17T15:16:32.3889391Z ------------------------------------------
2019-11-17T15:16:32.3889561Z stderr:
2019-11-17T15:16:32.3889561Z stderr:
2019-11-17T15:16:32.3889923Z ------------------------------------------
2019-11-17T15:16:32.3890103Z error: the crate `panic_abort` does not have the panic strategy `abort`
2019-11-17T15:16:32.3890567Z error: aborting due to previous error
2019-11-17T15:16:32.3890686Z 
2019-11-17T15:16:32.3890800Z 
2019-11-17T15:16:32.3891151Z ------------------------------------------
2019-11-17T15:16:32.3891151Z ------------------------------------------
2019-11-17T15:16:32.3891317Z 
2019-11-17T15:16:32.3891437Z 
2019-11-17T15:16:32.3891791Z ---- [ui] ui/panic-runtime/link-to-abort.rs stdout ----
2019-11-17T15:16:32.3891963Z 
2019-11-17T15:16:32.3892365Z error: test compilation failed although it shouldn't!
2019-11-17T15:16:32.3893759Z status: exit code: 1
2019-11-17T15:16:32.3894607Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/panic-runtime/link-to-abort.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panic-runtime/link-to-abort/a" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-C" "panic=abort" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panic-runtime/link-to-abort/auxiliary"
2019-11-17T15:16:32.3894941Z ------------------------------------------
2019-11-17T15:16:32.3894983Z 
2019-11-17T15:16:32.3895214Z ------------------------------------------
2019-11-17T15:16:32.3895261Z stderr:
2019-11-17T15:16:32.3895261Z stderr:
2019-11-17T15:16:32.3895470Z ------------------------------------------
2019-11-17T15:16:32.3895743Z error: the linked panic runtime `panic_abort` is not compiled with this crate's panic strategy `abort`
2019-11-17T15:16:32.3895856Z error: aborting due to previous error
2019-11-17T15:16:32.3895886Z 
2019-11-17T15:16:32.3895911Z 
2019-11-17T15:16:32.3896739Z ------------------------------------------
2019-11-17T15:16:32.3896739Z ------------------------------------------
2019-11-17T15:16:32.3896776Z 
2019-11-17T15:16:32.3896801Z 
2019-11-17T15:16:32.3897022Z ---- [ui] ui/panic-runtime/lto-abort.rs stdout ----
2019-11-17T15:16:32.3897074Z 
2019-11-17T15:16:32.3897296Z error: test compilation failed although it shouldn't!
2019-11-17T15:16:32.3897345Z status: exit code: 1
2019-11-17T15:16:32.3898233Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/panic-runtime/lto-abort.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panic-runtime/lto-abort/a" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-C" "lto" "-C" "panic=abort" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/panic-runtime/lto-abort/auxiliary"
2019-11-17T15:16:32.3898664Z ------------------------------------------
2019-11-17T15:16:32.3898700Z 
2019-11-17T15:16:32.3898914Z ------------------------------------------
2019-11-17T15:16:32.3898959Z stderr:
2019-11-17T15:16:32.3898959Z stderr:
2019-11-17T15:16:32.3899185Z ------------------------------------------
2019-11-17T15:16:32.3899239Z error: the crate `panic_abort` does not have the panic strategy `abort`
2019-11-17T15:16:32.3899335Z error: aborting due to previous error
2019-11-17T15:16:32.3899366Z 
2019-11-17T15:16:32.3899392Z 
2019-11-17T15:16:32.3899613Z ------------------------------------------
---
2019-11-17T15:16:32.3900509Z 
2019-11-17T15:16:32.3900908Z 1 error: building tests with panic=abort is not supported without `-Zpanic_abort_tests`
2019-11-17T15:16:32.3900958Z 2 
2019-11-17T15:16:32.3901159Z - error: aborting due to previous error
2019-11-17T15:16:32.3901229Z + error: the crate `panic_abort` does not have the panic strategy `abort`
2019-11-17T15:16:32.3901317Z + error: aborting due to 2 previous errors
2019-11-17T15:16:32.3901356Z 4 
2019-11-17T15:16:32.3901410Z 5 
2019-11-17T15:16:32.3901435Z 
2019-11-17T15:16:32.3901435Z 
2019-11-17T15:16:32.3901460Z 
2019-11-17T15:16:32.3901503Z The actual stderr differed from the expected stderr.
2019-11-17T15:16:32.3901824Z Actual stderr saved to /checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/test-panic-abort-disabled/test-panic-abort-disabled.stderr
2019-11-17T15:16:32.3902072Z To update references, rerun the tests and pass the `--bless` flag
2019-11-17T15:16:32.3902343Z To only update this specific test, also pass `--test-args test-panic-abort-disabled.rs`
2019-11-17T15:16:32.3902420Z error: 1 errors occurred comparing output.
2019-11-17T15:16:32.3902470Z status: exit code: 1
2019-11-17T15:16:32.3902470Z status: exit code: 1
2019-11-17T15:16:32.3903223Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/test-panic-abort-disabled.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/test-panic-abort-disabled" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--test" "-Cpanic=abort" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/test-panic-abort-disabled/auxiliary" "-A" "unused"
2019-11-17T15:16:32.3903545Z ------------------------------------------
2019-11-17T15:16:32.3903577Z 
2019-11-17T15:16:32.3903777Z ------------------------------------------
2019-11-17T15:16:32.3903838Z stderr:
2019-11-17T15:16:32.3903838Z stderr:
2019-11-17T15:16:32.3904041Z ------------------------------------------
2019-11-17T15:16:32.3904295Z error: building tests with panic=abort is not supported without `-Zpanic_abort_tests`
2019-11-17T15:16:32.3904331Z 
2019-11-17T15:16:32.3904398Z error: the crate `panic_abort` does not have the panic strategy `abort`
2019-11-17T15:16:32.3904471Z error: aborting due to 2 previous errors
2019-11-17T15:16:32.3904499Z 
2019-11-17T15:16:32.3904540Z 
2019-11-17T15:16:32.3904745Z ------------------------------------------
2019-11-17T15:16:32.3904745Z ------------------------------------------
2019-11-17T15:16:32.3904776Z 
2019-11-17T15:16:32.3904800Z 
2019-11-17T15:16:32.3905018Z ---- [ui] ui/test-panic-abort.rs stdout ----
2019-11-17T15:16:32.3905049Z 
2019-11-17T15:16:32.3905261Z error: test compilation failed although it shouldn't!
2019-11-17T15:16:32.3905412Z status: exit code: 1
2019-11-17T15:16:32.3906677Z command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/ui/test-panic-abort.rs" "-Zthreads=1" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/test-panic-abort/a" "-Crpath" "-O" "-Cdebuginfo=0" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--test" "-Cpanic=abort" "-Zpanic_abort_tests" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/test-panic-abort/auxiliary" "-A" "unused"
2019-11-17T15:16:32.3907061Z ------------------------------------------
2019-11-17T15:16:32.3907095Z 
2019-11-17T15:16:32.3907306Z ------------------------------------------
2019-11-17T15:16:32.3907369Z stderr:
2019-11-17T15:16:32.3907369Z stderr:
2019-11-17T15:16:32.3907578Z ------------------------------------------
2019-11-17T15:16:32.3907630Z error: the crate `panic_abort` does not have the panic strategy `abort`
2019-11-17T15:16:32.3907737Z error: aborting due to previous error
2019-11-17T15:16:32.3907766Z 
2019-11-17T15:16:32.3907791Z 
2019-11-17T15:16:32.3908002Z ------------------------------------------
---
2019-11-17T15:16:32.3914343Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:537:22
2019-11-17T15:16:32.3914419Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
2019-11-17T15:16:32.3920538Z 
2019-11-17T15:16:32.3920643Z 
2019-11-17T15:16:32.3922436Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--src-base" "/checkout/src/test/ui" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "ui" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-6.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "6.0.0\n" "--system-llvm" "--cc" "" "--cxx" "" "--cflags" "" "--llvm-components" "" "--llvm-cxxflags" "" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2019-11-17T15:16:32.3922688Z 
2019-11-17T15:16:32.3922735Z 
2019-11-17T15:16:32.3930923Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
2019-11-17T15:16:32.3931473Z Build completed unsuccessfully in 1:06:13
2019-11-17T15:16:32.3931473Z Build completed unsuccessfully in 1:06:13
2019-11-17T15:16:32.3991956Z == clock drift check ==
2019-11-17T15:16:32.4007670Z   local time: Sun Nov 17 15:16:32 UTC 2019
2019-11-17T15:16:32.6783249Z   network time: Sun, 17 Nov 2019 15:16:32 GMT
2019-11-17T15:16:32.6784656Z == end clock drift check ==
2019-11-17T15:16:33.3715099Z 
2019-11-17T15:16:33.3852967Z ##[error]Bash exited with code '1'.
2019-11-17T15:16:33.3891879Z ##[section]Starting: Checkout
2019-11-17T15:16:33.3893642Z ==============================================================================
2019-11-17T15:16:33.3893698Z Task         : Get sources
2019-11-17T15:16:33.3893863Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@Aaron1011
Copy link
Member Author

Would it be possible to move codegen_backend.init(sess) itself after parsing? Having two places to initialize a codegen backend can lead to confusion about when to use which place when adding some new thing to init.

@bjorn3: Unfortunately, no. Sadly, parsing the crate actually depends on LLVM being initialized, since we need to know LLVM's supported features for cfg-gating:

/// Adds `target_feature = "..."` cfgs for a variety of platform
/// specific features (SSE, NEON etc.).
///
/// This is performed by checking whether a whitelisted set of
/// features is available on the target machine, by querying LLVM.
pub fn add_configuration(
cfg: &mut ast::CrateConfig,
sess: &Session,
codegen_backend: &dyn CodegenBackend,
) {
let tf = sym::target_feature;
cfg.extend(
codegen_backend
.target_features(sess)
.into_iter()
.map(|feat| (tf, Some(feat))),
);
if sess.crt_static_feature() {
cfg.insert((tf, Some(Symbol::intern("crt-static"))));
}
}

This whole situation is far from ideal. However, I believe that this solution allows us to remove the bootstrap hack with a minimum of changes to the compiler.

@Aaron1011
Copy link
Member Author

I had to change my approach slightly. Checking for rustc_panic_abort_runtime immediately after parsing will cause us to miss the attribute, since it's inside a cfg_attr. We need to run the check after we've expanded cfg attrs, so that we know whether or not we're in bootstrap.

@Aaron1011
Copy link
Member Author

cc @alexcrichton: I believe this approach avoids your concerns with a custom environment variable/-Z flag

@alexcrichton
Copy link
Member

I personally believe that this is still not necessary and it's better to fix the build system rather than add features to the compiler which are likely to be somewhat difficult to maintain over time. This, unlike the previous PR, however is gated on nightly so we can change this later, and it seems fine to land if it's approved to me.

@Aaron1011
Copy link
Member Author

Aaron1011 commented Nov 19, 2019

I personally believe that this is still not necessary and it's better to fix the build system

By 'fix the build system', do you mean moving the hack to Xargo, or adjusting how we determine if libpanic_abort was built with the correct panic strategy (e.g. no longer requiring it to have a panic strategy distinct from the other crates in the build)?

If it's the former, I'm strongly opposed to perpetuating this kind of hack. I think any special handling of a crate should require something in the crate itself, whether it's an attribute or something in build.rs. I think hacks in bootstrap make it very easy to accidentally something that almost works, but breaks in some weird way due to a missing special case.

If it's the latter, I think that sounds fine, but I'm not sure the best way to go about that.

@Aaron1011
Copy link
Member Author

I'd appreciate it if someone more familiar with LLVM could weigh in on my approach to 'late-modification' of LLVM command line arguments. In my local testing, it appears to be doing the right thing w.r.t emscripten (I initially had it wrong, and the panic-related tests failed in weird ways). However, I don't know how likely this is to break in future LLVM releases, or if there's something subtle I'm overlooking.

@bjorn3
Copy link
Member

bjorn3 commented Nov 19, 2019

whether it's an attribute or something in build.rs.

Using something in build.rs would be the best I think, however cargo build scripts dont seem to be able to pass arbitrary flags to rustc yet.

@Alexendoo
Copy link
Member

Ping from triage, any updates? @nikomatsakis

@bors
Copy link
Contributor

bors commented Nov 29, 2019

☔ The latest upstream changes (presumably #66697) made this pull request unmergeable. Please resolve the merge conflicts.

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-6.0 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
2019-11-30T02:28:44.2829896Z ##[command]git remote add origin https://github.com/rust-lang/rust
2019-11-30T02:28:44.2990989Z ##[command]git config gc.auto 0
2019-11-30T02:28:44.3088281Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2019-11-30T02:28:44.3164694Z ##[command]git config --get-all http.proxy
2019-11-30T02:28:44.3324116Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/66489/merge:refs/remotes/pull/66489/merge
---
2019-11-30T02:31:46.7267259Z #########################################################                 79.9%
2019-11-30T02:31:46.7268627Z ######################################################################## 100.0%
2019-11-30T02:31:47.1534826Z extracting /checkout/obj/build/cache/2019-11-06/cargo-beta-x86_64-unknown-linux-gnu.tar.gz
2019-11-30T02:31:47.2410996Z     Updating crates.io index
2019-11-30T02:31:54.8383871Z error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
2019-11-30T02:31:54.8418675Z failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
2019-11-30T02:31:54.8465875Z Makefile:67: recipe for target 'prepare' failed
2019-11-30T02:31:54.8466515Z make: *** [prepare] Error 1
2019-11-30T02:31:55.8487755Z Command failed. Attempt 2/5:
2019-11-30T02:31:55.9753452Z     Updating crates.io index
2019-11-30T02:31:55.9753452Z     Updating crates.io index
2019-11-30T02:31:56.3499681Z error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
2019-11-30T02:31:56.3524037Z failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
2019-11-30T02:31:56.3570223Z make: *** [prepare] Error 1
2019-11-30T02:31:56.3573364Z Makefile:67: recipe for target 'prepare' failed
2019-11-30T02:31:58.3593476Z Command failed. Attempt 3/5:
2019-11-30T02:31:58.4891095Z     Updating crates.io index
2019-11-30T02:31:58.4891095Z     Updating crates.io index
2019-11-30T02:31:58.8271307Z error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
2019-11-30T02:31:58.8296106Z failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
2019-11-30T02:31:58.8343861Z make: *** [prepare] Error 1
2019-11-30T02:31:58.8344826Z Makefile:67: recipe for target 'prepare' failed
2019-11-30T02:32:01.8364778Z Command failed. Attempt 4/5:
2019-11-30T02:32:01.9612805Z     Updating crates.io index
2019-11-30T02:32:01.9612805Z     Updating crates.io index
2019-11-30T02:32:02.5724492Z error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
2019-11-30T02:32:02.5747708Z failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
2019-11-30T02:32:02.5793146Z Makefile:67: recipe for target 'prepare' failed
2019-11-30T02:32:02.5793574Z make: *** [prepare] Error 1
2019-11-30T02:32:07.3767882Z Command failed. Attempt 5/5:
2019-11-30T02:32:07.3768990Z     Updating crates.io index
2019-11-30T02:32:07.3768990Z     Updating crates.io index
2019-11-30T02:32:07.3770061Z error: the lock file /checkout/Cargo.lock needs to be updated but --locked was passed to prevent this
2019-11-30T02:32:07.3770704Z failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
2019-11-30T02:32:07.3771426Z make: *** [prepare] Error 1
2019-11-30T02:32:07.3771952Z Makefile:67: recipe for target 'prepare' failed
2019-11-30T02:32:07.3772242Z The command has failed after 5 attempts.
2019-11-30T02:32:07.3772397Z == clock drift check ==
2019-11-30T02:32:07.3772397Z == clock drift check ==
2019-11-30T02:32:07.3772586Z   local time: Sat Nov 30 02:32:07 UTC 2019
2019-11-30T02:32:07.3772748Z   network time: Sat, 30 Nov 2019 02:32:07 GMT
2019-11-30T02:32:07.3772903Z == end clock drift check ==
2019-11-30T02:32:19.1700950Z 
2019-11-30T02:32:19.1821154Z ##[error]Bash exited with code '1'.
2019-11-30T02:32:19.1854861Z ##[section]Starting: Checkout
2019-11-30T02:32:19.1856976Z ==============================================================================
2019-11-30T02:32:19.1857040Z Task         : Get sources
2019-11-30T02:32:19.1857130Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors
Copy link
Contributor

bors commented Dec 1, 2019

☔ The latest upstream changes (presumably #66908) made this pull request unmergeable. Please resolve the merge conflicts.

@bors
Copy link
Contributor

bors commented Dec 3, 2019

☔ The latest upstream changes (presumably #66947) made this pull request unmergeable. Please resolve the merge conflicts.

@bors
Copy link
Contributor

bors commented Dec 3, 2019

☔ The latest upstream changes (presumably #66997) made this pull request unmergeable. Please resolve the merge conflicts.

@Aaron1011
Copy link
Member Author

This is constantly getting broken due to changes to master. I'm going to hold off fixing the merge conflicts until after review of the main idea of this PR.

@Dylan-DPC-zz
Copy link

r? @oli-obk

Copy link
Contributor

@oli-obk oli-obk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've read through the superseded PRs comments and am still unclear on some things.

Is the main statement here that it will never be possible to move miri (and other xargo users) to cargo -Zbuild-std? If so, what could be changed with -Zbuild-std to support this use case? This does seem like a cargo issue and not a rustc issue (I was under the impression the goal is to retire xargo when cargo supports everything needed).

@@ -8,6 +8,7 @@
#![doc(html_root_url = "https://doc.rust-lang.org/nightly/",
issue_tracker_base_url = "https://github.com/rust-lang/rust/issues/")]
#![panic_runtime]
#![cfg_attr(not(bootstrap), rustc_panic_abort_runtime)]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't this in turn break cargo's -Zbuild-std which builds this crate with the unwind strategy?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Building this crate with the unwind strategy is the wrong behavior - it should always be built with the abort strategy. So, this will correctly change how -Zbuild-std builds this crate - but it's a bugfix, not breakage.

@Aaron1011
Copy link
Member Author

Is the main statement here that it will never be possible to move miri (and other xargo users) to cargo -Zbuild-std?

This change is correct regardless of what any xargousers (including Miri) choose to do.

In librustc_metadata, we expect libpanic_abort to always be built with the abort strategy. Doing anything else is flat out wrong - however, the fact that -Zbuild-std never uses a 'precompiled' libstd currently masks the issue.

While this change allows xargo to properly build libpanic_abort, I think we would want to make this kind of change even if xargo did not exist. Encoding this kind of logic in bootstrap is an awful hack (and a not very discoverable one at that).

@Aaron1011
Copy link
Member Author

@oli-obk: Rebased

@ehuss
Copy link
Contributor

ehuss commented Dec 21, 2019

and allows cargo -Z build-std to use a pre-built libstd without additional hacks.

I don't know what this means, build-std fundamentally does not use a pre-built libstd.

I also never saw an answer to my question above. Is there anything (other than time) blocking miri from using build-std?

@Aaron1011
Copy link
Member Author

Aaron1011 commented Dec 21, 2019

I don't know what this means, build-std fundamentally does not use a pre-built libstd.

@ehuss: Do you mean that build-std will never support a pre-built llibstd? That is, anyone using build-std will end up triggering a rebuild of libstd whenever they change a relevant option in their Cargo.toml (debug info level, optimization level, panic strategy, etc)? That seems really undesirable - for small crates, you would spend almost all of your time waiting on libstd

I also never saw an answer to my question above. Is there anything (other than time) blocking miri from using build-std?

Miri is currently using a pre-built libstd with Xargo. Switching away from a pre-built libstd would increase Miri run times and disk space usage (since every crate you run cargo miri on now needs its own libstd) without any benefit (as far as I can see)

@ehuss
Copy link
Contributor

ehuss commented Dec 21, 2019

Hm, I think I may be a little confused, as what I think of "pre-built libstd" is what ships with rustc. That's not what you're referring to, though? From what I understand xargo has a shared cache of sysroots in ~/.xargo, right?

Currently, build-std rebuilds the standard library for every project from scratch (stored in the target directory). That is intentional, as Cargo currently doesn't have a good story for shared artifact caching. This is why the panic settings aren't really relevant to build-std, because it recompiles the whole set with whatever panic strategy the user picks. It does not reuse unwind crates with an "abort" profile, like the standard library does today.

If I'm understanding that performance concern correctly, would it be sufficient for miri if Cargo had a shared dependency cache in a common location? Are there any other fundamental concerns? (FYI, I know next to nothing about miri or how it works, so I have no idea what it needs concerning the standard library crates.)

@bors
Copy link
Contributor

bors commented Dec 21, 2019

☔ The latest upstream changes (presumably #67485) made this pull request unmergeable. Please resolve the merge conflicts.

@Aaron1011
Copy link
Member Author

Aaron1011 commented Dec 21, 2019

Hm, I think I may be a little confused, as what I think of "pre-built libstd" is what ships with rustc. That's not what you're referring to, though? From what I understand xargo has a shared cache of sysroots in ~/.xargo, right?

Sorry, my use of 'pre-built' may have been unclear. I'm referring to you're calling 'shared artifact caching' - that is, building a libstd once in unwind mode, and then re-using that whenever a crate needs a libstd.

From what I understand xargo has a shared cache of sysroots in ~/.xargo, right?

That's correct

Currently, build-std rebuilds the standard library for every project from scratch (stored in the target directory). That is intentional, as Cargo currently doesn't have a good story for shared artifact caching.

Are there any plans to change this (even if it's far in the future)? I completely understand that there are more pressing concerns for -Z build-std at the moment. However, I don't think we should be requiring build-std users to repeatedly re-build libstd if we can find a way to avoid it.

Assuming that a 're-use a built libstd' feature eventually ends up in build-std, this PR will allow us to avoid the need for any additional hacks in cargo (namely, special-casing libpanic_abort)

If I'm understanding that performance concern correctly, would it be sufficient for miri if Cargo had a shared dependency cache in a common location?

Ideally, Miri would continue to be able to build libstd exactly once, and re-use it for every crate. Miri is already quite slow, and while building libstd may be a small percentage of the overall runtime, it would be quite unfortunate to make Miri even slower. (cc @RalfJung)

@Aaron1011 Aaron1011 closed this Dec 21, 2019
@Aaron1011 Aaron1011 reopened this Dec 21, 2019
Aaron1011 and others added 2 commits December 21, 2019 12:46
Supersedes rust-lang#66311

This replaces the hack in `bootstrap`, allowing the special handling for
`libpanic_abort` to be encoded into the crate source itself, rather than
existing as special knowledge in the build system. This will allow Miri
(and any other users of Xargo) to correctly build `libpanic_abort`
without relying on `bootstrap` or custom wrapepr hacks.

The trickeist part of this PR is how we handle LLVM. The `emscripten`
target family requires the "-enable-emscripten-cxx-exceptions" flag
to be passed to LLVM when the panic strategy is set to "unwind".

Unfortunately, the location of this emscripten-specific check ends up
creating a circular dependency between LLVM and attribute resoltion.
When we check the panic strategy, we need to have already parsed crate
attributes, so that we determine if `rustc_panic_abort_runtime` was set.
However, attribute parsing requires LLVM to be initialized, so that we
can proerly handle cfg-gating of target-specific features. However, the
whole point of checking the panic strategy is to determinne which flags
to use during LLVM initialization!

To break this circular dependency, we explicitly set the
"-enable-emscripten-cxx-exceptions" in LLVM's argument-parsing framework
(using public but poorly-documented APIs). While this approach is
unfortunate, it only affects emscripten, and only modifies a
command-line flag which is never used until much later (when we create a
`PassManager`).
Co-Authored-By: Thomas Lively <7121787+tlively@users.noreply.github.com>
@ehuss
Copy link
Contributor

ehuss commented Dec 21, 2019

Are there any plans to change this (even if it's far in the future)?

Yea, we want to get to a point where a shared artifact cache works well in Cargo. Some work was done over the past year, but there are some difficult things to deal with.

this PR will allow us to avoid the need for any additional hacks in cargo (namely, special-casing libpanic_abort)

This I don't agree with. There aren't any hacks currently in use, and AFAIK cargo doesn't need this. With build-std the user gets to choose the panic strategy, and the standard library will be built accordingly, without overrides or hacks.

@Aaron1011
Copy link
Member Author

With build-std the user gets to choose the panic strategy, and the standard library will be built accordingly, without overrides or hacks.

@ehuss: See my comment from before:

@ehuss: Do you mean that build-std will never support a pre-built llibstd? That is, anyone using build-std will end up triggering a rebuild of libstd whenever they change a relevant option in their Cargo.toml (debug info level, optimization level, panic strategy, etc)? That seems really undesirable - for small crates, you would spend almost all of your time waiting on libstd

@ehuss
Copy link
Contributor

ehuss commented Dec 21, 2019

Caching seems unrelated to whether or not Cargo needs this new attribute. If the artifacts are shared, it's a one-time cost. So if project A uses unwind, and B uses abort, you'll end up with two different copies of the standard library. If there is a shared cache, then project C will reuse either A or B's copy, depending on which it needs.

@Aaron1011
Copy link
Member Author

@ehuss: I see. So, there would be an initial cost to changing settings in your Cargo.toml due to the libstd rebuild, but you would only pay this cost once per unique combination of settings per machine (well, except for CI).

I believe that Miri will still want to build a libstd that's independent of the project's Cargo.toml - that is, we never have any reason to rebuilt it after cargo miri is first intalled/set up. However, @RalfJung will probably want to weigh in on this.

If Miri ends up switching to -Z build-std, and continues to use a pre-built libstd, then this PR will allow it to function correctly. Otherwise, this PR will not interact directly with -Z build-std, except for the fact that it will cause us to produce a non-broken (but still unused) libpanic_abort when we build libstd with panic=unwind.

Regardless of what Miri ends up doing, I don't think this kind of crate-specific logic belongs in bootstrap. It should either live in rustc, or (as @oli-obk proposed) live inside cargo itself.

@oli-obk
Copy link
Contributor

oli-obk commented Dec 21, 2019

I believe that Miri will still want to build a libstd that's independent of the project's Cargo.toml - that is, we never have any reason to rebuilt it after cargo miri is first intalled/set up. However, @RalfJung will probably want to weigh in on this.

Imo once -Zbuild-std works with caching we don't really care about having a single (unwind) libstd that hotswaps the panic runtime. The only reason we need this right now is that xargo would be trashing the globally cached libstd every time we switch between unwind and abort.

So basically the choices we have that would allow miri to work with panic=abort (rust-lang/miri#1058)

  • merge this PR, solves problem right now
    • other use cases do not benefit from this as all other xargo users don't switch panic strategies
  • wait until cargo allows sharing build artifacts
    • we may be waiting for a while. This disadvantage is obviously easily solved by helping in the effort
  • just trash the xargo build libstd whenever the panic strategy changes
    • sub-option: use sccache to make this essentially free

@bors
Copy link
Contributor

bors commented Dec 23, 2019

☔ The latest upstream changes (presumably #67540) made this pull request unmergeable. Please resolve the merge conflicts.

@joelpalmer
Copy link

Ping from Triage: any updates @Aaron1011?

@Aaron1011
Copy link
Member Author

@joelpalmer: This is blocked on a decision from @oli-obk, @ehuss, @alexcrichton and others on whether this is the right approach.

@JohnTitor JohnTitor added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 19, 2020
@RalfJung
Copy link
Member

@Aaron1011 does the fact that crates compiled for Miri no longer are codegen'd help with this?

@Aaron1011
Copy link
Member Author

@RalfJung: This PR addresses a different issue - the metadata that we write out for libpanic_unwind depends on the right flag being magically passed in from bootstrap. This not affected by whether or not we run codegen.

@Dylan-DPC-zz Dylan-DPC-zz added S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). and removed S-blocked Status: Blocked on something else such as an RFC or other implementation work. labels Feb 25, 2020
@oli-obk
Copy link
Contributor

oli-obk commented Mar 18, 2020

The current status is

  • @alexcrichton says this is not the right approach
  • @ehuss also seems hesitant about this approach
  • I gave a selection of approaches
    1. merge this PR, solves problem right now
      • other use cases do not benefit from this as all other xargo users don't switch panic strategies
    2. wait until cargo allows sharing build artifacts
      • we may be waiting for a while. This disadvantage is obviously easily solved by helping in the effort
    3. just trash the xargo build libstd whenever the panic strategy changes
      • sub-option: use sccache to make this essentially free

So, my suggestion is to

a) close this PR
b) start with option iii
c) work on option ii

@RalfJung
Copy link
Member

I agree leaving this PR open will not lead anywhere. Thus, let's close it.

The problem of supporting panic=abort in Miri is tracked at rust-lang/miri#1058.

@RalfJung RalfJung closed this Mar 18, 2020
@apiraino apiraino removed the S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). label May 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.