Apple Silicon: Exiting with SIGKILL when using custom build script

[p-kuen]

Problem
When using cargo check using the new apple silicon toolchain on any package with a build.rs file it ends with a "SIGKILL" abort:

patrickkuen@mbp-von-patrick test_library % cargo check
   Compiling test_library v0.1.0 (/Users/patrickkuen/Developer/personal/test_library)
error: failed to run custom build command for `test_library v0.1.0 (/Users/patrickkuen/Developer/personal/test_library)`

Caused by:
  process didn't exit successfully: `/Users/patrickkuen/Developer/personal/test_library/target/debug/build/test_library-deff025c5c03f0d3/build-script-build` (signal: 9, SIGKILL: kill)

rustup-toolchain:

beta-aarch64-apple-darwin (default)
rustc 1.49.0-beta.2 (bd26e4e54 2020-11-24)

Steps

1. Create a package using cargo new testlib --lib
2. Create a build.rs file with a simple main function in it
3. Run cargo check

Possible Solution(s)
When removing the file, the error appears, so it seems like the build.rs causes the sigkill. Maybe the main function expect any specific return?

Notes

Output of cargo version:
cargo 1.49.0-beta (b921683 2020-11-24)

[alexcrichton]

Is this perhaps related to Apple's latest changes around running untrusted executables? I'm not sure if there's anything Cargo can do about this since Cargo itself isn't sending SIGKILL or anything like that. I'm not entirely sure how toolchain-created-binaries are to be trusted on the new macOS...

[p-kuen]

Okay could be possible, but the only strange thing is that the SIGKILL only appears when using the build.rs file. When the binary would be untrusted, no build in general would be possible, not?

[alexcrichton]

Can you compile and run a simple program with just rustc? Something like:

$ cat foo.rs 
fn main() {}
$ rustc foo.rs
$ ./foo

or does that also die with SIGKILL?

[ehuss]

I believe others are having similar issues, see https://rust-lang.zulipchat.com/#narrow/stream/242906-t-compiler.2Farm/topic/macOS/near/213374421 and https://rust-lang.zulipchat.com/#narrow/stream/242906-t-compiler.2Farm/topic/Native.20aarch64-apple-darwin.20install.20trouble/near/217656988 and https://stackoverflow.com/questions/64830671/why-does-my-native-arm64-application-built-using-an-x86-64-build-system-fail-to.

I would check the Console app to see if there are any signature verification errors in the logs. Maybe also run with -vv to see if the build script has a chance to print anything (is your build script running clang or anything else?).

I don't think any of us have access to an arm64 mac, so it may be difficult for us to help. People working on that target are mostly hanging out on the #t-compiler/arm stream on Zulip, so you may get some more insight there.

[p-kuen]

Can you compile and run a simple program with just rustc? Something like:

$ cat foo.rs 
fn main() {}
$ rustc foo.rs
$ ./foo

or does that also die with SIGKILL?

No this works fine. As I said it also works fine when using cargo. Only when using a build.rs file for compilation, the SIGKILL appears.

Maybe also run with -vv

I tried this several times but the compiler does not print any useful data. It seems like it not even executes the main function inside build.rs.

I would check the Console app to see if there are any signature verification errors in the logs.

No, sadly there isn't any output.

See the following test:

patrickkuen@mbp-von-patrick test_library % ls -al
total 32
drwxr-xr-x   9 patrickkuen  staff  288 30 Nov 20:49 .
drwxr-xr-x   6 patrickkuen  staff  192 30 Nov 20:44 ..
drwxr-xr-x  10 patrickkuen  staff  320 29 Nov 10:54 .git
-rw-r--r--   1 patrickkuen  staff   19 29 Nov 10:53 .gitignore
-rw-r--r--   1 patrickkuen  staff  838 30 Nov 20:49 Cargo.lock
-rw-r--r--   1 patrickkuen  staff  506 29 Nov 10:56 Cargo.toml
-rw-r--r--   1 patrickkuen  staff   35 29 Nov 14:37 build.rs
drwxr-xr-x   3 patrickkuen  staff   96 29 Nov 10:53 src
drwxr-xr-x@  5 patrickkuen  staff  160 30 Nov 20:49 target
patrickkuen@mbp-von-patrick test_library % cargo build -vv
       Fresh unicode-xid v0.2.1
   Compiling test_library v0.1.0 (/Users/patrickkuen/Developer/personal/test_library)
     Running `/Users/patrickkuen/Developer/personal/test_library/target/debug/build/test_library-deff025c5c03f0d3/build-script-build`
error: failed to run custom build command for `test_library v0.1.0 (/Users/patrickkuen/Developer/personal/test_library)`

Caused by:
  process didn't exit successfully: `/Users/patrickkuen/Developer/personal/test_library/target/debug/build/test_library-deff025c5c03f0d3/build-script-build` (signal: 9, SIGKILL: kill)
patrickkuen@mbp-von-patrick test_library % cat build.rs   
fn main() {
    println!("test");
}%
patrickkuen@mbp-von-patrick test_library % rm build.rs
patrickkuen@mbp-von-patrick test_library % cargo build    
   Compiling test_library v0.1.0 (/Users/patrickkuen/Developer/personal/test_library)
    Finished dev [unoptimized + debuginfo] target(s) in 0.10s
patrickkuen@mbp-von-patrick test_library % 

[p-kuen]

I just found out another strange behaviour.

When using cargo build on a project with a build.rs in it, it throws the SIGKILL as we know. When rebooting the mac afterwords and running cargo build again it compiles successfully until removing the target folder.

[alexcrichton]

When you get SIGKILL, if you go to System Preferences under "Security & Privacy", is there anything there that's asking you to allow the binary in one way or another?

Similarly if you build a Rust program where the program spawns rustc build.rs, can you run that resulting binary? (it seems like you can run rustc and run the final binary, but I'm curious if you add a layer of indirection in there if it matters).

[p-kuen]

When you get SIGKILL, if you go to System Preferences under "Security & Privacy", is there anything there that's asking you to allow the binary in one way or another?

No sadly there is no option to allow anything. I know this message from downloading applications but in this case there is nothing to allow.

Similarly if you build a Rust program where the program spawns rustc build.rs, can you run that resulting binary? (it seems like you can run rustc and run the final binary, but I'm curious if you add a layer of indirection in there if it matters).

I'm not sure if you meant doing the following, but this also works:

patrickkuen@mbp-von-patrick test_library % rustc build.rs 
patrickkuen@mbp-von-patrick test_library % ./build 
test

I also tried the following:

patrickkuen@mbp-von-patrick test_library % cat build_spawner.rs   
use std::process::Command;

fn main() {
    Command::new("rustc")
        .arg("build.rs")
        .spawn()
        .expect("rustc command failed to start");
}%
patrickkuen@mbp-von-patrick test_library % rustc build_spawner.rs && ./build_spawner
patrickkuen@mbp-von-patrick test_library % ./build
test

After googling a bit, could this issue be of any help? neovim/neovim#13399

[alexcrichton]

Hm interesting! To confirm, what build targets are you working with? Are you running through the x86_64 emulation or native aarch64? Maybe the bug is with one or the other? (for example if you have an emulated executable produce an emulated executable, does that work? Or for that matter, do all permutations of executable producing executabale work or do some fail?)

[p-kuen]

I worked with the beta-aarch64-apple-darwin on a M1 Macbook Pro, so there would be no emulation. Yesterday I tried using the x86_64 toolchain, this works fine (probably it uses Rosetta?).

[alexcrichton]

So to confirm, using a native aarch64 toolchain produces these SIGKILL warnings while using the x86_64 toolchain does not? Can you test that at the command line? Does a bare rustc-produced executable fail/succeed depending on the architecture?

[ehuss]

As noted in https://rust-lang.zulipchat.com/#narrow/stream/242906-t-compiler.2Farm/topic/macOS.20build.20scripts.20and.20SIGKILL/near/218453006, you need to be extra careful to ensure the entire toolchain is arm64. Most notably, the rustup wrappers. Check file $(which rustup) and file $(which cargo) and file $(which rustc) to make sure there isn't an x86_64 version of rustup in the mix.

[kubkon]

If I can chime in here, did you check if this is not the result of the kernel caching the inodes? This results in similar behaviour where any change to the already signed and cached binary by the kernel will end up in an immediate SIGKILL. In Zig, to circumvent that when we do incremental linking (by modifying the existing binary in place), in addition to re-adding a valid code signature, it is also necessary to copy-rename the resultant binary so that the kernel doesn't see it as the already cached inode.

Another instance of this issue manifesting I've had when patching the output of LLD in Zig toolchain so that we can compute and append valid code signatures without requiring the toolchain to fall back to the system linker. In this case, after LLD generates the binary without the code signature, I open the binary for writing, calculate and embed an adhoc code signature, and then do the copy-rename dance on the binary. Here's the latest PR describing this in Zig: ziglang/zig#7273.

[p-kuen]

As noted in https://rust-lang.zulipchat.com/#narrow/stream/242906-t-compiler.2Farm/topic/macOS.20build.20scripts.20and.20SIGKILL/near/218453006, you need to be extra careful to ensure the entire toolchain is arm64. Most notably, the rustup wrappers. Check file $(which rustup) and file $(which cargo) and file $(which rustc) to make sure there isn't an x86_64 version of rustup in the mix.

You are right. I installed rustup-init with brew, but as the arm brew is still very unstable, I used the x86_64 brew over rosetta. Therefore rustup is also an x86_64 version. So you think that's the problem and the cause for the SIGKILL?

Sadly I don't really want to install brew for arm yet, so I fear I have to wait until it gets more stable to make rust for arm work.

[p-kuen]

I uninstalled rustup-init from brew and installed rustup the common way. I have the arm version of rustup now and the build of the test library completed successfully.
Therefore the solution is to have a complete arm toolchain (rustup, cargo and rustc) like @ehuss suggested.

It would be good if this case could be handled with an error message, but I will close this issue for now.
Thank you for your help!

[aircloud]

I still have this problem...
It looks like my rustc version is correct

file $(which rustup)
/Users/ad/.cargo/bin/rustup: Mach-O 64-bit executable arm64
file $(which cargo)
/Users/ad/.cargo/bin/cargo: Mach-O 64-bit executable arm64
file $(which rustc)
/Users/ad/.cargo/bin/rustc: Mach-O 64-bit executable arm64
rustc --version
rustc 1.50.0-nightly (6645da366 2020-12-01)

[sebboer]

@aircloud what terminal emulator are you using? running from alacritty under rosetta as x86 it's exiting with SIGKILL. running with macOs default terminal it completes the build.

[aircloud]

@aircloud what terminal emulator are you using? running from alacritty under rosetta as x86 it's exiting with SIGKILL. running with macOs default terminal it completes the build.

I tried macOS terminal(default in the system)
I found that sometimes cargo build is ok, but cargo test is not

error: failed to run custom build command for `pin-project-internal v0.4.27`

Caused by:
  process didn't exit successfully: `/Users/ad/work/my_work/target/debug/build/pin-project-internal-4a6a41908bd1b30e/build-script-build` (signal: 9, SIGKILL: kill)
warning: build failed, waiting for other jobs to finish...
error: build failed
child_process.js:656
    throw err;
    ^

Error: Command failed: cargo test

[frewsxcv]

I am also hitting this. I'm using Terminal.app and the latest Rust nightly.

coreyf@Coreys-MacBook-Air rgis % file $(which rustup)
/Users/coreyf/.cargo/bin/rustup: Mach-O 64-bit executable arm64
coreyf@Coreys-MacBook-Air rgis % file $(which cargo)
/Users/coreyf/.cargo/bin/cargo: Mach-O 64-bit executable arm64
coreyf@Coreys-MacBook-Air rgis % file $(which rustc)
/Users/coreyf/.cargo/bin/rustc: Mach-O 64-bit executable arm64
coreyf@Coreys-MacBook-Air rgis % rustc --version
rustc 1.50.0-nightly (0f6f2d681 2020-12-06)

[frewsxcv]

I ran the build script with lldb and here's the output, in case it's helpful:

lldb /Users/coreyf/tmp/log/target/debug/build/log-60446433c6d1ecd0/build-script-build
(lldb) target create "/Users/coreyf/tmp/log/target/debug/build/log-60446433c6d1ecd0/build-script-build"
Current executable set to '/Users/coreyf/tmp/log/target/debug/build/log-60446433c6d1ecd0/build-script-build' (arm64).
(lldb) run
error: process exited with status -1 (attach failed ((os/kern) invalid argument))

[frewsxcv]

Ah I see this is the tracking issue for MacBook M1, which has a workaround for the issue:

• Tracking issue for supporting macOS on Apple Silicon (a.k.a arm64, M1, M2, aarch64) rust#73908

Sorry for the noise!

[gcp]

This problem drove me nuts. The underlying issue was that I was running a shell from homebrew (bash), which was running under emulation, and confusing the architecture detection of everything that was running underneath.

arch -arm64e sh and going back to the default shells fixed this.

[mattrltrent]

Same issue. Tried a billion fixes, nothing worked. Ended up just re-running everything in Bash instead of Apple's native zsh and everything worked fine.