Illegal instruction when running on Raspberry Pi when building with RUSTFLAGS containing -C target_cpu=cortex-a72

[jupyterkat]

NOTE: this is compiled with -target-cpu=native and --release, and target triple stable-aarch64-unknown-linux-gnu. But it fails with default target-cpu anyways

backtrace:

Thread 1 "musicalcat" received signal SIGILL, Illegal instruction.
ring_core_0_17_7_aes_hw_set_encrypt_key ()
    at /home/katbot/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ring-0.17.7/pregenerated/aesv8-armx-linux64.S:122
122             aese    v6.16b,v0.16b
(gdb) bt full
#0  ring_core_0_17_7_aes_hw_set_encrypt_key ()
    at /home/katbot/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ring-0.17.7/pregenerated/aesv8-armx-linux64.S:122
No locals.
#1  0x0000005555f71bec in ring::aead::aes_gcm::init_256 ()
No symbol table info available.
#2  0x0000005555f6f72c in <ring::aead::unbound_key::UnboundKey as core::convert::From<ring::hkdf::Okm<&ring::aead::Algorithm>>>::from ()
No symbol table info available.
#3  0x0000005555f4e71c in rustls::tls13::key_schedule::KeySchedule::derive_decrypter::he91ce5cdc238f2b2 ()
No symbol table info available.
#4  0x0000005555f4db68 in rustls::tls13::key_schedule::KeyScheduleHandshakeStart::derive_client_handshake_secrets ()
No symbol table info available.
#5  0x0000005555f56a08 in rustls::client::tls13::handle_server_hello ()
No symbol table info available.
#6  0x0000005555f404a0 in <rustls::client::hs::ExpectServerHello as rustls::common_state::State<rustls::client::client_conn::ClientConnectionData>>::handle ()
No symbol table info available.
#7  0x0000005555f409f0 in <rustls::client::hs::ExpectServerHelloOrHelloRetryRequest as rustls::common_state::State<rustls::client::client_conn::ClientConnectionData>>::handle ()
No symbol table info available.
#8  0x0000005555eb7950 in rustls::common_state::CommonState::process_main_protocol ()

lscpu:

Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
Vendor ID:                       ARM
Model:                           3
Model name:                      Cortex-A72
Stepping:                        r0p3
CPU max MHz:                     1500.0000
CPU min MHz:                     600.0000
BogoMIPS:                        108.00
L1d cache:                       128 KiB
L1i cache:                       192 KiB
L2 cache:                        1 MiB
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Vulnerable
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm crc32 cpuid

[briansmith]

@jupyterkat Thanks for the bug report. Which operating system (kernel version and distribution version) are you using?

[jupyterkat]

It's kernel 6.1.21-v8+ on debian 11

[briansmith]

@jupyterkat Could you help with some additional info:

• What is the exact cargo/rustc version you are using (cargo --version / rustc --version)? Is this the standard toolchain you get from rustup.sh or a custom one?
• What is the output of rustc --target=aarch64-unknown-linux-gnu --print cfg using the rustc with which you are experiencing this issue?
• What hardware is this? Cortex-A72 usually does have AES enabled but not for Raspberry Pi and a few others.
• Do you have any idea how I could emulate this configuration using QEMU?

[jupyterkat]

• 1.74.1, standard toolchain from rustup.sh
• print cfg returns below

debug_assertions
panic="unwind"
target_arch="aarch64"
target_endian="little"
target_env="gnu"
target_family="unix"
target_feature="neon"
target_has_atomic="128"
target_has_atomic="16"
target_has_atomic="32"
target_has_atomic="64"
target_has_atomic="8"
target_has_atomic="ptr"
target_os="linux"
target_pointer_width="64"
target_vendor="unknown"
unix

• This is a Raspberry Pi 4 Model B Rev 1.1 with a BCM2835

[jupyterkat]

not sure about emulating raspi4, qemu should already support it, or there is a patch to that.

[kryptan]

I also get illegal instruction on Raspberry Pi 4 but only when using target-cpu=native.

Code:

use ring::signature::{EcdsaKeyPair, ECDSA_P256_SHA256_ASN1_SIGNING};

fn main() {
    let rng = ring::rand::SystemRandom::new();
    let pkcs8 = EcdsaKeyPair::generate_pkcs8(&ECDSA_P256_SHA256_ASN1_SIGNING, &rng).unwrap();
    EcdsaKeyPair::from_pkcs8(&ECDSA_P256_SHA256_ASN1_SIGNING, pkcs8.as_ref(), &rng).unwrap();
}

Run with RUSTFLAGS='-C target-cpu=native' cargo run --release

Result from gdb:

Program terminated with signal SIGILL, Illegal instruction.
#0  sha256_block_armv8 () at /home/pi/.cargo/registry/src/index.crates.io-6f17d22bba15001f/ring-0.17.7/pregenerated/sha256-armv8-linux64.S:1087
1087    .inst   0x5e2828a4      //sha256su0 v4.16b,v5.16b

pi@raspberrypi:~/dev/server $ rustc --version
rustc 1.77.0-nightly (fb5ed726f 2023-12-28)

pi@raspberrypi:~/dev/server $ rustc --target=aarch64-unknown-linux-gnu --print cfg
debug_assertions
overflow_checks
panic="unwind"
relocation_model="pic"
target_abi=""
target_arch="aarch64"
target_endian="little"
target_env="gnu"
target_family="unix"
target_feature="neon"
target_has_atomic
target_has_atomic="128"
target_has_atomic="16"
target_has_atomic="32"
target_has_atomic="64"
target_has_atomic="8"
target_has_atomic="ptr"
target_has_atomic_equal_alignment="128"
target_has_atomic_equal_alignment="16"
target_has_atomic_equal_alignment="32"
target_has_atomic_equal_alignment="64"
target_has_atomic_equal_alignment="8"
target_has_atomic_equal_alignment="ptr"
target_has_atomic_load_store
target_has_atomic_load_store="128"
target_has_atomic_load_store="16"
target_has_atomic_load_store="32"
target_has_atomic_load_store="64"
target_has_atomic_load_store="8"
target_has_atomic_load_store="ptr"
target_os="linux"
target_pointer_width="64"
target_thread_local
target_vendor="unknown"
unix

pi@raspberrypi:~/dev/server $ cat /proc/cpuinfo
processor       : 0
BogoMIPS        : 108.00
Features        : fp asimd evtstrm crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd08
CPU revision    : 3

...

Model           : Raspberry Pi 4 Model B Rev 1.4

Operating System: Debian GNU/Linux 12 (bookworm)
Kernel: Linux 6.1.0-rpi7-rpi-v8

[briansmith]

I also get illegal instruction on Raspberry Pi 4 but only when using target-cpu=native.

Thank you for solving the problem. target-cpu=native seems to do the wrong thing on Raspberry Pi on a Cortex-A72 CPU:

$ rustc --print cfg --target=aarch64-unknown-linux-gnu -C target_cpu=cortex-a72 | grep feature
target_feature="aes"
target_feature="crc"
target_feature="neon"
target_feature="pmuv3"
target_feature="sha2"

Without -C target_cpu=cortex-a72 we get the correct feature flags:

$ rustc --print cfg --target=aarch64-unknown-linux-gnu | grep feature
target_feature="neon"

This is a bug in rustc/LLVM or user error.

[briansmith]

The LLVM bug is llvm/llvm-project#90365.

[briansmith]

Coincidentally, for aarch64-apple targets, we always force dynamic detection of SHA512, and coincidentally PR #2040 generalizes that mechanism to allow us to force dynamic detection of any set of features on a per-OS basis. One way we could work around this issue with the compiler would be to merge PR #2040, then change the src/cpu/arm/linux.rs to have const FORCE_DYNAMIC_DETECTION: u32 = !NEON.mask; (or whatever works) to force all features except NEON to be dynamically detected for Linux and Android. If we were to do that then we'd need to document in a comment above that definition why we are doing this.

[briansmith]

PR #2050 implements the aforementioned workaround for this. PTAL.

[briansmith]

Related issues:

• Similar issue reported against BoringSSL: https://bugs.chromium.org/p/boringssl/issues/detail?id=460 ...
• ...which triggered https://bugzilla.yoctoproject.org/show_bug.cgi?id=14641
• ...and raspberrypi4-64 assumes crypto extension tuning, but CPU does not provide it agherzan/meta-raspberrypi#964

[briansmith]

PR #2050 has been merged. It would be great if somebody could confirm that this fixes the issue for them.

[briansmith]

Fixed by #2050 (and #2051).