FreeBSD segfault when capturing backtrace with RUST_BACKTRACE=1

[asomers]

Problem

With recent nightly toolchains, cargo segfaults on FreeBSD x86_64. The 2024-10-17 toolchain worked fine, but the bug was present in the 2024-10-24 toolchain . The segfault is dependent on the RUST_BACKTRACE=1 environment variable being set. This bug is causing rust-lang/libc's CI to fail.

An example CI failure:
https://github.com/rust-lang/libc/runs/32002285710

The stack trace of the segfault:

Program received signal SIGSEGV, Segmentation fault.
Address not mapped to object.
libunwind::CFI_Parser<libunwind::LocalAddressSpace>::parseCIE (addressSpace=..., cie=18446744073501904251, cieInfo=0x7fffffffacb0)
    at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/DwarfParser.hpp:332
332	  pint_t cieLength = (pint_t)addressSpace.get32(p);
(gdb) bt
#0  libunwind::CFI_Parser<libunwind::LocalAddressSpace>::parseCIE (addressSpace=..., cie=18446744073501904251, cieInfo=0x7fffffffacb0)
    at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/DwarfParser.hpp:332
rust-lang/cargo#1  0x0000000802b4ea0c in libunwind::CFI_Parser<libunwind::LocalAddressSpace>::findFDE (addressSpace=..., pc=pc@entry=19184556, ehSectionStart=41958640, sectionLength=<optimized out>, 
    fdeHint=fdeHint@entry=0, fdeInfo=fdeInfo@entry=0x7ffffffface8, cieInfo=0x7fffffffacb0) at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/DwarfParser.hpp:264
rust-lang/cargo#2  0x0000000802b4e6a2 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::getInfoFromDwarfSection (this=this@entry=0x7fffffffb330, pc=pc@entry=19184556, 
    sects=..., fdeSectionOffsetHint=fdeSectionOffsetHint@entry=0) at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/UnwindCursor.hpp:1689
rust-lang/cargo#3  0x0000000802b4bb60 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister (this=0x7fffffffb330, isReturnAddress=true)
    at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/UnwindCursor.hpp:2634
rust-lang/cargo#4  0x0000000802b4b9b2 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::step (this=0x7fffffffb330, stage2=<optimized out>)
    at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/UnwindCursor.hpp:2946
rust-lang/cargo#5  0x0000000802b48ef9 in _Unwind_Backtrace (callback=0x25720d0 <std::backtrace_rs::backtrace::libunwind::trace::trace_fn>, ref=0x7fffffffb4d8)
    at /usr/home/somers/src/freebsd.org/src/contrib/llvm-project/libunwind/src/UnwindLevel1-gcc-ext.c:156
rust-lang/cargo#6  0x0000000002558fd5 in std::backtrace_rs::backtrace::libunwind::trace () at std/src/../../backtrace/src/backtrace/libunwind.rs:116
rust-lang/cargo#7  std::backtrace_rs::backtrace::trace_unsynchronized<std::backtrace::{impl#4}::create::{closure_env#0}> () at std/src/../../backtrace/src/backtrace/mod.rs:66
rust-lang/cargo#8  std::backtrace::Backtrace::create () at std/src/backtrace.rs:331
rust-lang/cargo#9  0x0000000002558e40 in std::backtrace::Backtrace::capture () at std/src/backtrace.rs:296
rust-lang/cargo#10 0x00000000011d5c08 in <anyhow::Error as core::convert::From<clap_builder::error::Error>>::from ()
rust-lang/cargo#11 0x00000000018a1f26 in <cargo::util::errors::CliError as core::convert::From<clap_builder::error::Error>>::from ()
rust-lang/cargo#12 0x00000000012de941 in cargo::cli::main ()
rust-lang/cargo#13 0x00000000012d14e4 in cargo::main ()
rust-lang/cargo#14 0x00000000012c98e3 in std::sys::backtrace::__rust_begin_short_backtrace::<fn(), ()> ()
rust-lang/cargo#15 0x00000000012eebf9 in std::rt::lang_start::<()>::{closure#0} ()
rust-lang/cargo#16 0x0000000002558047 in core::ops::function::impls::{impl#2}::call_once<(), (dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)>
    () at core/src/ops/function.rs:284
rust-lang/cargo#17 std::panicking::try::do_call<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> () at std/src/panicking.rs:557
rust-lang/cargo#18 std::panicking::try<i32, &(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe)> () at std/src/panicking.rs:520
rust-lang/cargo#19 std::panic::catch_unwind<&(dyn core::ops::function::Fn<(), Output=i32> + core::marker::Sync + core::panic::unwind_safe::RefUnwindSafe), i32> () at std/src/panic.rs:358
rust-lang/cargo#20 std::rt::lang_start_internal::{closure#1} () at std/src/rt.rs:174
rust-lang/cargo#21 std::panicking::try::do_call<std::rt::lang_start_internal::{closure_env#1}, isize> () at std/src/panicking.rs:557
rust-lang/cargo#22 std::panicking::try<isize, std::rt::lang_start_internal::{closure_env#1}> () at std/src/panicking.rs:520
rust-lang/cargo#23 std::panic::catch_unwind<std::rt::lang_start_internal::{closure_env#1}, isize> () at std/src/panic.rs:358
rust-lang/cargo#24 std::rt::lang_start_internal () at std/src/rt.rs:174
rust-lang/cargo#25 0x00000000012d58d5 in main ()

Steps

In a FreeBSD x86_64 environment with the latest nightly toolchain installed:

1. git clone git@github.com:rust-lang/libc.git
2. cd libc
3. env RUST_BACKTRACE=1 cargo test --features extra_traits --manifest-path libc-test/Cargo.toml --target x86_64-unknown-freebsd

Possible Solution(s)

No response

Notes

No response

Version

> cargo version --verbose
cargo 1.84.0-nightly (cf53cc54b 2024-10-18)
release: 1.84.0-nightly
commit-hash: cf53cc54bb593b5ec3dc2be4b1702f50c36d24d5
commit-date: 2024-10-18
host: x86_64-unknown-freebsd
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: FreeBSD 15.0-CURRENT [64-bit]

[weihanglo]

Thanks for the report. rust-lang/cargo#14712 might be related. Do you happen to know the RAM size of the CI machine?

[asomers]

Thanks for the report. #14712 might be related. Do you happen to know the RAM size of the CI machine?

I don't. But I can also reproduce the problem locally with 64 GB of RAM.

[weihanglo]

Cargo on nightly hasn't changed since nightly-2024-10-20. It is on the commit cf53cc54bb593b5ec3dc2be4b1702f50c36d24d5.

The last submodule update is #131910.

I can't tell which PR is relevant to this backtrace 😞.

rust-lang/cargo#10 0x00000000011d5c08 in <anyhow::Error as core::convert::From<clap_builder::error::Error>>::from ()
rust-lang/cargo#11 0x00000000018a1f26 in <cargo::util::errors::CliError as core::convert::From<clap_builder::error::Error>>::from ()
rust-lang/cargo#12 0x00000000012de941 in cargo::cli::main ()
rust-lang/cargo#13 0x00000000012d14e4 in cargo::main ()

[weihanglo]

Thanks for the report. #14712 might be related. Do you happen to know the RAM size of the CI machine?

I don't. But I can also reproduce the problem locally with 64 GB of RAM.

If you are able to reproduce locally, would you mind building rust-lang/cargo@8c30ce5 cargo and running with rustc from 2024-10-24? Setting RUSTC=~/.rustup/toolchains/x86_64-unknown-freebsd/bin/rustc env var should help choose the correct compiler.

[asomers]

I've managed to bisect the problem. It was introduced by 6d88158 . I don't know why yet.

[asomers]

@linyihai I think you actually ran into the other bug: rust-lang/cargo#14712 , judging by the "memory allocation ... bytes failed" message.

[ehuss]

Transferred to rust-lang/rust since this is not a cargo issue. It reproduces with a fairly basic program like:

fn main() {
std::backtrace::Backtrace::capture();
}

I bisected to #120869. cc @devnexen

[asomers]

@ehuss what do you mean by "backtraced" as a verb?

[ehuss]

Sorry, that was a mis-type. I meant "bisected" (via cargo-bisect-rustc).

[devnexen]

This also makes the binaries from previous FreeBSD versions incompatible with 13.0-RELEASE

@asomers could it be some ABI incompatibility with backtrace for example ?

[devnexen]

So I can reproduce the segfault ; however I can also reproduce the segfault with the stable 1.82 version from rustup.

[ehuss]

@devnexen Just to confirm, what were your reproduction steps?

I have a virtual machine with FreeBSD 14.1 (amd64). The reproduction for me is:

1. cargo new foo
2. cd foo
3. RUST_BACKTRACE=1 cargo +stable build
4. RUST_BACKTRACE=1 cargo +nightly build

With +stable I get a normal build, with nightly it core dumps (libunwind::LocalAddressSpace::get32).

[devnexen]

nvm I ran the wrong rustc binary, yes it works with stable..

[asomers]

I can no longer reproduce this issue when using a locally built toolchain on FreeBSD 15. I could before, but can't now, which suggests that I was probably doing something wrong before.

I did find a related issue in LLVM: llvm/llvm-project#76957 . That issue can cause crashes of the sort we're seeing. Crucially, it affect the version of LLVM included in FreeBSD 13.2 and 13.3, but not 13.4. Note that the fact that Rust bundles its own version of LLVM is irrelevant, because in this stack trace we're accessing LLVM via libunwind, not directly from Rust. So perhaps #120869 caused the official build to link to a buggy version of LLVM. If so, the solution would be to update the build image to FreeBSD 13.4. HOWEVER, that could potentially break the rustup-distributed toolchain for users of FreeBSD 13.3, which is officially supported until 31-Dec of this year. So it would be better to revert PR #120869 for now, but reapply it (using FreeBSD 13.4 as the image instead of 13.2) in early 2025.

But that's just my theory. To test it, we would need to build a set of x86_64 FreeBSD artifacts through Rust's CI pipeline. Is there a way to do that without merging a PR?

[ehuss]

Is there a way to do that without merging a PR?

Open a PR that updates to 13.4, in the PR description, include a line that says try-job: dist-x86_64-freebsd, and ask someone with permissions to do a try build (I can do that). You can then download the artifacts.

[workingjubilee]

I thought that the string to use was try-job?

[ehuss]

Yep, edited my comment.

[asomers]

Can we please reopen this issue? PR #132228 wasn't a permanent solution.