Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"memory allocation of .. bytes failed" appears when using cargo run test to run specific test name #14712

Closed
chentodd opened this issue Oct 21, 2024 · 10 comments · Fixed by #14751
Assignees
Labels
A-dep-info Area: dep-info, .d files C-bug Category: bug regression-from-stable-to-nightly Regression in nightly that previously worked in stable. S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted. Z-checksum-freshness Nightly: rebuild detection on file checksum instead of mtime

Comments

@chentodd
Copy link
Contributor

Problem

cargo run test shows "memory allocation of .. bytes failed" when I run specific test case:

Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.20s
     Running `target/debug/cargo test source_replacement`
memory allocation of 63063237072 bytes failed
Aborted (core dumped)

I expect there are no issues when I use cargo run test to run the tests.

Steps

  1. Run rustup update to update needed tools to build cargo
  2. Clone cargo repository
  3. cd into cargo repository
  4. Change to master branch
  5. Run cargo check --workspace
  6. Run cargo build
  7. Run cargo run test source_replacement

here is the test video that contains step.5 to step.7 for the reference: 20241021_cargo_test_video.zip

Possible Solution(s)

No response

Notes

The error message in this issue is similar to #7543

Version

cargo 1.84.0 (edd36eb 2024-10-21)
release: 1.84.0
commit-hash: edd36eb
commit-date: 2024-10-21
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.5.0 (sys:0.4.74+curl-8.9.0 system ssl:OpenSSL/3.0.13)
os: Ubuntu 24.4.0 (noble) [64-bit]

additional hardware information of test machine:
Memory: 16 GB

@chentodd chentodd added C-bug Category: bug S-triage Status: This issue is waiting on initial triage. labels Oct 21, 2024
@weihanglo weihanglo added the A-testing-cargo-itself Area: cargo's tests label Oct 21, 2024
@epage
Copy link
Contributor

epage commented Oct 22, 2024

I wasn't able to reproduce. Unsure what the difference is.

@weihanglo
Copy link
Member

The RAM size is critical.

additional hardware information of test machine:
Memory: 16 GB

I reproduced it on a machine with 4 GiB RAM.

(24-10-24 17:12:24) <0> [~/cargo]  
%  cargo -vV
cargo 1.82.0 (8f40fc59f 2024-08-21)
release: 1.82.0
commit-hash: 8f40fc59fb0c8df91c97405785197f3c630304ea
commit-date: 2024-08-21
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Amazon Linux AMI 2.0.0 [64-bit]

(24-10-24 17:12:30) <0> [~/cargo]  
%  free -h
              total        used        free      shared  buff/cache   available
Mem:           3.6G        520M        1.6G        1.7M        1.5G        2.8G
Swap:            0B          0B          0B

(24-10-24 17:12:35) <0> [~/cargo]  
%  cargo run -F all-static -- test -F all-static source_replacement
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.23s
     Running `target/debug/cargo test -F all-static source_replacement`
memory allocation of 63063237072 bytes failed
zsh: abort      cargo run -F all-static -- test -F all-static source_replacement

Testing other combinations:

  • cargo test -F all-static source_replacement
  • cargo build -F all-static source_replacement
  • ./target/debug/cargo build -F all-static
  • ./target/release/cargo build -F all-static
  • ./target/debug/cargo test -F all-static
  • ./target/release/cargo test -F all-static
  • cp ./target/debug/cargo dargo && cargo clean && ./dargo test -F all-static
  • cp ./target/release/cargo rargo && cargo clean && ./rargo test -F all-static
  • cp ./target/debug/cargo dargo && ./dargo build -F all-static
  • cp ./target/debug/cargo dargo && ./dargo build -F all-static
  • cp ./target/debug/cargo dargo && ./dargo test -F all-static
  • cp ./target/debug/cargo dargo && ./dargo test -F all-static

So it looks like the requirements are

  • With existing build cache
  • Using the newly built cargo binary
  • Running cargo test

to trigger a core dump.

I may investigate it more when having time. At least we need to be sure that it is really an edge case for cargo itself.

@weihanglo weihanglo added S-needs-info Status: Needs more info, such as a reproduction or more background for a feature request. P-low Priority: Low and removed S-triage Status: This issue is waiting on initial triage. labels Oct 24, 2024
@weihanglo
Copy link
Member

Did a cargo t against rust-lang/crates.io with both debug/release built cargo from master. It works alright with 4GiB RAM (c5.large AWS EC2 instance). So, probably this is probably a Cargo-specific issue.

Labeled as low priority though feel free to help debug it!

@AlexTMjugador
Copy link
Contributor

AlexTMjugador commented Oct 26, 2024

I got hit by this bug when running cargo check on this commit (cargo 1.84.0-nightly (cf53cc54b 2024-10-18), rustc 1.84.0-nightly (c1db4dc24 2024-10-25)). My box is also x86_64-unknown-linux-gnu, but it has 32 GiB of RAM, so throwing more RAM at the problem does not help either. I didn't have to set RUST_BACKTRACE to any value.

Edit: doing a cargo clean fixed it. Looks like doing a rustup update while having an older Rust version build cache is troublesome, is this a supported use case or does it happen to mostly work in practice in spite of it not being supported? For what it's worth, it's not the first time I do a rustup update with an already existing build cache.

@weihanglo weihanglo removed the P-low Priority: Low label Oct 26, 2024
@linyihai
Copy link
Contributor

Edit: doing a cargo clean fixed it. Looks like doing a rustup update while having an older Rust version build cache is troublesome, is this a supported use case or does it happen to mostly work in practice in spite of it not being supported? For what it's worth, it's not the first time I do a rustup update with an already existing build cache.

I encountered this problem recently. And I did cargo clean and rustup update to workaround it.

@weihanglo
Copy link
Member

so throwing more RAM at the problem does not help either

It does help if you have more than 63063237072 bytes available in your RAM 😆.
I do have a machine with 128 GiB RAM and can't reproduce this issue on it.


Since there is a non Cargo project affected, I removed the P-low label.

@weihanglo weihanglo added the regression-from-stable-to-nightly Regression in nightly that previously worked in stable. label Oct 27, 2024
@weihanglo weihanglo self-assigned this Oct 27, 2024
@weihanglo
Copy link
Member

I compiled master Cargo (e75214e) with -Zoom=panic and got this backtrace.

RUSTC_BOOTSTRAP=1 RUSTFLAGS=-Zoom=panic cargo b -F all-static
RUST_BACKTRACE=1 ./target/debug/cargo t
Backtrace

thread 'main' panicked at std/src/alloc.rs:355:9:
memory allocation of 63063237072 bytes failed
stack backtrace:
   0: rust_begin_unwind
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/panicking.rs:74:14
   2: std::alloc::default_alloc_error_hook
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/std/src/alloc.rs:355:9
   3: std::alloc::rust_oom
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/std/src/alloc.rs:375:5
   4: __rg_oom
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/std/src/alloc.rs:371:1
   5: alloc::alloc::handle_alloc_error::rt_error
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/alloc.rs:403:13
   6: alloc::alloc::handle_alloc_error
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/alloc.rs:409:9
   7: alloc::raw_vec::handle_error
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/raw_vec.rs:793:38
   8: alloc::raw_vec::RawVecInner<A>::with_capacity_in
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/raw_vec.rs:431:25
   9: alloc::raw_vec::RawVec<T,A>::with_capacity_in
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/raw_vec.rs:194:20
  10: alloc::vec::Vec<T,A>::with_capacity_in
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/mod.rs:803:20
  11: alloc::vec::Vec<T>::with_capacity
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/mod.rs:483:9
  12: cargo::core::compiler::fingerprint::EncodedDepInfo::parse
             at ./src/cargo/core/compiler/fingerprint/mod.rs:2272:23
  13: cargo::core::compiler::fingerprint::parse_dep_info
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1928:22
  14: cargo::core::compiler::fingerprint::LocalFingerprint::find_stale_item
             at ./src/cargo/core/compiler/fingerprint/mod.rs:835:34
  15: cargo::core::compiler::fingerprint::Fingerprint::check_filesystem
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1236:33
  16: cargo::core::compiler::fingerprint::calculate
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1432:5
  17: cargo::core::compiler::fingerprint::DepFingerprint::new
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1315:27
  18: cargo::core::compiler::fingerprint::calculate_normal::{{closure}}
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1467:24
  19: core::iter::adapters::map::map_try_fold::{{closure}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/map.rs:95:28
  20: core::iter::adapters::filter::filter_try_fold::{{closure}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/filter.rs:86:44
  21: <alloc::vec::into_iter::IntoIter<T,A> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/into_iter.rs:346:25
  22: <core::iter::adapters::filter::Filter<I,P> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/filter.rs:151:9
  23: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/map.rs:121:9
  24: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/mod.rs:191:9
  25: <I as alloc::vec::in_place_collect::SpecInPlaceCollect<T,I>>::collect_in_place
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:380:13
  26: alloc::vec::in_place_collect::from_iter_in_place
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:271:9
  27: alloc::vec::in_place_collect::from_iter_in_place{{reify.shim}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:251:1
  28: alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:246:9
  29: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/mod.rs:3410:9
  30: core::iter::traits::iterator::Iterator::collect
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/traits/iterator.rs:2001:9
  31: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/result.rs:1979:51
  32: core::iter::adapters::try_process
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/mod.rs:160:17
  33: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/result.rs:1979:9
  34: core::iter::traits::iterator::Iterator::collect
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/traits/iterator.rs:2001:9
  35: cargo::core::compiler::fingerprint::calculate_normal
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1464:24
  36: cargo::core::compiler::fingerprint::calculate
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1425:9
  37: cargo::core::compiler::fingerprint::DepFingerprint::new
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1315:27
  38: cargo::core::compiler::fingerprint::calculate_normal::{{closure}}
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1467:24
  39: core::iter::adapters::map::map_try_fold::{{closure}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/map.rs:95:28
  40: core::iter::adapters::filter::filter_try_fold::{{closure}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/filter.rs:86:44
  41: <alloc::vec::into_iter::IntoIter<T,A> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/into_iter.rs:346:25
  42: <core::iter::adapters::filter::Filter<I,P> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/filter.rs:151:9
  43: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/map.rs:121:9
  44: <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/mod.rs:191:9
  45: <I as alloc::vec::in_place_collect::SpecInPlaceCollect<T,I>>::collect_in_place
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:380:13
  46: alloc::vec::in_place_collect::from_iter_in_place
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:271:9
  47: alloc::vec::in_place_collect::from_iter_in_place{{reify.shim}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:251:1
  48: alloc::vec::in_place_collect::<impl alloc::vec::spec_from_iter::SpecFromIter<T,I> for alloc::vec::Vec<T>>::from_iter
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/in_place_collect.rs:246:9
  49: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/alloc/src/vec/mod.rs:3410:9
  50: core::iter::traits::iterator::Iterator::collect
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/traits/iterator.rs:2001:9
  51: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter::{{closure}}
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/result.rs:1979:51
  52: core::iter::adapters::try_process
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/adapters/mod.rs:160:17
  53: <core::result::Result<V,E> as core::iter::traits::collect::FromIterator<core::result::Result<A,E>>>::from_iter
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/result.rs:1979:9
  54: core::iter::traits::iterator::Iterator::collect
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/iter/traits/iterator.rs:2001:9
  55: cargo::core::compiler::fingerprint::calculate_normal
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1464:24
  56: cargo::core::compiler::fingerprint::calculate
             at ./src/cargo/core/compiler/fingerprint/mod.rs:1425:9
  57: cargo::core::compiler::fingerprint::prepare_target
             at ./src/cargo/core/compiler/fingerprint/mod.rs:428:23
  58: cargo::core::compiler::compile
             at ./src/cargo/core/compiler/mod.rs:194:23
  59: cargo::core::compiler::build_runner::BuildRunner::compile
             at ./src/cargo/core/compiler/build_runner/mod.rs:192:13
  60: cargo::ops::cargo_compile::compile_ws
             at ./src/cargo/ops/cargo_compile/mod.rs:162:9
  61: cargo::ops::cargo_compile::compile_with_exec
             at ./src/cargo/ops/cargo_compile/mod.rs:141:5
  62: cargo::ops::cargo_compile::compile
             at ./src/cargo/ops/cargo_compile/mod.rs:128:5
  63: cargo::ops::cargo_test::compile_tests
             at ./src/cargo/ops/cargo_test.rs:105:27
  64: cargo::ops::cargo_test::run_tests
             at ./src/cargo/ops/cargo_test.rs:68:23
  65: cargo::commands::test::exec
             at ./src/bin/cargo/commands/test.rs:114:5
  66: cargo::cli::Exec::exec
             at ./src/bin/cargo/cli.rs:474:36
  67: cargo::cli::main
             at ./src/bin/cargo/cli.rs:96:9
  68: cargo::main
             at ./src/bin/cargo/main.rs:57:9
  69: core::ops::function::FnOnce::call_once
             at /rustc/c1db4dc24267a707409c9bf2e67cf3c7323975c8/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

It was this dep info file target/debug/.fingerprint/cargo-test-macro-3b2e77deed00ae97/dep-lib-cargo_test_macro during decoding, it got checksum=1, and thought it had to support checksum freshness.

let checksum_info = has_checksum

As a result, Cargo then wrongly decoded nenv as "1313817439", and then pushed nenv number of (String, Option<String>) to the Vec which caused OOM-killed.

let nenv = read_usize(bytes)?;
let mut env = Vec::with_capacity(nenv);
for _ in 0..nenv {

Feel like decoding a bool also for old dep-info is too fragile, though I may not be available in the next few hours to come up with a fix.

@weihanglo weihanglo added S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted. A-dep-info Area: dep-info, .d files Z-checksum-freshness Nightly: rebuild detection on file checksum instead of mtime and removed S-needs-info Status: Needs more info, such as a reproduction or more background for a feature request. A-testing-cargo-itself Area: cargo's tests labels Oct 27, 2024
@weihanglo
Copy link
Member

I got hit by this bug when running cargo check on this commit (cargo 1.84.0-nightly (cf53cc54b 2024-10-18), rustc 1.84.0-nightly (c1db4dc24 2024-10-25)). My box is also x86_64-unknown-linux-gnu, but it has 32 GiB of RAM, so throwing more RAM at the problem does not help either. I didn't have to set RUST_BACKTRACE to any value.

Edit: doing a cargo clean fixed it. Looks like doing a rustup update while having an older Rust version build cache is troublesome, is this a supported use case or does it happen to mostly work in practice in spite of it not being supported? For what it's worth, it's not the first time I do a rustup update with an already existing build cache.

@AlexTMjugador would you mind sharing steps to reproduce? Like which version of Cargo you used for the first build, and which cargo you use that encountered a core-dumped.

I have a hacky fix at hand, but unsure if it is worthy. Fingerprints generally are not shared between different toolchains. Unless you manually build cargo or have a pretty specialized toolchain setup, I still feel like normal users won't be bitten by this.

@AlexTMjugador
Copy link
Contributor

AlexTMjugador commented Oct 29, 2024

@AlexTMjugador would you mind sharing steps to reproduce? Like which version of Cargo you used for the first build, and which cargo you use that encountered a core-dumped.

I have a hacky fix at hand, but unsure if it is worthy. Fingerprints generally are not shared between different toolchains. Unless you manually build cargo or have a pretty specialized toolchain setup, I still feel like normal users won't be bitten by this.

Sure! Please find the exact steps to reproduce this issue attached below. To ensure a clean and consistent environment, I used a temporary Docker container based on the rustlang/rust:nightly image, which currently refers to this digest.

Terminal session of reproduction steps
$ docker run -it --rm rustlang/rust:nightly # Digest: https://hub.docker.com/layers/rustlang/rust/nightly/images/sha256-eceb4b2464041d21977d3699e4a085e0859a13c2c5227839eb7e9219f53f26d6?context=explore
root@3fde24c67f0d:/# rustc --version # The rustc version used by default
rustc 1.84.0-nightly (df4ca44d3 2024-10-27)
root@3fde24c67f0d:/# rustup toolchain install nightly-2024-09-15 # The previous rustc version, chosen arbitrarily to be approximately as old as I remember having before updating
info: syncing channel updates for 'nightly-2024-09-15-x86_64-unknown-linux-gnu'
info: latest update on 2024-09-15, rust version 1.83.0-nightly (9b72238eb 2024-09-14)
info: downloading component 'cargo'
  8.5 MiB /   8.5 MiB (100 %)   2.6 MiB/s in  1s ETA:  0s
info: downloading component 'rust-std'
 28.1 MiB /  28.1 MiB (100 %)   5.6 MiB/s in  5s ETA:  0s
info: downloading component 'rustc'
 72.1 MiB /  72.1 MiB (100 %) 758.2 KiB/s in 32s ETA:  0s    
info: installing component 'cargo'
info: installing component 'rust-std'
 28.1 MiB /  28.1 MiB (100 %)  21.1 MiB/s in  1s ETA:  0s
info: installing component 'rustc'
 72.1 MiB /  72.1 MiB (100 %)  23.0 MiB/s in  3s ETA:  0s

  nightly-2024-09-15-x86_64-unknown-linux-gnu installed - rustc 1.83.0-nightly (9b72238eb 2024-09-14)

info: checking for self-update
root@3fde24c67f0d:/# cd tmp
root@3fde24c67f0d:/tmp# git clone https://github.com/ComunidadAylas/PackSquash.git
Cloning into 'PackSquash'...
remote: Enumerating objects: 20748, done.
remote: Counting objects: 100% (517/517), done.
remote: Compressing objects: 100% (168/168), done.
remote: Total 20748 (delta 302), reused 471 (delta 277), pack-reused 20231 (from 1)
Receiving objects: 100% (20748/20748), 6.74 MiB | 15.57 MiB/s, done.
Resolving deltas: 100% (13003/13003), done.
root@3fde24c67f0d:/tmp# cd PackSquash/
root@3fde24c67f0d:/tmp/PackSquash# git reset --hard f973bd3
HEAD is now at f973bd38 Revert "chore: ignore broken `clippy::needless_return` lint in latest nightlies"
root@3fde24c67f0d:/tmp/PackSquash# cargo +nightly-2024-09-15 check # Uses an older nightly toolchain
<Normal cargo check output, works fine>
root@3fde24c67f0d:/tmp/PackSquash# cargo check # Uses a recent nightly toolchain
memory allocation of 67045949232 bytes failed
Aborted (core dumped)

bors added a commit that referenced this issue Oct 29, 2024
docs(fingerprint): document the encoding of Cargo's depinfo

### What does this PR try to resolve?

When investigating #14712,
I figured that it lacks of docs for the Cargo's depinfo format.
Hence documenting them.

This PR also move dep info files related items to a separate module.

### How should we test and review this PR?

No behavior change, though I may submit a hack of #14712 after this.

I didn't move dep-info specific documentation into that new module.
It feels more nature for keeping them all in the fingerprint module.
@weihanglo
Copy link
Member

Thanks @AlexTMjugador for the reproduction! It was really helpful :)

Confirmed that this issue is an incompatibility bug introduced by -Zchecksum-freshess.

The way it happened to Cargo itself is obvious:
When running a cargo built from source with old build cache not aware of the new -Zchecksum-freshness fingerprint dep-info format,
Cargo itself couldn't distinguish the difference.
Instead it tried to read dep-info from the same location,
and then got an OOM because the format is not compatible.

For the PackSquash repo, it is a bit subtle. The reproduction uses two different nightly versions. Cargo passes -C metadata (the hash appended after <pkg>-<ver> coordinate) to distinguish different output artifact for different set of flags and features. However, if you're using nightly, Cargo doesn't really separate different versions of nightly. It ends up putting in the same directory. This is somehow intentional however depending on fingerprint to correctly detect a rebuild is needed. Before fingerprint ever got started, Cargo loaded the fingerprint dep-info files and boo!

BTW, the first bad dep-info files in PackSquash is target/debug/.fingerprint/packsquash_cli-5a0a48e3ab0b2ec9/dep-build-script-build-script-build, which in hex view looks like

Image

@bors bors closed this as completed in de5832f Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-dep-info Area: dep-info, .d files C-bug Category: bug regression-from-stable-to-nightly Regression in nightly that previously worked in stable. S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted. Z-checksum-freshness Nightly: rebuild detection on file checksum instead of mtime
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants