Multiple simultaneous cargo builds that depend on the same git repo can fail

[jgopel]

Problem

I am building multiple binaries from the same repo for docker compose services with docker RUN cache mounting. I depend on a crate as a git repository. When these services are built simultaneously, they share the same /usr/local/cargo directory. The builds fail for all but the last service to finish downloading the dependency crate repo with the error "error: failed to get <dep crate> as a dependency package of <my crate> - I can share the full error if it's helpful.

I have also reproduced this issue outside of docker by manually starting 2 builds in close succession, and I see the same behavior.

Steps

Manual repro

1. Setup a project that depends on a crate via git - the longer the dependency takes to download, the easier this will be to repro.
2. Clean all cargo files - I am using sudo rm -rf /usr/local/cargo/git/ /usr/local/cargo/registry/ /usr/local/cargo/.crates* && cargo clean
3. Run cargo build twice simultaneously - the intent here is to get the git clones to overlap

Expected outcome: One cargo build does the clone, the other waits for it to finish and then uses the result, as with other dependency types.

Actual outcome: Both cargo builds try to do the git clone at the same time, the first one ends up getting stomped on by the second one and so the first cargo build fails while the second one succeeds.

Possible Solution(s)

In my docker case, I am able to switch the sharing type to private to spend some disk space to keep parallel builds. That looks something like this:

RUN --mount=type=cache,target=/usr/local/cargo/git/,sharing=private \
    --mount=type=cache,target=/usr/local/cargo/registry/,sharing=private \
    --mount=type=cache,target=/app/repo/target/ \
    cargo build

Notes

No response

Version

$ cargo version --verbose
cargo 1.85.0 (d73d2caf9 2024-12-31)
release: 1.85.0
commit-hash: d73d2caf9e41a39daf2a8d6ce60ec80bf354d2a7
commit-date: 2024-12-31
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Ubuntu 24.4.0 (noble) [64-bit]

[epage]

I tried to script the reproduction steps (haven't gotten far enough to clear out my CARGO_HOME yet) but I'm not able to reproduce this. Instead one process blocks while the other one fetches.

#!/usr/bin/env bash
rm -rf cargo-15267-1
rm -rf cargo-15267-2
cargo new cargo-15267-1
cargo new cargo-15267-2
#echo "qcmx = { git = \"https://github.com/mozilla/gecko-dev.git\" }" >> cargo-15267-1/Cargo.toml
#echo "qcmx = { git = \"https://github.com/mozilla/gecko-dev.git\" }" >> cargo-15267-2/Cargo.toml
echo "anthropic = { git = \"https://github.com/zed-industries/zed.git\" }" >> cargo-15267-1/Cargo.toml
echo "anthropic = { git = \"https://github.com/zed-industries/zed.git\" }" >> cargo-15267-2/Cargo.toml
cargo check --manifest-path cargo-15267-1/Cargo.toml &
cargo check --manifest-path cargo-15267-2/Cargo.toml
fg 1

$ cargo --version --verbose
cargo 1.85.0 (d73d2caf9 2024-12-31)
release: 1.85.0
commit-hash: d73d2caf9e41a39daf2a8d6ce60ec80bf354d2a7
commit-date: 2024-12-31
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Pop!_OS 22.4.0 (jammy) [64-bit]

[jgopel]

Thank you for looking in to this - I really appreciate it. I've run your script across 2 systems and a number of configurations and it seems like the issue is isolated to a specific system - maybe even to a specific config. I will try to see if I can narrow in on a more specific repro and post an update here.