cargo doesn't respect the maximum number of jobs

[glandium]

Problem
When running cargo with an explicit -jn, or via GNU make with an explicit -jn, the number of jobs can actually exceed the limit given. Presumably, the same happens without -jn, but now that I have 32 cores/64 threads, I can't find a crate that saturates them.

Steps

1. clone the cargo repository
2. run cargo build -j4 -Z timings
3. open cargo-timings.html

You'll notice that at some point the number of active jobs goes over 4. This report (cargo-timing.zip) goes up to 7.

Notes

Output of cargo version:

cargo 1.42.0-nightly (86134e766 2019-12-23)

[alexcrichton]

This is a known bug with the cc crate where it releases its implicit process token which can be acquired by something else to spawn more jobs, but I don't think this is an issue with Cargo so I'm going to close this.

[glandium]

This is a known bug with the cc crate

Do you have a link to an open issue?

[alexcrichton]

There is not currently an open issue on the cc crate.

[glandium]

Should this one be redirected there, then, with more details about the "known bug"? Because it's not clear to me how cc releases its process token, and why that ends up bumping the number of active jobs in cargo.

[alexcrichton]

Er well I'm pointing out that this issue you're reporting has nothing to do with Cargo itself, so this isn't the proper place for this issue. This is likely very related to the behavior of the cc crate, rather.

I don't really consider the behavior a bug on the cc crate myself so there's not an issue for it, so calling this a "known bug" was probably not a great usage of words on my part.

[glandium]

I still don't understand how cc messes with the number of active job...

[ehuss]

@glandium There is more detail at #7689 (comment) and #7344 (comment). The jobserver protocol is a very simple queue of tokens, and there is no protection that a client pushes extra tokens into the queue. The cc crate is releasing its implicit token to simplify its implementation. This will result in extra jobs, but they are not consuming CPU time since they are just waiting to reacquire their implicit token.