Names of intermediate files collide when running rustc in parallel

[SiegeLord]

For example, given an empty lib.rs, this happens:

rustc --lib lib.rs & rustc --rlib lib.rs
error: linking with `cc` failed: exit code: 1
note: cc arguments: '-m64' '-L/usr/local/lib/rustc/x86_64-unknown-linux-gnu/lib' '-o' 'liblib-9ed81b85-0.0.so' 'lib.o' 'lib.metadata.o' '-Wl,--as-needed' '-L/tmp/.rust' '-L/tmp' '-L/usr/local/lib/rustc/x86_64-unknown-linux-gnu/lib' '-lstd-04ff901e-0.9-pre' '-lrt' '-ldl' '-lm' '-lpthread' '-lstdc++' '-shared' '-lmorestack' '-Wl,-rpath,$ORIGIN/../usr/local/lib/rustc/x86_64-unknown-linux-gnu/lib' '-Wl,-rpath,/usr/local/lib/rustc/x86_64-unknown-linux-gnu/lib'
note: /usr/bin/ld: reopening lib.o: No such file or directory

/usr/bin/ld: final link failed: No such file or directory
collect2: error: ld returned 1 exit status

error: aborting due to previous error
task 'rustc' failed at 'explicit failure', /home/siege/src/rust2/src/libsyntax/diagnostic.rs:102
task '<main>' failed at 'explicit failure', /home/siege/src/rust2/src/librustc/lib.rs:398

This could be fixed in two main ways.

1. Add an option (e.g. --temps-path) to store the intermediates in a specific location
2. Add some kind of hash to the names of the intermediates to stop them from colliding. Can't just be the pkgid hash, as it'd still fail the above test.

[metajack]

This is sort of a duplicate of #10922.

Does these libraries have link metadata that isn't converted to pkgids?

[SiegeLord]

lib.rs is an empty file, so there is no custom metadata.

[metajack]

Then it's not really a dupe of that bug as this behavior existed before the change to pkgids.

One workaround would be to change crate name inference to use the something like the last component of the path of the dirname of the file if the file has one of hte well known names (lib.rs, test.rs, main.rs and bench.rs).

[SiegeLord]

Well, note that this only affects the intermediate files. If I do this:

rustc --lib lib.rs && rustc --rlib lib.rs

I get the two files (liblib-9ed81b85-0.0.rlib and liblib-9ed81b85-0.0.so) as expected. I am perfectly content with the library names being what they are. The issue is that the intermediate files are named lib.o, lib.bc and so on, for both calls to rustc. Therefore, running them in parallel leads to a race condition. In C/C++ I would have done this:

gcc -fPIC -c lib.c -o lib_shared/lib.o & gcc -c lib.c -o lib_static/lib.o
...

In D, which has a similar situation as Rust, I'd have used the equivalent of the --temps-path idea:

ldc2 -relocation-model=pic --shared lib.d -od=shared_lib_objects & ldc2 --lib lib.d -od=static_lib_objects

Where -od does what --temps-path would do.

[steveklabnik]

A lot has changed here, but with an empty lib.rs, I get

$ rustc --crate-type=rlib lib.rs & rustc --crate-type=staticlib lib.rs
[1] 5798
error: failed to remove lib.0.o: couldn't unlink path (no such file or directory (No such file or directory); path=lib.0.o)
error: aborting due to previous error

Which seems like a similar error.

[steveklabnik]

Triage: a slightly different error today:

$ touch lib.rs
$ rustc --crate-type=rlib lib.rs & rustc --crate-type=staticlib lib.rs
[1] 17820
error: failed to build archive: No such file or directory
[1]+  Done                    rustc --crate-type=rlib lib.rs

[steveklabnik]

Triage: apparently PowerShell doesn't make this easy, so I cannot attempt to reproduce. Can anyone else?

[SiegeLord]

The error from 2016 still happens on Linux.

[torhovland]

@rustbot claim

[jyn514]

The error from 2016 still happens on Linux.

I had trouble reproducing with --crate-type lib and --crate-type rlib, but changing rlib to cdylib consistently gives

error: linking with `cc` failed: exit status: 1
  = note: cc: error: lib.lib.3a1fbbbh-cgu.0.rcgu.o: No such file or directory

[jyn514]

Add some kind of hash to the names of the intermediates to stop them from colliding. Can't just be the pkgid hash, as it'd still fail the above test.

I think hashing the command line arguments would work.

[torhovland]

I had trouble reproducing with --crate-type lib and --crate-type rlib, but changing rlib to cdylib consistently gives

error: linking with `cc` failed: exit status: 1
  = note: cc: error: lib.lib.3a1fbbbh-cgu.0.rcgu.o: No such file or directory

OK, I just checked that my PR does fix that (using --temps-dir).