GPUCompiler.reset_runtime() race condition

[simonbyrne]

Describe the bug

If multiple processes attempt to precompile CUDA.jl, they might call GPUCompiler.reset_runtime(), which due to a bug in Julia 1.9.3 and earlier, can trigger a race condition in recursive rm. See log here:
https://buildkite.com/clima/climaatmos-ci/builds/14843#018bc9e6-4dcd-4a53-87e4-60467b240fda/162-168

The Julia bug is fixed in JuliaLang/julia#50842, but this has not made it to a release version yet.

(I'm trying to figure out why they were getting recompiled, but @vchuravy suggested I open an issue for this).

To reproduce

Honestly, I can't figure out a way to reproduce it, but I have seen it several times.

[maleadt]

Ah, good catch. What do you suggest as workaround or fix? Using a pidlock seems heavyweight; I wonder if we could first do an atomic rename of the directory before wiping it.

[simonbyrne]

atomic rename before and after writing seems like the easiest fix?

[simonbyrne]

Any idea why we might be getting the compilecache call? Or any suggestions to try (i can't reliably reproduce it, unfortunately)

[simonbyrne]

I think this might be related to segfaults I'm seeing on loading of CUDA_Runtime_jll:
https://buildkite.com/clima/climacore-ci/builds/2842#018bd01c-5d70-4661-bc81-5cf000add4a3/161-422