Avoid race condition in cleaning up cache files #45214
Conversation
If multiple processes attempt to clean up cache files at the same time, a race condition can result, e.g. https://buildkite.com/clima/climaatmos-ci/builds/812#6a961e99-e2a1-488b-a116-2a45dee26d38/102-104
|
Out of curiosity, on CI, is the issue that you have two different Buildkite agents writing to the same depot at the same time? Or do you have a single Buildkite agent, and there are two different Julia processes, but they are part of the same Buildkite job? |
|
If it's the latter, wouldn't it be better to first do a single |
|
They're separate buildkite jobs sharing the same depot. |
There was a problem hiding this comment.
They're separate buildkite jobs sharing the same depot.
Ah, in that case, I don't recommend doing this PR; instead, I would recommend giving each Buildkite job its own depot. This is what we do for Base Julia CI, and it works very well. There are all kinds of issues that you can run into when multiple Buildkite agents are trying to write to the same depot at the same time.
|
Yeah, we do that for other cases but in this particular workflow we're trying to reduce the instantiation time. Is there any downside of merging this PR anyway? |
|
Note that this situation can arise in any case where multiple Julia processes are attempting to precompile at the same time, it just so happens that this particular job happens to trigger it regularly. |
I think a pull request should be evaluated on its merit, not the circumstances which motivated it. |
simonbyrne
added
compiler:precompilation
If multiple processes attempt to clean up cache files at the same time, a race condition can result, e.g. https://buildkite.com/clima/climaatmos-ci/builds/812#6a961e99-e2a1-488b-a116-2a45dee26d38/102-104 (cherry picked from commit d4acead)
If multiple processes attempt to clean up cache files at the same time, a race condition can result, e.g. https://buildkite.com/clima/climaatmos-ci/builds/812#6a961e99-e2a1-488b-a116-2a45dee26d38/102-104 (cherry picked from commit d4acead)
If multiple processes attempt to clean up cache files at the same time, a race condition can result, e.g. https://buildkite.com/clima/climaatmos-ci/builds/812#6a961e99-e2a1-488b-a116-2a45dee26d38/102-104 (cherry picked from commit d4acead)
If multiple processes attempt to clean up cache files at the same time, a race condition can result, e.g. https://buildkite.com/clima/climaatmos-ci/builds/812#6a961e99-e2a1-488b-a116-2a45dee26d38/102-104 (cherry picked from commit d4acead)
If multiple processes attempt to clean up cache files at the same time, a race condition can result, e.g.
https://buildkite.com/clima/climaatmos-ci/builds/812#6a961e99-e2a1-488b-a116-2a45dee26d38/102-104