Concurrent GAP package installation

[rbehrends]

Currently, it appears that concurrent installation of GAP packages can lead to race conditions. This has occurred previously when running multiple CI tests that both use GAP.jl at the same time.

What complicates the issue are the following two points:

1. This can happen when multiple Julia processes are running concurrently, not just from multiple threads within Julia. Thus, this does not just require a mutex, but a file lock.
2. The issue can probably also be triggered by doing the same in two GAP processes at the same time. Thus, I'm not sure if this is actually something that should be fixed in GAP.jl or in GAP proper.

[ThomasBreuer]

Then the right place for fixing this problem would be GAP's PackageManager package.

[rbehrends]

Possibly, but I'm not sure how easy it would be to do file locking in PackageManager. Correct me if I'm wrong, but I don't think GAP currently supports file locking primitives? If not, then doing it in GAP.jl might be the pragmatic approach for the time being.

[ThomasBreuer]

I thought about adding some kind of preprocessing to the installation:
Before starting to download, unpack, and compile the package in question, the process could check for a file with a reserved name in the package directory; if such a file is there then someone else is currently installing this package; if the file is not there then write it, and afterwards check whether the file is there and contains what it should be in, in order to make sure that the file was not written by someone else; if the file is o.k. then install the package and finally remove the file in question.
(Of course one has to deal with the case that the process crashes before the final step, for example via a reasonable timestamp heuristic.)
Is this perhaps too naive?

[rbehrends]

There are a number of issues that make this tricky:

1. If the process crashes during installation, the lock file will be left behind, preventing further installations; I don't think a timestamp heuristic can fix that reliably.
2. You can mitigate it by storing the PID in the lockfile and testing whether the process is alive, but doing this atomically without file locking is not easy.

It's not impossible (see Pidfile.jl), but it's much easier with fcntl().