file locking

[StefanKarpinski]

Sometimes you want to make sure that multiple processes don't update the same file at the same time. For example, Pkg should lock the ~/.julia/REQUIRE file when installing or removing packages, and when writing the the ~/.julia_history` file, the REPL should lock it (or use atomic operations like seek-and-write in a true append mode).

[StefanKarpinski]

I'm not sure what the status of support for this on Windows is, but there is both advisory and mandatory file locking. I prefer advisory file locking myself, but maybe we should support both in a portable way. There also seem to be distributed file system considerations.

[StefanKarpinski]

Possibly relevant or at least interesting: https://lwn.net/Articles/586904/.

[jakebolewski]

We'll want this as part of the Pkg revamp.

[jakebolewski]

Relevant libuv issue: joyent/libuv#1533. It seems like it is impossible to do this in a cross-platform way.

[ihnorton]

Our userbase probably has a higher % of NFS usage than the general population due to research clusters / shared computing resources, so the issues on NFS would be a major concern.

Relevant:

• this FAQ item from SQLite: http://www.sqlite.org/faq.html#q5
• http://0pointer.de/blog/projects/locking.html
• https://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html (linked from above)

[StefanKarpinski]

This would make an excellent up for grabs project, requiring mostly research into how to do locking on various platforms.

[amitmurthy]

This may just work. Use a named socket as a lock file.

julia> a=listen("/tmp/Modname.lock")
Base.PipeServer(active)

julia> b=listen("/tmp/Modname.lock")
ERROR: ArgumentError: could not listen on path /tmp/Modname.lock

Closing the named socket releases the lock. The path can be path/to/cachedir/module.sock.
Processes waiting on the lock would need to poll to acquire the same.

[amitmurthy]

I was commenting in the context of parallel loading, but the locking need is the same.

[amitmurthy]

Will not work if the cache/pkg is on a shared directory, but for that we would need a truly distributed lock anyway.

[yuyichao]

what's wrong with flock?

[amitmurthy]

Don't know the reasons but the linked libuv issue mentions that there is no cross-platform way to implement file locking.

[yuyichao]

The issue seems to be mostly about nfs or similar shared fs. Using a socket wont do much better either. Using a socket also has the issue that the lock wont be fully released if the owner crashed

[amitmurthy]

One issue with using flock directly is that since it is not supported via libuv, the waiting process is effectively blocked on the ccall.

For us to manage locking via a file using libuv's open call via FS.open with flags set to O_EXCL | O_CREAT, along with polling for acquiring the lock would have the same issue on how to deal with owner crashes.

If we ignore the shared fs scenario for now, we could still go with the unix domain socket option that the acquirer can unlink if unable to connect to it. So the polling mechanism would be

1. try opening the UD socket
2. If bind error try connect
3. If connect fails, unlink socket and try opening again
4. If connect succeeds, close socket, wait for awhile and goto step 1.

[amitmurthy]

Can use flock with non-blocking option.

Requesting a lock on an object that is already locked normally causes the caller to be blocked 
until the lock may be acquired.  If LOCK_NB is included in operation, then this will not happen;
instead the call will fail and the error EWOULDBLOCK will be returned.

A solution for non-shared filesystems would be a good start for now.

[stevengj]

#5622 says that libgit2 provides this functionality in a cross-platform way, so maybe we just need to export it?

[ihnorton]

@stevengj: not sure why the comment there refers to atomic symlinks. As far as I can tell libgit2 uses plain old lockfiles and open/_wopen:

https://github.com/libgit2/libgit2/blob/79194bcdc956406979cd27ac99198826860d3f20/src/fileops.c#L66-L77
https://github.com/libgit2/libgit2/blob/77394a27af283b366fa8bb444d29670131bfa104/src/posix.c#L105-L118
(worse, it doesn't seem to even unlink the file, so stale lockfiles can easily happen on unclean exit).

[vtjnash]

I've created the Pidfile.jl package for this. We should probably keep this open until we move that into Base (for use with compilecache, as well as Pkg, and perhaps other things).

[simonbyrne]

If we are to bring in Pidfile.jl, would it need to live in Base, or could it be a stdlib? I guess if it is used at the Pkg.precompile level instead of Base.compilecache, it would be fine as a stdlib?

[KristofferC]

Pidfile is now in Base so I think this can be closed.

[simonbyrne]

More precisely, it's part of the FileWatching stdlib (#44367)