Lock package directory when executing Pkg commands

[jakebolewski]

Possible fix for #5622. Write a simple lock file to the package directory when executing Pkg commands to prevent multiple instances of Pkg clobbering the same directory.

[jiahao]

I think this could work, but one would want the ability to guard against a stale lock. (What if I ran Pkg.add and got closed my terminal window by accident, thus terminating Julia, for example...)

[jakebolewski]

You can remove the stale lock with Pkg.rmlock() the next time you want to do anything. This is how Arch's pacman package manager does it, it seems to work well in practice.

[jiahao]

True, I missed that on my first read-through. I'll shut up now.

[simonster]

Not sure if it's worth putting time into this, but there are things you can do to avoid a stale lock. On *nix, Firefox uses fcntl to lock a file, or creates symlink whose target contains the pid if fcntl locking isn't supported (e.g. on old NFS) (presumably it's a symlink because creating a symlink is atomic whereas opening and writing to a file is not). If the process is killed, the fcntl lock gets released, or the symlink hopefully doesn't match a running pid. On Windows, Firefox creates a file using CreateFile with dwShareMode = 0, which has the same characteristics as fcntl-based locking. Code here.

[jakebolewski]

@simonster that seems like quite the jump in complexity. I could work on that though it people feel it is worth it.

[simonster]

Probably not worth it for now. I doubt stale locks will arise too often in practice.

[StefanKarpinski]

We should really use something like flock which was designed for this and has OS support. Getting this kind of thing to actually be atomic with normal file-system operations like this is almost impossible.

[StefanKarpinski]

Also has the advantage of automatically being cleaned up when a process exits.

[simonster]

I think fcntl is a better approach than flock; the flock man page says it does not work under NFS but fcntl does.

[StefanKarpinski]

Yeah, I don't really care which one, but we should not do this by moving files around.

[jakebolewski]

@staticfloat brought up a good point. It would be better to hold off and leverage libgit2 for this.

[StefanKarpinski]

Git will only prevent us from modifying the same git repository but there are Pkg-level operations that will stomp all over each other too. I think that we should lock on the REQUIRE file.

[staticfloat]

Of course: forgot that we don't treat ~/.julia as a giant got repo anymore.
;)

Yeah, I agree. Using libuv to lock REQUIRE should work just fine.
On Jan 31, 2014 5:52 PM, "Stefan Karpinski" notifications@github.com
wrote:

Git will only prevent us from modifying the same git repository but there
are Pkg-level operations that will stomp all over each other too. I think
that we should lock on the REQUIRE file.

Reply to this email directly or view it on GitHubhttps://github.com//pull/5628#issuecomment-33859964
.

[vtjnash]

bump. Sounds like we should lock the REQUIRE file using the system locks

[StefanKarpinski]

Depends on #7176.

[JeffBezanson]

This seems like a nice-to-have, but not crucial. Does it need to be in 0.3?

[StefanKarpinski]

This definitely doesn't need to be in 0.3.

[StefanKarpinski]

Why'd you close this, jake?

[jakebolewski]

This proposed fix was wrong, old and stale. #5628 captures the underlying problem. We will have to write this functionality ourselves as libuv has decided not support any type of file locking functionality.