jj-autosync daemon to auto-sync a repos across dev machines

[colemickens]

Hi,

I saw another user make a casual reference to auto-syncing or some sort of auto-fast-forward functionality and I've been thinking about writing this post ever since. After a small data-blip incident today and re-evaluating my current data/code syncing strategy, I decided to loop back here again and ask about this.

Given that jj auto-commits/tracks my working copy, I keep thinking about how this could be leveraged to provide an auto-sync functionality, at least across a set of machines for maybe a single developer.

In my scenario, I'd be pairing configuring and running the service via NixOS, giving me the final goal of being able to write a manifest file that a theoretical jj-powered daemon would consume and use to know what repos to clone, and manage auto-fast-forwarding for me.

Is this something someone has given much thought to? Is it something others are interested in ... using? building? etc?

[martinvonz]

Interesting idea. My first thought was that you could just use rsync, but I suppose you also want versioning? Will you sometimes be making explicit commits too (not just relying on automatic committing of the working copy)?

I have thought about how to use jj with Google's internal distributed file system ¹. I suspect the user-visible behavior you have in mind is similar to what I had imagined for that. What I had imagined was that the user would be able to work on any machine and any changes to the repo and the working copy would be reflected on all machines (after some replication delay, of course). Concurrent modifications (i.e. modifications within the replication delay) would result in divergent commits (i.e. two commits with the same change ID) instead of changes being lost. That would depend on the working copy getting automatically committed, of course, which would happens when you run a jj command but could also be done by a file-system watcher. I suppose the replication delay might be larger in your setup but otherwise it could probably behave the way I described here. Does that make sense to you?

I think the work I'm currently doing for automatic syncing with git and for supporting multiple working copies would both provide some useful building blocks.

jj currently doesn't have any native wire protocol and exchanging over the git protocol loses some information (in particular the change ID and anonymous heads). It will take a looong time before we have native wire protocol support (probably not until someone other than me invests time in it). However, jj's file formats are designed to work on a distributed file system, so you should be able to just rsync the who .jj/ directory. One caveat is that git does not have that property, so the git directory in .jj/store/git/ is not guaranteed to be correct. I think it will work in practice, though (the only issue I could think of is that git notes can be lost, but I recently replaced the git-notes-based storage by a custom format so that's no longer an issue).

Footnotes

1. Reminder that this project is just my hobby/20% project and not endorsed by my team :) ↩

[colemickens]

Thanks for the, again, very helpful, detailed reply @martinvonz!

Regarding, rsync, yeah -- I'm pretty naively using SyncThing right now:

• for ~/code, the folder watch and/or trigger delay is turned down as low as it goes (1 sec)
• just blindly syncing all of ~/code - including huge target/ directories, .git directories, etc (I could use .stignore but feels like effort in the wrong place)
• all sync targets are local or a short gigabit link away

It's been working completely fine, I'm blessed with fast internet and there's only one of me so there aren't really conflicts. However, it feels a bit of a large, blunt hammer, when at the core what I care about comes down to SCM primitives.

I don't know Google's FS, but I do think/get the impression that my dream model has been inspired by things I've heard of some google dev envs over time. And what you describe pretty accurately captures what I've thought about, and then what I haven't known to think about. For example, it hadn't occurred to me that jj would need a wire protocol, or that exporting to git being lossy.

However, jj's file formats are designed to work on a distributed file system, so you should be able to just rsync the who .jj/ directory. One caveat is that git does not have that property, so the git directory in .jj/store/git/ is not guaranteed to be correct. I think it will work in practice, though (the only issue I could think of is that git notes can be lost, but I recently replaced the git-notes-based storage by a custom format so that's no longer an issue).

Maybe it doesn't matter, but I'm not sure that I follow - is this just because something like running git gc concurrently on two machines could be bad? I'm having a bit of a hard time wrapping my head (lack of experience with jj or lacking deep enough knowledge of git maybe) around the implications of being able to just sync .jj in terms of what that would mean for the working copy, etc.

[martinvonz]

Thanks for the, again, very helpful, detailed reply @martinvonz!

No problem :) I'm just happy that someone's interested in my project.

It's been working completely fine, I'm blessed with fast internet and there's only one of me so there aren't really conflicts. However, it feels a bit of a large, blunt hammer, when at the core what I care about comes down to SCM primitives.

Makes sense.

For example, it hadn't occurred to me that jj would need a wire protocol, or that exporting to git being lossy.

Yes, jj is its own DVCS so it needs its own wire protocol, it just has a bunch of Git interop. (I don't know if you saw on the other thread but I hope to make it much easier to use jj and git together by letting them share a working copy.)

However, jj's file formats are designed to work on a distributed file system, so you should be able to just rsync the who .jj/ directory. One caveat is that git does not have that property, so the git directory in .jj/store/git/ is not guaranteed to be correct. I think it will work in practice, though (the only issue I could think of is that git notes can be lost, but I recently replaced the git-notes-based storage by a custom format so that's no longer an issue).

Maybe it doesn't matter, but I'm not sure that I follow - is this just because something like running git gc concurrently on two machines could be bad? I'm having a bit of a hard time wrapping my head (lack of experience with jj or lacking deep enough knowledge of git maybe) around the implications of being able to just sync .jj in terms of what that would mean for the working copy, etc.

No, that's true even without GC. Git's object store is safe to use on a distributed file system, but ref updates are not safe. If you move the main branch from commit X to commit Y on one machine and move it (from commit X) to commit Z on another machine, then one of those will be lost. Since Git notes are stored in objects pointed to by refs, that means notes can also get lost that way (again, jj used Git notes internally until recently).

The concurrency problem is what jj's "operation log" was designed to solve (it just happened that it also provides undo functionality :)). I know it's weird, but just like conflicting files (and symlinks and trees) can be in a conflicted state, so can branches. In the case above, you'd see main? (with the question mark indicating conflict/uncertainty) on both commits and Y and Z until you resolve it. By the way, note that concurrent access from two processes is very similar to concurrent access on two different machines. The difference is that you can use (local) locking in the former case. However, since the operation log solves the problem generally, jj doesn't need locking to prevent concurrent local access. The exception is the working copy state (the "index"), but no content is lost even if there's a lost write to that.

The bit about just syncing the .jj/ was incorrect -- you'd also need to sync the working copy.

[colemickens]

With this information, it seems that maybe there isn't really too much of a need for a tool to be "integrated" with jj. It seems that my original idea is mostly accomplished by:

1. fswatching my code directories (ignoring per .gitignore)
2. invoking some sort of jj touch (or maybe jj status is already fastest for this operation?) on notification of changes in a dir containing a .jj (todo: validate assumptions regarding multiple jj workdirs for a repo)
3. effectively just mirror all notified writes across to sync targets

And then initial and periodic sync could just be some form of rsync --filter=':- .gitignore' ...

I think my original goal was to avoid "dumb" syncing and use the wire protocol where possible. Rather than syncing .git ever, I would exclusively use git push/pull to replicate "shadow" branches around (and then backdoor fast-forward if possible). But this seems unnecessary and maybe not worth much, given the (extra) concurrency assurances jj is providing (even if there was a native jj wire protocol right now).

Hm, I probably should experiment and think more -- if things diverge on two machines, jj might be able to handle it internally, but the working copy will still wind up in some sort of awkward state due to the "dumb" syncing happening below it... either requiring manual intervention, or worse, last-wins, and then the user winds up running a jj command and records an old-revision-synced-over as the new revision, thereby shadowing local changes. But maybe the point is that jj could detect this because of the concurrency internals it provides? I'm not sure.

I should experiment more so I can understand more and then revisit this. And/or let me know if you have some other ideas about achieving the descried sort of UX with fully-OSS solutions.

[martinvonz]

With this information, it seems that maybe there isn't really too much of a need for a tool to be "integrated" with jj. It seems that my original idea is mostly accomplished by:

1. fswatching my code directories (ignoring per .gitignore)
2. invoking some sort of jj touch (or maybe jj status is already fastest for this operation?) on notification of changes in a dir containing a .jj (todo: validate assumptions regarding multiple jj workdirs for a repo)
3. effectively just mirror all notified writes across to sync targets

And then initial and periodic sync could just be some form of rsync --filter=':- .gitignore' ...

Sounds like that should work and yes, jj status should be good enough (jj log -r @ -T commit_id is faster if you need it).

I think my original goal was to avoid "dumb" syncing and use the wire protocol where possible. Rather than syncing .git ever, I would exclusively use git push/pull to replicate "shadow" branches around (and then backdoor fast-forward if possible). But this seems unnecessary and maybe not worth much, given the (extra) concurrency assurances jj is providing (even if there was a native jj wire protocol right now).

I agree that it would be nice to do it with a proper push/pull because it would minimize bandwidth usage but it would probably actually be slower in most cases.

Hm, I probably should experiment and think more -- if things diverge on two machines, jj might be able to handle it internally, but the working copy will still wind up in some sort of awkward state due to the "dumb" syncing happening below it... either requiring manual intervention, or worse, last-wins, and then the user winds up running a jj command and records an old-revision-synced-over as the new revision, thereby shadowing local changes. But maybe the point is that jj could detect this because of the concurrency internals it provides? I'm not sure.

You are correct that it could get messy if the working copy is also synced. That's why I was initially that it's best to sync only .jj/ but I wasn't sure how to best get the working copy updated in that case. Having thought a bit more about it, I think it would be ideal to do something like this:

1. Run jj st to commit any local changes to the working copy.
2. Run rsync on .jj/ to copy recent operations from the other machine
3. Update the working copy to where the new operation says it should point.

The problem is that there's no good way of doing step 3 yet. You could do it by running jj --at-op=<latest operation synced over> log -r @to find what the working copy is supposed to point to, thenjj restore --from to restore the working copy to that state, but that's racy for two reasons. First, any working copy changes that happen between the sync and thejj restorecommand will be committed and then overwritten byjj restore. Second, there's currently no protection against overwriting changes that happen [while the checkout is happening](https://github.com/martinvonz/jj/blob/d451c1adf8494664b22c165ebd64736d978a7eae/lib/src/working_copy.rs#L602. I don't think you can completely protect against that second kind of race (the jj restore` process and your editor could be writing to the file at the same time). So if you're thinking of doing the auto-syncing in the background, I think there's always going to be a small risk of data loss. Oh, I suppose we could use flock or something to protect against that, but I'm not sure it's worth it.

I should experiment more so I can understand more and then revisit this. And/or let me know if you have some other ideas about achieving the descried sort of UX with fully-OSS solutions.

I should also say that there's no GC yet (#12), so don't expect .jj/ to remain small.