full resync fails

[anarcat]

I had a lot ofdifficulty and uncertainty with the offlineimap migration procedure, for example with #7 but also in general with a lot of inconsistencies between the two mailboxes. I pull would download ~600 new emails and push delete another 500, which means about 100 emails (or more?) might have been duplicated, or 500 were lost and 600 duplicated... it's hard to tell and scary.

So I decided to try out a fresh sync, ignoring previously existing OfflineIMAP folders. I figured I would use this simple configuration:

    MAILBOX=Maildir
    EXCLUDE="Maildir/.notmuch/hooks/* Maildir/.notmuch/xapian/*"

No translation layer necessary! I first stripped the offlineimap headers on the server (because, yes, there were some of those in the mailbox - go figure) with the new strip-header script, and then copied over the entire mailbox with:

mv Maildir Maildir-offlineimap
ssh example.net "tar cf - Maildir/ | pv -s 13G | gzip -c" | tar xf -

I had to cleanup some Dovecot cruft like that:

( cd Maildir
    find \( -name 'dovecot.index*' -o -name 'dovecot-uidlist' -o -name 'dovecot-keywords' \) -a -delete
    rm dovecot-uidvalidity* dovecot.mailbox.log subscriptions )

... and off we go with smd-pull --dry-run. The problem is I end up with confusing error messages like this:

    smd-server: ERROR: Client aborted, removing /home/anarcat/.smd/curie-anarcat__Maildir.db.txt.new and /home/anarcat/.smd/curie-anarcat__Maildir.db.txt.mtime.new
    smd-client: ERROR: Failed to add Maildir/.bresil/cur/1190657273_35.6763.mumia,U=10,FMD5=dbed73e537122b2e2505089b78e7b27c:2,FS since a file with the same name
    smd-client: ERROR: exists but its content is different.
    smd-client: ERROR: To fix this problem you should rename Maildir/.bresil/cur/1190657273_35.6763.mumia,U=10,FMD5=dbed73e537122b2e2505089b78e7b27c:2,FS
    smd-client: ERROR: Executing `cd; mv -n "Maildir/.bresil/cur/1190657273_35.6763.mumia,U=10,FMD5=dbed73e537122b2e2505089b78e7b27c:2,FS" "Maildir/.bresil/cur/1527001064.1.localhost"` should work.
    default: smd-client@localhost: TAGS: error::context(mail-addition) probable-cause(concurrent-mailbox-edit) human-intervention(necessary) suggested-actions(run(mv -n "/home/anarcat/.smd/workarea/Maildir/.bresil/cur/1190657273_35.6763.mumia,U=10,FMD5=dbed73e537122b2e2505089b78e7b27c:2,FS" "/home/anarcat/.smd/workarea/Maildir/.bresil/tmp/1190657273_35.6763.mumia,U=10,FMD5=dbed73e537122b2e2505089b78e7b27c:2,FS"))

The first problem is this error message is incredibly confusing: there are 6 lines of error messages including two different commands to run:

cd; mv -n "Maildir/.bresil/cur/1190657273_35.6763.mumia,U=10,FMD5=dbed73e537122b2e2505089b78e7b27c:2,FS" "Maildir/.bresil/cur/1527001064.1.localhost"

or:

mv -n "/home/anarcat/.smd/workarea/Maildir/.bresil/cur/1190657273_35.6763.mumia,U=10,FMD5=dbed73e537122b2e2505089b78e7b27c:2,FS" "/home/anarcat/.smd/workarea/Maildir/.bresil/tmp/1190657273_35.6763.mumia,U=10,FMD5=dbed73e537122b2e2505089b78e7b27c:2,FS"

Should the two commands be ran?

Now, even if I do run those commands, I only stumble upon the next roadblock: SMD asks me again to rename yet another file:

default: smd-client@localhost: TAGS: error::context(mail-addition) probable-cause(concurrent-mailbox-edit) human-intervention(necessary) suggested-actions(run(mv -n "/home/anarcat/.smd/workarea/Maildir/.cmaq/cur/1190657303_7.6763.mumia,U=59,FMD5=6073e73e1fcf82f8a2cb5d37f8f05209:2,FSb" "/home/anarcat/.smd/workarea/Maildir/.cmaq/tmp/1190657303_7.6763.mumia,U=59,FMD5=6073e73e1fcf82f8a2cb5d37f8f05209:2,FSb"))

Those files were not changed on either side: those are old email archives that never change and it's basically impossible those emails were added, removed or renamed after the first large sync.

It sure feels like SMD will make me rename all of those files one at a time. I know there's a smd-rename command, but that is designed to deal with offlineimap idiosyncracies: this is pure SMD, working on an exact copy of the original mailbox. Why is it failing of those messages? It should just say "0 emails changed" here.

I've tried really hard to make this software work for me: the thing is promising: i get at least two-fold performance improvements over offlienimap, and an SSH transport which I find really useful. But unfortunately, nothing seems to work here: i'm down to trying a full sync from scratch which the README recommends against, and I suspect would take a long time.

How are we supposed to sync a mailbox from scratch? Am I doing this right? Note that my full procedure actually starts with a working copy in Maildir-smd/ on both the server and client. Only when things look right (and they did!) do I swap the folders around on the client (mv Maildir-smd Maildir) and try the sync again. That's when things go haywire. If I rename the local folder (mv Maildir Maildir-smd) things become reasonable again. It feels there's some state that kept that gets SMD confused somewhere...

My full procedure is documented here:

https://anarc.at/services/mail/offlineimap2smd/#clean-slate-procedure

I get stuck at step 13. It's also pretty annoying that every smd-pull needs to rebuild the SHA checksum database, which takes a loooong time...
Thanks for the help...

[anarcat]

sigh... turns out the problem was that I had ran the strip-header on the server, but forgot to apply the changes by moving the folder into place... gah.

[anarcat]

so the full procedure now works correctly, sorry for the noise.

[anarcat]

for what it's worth, my final migration procedure was the following:

https://anarc.at/services/mail/offlineimap2smd/#clean-slate-procedure