doveadm fts rescan removes all indices

[leezu]

doveadm fts rescan -A deletes all indices, ie. all folders and files in the xapian-indexes are deleted. However, according to man doveadm fts, the rescan command should only

Scan what mails exist in the full text search index and compare those to what
actually exist in mailboxes. This removes mails from the index that have already
been expunged and makes sure that the next doveadm index will index all the
missing mails (if any).

Deleting all indices does not seem to be the intended action, especially as constructing the index anew may take very long on large mailboxes.

[grosjo]

I asked the dovecot mailing list, becuase this is a limitation from the design

[grosjo]

Below the answer from Timo.

I find then command "doveadm index -A -q *" rather handy, instead of "doveadm fts rescan"

-------- Original Message --------
Subject: Re: [grosjo/fts-xapian] doveadm fts rescan removes all indices (#15)
Date: 2019-02-14 16:24
From: Timo Sirainen via dovecot dovecot@dovecot.org
To: Joan Moreau jom@grosjo.net
Cc: dovecot@dovecot.org
Reply-To: Timo Sirainen tss@iki.fi, Dovecot Mailing List dovecot@dovecot.org

Hi,

The rescan() function is a bit badly designed. Currently what you could do what fts-lucene does and:

• Get list of UIDs for all mails in each folder
• If Xapian has UID that doesn't exist -> delete it from Xapian
• If UID is missing from Xapian -> expunge the rest of the UIDs in that folder, so the next indexing will cause them to be indexed

The expunging of rest of the mails is rather ugly, yes.. A better API would be if backend simply had a way to iterate all mails in the index, preferrably sorted by folder. Then a more generic code could go through them and expunge the necessary mails and index the missing mails. Although not all FTS backends support indexing in the middle. Anyway, we don't really have time to implement this new API soon.

I'm not sure if this is a big problem though. I don't think most people running FTS have ever run rescan.

[grosjo]

No real care from dovecot team about the matter.

CLosing this