Imap Search on large mailbox not returning any results with Gmail app on Android

[matidau]

While testing large mailboxes, found that the search functionality works for a smaller mailbox but not for a large mailbox.

possible fix: https://forum.kopano.io/topic/1922/fix-or-improvement-in-imap-search-with-gmail-app

needs further investigation and testing.

[jvolkenant]

I've found that a FTS engine is needed to get any quality results from searching via IMAP or Activesync. In 18.04 I used Solr, but now there is fts-xapian which works well without extra daemons needed to be run. Inital indexing took a while but results come back in 5-10sec. https://github.com/jvolkenant/mailinabox/tree/fts-xapian

[matidau]

Sounds promising :)

Do you know if there is any appetite for this to be included in the mailinabox upstream?

[jvolkenant]

I don't think there will be any new features accepted upstream anymore. I tried upstreaming Solr FTS for 18.04 but it was never merged. I feel MIAB is really in a maintenance mode now a days.

I just run an unpushed branch of changes on my server and then rebase when upstream is updated. I.e. for now it's https://github.com/jvolkenant/mailinabox/tree/fts-xapian and https://github.com/matidau/Z-Push/releases/tag/2.6.4-php8-testing-0.3

Regarding https://github.com/jvolkenant/mailinabox/tree/fts-xapian Indexing worked ok, but really hammered the CPU to the point that my provider started to throttle me (I might have even upsized the VM to finish indexing and then downsized when it completed), and I had to disable a night's backups otherwise dovecot gets stopped in the middle of an indexing. So there's at least some edge cases that I don't handle.

But since It won't be upstreammed It's working ok so far.

[cbren]

You really need one of the FTS engines to provide useful search results, especially on large inboxes. I'll echo the choice of a Xapian based engine, unless you have the resources to run solr.

There's another option to look at as well. fts-flatcurve: https://github.com/slusarz/dovecot-fts-flatcurve but it has some specific dovecot requirements that depending on your distro might require building dovecot form source. If you're using EL9 I've got a fork with some adjustments to make it compatible with the distro provided package.

The upside is It's indexing is much improved: significantly faster and less resource intensive(both CPU time for indexing and storage of the index) compared to fts-xapain (at least in my testing.) It's being upstremed into the dovecot 2.4 branch so maybe better support going forward. Works great with z-push and android clients.

[matidau]

Both Xapian and flatcurve look like good solutions.

I try to achieve a Zero Inbox and tend not to do a lot of searching. So for my personal use case I don't need to set this up on my production.

This thread has some really good info, worth sharing around to any that are curious and what to implement it on their setups.

Will close this off as not firectly a z-push problem or fix.