Getting SIGSEGV on doveadm index

[tdegrunt]

Hello,

This happens upon indexing a mailbox, but also with dovecot running indices itself:
2019-07-02T13:04:11.970270+00:00 g02ms01 dovecot: indexer-worker(some@email.com)<7994>srt6YbKMdtZU8cwR:eZiUOUtWG106HwAADIg0GQ: Fatal: master: service(indexer-worker): child 7994 killed with signal 11 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps)

After this I ran an individual index of the mailbox and here too I received a SIGSEGV.
Below is the backtrace.

Kind Regards,
Tom

OS: Ubuntu 14.04
Dovecot 2.3.6 (7eab80676)

Here is the backtrace:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff49309eb in std::string::assign(char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt full
#0 0x00007ffff49309eb in std::string::assign(char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#1 0x00007ffff52f1536 in assign (__s=0x555555823980 "return-path", this=0x5555558468c8) at /usr/include/c++/4.8/bits/basic_string.h:1131
No locals.
#2 operator= (__s=0x555555823980 "return-path", this=0x5555558468c8) at /usr/include/c++/4.8/bits/basic_string.h:555
No locals.
#3 fts_backend_xapian_update_set_build_key (_ctx=0x5555558468a0, key=0x7fffffffde10) at fts-backend-xapian.cpp:307
ctx = 0x5555558468a0
i =
f2 = 0x555555823980 "return-path"
backend =
j = 11
field = 0x555555846e26 "RETURN-PATH"
#4 0x00007ffff65a8064 in fts_backend_update_set_build_key () from /usr/lib/dovecot/modules/lib20_fts_plugin.so
No symbol table info available.
#5 0x00007ffff65a9377 in fts_build_mail () from /usr/lib/dovecot/modules/lib20_fts_plugin.so
No symbol table info available.
#6 0x00007ffff65afae2 in ?? () from /usr/lib/dovecot/modules/lib20_fts_plugin.so
No symbol table info available.
#7 0x00007ffff767816e in mail_precache () from /usr/lib/dovecot/libdovecot-storage.so.0
No symbol table info available.
#8 0x0000555555584eaf in ?? ()
No symbol table info available.
#9 0x0000555555585400 in ?? ()
No symbol table info available.
#10 0x000055555557f7d7 in ?? ()
No symbol table info available.
#11 0x0000555555580318 in ?? ()
No symbol table info available.
#12 0x000055555558115d in doveadm_cmd_ver2_to_mail_cmd_wrapper ()
No symbol table info available.
#13 0x00005555555911ca in doveadm_cmd_run_ver2 ()
No symbol table info available.
#14 0x00005555555912f7 in doveadm_cmd_try_run_ver2 ()
---Type to continue, or q to quit---
No symbol table info available.
#15 0x000055555557050c in main ()
No symbol table info available.

[grosjo]

Hi
Are you using git version ?
I ma de a small fix in field selection. Can you test again ?

[tdegrunt]

Hello,

Thanks for your answer!

I'm indeed using the git version. I've updated git, ran make clean, make and make install.
Sadly seeing SIGSEGV though, see below for the backtrace.

Problem line seems the same, other line position, so surely the new code.

Regards,
Tom

Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error
warning: File "/lib/x86_64-linux-gnu/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
	add-auto-load-safe-path /lib/x86_64-linux-gnu/libthread_db-1.0.so
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff49309eb in std::string::assign(char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x00007ffff49309eb in std::string::assign(char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007ffff52f1503 in assign (__s=0x5555558239a0 "return-path", this=0x5555558468e8) at /usr/include/c++/4.8/bits/basic_string.h:1131
#2  operator= (__s=0x5555558239a0 "return-path", this=0x5555558468e8) at /usr/include/c++/4.8/bits/basic_string.h:555
#3  fts_backend_xapian_update_set_build_key (_ctx=0x5555558468c0, key=0x7fffffffdde0) at fts-backend-xapian.cpp:304
#4  0x00007ffff65a8064 in fts_backend_update_set_build_key () from /usr/lib/dovecot/modules/lib20_fts_plugin.so
#5  0x00007ffff65a9377 in fts_build_mail () from /usr/lib/dovecot/modules/lib20_fts_plugin.so
#6  0x00007ffff65afae2 in ?? () from /usr/lib/dovecot/modules/lib20_fts_plugin.so
#7  0x00007ffff767816e in mail_precache () from /usr/lib/dovecot/libdovecot-storage.so.0
#8  0x0000555555584eaf in ?? ()
#9  0x0000555555585400 in ?? ()
#10 0x000055555557f7d7 in ?? ()
#11 0x0000555555580318 in ?? ()
#12 0x000055555558115d in doveadm_cmd_ver2_to_mail_cmd_wrapper ()
#13 0x00005555555911ca in doveadm_cmd_run_ver2 ()
#14 0x00005555555912f7 in doveadm_cmd_try_run_ver2 ()
#15 0x000055555557050c in main ()

[grosjo]

I don't see why assigning a string to a std::string would cause a core dump.

Maybe memory limit ? Do you have "default_vsz_limit = 2GB // or above" in your dovecot config ?

Can you share the doveconf -n output ?

[tdegrunt]

That's indeed the first thing I thought.
I have this, like in the README: default_vsz_limit = 2GB

This is the output of the doveconf command:

# 2.3.6 (7eab80676): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.6 (92dc263a)
doveconf: Warning: service auth { client_limit=1000 } is lower than required under max. load (6144)
doveconf: Warning: service anvil { client_limit=1000 } is lower than required under max. load (4099)
# OS: Linux 3.13.0-135-generic x86_64 Ubuntu 14.04.6 LTS
# Hostname: g02ms01.example.com
auth_mechanisms = plain login
default_process_limit = 2048
default_vsz_limit = 2 G
dict {
  sieve = pgsql:/etc/dovecot/dict-sieve-sql.conf.ext
}
disable_plaintext_auth = no
doveadm_password = # hidden, use -P to show it
doveadm_port = 2143
hostname = g02ms01.example.com
lmtp_save_to_detail_mailbox = yes
login_trusted_networks = 10.10.0.0/16
mail_fsync = never
mail_location = sdbox:~/sdbox
mail_plugins = fts fts_xapian notify replication zlib mailbox_alias
namespace inbox {
  inbox = yes
  location =
  mailbox Archive {
    auto = no
    special_use = \All
  }
  mailbox Drafts {
    auto = create
    special_use = \Drafts
  }
  mailbox Sent {
    auto = create
    special_use = \Sent
  }
  mailbox Spam {
    auto = create
    special_use = \Junk
  }
  mailbox Trash {
    auto = create
    special_use = \Trash
  }
  prefix =
  separator = /
}
passdb {
  args = /etc/secure/dovecot/dovecot-sql.conf
  driver = sql
}
plugin {
  fts = xapian
  fts_enforced = yes
  fts_xapian = partial=2 full=20
  mail_replica = tcp:10.10.2.39
  mailbox_alias_new = Sent Messages
  mailbox_alias_new2 = Sent Items
  mailbox_alias_new3 = Verzonden Items
  mailbox_alias_new4 = Verzonden
  mailbox_alias_new5 = Gesendet
  mailbox_alias_new6 = Junk
  mailbox_alias_old = Sent
  mailbox_alias_old2 = Sent
  mailbox_alias_old3 = Sent
  mailbox_alias_old4 = Sent
  mailbox_alias_old5 = Sent
  mailbox_alias_old6 = Spam
  plugin = fts fts_xapian
  sieve = dict:proxy::sieve;name=active
  sieve_before = /data/sieve/general/process-spam.sieve
  sieve_dir = /data/sieve/%d/%n/
  sieve_extensions = +editheader
  sieve_max_redirects = 6
  zlib_save = gz
  zlib_save_level = 6
}
postmaster_address = postmaster@example.com
protocols = imap lmtp
service aggregator {
  fifo_listener replication-notify-fifo {
    user = vmail
  }
  unix_listener replication-notify {
    user = vmail
  }
}
service auth {
  inet_listener {
    address = 10.10.2.38
    port = 1143
  }
  unix_listener auth-userdb {
    mode = 0600
    user = vmail
  }
}
service dict {
  unix_listener dict {
    mode = 0600
    user = vmail
  }
}
service doveadm {
  inet_listener {
    port = 2143
  }
  vsz_limit = 2 G
}
service imap {
  vsz_limit = 768 M
}
service index-worker {
  vsz_limit = 2 G
}
service indexer-worker {
  vsz_limit = 2 G
}
service lmtp {
  inet_listener lmtp {
    address = 10.10.2.38
    port = 24
  }
  user = vmail
  vsz_limit = 2 G
}
service replicator {
  unix_listener replicator-doveadm {
    mode = 0600
    user = vmail
  }
}
ssl_cert = </etc/secure/certificates/certificate.pem
ssl_client_ca_dir = /etc/ssl/certs
ssl_dh = # hidden, use -P to show it
ssl_key = # hidden, use -P to show it
userdb {
  driver = prefetch
}
userdb {
  args = /etc/secure/dovecot/dovecot-sql.conf
  driver = sql
}
protocol imap {
  mail_max_userip_connections = 50
  mail_plugins = fts fts_xapian notify replication zlib mailbox_alias imap_zlib
}
protocol lmtp {
  mail_fsync = optimized
  mail_plugins = fts fts_xapian notify replication zlib mailbox_alias sieve
}

[grosjo]

I made a slight change in the code, to figure out maybe better who is the culprit Can you git pull again ?

[tdegrunt]

Just tried this, it's again the assignment line.

I can't think of anything besides: not null terminated string, not enough memory, unintialized pointer or having a dereferenced pointer.

Here's the backtrace:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff48d16ad in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x00007ffff48d16ad in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007ffff4931529 in std::string::assign(std::string const&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007ffff52f14e9 in operator= (__str="return-path", this=0x5555558468b8) at /usr/include/c++/4.8/bits/basic_string.h:547
#3  fts_backend_xapian_update_set_build_key (_ctx=0x555555846890, key=0x7fffffffdde0) at fts-backend-xapian.cpp:306
#4  0x00007ffff65a8064 in fts_backend_update_set_build_key () from /usr/lib/dovecot/modules/lib20_fts_plugin.so
#5  0x00007ffff65a9377 in fts_build_mail () from /usr/lib/dovecot/modules/lib20_fts_plugin.so
#6  0x00007ffff65afae2 in ?? () from /usr/lib/dovecot/modules/lib20_fts_plugin.so
#7  0x00007ffff767816e in mail_precache () from /usr/lib/dovecot/libdovecot-storage.so.0
#8  0x0000555555584eaf in ?? ()
#9  0x0000555555585400 in ?? ()
#10 0x000055555557f7d7 in ?? ()
#11 0x0000555555580318 in ?? ()
#12 0x000055555558115d in doveadm_cmd_ver2_to_mail_cmd_wrapper ()
#13 0x00005555555911ca in doveadm_cmd_run_ver2 ()
#14 0x00005555555912f7 in doveadm_cmd_try_run_ver2 ()
#15 0x000055555557050c in main ()

[grosjo]

Actually the plugin works in production on my own servers without any coredump. So, maybe we should look into the difference between your setup and mine. in the configuration file you sent , it says "2 G" , instead of "2G" , maybe remove the space to make sure the memory allocated is really 2giga and not 2 bytes

[tdegrunt]

Agreed, it’s the Ubuntu 14.04 machines that have the problem. I have the plugin running on Ubuntu 18.04 without issues. Dovecot is the same version.
I’ve checked, the config file really says “default_vsz_limit = 2GB”, the output of doveconf makes this look weird.
Perhaps the libstdc++ is different.

[grosjo]

maybe a "apt-get update / upgrade" on the Ubuntu 14.04 will solve it ?

[tdegrunt]

Now trying to upgrade away from 14.04, that will probably solve the issue, thanks!

[grosjo]

Problem solved ?

[tdegrunt]

We’re running the plugin with 18.04 in production already. So let’s consider it solved, thanks!