Galera Load Balancer connections crush

[smartdoc]

Hi there!

I try to connect more than 2 clients to galera cluster through glb.
What have i got:

root@galera1:~/glb-1.0.1# glbd -w exec:"/root/glb-1.0.1/files/mysql.sh -uroot -proot" -m 512 -t 3 3306 galera3:3308 galera2:3308 galera1:3308

Watchdog:

Address : exp setw state lat curw
192.168.55.115:3308 : + 1.000 READY 0.18607 1.000
192.168.55.114:3308 : + 1.000 READY 0.17800 1.000
127.0.1.1:3308 : + 1.000 READY 0.17787 1.000
Destinations: 3

Router:
Address : weight usage map conns
127.0.1.1:3308 : 1.000 0.500 N/A 1
192.168.55.114:3308 : 1.000 0.500 N/A 1
192.168.55.115:3308 : 1.000 0.500 N/A 1
Destinations: 3, total connections: 3 of 512 max

Pool: connections per thread: 1 1 1

glbd: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
INFO: glb_signal.c:42: Received signal 6. Terminating.
Aborted

I can send you virtualbox vm with galera cluster and glb.
Please, fix it.
Sincerely,
from Russia, with love.

[ayurchen]

Sorry for late reply. As you can see, it looks very much like a system issue, at least it reports assertion failure in malloc code. This of course is very unlikely, but to find the offending code in glbd we need a backtrace.

For that, please

1. build glbd with debug option: --with-debug
2. enable core dumps in the shell session:
   $ ulimit -c unlimited
3. start glbd in that session
4. when it aborts, locate the core file (usually 'core' in the working directory)
5. run
   $ gdb -c -batch -ex "thr apply all bt" > stack.dump

Post stack.dump here, together with the glbd command line.

Thanks,
Alex

[ayurchen]

Don't remove the core file, it may be needed for further investigation.

[smartdoc]

Hi, Alex.
Regarding to this problem.
i reconfigure glb, and install it.
Then, try to start it:
root@galera1:~/glb-1.0.1# glbd -w exec:"/root/glb-1.0.1/files/mysql.sh -uroot -proot" -t 3 0.0.0.0:3306 galera1:3308 galera2:3308 galera3:3308
glb v1.0.1 (epoll)
Incoming address: 0.0.0.0:3306, control FIFO: /tmp/glbd.fifo
Control address: none
Number of threads: 3, max conn: 493, nodelay: ON, keepalive: ON, defer accept: OFF, linger: OFF, daemon: NO, lat.count: 0, policy: 'least connected', top: NO, verbose: NO
Destinations: 3
0: 127.0.1.1:3308 , w: 1.000
1: 192.168.55.114:3308 , w: 1.000
2: 192.168.55.115:3308 , w: 1.000
Watchdog:
Address : exp setw state lat curw
127.0.1.1:3308 : + 1.000 READY 0.21660 1.000
192.168.55.114:3308 : + 1.000 NOTFOUND 0.00000 -1.000
192.168.55.115:3308 : + 1.000 NOTFOUND 0.00000 -1.000
Destinations: 3

Router:
Address : weight usage map conns
127.0.1.1:3308 : 1.000 0.000 N/A 0
Destinations: 1, total connections: 0 of 493 max

Pool: connections per thread: 0 0 0

*** glibc detected *** glbd: realloc(): invalid pointer: 0x095961c0 ***
======= Backtrace: =========
/lib/i386-linux-gnu/i686/cmov/libc.so.6(+0x70f01)[0xb7673f01]
/lib/i386-linux-gnu/i686/cmov/libc.so.6(realloc+0x2bb)[0xb7679acb]
glbd[0x804d15f]
======= Memory map: ========
08048000-0805c000 r-xp 00000000 08:01 13804 /usr/sbin/glbd
0805c000-0805d000 rw-p 00013000 08:01 13804 /usr/sbin/glbd
09588000-095a9000 rw-p 00000000 00:00 0 [heap]
b2d08000-b2d24000 r-xp 00000000 08:01 129301 /lib/i386-linux-gnu/libgcc_s.so.1
b2d24000-b2d25000 rw-p 0001b000 08:01 129301 /lib/i386-linux-gnu/libgcc_s.so.1
b2d2a000-b2d2b000 ---p 00000000 00:00 0
b2d2b000-b352b000 rw-p 00000000 00:00 0
b352b000-b352c000 ---p 00000000 00:00 0
b352c000-b3d2c000 rw-p 00000000 00:00 0
b3d2c000-b3d2d000 ---p 00000000 00:00 0
b3d2d000-b452f000 rw-p 00000000 00:00 0
b452f000-b4530000 ---p 00000000 00:00 0
b4530000-b4d30000 rw-p 00000000 00:00 0
b4d30000-b4d31000 ---p 00000000 00:00 0
b4d31000-b5531000 rw-p 00000000 00:00 0
b5531000-b5532000 ---p 00000000 00:00 0
b5532000-b5d32000 rw-p 00000000 00:00 0
b5d32000-b5d33000 ---p 00000000 00:00 0
b5d33000-b6533000 rw-p 00000000 00:00 0
b6533000-b6534000 ---p 00000000 00:00 0
b6534000-b6d34000 rw-p 00000000 00:00 0
b6d34000-b6d35000 ---p 00000000 00:00 0
b6d35000-b75f6000 rw-p 00000000 00:00 0
b75f6000-b7600000 r-xp 00000000 08:01 134570 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
b7600000-b7601000 r--p 00009000 08:01 134570 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
b7601000-b7602000 rw-p 0000a000 08:01 134570 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
b7602000-b7603000 rw-p 00000000 00:00 0
b7603000-b775f000 r-xp 00000000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b775f000-b7760000 ---p 0015c000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b7760000-b7762000 r--p 0015c000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b7762000-b7763000 rw-p 0015e000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b7763000-b7767000 rw-p 00000000 00:00 0
b7767000-b777c000 r-xp 00000000 08:01 134565 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
b777c000-b777d000 r--p 00014000 08:01 134565 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
b777d000-b777e000 rw-p 00015000 08:01 134565 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
b777e000-b7780000 rw-p 00000000 00:00 0
b7780000-b7782000 r-xp 00000000 08:01 134569 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
b7782000-b7783000 r--p 00001000 08:01 134569 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
b7783000-b7784000 rw-p 00002000 08:01 134569 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
b7784000-b778b000 rw-p 00000000 00:00 0
b778b000-b778c000 r-xp 00000000 00:00 0 [vdso]
b778c000-b77a8000 r-xp 00000000 08:01 129352 /lib/i386-linux-gnu/ld-2.13.so
b77a8000-b77a9000 r--p 0001b000 08:01 129352 /lib/i386-linux-gnu/ld-2.13.so
b77a9000-b77aa000 rw-p 0001c000 08:01 129352 /lib/i386-linux-gnu/ld-2.13.so
bfa15000-bfa36000 rw-p 00000000 00:00 0 [stack]
INFO: glb_signal.c:42: Received signal 6. Terminating.
Aborted (core dumped)

It's created core dump file.
I make analyze it and get the next thing:

root@galera1:/glb-1.0.1# gdb -c core -batch -ex "thr apply all bt" > stack.dump
root@galera1:/glb-1.0.1# more stack.dump
[New LWP 18833]
[New LWP 18874]
[New LWP 18875]
[New LWP 18818]
[New LWP 18819]
[New LWP 18820]
[New LWP 18821]
[New LWP 18822]
[New LWP 18824]
[New LWP 18828]
Core was generated by `glbd -w exec /root/glb-1.0.1/files/mysql.sh -uroot -proot -t 3 0.0.
0.0:3306 gal'.
Program terminated with signal 6, Aborted.
#0 0xb778b422 in __kernel_vsyscall ()

Thread 10 (LWP 18828):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb7771703 in ?? ()
#2 0x00000000 in ?? ()

Thread 9 (LWP 18824):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb7771703 in ?? ()
#2 0x00000000 in ?? ()

Thread 8 (LWP 18822):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb7771703 in ?? ()
#2 0x00000000 in ?? ()

Thread 7 (LWP 18821):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb76d91f6 in ?? ()
#2 0x00000000 in ?? ()

Thread 6 (LWP 18820):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb76d91f6 in ?? ()
#2 0x00000000 in ?? ()

Thread 5 (LWP 18819):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb76d91f6 in ?? ()
#2 0x00000000 in ?? ()

Thread 4 (LWP 18818):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb76a3326 in ?? ()
#2 0x00000000 in ?? ()

Thread 3 (LWP 18875):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb76da3f8 in ?? ()
#2 0xb352a334 in ?? ()
#3 0xb76d878e in ?? ()

Thread 2 (LWP 18874):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb76cad96 in ?? ()
#2 0x000003e8 in ?? ()
#3 0x00000001 in ?? ()
#4 0x09596200 in ?? ()
#5 0x0804ee40 in ?? ()
#6 0x09596224 in ?? ()
#7 0xb76d878e in ?? ()

Thread 1 (LWP 18833):
#0 0xb778b422 in __kernel_vsyscall ()
#1 0xb762d941 in ?? ()
#2 0xb7761ff4 in ?? ()
#3 0xb7630d02 in ?? ()
#4 0x00000006 in ?? ()
#5 0xb452b924 in ?? ()
#6 0xb7669e15 in ?? ()
#7 0x00000016 in ?? ()
#8 0xb452bb70 in ?? ()
#9 0xb7673f01 in ?? ()
#10 0x00000002 in ?? ()
#11 0xb7741e70 in ?? ()
#12 0xbfa35e22 in ?? ()
#13 0xb7679acb in ?? ()
#14 0x095961c0 in ?? ()
#15 0x0804d15f in ?? ()
#16 0x095961c0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

That's all. Don't hesitate to push me if you need something else ;)
I wish you a Merry Christmas and happy holidays :)

Regards,
V.

[ayurchen]

Hi!

1. You forgot to include path to glbd binary in tha arguments to gdb. As a result it failed to find symbols definitions.
2. installing glbd may lead to stripping the debug info from it. Don't install. Run glbd from where it was built. E.g.

root@galera1:/glb-1.0.1# ./src/glbd -w exec:"/root/glb-1.0.1/files/mysql.sh -uroot -proot" -t 3 0.0.0.0:3306 galera1:3308 galera2:3308 galera3:3308
root@galera1:/glb-1.0.1# gdb ./src/glbd -c core -batch -ex "thr apply all bt" > stack.dump

3. You don't have to be root for that. And you better not be root.

Regards,
Alex

[smartdoc]

Sorry for late response. I was on vacation and throw out computer :)
Well, about glb.
3rd point doesn't matter, cause of i use virtual machines.
What have i got using your recommendations:

root@galera1:/glb-1.0.1# ulimit -c unlimited
root@galera1:/glb-1.0.1# ./src/glbd -w exec:"/root/glb-1.0.1/files/mysql.sh -uroot -proot" -t 3 0.0.0.0:3306 galera1:3308 galera2:3308 galera3:3308
glb v1.0.1 (epoll)
Incoming address: 0.0.0.0:3306, control FIFO: /tmp/glbd.fifo
Control address: none
Number of threads: 3, max conn: 493, nodelay: ON, keepalive: ON, defer accept: OFF, linger: OFF, daemon: NO, lat.count: 0, policy: 'least connected', top: NO, verbose: NO
Destinations: 3
0: 127.0.1.1:3308 , w: 1.000
1: 192.168.55.114:3308 , w: 1.000
2: 192.168.55.115:3308 , w: 1.000
Watchdog:

    Address       : exp  setw     state    lat     curw
  127.0.1.1:3308  :  +   1.000    READY  0.20919   1.000

192.168.55.114:3308 : + 1.000 READY 0.21450 1.000
192.168.55.115:3308 : + 1.000 NOTFOUND 0.00000 -1.000

Destinations: 3

Router:

    Address       :   weight   usage    map  conns

192.168.55.114:3308 : 1.000 0.000 N/A 0
127.0.1.1:3308 : 1.000 0.000 N/A 0

Destinations: 2, total connections: 0 of 493 max

Pool: connections per thread: 0 0 0

*** glibc detected *** ./src/glbd: realloc(): invalid pointer: 0x09481208 ***
===== Backtrace: =====
/lib/i386-linux-gnu/i686/cmov/libc.so.6(+0x70f01)[0xb761ff01]
/lib/i386-linux-gnu/i686/cmov/libc.so.6(realloc+0x2bb)[0xb7625acb]
./src/glbd[0x804cd75]
./src/glbd[0x805051e]
./src/glbd[0x80507d7]
/lib/i386-linux-gnu/i686/cmov/libpthread.so.0(+0x5c39)[0xb7718c39]
/lib/i386-linux-gnu/i686/cmov/libc.so.6(clone+0x5e)[0xb768478e]
===== Memory map: =====
08048000-0805d000 r-xp 00000000 08:01 135397 /root/glb-1.0.1/src/glbd
0805d000-0805e000 rw-p 00014000 08:01 135397 /root/glb-1.0.1/src/glbd
09473000-09494000 rw-p 00000000 00:00 0 [heap]
b2cb4000-b2cd0000 r-xp 00000000 08:01 129301 /lib/i386-linux-gnu/libgcc_s.so.1
b2cd0000-b2cd1000 rw-p 0001b000 08:01 129301 /lib/i386-linux-gnu/libgcc_s.so.1
b2cd6000-b2cd7000 ---p 00000000 00:00 0
b2cd7000-b34d7000 rw-p 00000000 00:00 0
b34d7000-b34d8000 ---p 00000000 00:00 0
b34d8000-b3cd8000 rw-p 00000000 00:00 0
b3cd8000-b3cd9000 ---p 00000000 00:00 0
b3cd9000-b44db000 rw-p 00000000 00:00 0
b44db000-b44dc000 ---p 00000000 00:00 0
b44dc000-b4cdc000 rw-p 00000000 00:00 0
b4cdc000-b4cdd000 ---p 00000000 00:00 0
b4cdd000-b54dd000 rw-p 00000000 00:00 0
b54dd000-b54de000 ---p 00000000 00:00 0
b54de000-b5cde000 rw-p 00000000 00:00 0
b5cde000-b5cdf000 ---p 00000000 00:00 0
b5cdf000-b64df000 rw-p 00000000 00:00 0
b64df000-b64e0000 ---p 00000000 00:00 0
b64e0000-b6ce0000 rw-p 00000000 00:00 0
b6ce0000-b6ce1000 ---p 00000000 00:00 0
b6ce1000-b75a2000 rw-p 00000000 00:00 0
b75a2000-b75ac000 r-xp 00000000 08:01 134570 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
b75ac000-b75ad000 r--p 00009000 08:01 134570 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
b75ad000-b75ae000 rw-p 0000a000 08:01 134570 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
b75ae000-b75af000 rw-p 00000000 00:00 0
b75af000-b770b000 r-xp 00000000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b770b000-b770c000 ---p 0015c000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b770c000-b770e000 r--p 0015c000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b770e000-b770f000 rw-p 0015e000 08:01 134576 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
b770f000-b7713000 rw-p 00000000 00:00 0
b7713000-b7728000 r-xp 00000000 08:01 134565 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
b7728000-b7729000 r--p 00014000 08:01 134565 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
b7729000-b772a000 rw-p 00015000 08:01 134565 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
b772a000-b772c000 rw-p 00000000 00:00 0
b772c000-b772e000 r-xp 00000000 08:01 134569 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
b772e000-b772f000 r--p 00001000 08:01 134569 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
b772f000-b7730000 rw-p 00002000 08:01 134569 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
b7730000-b7737000 rw-p 00000000 00:00 0
b7737000-b7738000 r-xp 00000000 00:00 0 [vdso]
b7738000-b7754000 r-xp 00000000 08:01 129352 /lib/i386-linux-gnu/ld-2.13.so
b7754000-b7755000 r--p 0001b000 08:01 129352 /lib/i386-linux-gnu/ld-2.13.so
b7755000-b7756000 rw-p 0001c000 08:01 129352 /lib/i386-linux-gnu/ld-2.13.so
bfc54000-bfc75000 rw-p 00000000 00:00 0 [stack]
INFO: glb_signal.c:42: Received signal 6. Terminating.
Aborted (core dumped)

After it, dumped core was delivered to gdb.

root@galera1:~/glb-1.0.1# gdb ./src/glbd -c core -batch -ex "thr apply all bt" > stack.dump

warning: Can't read pathname for load map: Input/output error.
root@galera1:~/glb-1.0.1# cat stack.dump
[New LWP 13402]
[New LWP 13446]
[New LWP 13447]
[New LWP 13389]
[New LWP 13390]
[New LWP 13391]
[New LWP 13392]
[New LWP 13393]
[New LWP 13395]
[New LWP 13399]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `./src/glbd -w exec /root/glb-1.0.1/files/mysql.sh -uroot -proot -t 3 0.0.0.0:33'.
Program terminated with signal 6, Aborted.
#0 0xb7737422 in __kernel_vsyscall ()

Thread 10 (Thread 0xb4cdbb70 (LWP 13399)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb771d703 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#2 0x080519f4 in exec_thread (arg=0x947f920) at glb_wdog_exec.c:281
#3 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#4 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 9 (Thread 0xb54dcb70 (LWP 13395)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb771d703 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#2 0x080519f4 in exec_thread (arg=0x947e408) at glb_wdog_exec.c:281
#3 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#4 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 8 (Thread 0xb5cddb70 (LWP 13393)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb771d703 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#2 0x080519f4 in exec_thread (arg=0x947d1d8) at glb_wdog_exec.c:281
#3 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#4 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 7 (Thread 0xb64deb70 (LWP 13392)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb76851f6 in epoll_wait () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0x080536c0 in pool_fds_wait (pool=0xb75611a8) at glb_pool.c:261
#3 0x08054c74 in pool_thread (arg=0xb75611a8) at glb_pool.c:829
#4 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#5 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 6 (Thread 0xb6cdfb70 (LWP 13391)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb76851f6 in epoll_wait () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0x080536c0 in pool_fds_wait (pool=0xb75210f0) at glb_pool.c:261
#3 0x08054c74 in pool_thread (arg=0xb75210f0) at glb_pool.c:829
#4 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#5 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 5 (Thread 0xb74e0b70 (LWP 13390)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb76851f6 in epoll_wait () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0x080536c0 in pool_fds_wait (pool=0xb74e1038) at glb_pool.c:261
#3 0x08054c74 in pool_thread (arg=0xb74e1038) at glb_pool.c:829
#4 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#5 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 4 (Thread 0xb75ae6c0 (LWP 13389)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb764f326 in nanosleep () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0xb764f150 in sleep () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#3 0x08056b37 in main (argc=9, argv=0xbfc74754) at glb_main.c:217

Thread 3 (Thread 0xb34d6b70 (LWP 13447)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb76863f8 in accept4 () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0x08055912 in listener_thread (arg=0x9480ea0) at glb_listener.c:50
#3 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#4 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 2 (Thread 0xb3cd7b70 (LWP 13446)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb7676d96 in poll () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0x0804ea7b in ctrl_thread (arg=0x9481288) at glb_control.c:219
#3 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#4 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

Thread 1 (Thread 0xb44d8b70 (LWP 13402)):
#0 0xb7737422 in __kernel_vsyscall ()
#1 0xb75d9941 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0xb75dcd02 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#3 0xb7615e15 in ?? () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#4 0xb761ff01 in ?? () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#5 0xb7625acb in realloc () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#6 0x0804cd75 in glb_router_change_dst (router=0x9473d18, dst=0xb44d82e0, probe_ctx=0x947f920) at glb_router.c:280
#7 0x0805051e in wdog_collect_results (wdog=0x9473678) at glb_wdog.c:563
#8 0x080507d7 in wdog_main_loop (arg=0x9473678) at glb_wdog.c:645
#9 0xb7718c39 in start_thread () from /lib/i386-linux-gnu/i686/cmov/libpthread.so.0
#10 0xb768478e in clone () from /lib/i386-linux-gnu/i686/cmov/libc.so.6

I hope, it helps,
Regards,
V.