encfs does not want to exit by itself after inactivity

[benrubson]

Hello,

Sometimes, I encounter the following behavior when using inactivity auto-unmount :

Feb 16 21:15:07 s1 encfs/usr1[5293]: INFO Filesystem mounted: /home/usr1/dec/
Feb 16 21:23:18 s1 encfs/usr1[5293]: INFO Filesystem inactive, unmounted: /home/usr1/dec/

Filesystem is properly unmounted, however, encfs process is still running :

# ps aux | grep -E "^USER|encfs"
USER  PID %CPU %MEM   VSZ   RSS TT STAT STARTED    TIME COMMAND
usr1 5293  0.0  0.1 86928 55688  - Is   Fri21   1:50.86 /usr/local/bin/encfs -S -i 3 -t
    encfs/usr1 --no-default-flags -o use_ino -o direct_io /home/usr1/enc /home/usr1/dec

# lsof -p 5293
COMMAND  PID USER   FD   TYPE             DEVICE   SIZE/OFF NODE NAME
encfs   5293 usr1  cwd   VDIR      42,4090757305         24    4 /
encfs   5293 usr1  rtd   VDIR      42,4090757305         24    4 /
encfs   5293 usr1    0u  VCHR               0,23        0t0   23 /dev/null
encfs   5293 usr1    1u  VCHR               0,23        0t0   23 /dev/null
encfs   5293 usr1    2u  VCHR               0,23        0t0   23 /dev/null
encfs   5293 usr1    3u  unix 0xfffff801377976c0        0t0      ->0xfffff8017053aa20
encfs   5293 usr1    4u  VCHR               0,74 0x34bf834d   74 /dev/fuse

I already had this issue from time to time with 1.9.1, still the same with 1.9.4.

Any idea why ?
A race at the end of the code path ?

Many thanks 👍

Ben

[benrubson]

I've put some log around pthread_* calls to try to understand what happens next time issue occurs.

[benrubson]

So issue reproduced last night.

According to my debug log lines, main thread seems to still be in :
int res = fuse_main(encfsArgs->fuseArgc, ...
https://github.com/vgough/encfs/blob/v1.9.4/encfs/main.cpp#L683
The debug log line just after does not show-up in syslog.

# grep 87701 /var/log/encfs
Mar 19 21:15:06 s1 encfs/usr1[87701]: INFO Filesystem mounted: /home/usr1/dec/
Mar 19 21:26:17 s1 encfs/usr1[87701]: INFO Filesystem inactive, unmounted: /home/usr1/dec/

# lsof -p 87701
COMMAND   PID USER   FD   TYPE             DEVICE   SIZE/OFF NODE NAME
encfs   87701 usr1  cwd   VDIR      42,4090757305         24    4 /
encfs   87701 usr1  rtd   VDIR      42,4090757305         24    4 /
encfs   87701 usr1    0u  VCHR               0,21        0t0   21 /dev/null
encfs   87701 usr1    1u  VCHR               0,21        0t0   21 /dev/null
encfs   87701 usr1    2u  VCHR               0,21        0t0   21 /dev/null
encfs   87701 usr1    3u  unix 0xfffff809574f1360        0t0      ->0xfffff8013882fa20
encfs   87701 usr1    4u  VCHR               0,73 0x2e65a09e   73 /dev/fuse

# ps aux -H | grep -E "^USER|encfs"
USER   PID %CPU %MEM   VSZ   RSS TT STAT STARTED  TIME COMMAND
usr1 87701  0.0  0.1 87848 56996  - Is   21:15 0:00.00 /usr/local/bin/encfs -S -i 3 -t encfs/usr1 --no-default-flags -o use_ino -o direct_io /home/usr1/enc /home/usr1/dec
usr1 87701  0.0  0.1 87848 56996  - Is   21:15 0:33.90 /usr/local/bin/encfs -S -i 3 -t encfs/usr1 --no-default-flags -o use_ino -o direct_io /home/usr1/enc /home/usr1/dec
usr1 87701  0.0  0.1 87848 56996  - Is   21:15 0:21.91 /usr/local/bin/encfs -S -i 3 -t encfs/usr1 --no-default-flags -o use_ino -o direct_io /home/usr1/enc /home/usr1/dec
usr1 87701  0.0  0.1 87848 56996  - Is   21:15 0:00.38 /usr/local/bin/encfs -S -i 3 -t encfs/usr1 --no-default-flags -o use_ino -o direct_io /home/usr1/enc /home/usr1/dec

[rfjakob]

Can you attach gdb and get a backtrace?

[rfjakob]

I wonder what we are actually waiting for here

encfs/encfs/main.cpp

Line 765 in c9d6ec8

pthread_cond_wait(&ctx->wakeupCond, &ctx->wakeupMutex);

the FS has been umounted, couldn't we just exit right there?

[benrubson]

the FS has been umounted, couldn't we just exit right there?

Yes we could, as we are just waiting to be waken-up by the main thread to exit...
I think we could simply remove this line.

Can you attach gdb and get a backtrace?

https://pastebin.com/raw/d520L5Hc

[benrubson]

Issue reproduced again.
(even with the idleMonitor thread exited, #505, but it's not a surprise).

Again, main thread seems to hang in fuse_main().

Here are several backtraces :
https://pastebin.com/raw/ubxzKck8
https://pastebin.com/raw/FXRy2u22
https://pastebin.com/raw/7CG9BuDq

[rfjakob]

Alll threads hanging in pthread_cleanup_pop. Is it possible that this is a bug in FreeBSD's /lib/libthr.so.3?

[benrubson]

I wonder if this could effectively be a FreeBSD bug.
All threads are stuck in pthread_cleanup_pop, main thread then hanging in fuse_main.
Sounds then like to me it may be out of EncFS control.
I'll open a FreeBSD bug and will see if we can go further 👍
Edit : FreeBSD bug report : https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=227826