OvenMediaEngine crashes with segmentation fault (signal 11)

[stefanomazzoni]

Describe the bug
OvenMediaEngine unexpectedly crashes with 400-500 users with segmentation fault signal

To Reproduce
Steps to reproduce the behavior:

1. Run OvenMediaEngine with default configurations
2. Start OBS with version 25.0.8
3. Connect 400+ users
4. Wait for 40 minutes or more (it crashed several times, one after 40 minutes, one after 10 minutes and another after 2 hours and 30 minutes)

Expected behavior
OvenMediaEngine stops.

Logs

***** Crash dump *****
OvenMediaEngine v0.10.1 (pid: 4018, tid: 4082)
Signal 11 (SIGSEGV) 2020-05-13T00:22:02+0000
#0 /lib/x86_64-linux-gnu/libpthread.so.0 0x7f7626798890 ? + 0x12890

Server (please complete the following information):

• OS: Ubuntu 18.04
• OvenMediaEngine Version: v0.10.2 (even though crash dump shows v0.10.1 it is 0.10.2)

[getroot]

Could you upload full crash log and last 100 lines of OvenMediaEngine.log (or full log)?

[stefanomazzoni]

Ok, I attached full crash log of 2020-05-13 and ovenmediaengine log of 2020-05-13 hour 00.

***** Crash dump *****
OvenMediaEngine v0.10.1 (pid: 4018, tid: 4082)
Signal 11 (SIGSEGV) 2020-05-13T00:22:02+0000
#0 /lib/x86_64-linux-gnu/libpthread.so.0 0x7f7626798890 ? + 0x12890
***** Crash dump *****
OvenMediaEngine v0.10.1 (pid: 10171, tid: 10238)
Signal 11 (SIGSEGV) 2020-05-13T00:29:55+0000
#0 /lib/x86_64-linux-gnu/libpthread.so.0 0x7f14eef55890 ? + 0x12890
***** Crash dump *****
OvenMediaEngine v0.10.1 (pid: 13467, tid: 13533)
Signal 11 (SIGSEGV) 2020-05-13T00:36:49+0000
#0 /lib/x86_64-linux-gnu/libpthread.so.0 0x7fa39d770890 ? + 0x12890
***** Crash dump *****
OvenMediaEngine v0.10.1 (pid: 17029, tid: 17165)
Signal 11 (SIGSEGV) 2020-05-13T00:42:41+0000
#0 /lib/x86_64-linux-gnu/libpthread.so.0 0x7fb32d52a890 ? + 0x12890
#1 /opt/ovenmediaengine/lib/libssl.so.1.1 0x7fb32c3928b1 ? + 0x268b1
#2 /opt/ovenmediaengine/lib/libssl.so.1.1 0x7fb32c3b51eb ? + 0x491eb
#3 /opt/ovenmediaengine/lib/libssl.so.1.1 0x7fb32c3a78bd ? + 0x3b8bd
#4 /opt/ovenmediaengine/lib/libssl.so.1.1 0x7fb32c39ee45 ? + 0x45
#5 /usr/bin/OvenMediaEngine 0x55d17dcd7abf ? + 0x212abf
#6 /usr/bin/OvenMediaEngine 0x55d17dc69eea ? + 0x1a4eea
#7 /usr/bin/OvenMediaEngine 0x55d17dc6aba8 ? + 0x1a5ba8
#8 /usr/bin/OvenMediaEngine 0x55d17dc67370 ? + 0x1a2370
#9 /usr/bin/OvenMediaEngine 0x55d17dcaf6f4 ? + 0x1ea6f4
#10 /usr/lib/x86_64-linux-gnu/libstdc++.so.6 0x7fb32b3176df ? + 0xbd6df
#11 /lib/x86_64-linux-gnu/libpthread.so.0 0x7fb32d51f6db ? + 0x76db
#12 /lib/x86_64-linux-gnu/libc.so.6 0x7fb32a9d488f ? + 0x3f
***** Crash dump *****
OvenMediaEngine v0.10.1 (pid: 21477, tid: 21539)
Signal 11 (SIGSEGV) 2020-05-13T00:57:45+0000
#0 /lib/x86_64-linux-gnu/libpthread.so.0 0x7f0a3ee3b890 ? + 0x12890
#1 /opt/ovenmediaengine/lib/libjemalloc.so.2 0x7f0a3cf4a5b9 ? + 0x565b9
#2 /opt/ovenmediaengine/lib/libjemalloc.so.2 0x7f0a3cf0248c ? + 0xe48c
#3 /opt/ovenmediaengine/lib/libssl.so.1.1 0x7f0a3dc9c19c ? + 0x1f19c
#4 /opt/ovenmediaengine/lib/libssl.so.1.1 0x7f0a3dc99352 ? + 0x1c352
#5 /opt/ovenmediaengine/lib/libssl.so.1.1 0x7f0a3dcb0f49 ? + 0x279
#6 /usr/bin/OvenMediaEngine 0x56023447b445 ? + 0x212445
#7 /usr/bin/OvenMediaEngine 0x56023440b4a0 ? + 0x1a24a0
#8 /usr/bin/OvenMediaEngine 0x5602342e5d2b ? + 0x7cd2b
#9 /usr/bin/OvenMediaEngine 0x56023444f858 ? + 0x1e6858
#10 /usr/bin/OvenMediaEngine 0x56023444fa4c ? + 0x1e6a4c
#11 /usr/bin/OvenMediaEngine 0x5602342d0568 ? + 0x67568
#12 /usr/bin/OvenMediaEngine 0x5602343bf601 ? + 0x156601
#13 /usr/bin/OvenMediaEngine 0x5602343bfc19 ? + 0x156c19
#14 /usr/bin/OvenMediaEngine 0x5602344038da ? + 0x19a8da
#15 /usr/bin/OvenMediaEngine 0x5602343f54e4 ? + 0x18c4e4
#16 /usr/bin/OvenMediaEngine 0x56023445db17 ? + 0x1f4b17
#17 /usr/bin/OvenMediaEngine 0x56023446f173 ? + 0x206173
#18 /usr/bin/OvenMediaEngine 0x56023446d2da ? + 0x2042da
#19 /usr/bin/OvenMediaEngine 0x56023446e7ae ? + 0x2057ae
#20 /usr/bin/OvenMediaEngine 0x56023445bd96 ? + 0x1f2d96
#21 /usr/lib/x86_64-linux-gnu/libstdc++.so.6 0x7f0a3cc286df ? + 0xbd6df
#22 /lib/x86_64-linux-gnu/libpthread.so.0 0x7f0a3ee306db ? + 0x76db
#23 /lib/x86_64-linux-gnu/libc.so.6 0x7f0a3c2e588f ? + 0x3f
***** Crash dump *****
OvenMediaEngine v0.10.1 (pid: 27776, tid: 27913)
Signal 11 (SIGSEGV) 2020-05-13T01:20:38+0000
#0 /lib/x86_64-linux-gnu/libpthread.so.0 0x7f31f486e890 ? + 0x12890
#1 /opt/ovenmediaengine/lib/libssl.so.1.1 0x7f31f36f24ba ? + 0x424ba
#2 /opt/ovenmediaengine/lib/libssl.so.1.1 0x7f31f36c857d ? + 0x1857d
#3 /opt/ovenmediaengine/lib/libssl.so.1.1 0x7f31f36f9b20 ? + 0x49b20
#4 /opt/ovenmediaengine/lib/libssl.so.1.1 0x7f31f36eb5ee ? + 0x3b5ee
#5 /opt/ovenmediaengine/lib/libssl.so.1.1 0x7f31f36e2e45 ? + 0x45
#6 /usr/bin/OvenMediaEngine 0x5565985b9abf ? + 0x212abf
#7 /usr/bin/OvenMediaEngine 0x55659854beea ? + 0x1a4eea
#8 /usr/bin/OvenMediaEngine 0x55659854cba8 ? + 0x1a5ba8
#9 /usr/bin/OvenMediaEngine 0x556598549370 ? + 0x1a2370
#10 /usr/bin/OvenMediaEngine 0x5565985916f4 ? + 0x1ea6f4
#11 /usr/lib/x86_64-linux-gnu/libstdc++.so.6 0x7f31f265b6df ? + 0xbd6df
#12 /lib/x86_64-linux-gnu/libpthread.so.0 0x7f31f48636db ? + 0x76db
#13 /lib/x86_64-linux-gnu/libc.so.6 0x7f31f1d1888f ? + 0x3f

ovenmediaengine.log.20200513_00.zip

[getroot]

We recently modified this in the Dev branch. Please let me know if your problem is solved.

[getroot]

I reproduced this problem and found and solved the possibility of the problem. (4d90c66)
It was no longer reproduced in my environment.
Please check in your environment.

[getroot]

This issue has been reported as being resolved.
If the problem is reproduced, please reopen this issue.
Thanks for your contribution.