Crash on Resume from Suspend

[Riolku]

Ran into an issue today on sway + Arch (running waybar 0.9.19) where it crashed often on system resume.

Steps

• Run waybar
• Disable the output, re-enable it, and suspend:
  swaymsg output eDP-1 disable; swaymsg output eDP-1 enable; systemctl suspend
• Resume the laptop

The issue seems to be related to some sort of mutex race condition. The errors vary, but all have to do with wakeUp.

I managed to reproduce with a config of:

{
    "ipc": true,
}

But the error is much more consistent with many modules. I failed to reproduce it with an empty config.

I built a debug binary from master and tested, applying the following diff to avoid catching the exception that kills waybar:

diff --git a/src/main.cpp b/src/main.cpp
index ff446ff..fe1a194 100644
--- a/src/main.cpp
+++ b/src/main.cpp
@@ -108,9 +108,6 @@ int main(int argc, char* argv[]) {
 
     delete client;
     return ret;
-  } catch (const std::exception& e) {
-    spdlog::error("{}", e.what());
-    return 1;
   } catch (const Glib::Exception& e) {
     spdlog::error("{}", static_cast<std::string>(e.what()));
     return 1;

This allowed me some intuition behind the error.

Here is a GDB log of reproducing the error with the minimal config above:

Temporary breakpoint 1 at 0xa0aff: file ../src/main.cpp, line 74.
Starting program: /home/keenan/Projects/Waybar/build/waybar 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

Temporary breakpoint 1, main (argc=1, argv=0x7fffffffe1d8) at ../src/main.cpp:74
74	   auto client = waybar::Client::inst();
Continuing.
[New Thread 0x7ffff27b86c0 (LWP 62592)]
[New Thread 0x7ffff1bff6c0 (LWP 62593)]
[New Thread 0x7ffff13fe6c0 (LWP 62594)]
[New Thread 0x7ffff0bfd6c0 (LWP 62595)]
[New Thread 0x7fffe3fff6c0 (LWP 62596)]
[New Thread 0x7fffe37fe6c0 (LWP 62597)]
[Thread 0x7ffff0bfd6c0 (LWP 62595) exited]

Thread 1 "waybar" received signal SIGINT, Interrupt.
0x00007ffff6313c0f in poll () from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.1.1/../../../../lib64/libc.so.6
Quit
quit
Temporary breakpoint 1 at 0xa0aff: file ../src/main.cpp, line 74.
Starting program: /home/keenan/Projects/Waybar/build/waybar 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

Temporary breakpoint 1, main (argc=1, argv=0x7fffffffe1d8) at ../src/main.cpp:74
74	   auto client = waybar::Client::inst();
Continuing.
[New Thread 0x7ffff27b86c0 (LWP 63152)]
[New Thread 0x7ffff1bff6c0 (LWP 63153)]
[New Thread 0x7ffff13fe6c0 (LWP 63154)]
[New Thread 0x7ffff0bfd6c0 (LWP 63155)]
[New Thread 0x7fffe3fff6c0 (LWP 63156)]
[New Thread 0x7fffdb7fe6c0 (LWP 63157)]
[New Thread 0x7fffe37fe6c0 (LWP 63158)]
[Thread 0x7fffe37fe6c0 (LWP 63158) exited]
[New Thread 0x7fffe37fe6c0 (LWP 63166)]
[Thread 0x7ffff0bfd6c0 (LWP 63155) exited]
[Thread 0x7fffe37fe6c0 (LWP 63166) exited]
[New Thread 0x7fffe37fe6c0 (LWP 63254)]

Thread 1 "waybar" received signal SIGABRT, Aborted.
0x00007ffff629f26c in ?? () from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.1.1/../../../../lib64/libc.so.6
warning: Memory read failed for corefile section, 4096 bytes at 0xffffffffff600000.
Saved corefile core.63146
quit

I've uploaded the core file of this run to my own server because it's too large for github.

[helloworld1]

I think I have the same crash. After resume from suspend, waybar crashed

Jul 12 21:32:55 liberty-pc systemd-coredump[47771]: Process 748 (waybar) of user 1000 dumped core.
                                                    
                                                    Stack trace of thread 748:
                                                    #0  0x00007f8333b8d26c n/a (libc.so.6 + 0x8926c)
                                                    #1  0x00007f8333b3da08 raise (libc.so.6 + 0x39a08)
                                                    #2  0x00007f8333b26538 abort (libc.so.6 + 0x22538)
                                                    #3  0x00007f8333b272db n/a (libc.so.6 + 0x232db)
                                                    #4  0x00007f8333b36227 n/a (libc.so.6 + 0x32227)
                                                    #5  0x00007f8333b938a1 n/a (libc.so.6 + 0x8f8a1)
                                                    #6  0x00007f8333b8e1ff n/a (libc.so.6 + 0x8a1ff)
                                                    #7  0x000056154e339f67 n/a (waybar + 0x66f67)
                                                    #8  0x000056154e35616d n/a (waybar + 0x8316d)
                                                    #9  0x000056154e3563a2 n/a (waybar + 0x833a2)
                                                    #10 0x00007f83346301f8 n/a (libgio-2.0.so.0 + 0xff1f8)
                                                    #11 0x00007f8334355a31 g_main_context_dispatch (libglib-2.0.so.0 + 0x5aa31)
                                                    #12 0x00007f83343b2cc9 n/a (libglib-2.0.so.0 + 0xb7cc9)
                                                    #13 0x00007f83343530e2 g_main_context_iteration (libglib-2.0.so.0 + 0x580e2)
                                                    #14 0x00007f833460eed6 g_application_run (libgio-2.0.so.0 + 0xdded6)
                                                    #15 0x000056154e31e127 n/a (waybar + 0x4b127)
                                                    #16 0x000056154e31cd45 main (waybar + 0x49d45)
                                                    #17 0x00007f8333b27850 n/a (libc.so.6 + 0x23850)
                                                    #18 0x00007f8333b2790a __libc_start_main (libc.so.6 + 0x2390a)
                                                    #19 0x000056154e3238d5 n/a (waybar + 0x508d5)
                                                    
                                                    Stack trace of thread 789:
                                                    #0  0x00007f8333c072ed syscall (libc.so.6 + 0x1032ed)
                                                    #1  0x00007f83343aaca7 g_cond_wait (libglib-2.0.so.0 + 0xafca7)
                                                    #2  0x00007f8334320144 n/a (libglib-2.0.so.0 + 0x25144)
                                                    #3  0x00007f833438a2fe n/a (libglib-2.0.so.0 + 0x8f2fe)
                                                    #4  0x00007f8334387d75 n/a (libglib-2.0.so.0 + 0x8cd75)
                                                    #5  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #6  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 861:
                                                    #0  0x00007f8333c10bcc recv (libc.so.6 + 0x10cbcc)
                                                    #1  0x000056154e372b80 n/a (waybar + 0x9fb80)
                                                    #2  0x000056154e37482e n/a (waybar + 0xa182e)
                                                    #3  0x000056154e374908 n/a (waybar + 0xa1908)
                                                    #4  0x000056154e33a051 n/a (waybar + 0x67051)
                                                    #5  0x00007f8333ee1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                    #6  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #7  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 790:
                                                    #0  0x00007f8333c01c0f __poll (libc.so.6 + 0xfdc0f)
                                                    #1  0x00007f83343b2c2f n/a (libglib-2.0.so.0 + 0xb7c2f)
                                                    #2  0x00007f83343530e2 g_main_context_iteration (libglib-2.0.so.0 + 0x580e2)
                                                    #3  0x00007f8334353132 n/a (libglib-2.0.so.0 + 0x58132)
                                                    #4  0x00007f8334387d75 n/a (libglib-2.0.so.0 + 0x8cd75)
                                                    #5  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #6  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 793:
                                                    #0  0x00007f8333c01c0f __poll (libc.so.6 + 0xfdc0f)
                                                    #1  0x00007f83343b2c2f n/a (libglib-2.0.so.0 + 0xb7c2f)
                                                    #2  0x00007f83343530e2 g_main_context_iteration (libglib-2.0.so.0 + 0x580e2)
                                                    #3  0x00007f8331eeafde n/a (libdconfsettings.so + 0x5fde)
                                                    #4  0x00007f8334387d75 n/a (libglib-2.0.so.0 + 0x8cd75)
                                                    #5  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #6  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 863:
                                                    #0  0x00007f8333c10bcc recv (libc.so.6 + 0x10cbcc)
                                                    #1  0x000056154e372b80 n/a (waybar + 0x9fb80)
                                                    #2  0x000056154e37482e n/a (waybar + 0xa182e)
                                                    #3  0x000056154e37b938 n/a (waybar + 0xa8938)
                                                    #4  0x000056154e33a051 n/a (waybar + 0x67051)
                                                    #5  0x00007f8333ee1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                    #6  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #7  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 792:
                                                    #0  0x00007f8333c01c0f __poll (libc.so.6 + 0xfdc0f)
                                                    #1  0x00007f83343b2c2f n/a (libglib-2.0.so.0 + 0xb7c2f)
                                                    #2  0x00007f8334354fef g_main_loop_run (libglib-2.0.so.0 + 0x59fef)
                                                    #3  0x00007f833464128c n/a (libgio-2.0.so.0 + 0x11028c)
                                                    #4  0x00007f8334387d75 n/a (libglib-2.0.so.0 + 0x8cd75)
                                                    #5  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #6  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 860:
                                                    #0  0x00007f8333c10bcc recv (libc.so.6 + 0x10cbcc)
                                                    #1  0x000056154e372b80 n/a (waybar + 0x9fb80)
                                                    #2  0x000056154e37482e n/a (waybar + 0xa182e)
                                                    #3  0x000056154e37b998 n/a (waybar + 0xa8998)
                                                    #4  0x000056154e33a051 n/a (waybar + 0x67051)
                                                    #5  0x00007f8333ee1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                    #6  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #7  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 866:
                                                    #0  0x00007f8333b87f0e n/a (libc.so.6 + 0x83f0e)
                                                    #1  0x00007f8333b8ad85 pthread_cond_clockwait (libc.so.6 + 0x86d85)
                                                    #2  0x000056154e3ff3f4 n/a (waybar + 0x12c3f4)
                                                    #3  0x000056154e33a051 n/a (waybar + 0x67051)
                                                    #4  0x00007f8333ee1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                    #5  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #6  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 864:
                                                    #0  0x00007f8333c01c0f __poll (libc.so.6 + 0xfdc0f)
                                                    #1  0x00007f83342658c7 n/a (libpulse.so.0 + 0x338c7)
                                                    #2  0x00007f833424f46c pa_mainloop_poll (libpulse.so.0 + 0x1d46c)
                                                    #3  0x00007f833425942c pa_mainloop_iterate (libpulse.so.0 + 0x2742c)
                                                    #4  0x00007f83342594e1 pa_mainloop_run (libpulse.so.0 + 0x274e1)
                                                    #5  0x00007f8334269c02 n/a (libpulse.so.0 + 0x37c02)
                                                    #6  0x00007f8333176c67 n/a (libpulsecommon-16.1.so + 0x5bc67)
                                                    #7  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #8  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 865:
                                                    #0  0x00007f8333b87f0e n/a (libc.so.6 + 0x83f0e)
                                                    #1  0x00007f8333b8ad85 pthread_cond_clockwait (libc.so.6 + 0x86d85)
                                                    #2  0x000056154e3ff3f4 n/a (waybar + 0x12c3f4)
                                                    #3  0x000056154e33a051 n/a (waybar + 0x67051)
                                                    #4  0x00007f8333ee1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                    #5  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #6  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 870:
                                                    #0  0x00007f8333b87f0e n/a (libc.so.6 + 0x83f0e)
                                                    #1  0x00007f8333b8ad85 pthread_cond_clockwait (libc.so.6 + 0x86d85)
                                                    #2  0x000056154e3ff3f4 n/a (waybar + 0x12c3f4)
                                                    #3  0x000056154e33a051 n/a (waybar + 0x67051)
                                                    #4  0x00007f8333ee1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                    #5  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #6  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 871:
                                                    #0  0x00007f8333b87f0e n/a (libc.so.6 + 0x83f0e)
                                                    #1  0x00007f8333b8ad85 pthread_cond_clockwait (libc.so.6 + 0x86d85)
                                                    #2  0x000056154e3ff3f4 n/a (waybar + 0x12c3f4)
                                                    #3  0x000056154e33a051 n/a (waybar + 0x67051)
                                                    #4  0x00007f8333ee1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                    #5  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #6  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 872:
                                                    #0  0x00007f8333b87f0e n/a (libc.so.6 + 0x83f0e)
                                                    #1  0x00007f8333b8ad85 pthread_cond_clockwait (libc.so.6 + 0x86d85)
                                                    #2  0x000056154e3ff3f4 n/a (waybar + 0x12c3f4)
                                                    #3  0x000056154e33a051 n/a (waybar + 0x67051)
                                                    #4  0x00007f8333ee1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                    #5  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #6  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 786:
                                                    #0  0x00007f8333b3e76a __sigtimedwait (libc.so.6 + 0x3a76a)
                                                    #1  0x00007f8333b3de3c sigwait (libc.so.6 + 0x39e3c)
                                                    #2  0x000056154e343e7c n/a (waybar + 0x70e7c)
                                                    #3  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #4  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 867:
                                                    #0  0x00007f8333b87f0e n/a (libc.so.6 + 0x83f0e)
                                                    #1  0x00007f8333b8ad85 pthread_cond_clockwait (libc.so.6 + 0x86d85)
                                                    #2  0x000056154e3ff3f4 n/a (waybar + 0x12c3f4)
                                                    #3  0x000056154e33a051 n/a (waybar + 0x67051)
                                                    #4  0x00007f8333ee1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                    #5  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #6  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 878:
                                                    #0  0x00007f8333c0f266 epoll_wait (libc.so.6 + 0x10b266)
                                                    #1  0x000056154e3a3c0d n/a (waybar + 0xd0c0d)
                                                    #2  0x000056154e33a051 n/a (waybar + 0x67051)
                                                    #3  0x00007f8333ee1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                    #4  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #5  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 877:
                                                    #0  0x00007f8333b87f0e n/a (libc.so.6 + 0x83f0e)
                                                    #1  0x00007f8333b8ad85 pthread_cond_clockwait (libc.so.6 + 0x86d85)
                                                    #2  0x000056154e3ff3f4 n/a (waybar + 0x12c3f4)
                                                    #3  0x000056154e33a051 n/a (waybar + 0x67051)
                                                    #4  0x00007f8333ee1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                    #5  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #6  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    
                                                    Stack trace of thread 879:
                                                    #0  0x00007f8333b87f0e n/a (libc.so.6 + 0x83f0e)
                                                    #1  0x00007f8333b8ad85 pthread_cond_clockwait (libc.so.6 + 0x86d85)
                                                    #2  0x000056154e3ff3f4 n/a (waybar + 0x12c3f4)
                                                    #3  0x000056154e33a051 n/a (waybar + 0x67051)
                                                    #4  0x00007f8333ee1943 execute_native_thread_routine (libstdc++.so.6 + 0xe1943)
                                                    #5  0x00007f8333b8b44b n/a (libc.so.6 + 0x8744b)
                                                    #6  0x00007f8333c0ee40 n/a (libc.so.6 + 0x10ae40)
                                                    ELF object binary architecture: AMD x86-64

[manfred3000]

@Riolku Thanks for the detailed report (maybe consider changing the title - the 'mutex' part may prevent other users with a similar issue from reading here).

I'm experiencing a similar issue running waybar 0.9.19 on Arch Linux (wlroots based compositors sway, wayfire). waybar keeps crashing silently or just freezes after waking up from suspend and is pretty much unusable. I tried with disabling modules to no avail.

@Alexays Could this commit be related?

[Riolku]

git bisect claims that 070110a is the first bad commit.

[Riolku]

Unsure if git bisect is finding a different segfault.

[mysfitt]

I believe that issue #2310 is a dup of this same bug. Please see my comment there as I did a backtrace to isolate the bug to a specific commit in the 0.9.19 release. The quick fix is to roll back to version 0.9.18.

[Alexays]

@xytovl Maybe you can check, seem related to your work

[xytovl]

With the given command
swaymsg output eDP-1 disable; swaymsg output eDP-1 enable; systemctl suspend
I could reproduce, I have made a commit in #2324 that fixes it for me. Can you verify this works for you?

[Riolku]

Seems to work