Assertion failed with vlc in fullscreen [experimental-backends]

[tryone144]

Platform

ArchLinux with linux 5.6.7.arch1-1 on amd64

GPU, drivers, and screen setup

• NVidia GTX 660Ti with nvidia 440.82-8, nvidia-utils 440.82-1, mesa 20.0.5-1
• Two monitors side-by-side with xrandr (2x 1920x1200).
• glxinfo: https://pastebin.com/8Fehjmsq

Environment

• WM: i3-gaps 4.18.1-1
• Tested with vlc 3.0.9.2-2

picom version

**Version:** vgit-0b377

### Extensions:

* Shape: Yes
* XRandR: Yes
* Present: Present

### Misc:

* Use Overlay: Yes* Config file used: picom.conf

### Drivers (inaccurate):

NVIDIA  

Configuration:

• Configuration: https://pastebin.com/y2QsBa00

Steps of reproduction

1. Start picom with --experimental-backends
2. Play a video in vlc and enter fullscreen mode
3. Toggle visibility of the playback controls a few times either by moving the mouse, opening/closing the context menu, entering/leaving the window

Expected behavior

no crash 😉

Current Behavior

Picom suddenly crashes with SIGABRT on

picom: ../src/backend/backend.c:270: paint_all_new: Assertion »!(w->flags & WIN_FLAGS_SHADOW_NONE)« failed.

OR

picom: ../src/win.c:318: win_release_images: Assertion »!win_check_flags_all(w, WIN_FLAGS_SHADOW_STALE)« failed.

Stack trace

gef➤  bt
#0  0x00007fcecbf4ece5 in raise () from /usr/lib/libc.so.6
#1  0x00007fcecbf38857 in abort () from /usr/lib/libc.so.6
#2  0x00007fcecbf38727 in __assert_fail_base.cold () from /usr/lib/libc.so.6
#3  0x00007fcecbf47426 in __assert_fail () from /usr/lib/libc.so.6
#4  0x0000564e19cf19d9 in paint_all_new (ps=0x564e19e05390, t=0x564e19e987a0, ignore_damage=0x0) at ../src/backend/backend.c:270
#5  0x0000564e19cba82a in _draw_callback (loop=0x7fcecc8cd720, ps=0x564e19e05390, revents=0x2000) at ../src/picom.c:1443
#6  0x0000564e19cba946 in draw_callback (loop=0x7fcecc8cd720, w=0x564e19e05450, revents=0x2000) at ../src/picom.c:1467
#7  0x00007fcecc8c1033 in ev_invoke_pending () from /usr/lib/libev.so.4
#8  0x00007fcecc8c4902 in ev_run () from /usr/lib/libev.so.4
#9  0x0000564e19cbda92 in session_run (ps=0x564e19e05390) at ../src/picom.c:2323
#10 0x0000564e19cbdd89 in main (argc=0x4, argv=0x7ffff0d6e528) at ../src/picom.c:2424

OR

gef➤  bt
#0  0x00007f53d43ccce5 in raise () from /usr/lib/libc.so.6
#1  0x00007f53d43b6857 in abort () from /usr/lib/libc.so.6
#2  0x00007f53d43b6727 in __assert_fail_base.cold () from /usr/lib/libc.so.6
#3  0x00007f53d43c5426 in __assert_fail () from /usr/lib/libc.so.6
#4  0x00005624ae4f2710 in win_release_images (backend=0x5624aeefaf60, w=0x5624aeef7be0) at ../src/win.c:318
#5  0x00005624ae4f84d1 in unmap_win_finish (ps=0x5624aee63390, w=0x5624aeef7be0) at ../src/win.c:1703
#6  0x00005624ae4f862f in destroy_win_finish (ps=0x5624aee63390, w=0x5624aeef7be0) at ../src/win.c:1733
#7  0x00005624ae4f96bd in win_check_fade_finished (ps=0x5624aee63390, w=0x5624aeef7be0) at ../src/win.c:1979
#8  0x00005624ae4eb096 in paint_preprocess (ps=0x5624aee63390, fade_running=0x7ffe2b33073d) at ../src/picom.c:658
#9  0x00005624ae4ed689 in _draw_callback (loop=0x7f53d4d4b720, ps=0x5624aee63390, revents=0x2000) at ../src/picom.c:1414
#10 0x00005624ae4ed946 in draw_callback (loop=0x7f53d4d4b720, w=0x5624aee63450, revents=0x2000) at ../src/picom.c:1467
#11 0x00007f53d4d3f033 in ev_invoke_pending () from /usr/lib/libev.so.4
#12 0x00007f53d4d42902 in ev_run () from /usr/lib/libev.so.4
#13 0x00005624ae4f0a92 in session_run (ps=0x5624aee63390) at ../src/picom.c:2323
#14 0x00005624ae4f0d89 in main (argc=0xa, argv=0x7ffe2b3309d8) at ../src/picom.c:2424

• Coredump 1 (paint_all_new) and 2 (win_release_images): picom-core.zip

Other details

Might be related to #390. See also #393, which seems to be related as well.
I currently can't reproduce with the xrender backend. See comment for crashlog with the xrender backend.

EDIT: Bad timing, oh well... 😆

[tryone144]

I managed to capture a coredump/log of the assertion in win_release_images() with the xrender backend as well: picom-core-xrender.zip