Suspicious nftables chain priority seems to cause undesired forward packet dropping

[yourcase]

Qubes OS release R4.2.3

Brief summary

Following instructions for networking between AppVMs as described here, doesn't necessarily lead to success. Modifying the suspicious-looking nftables chain qubes-firewall forward resolves the issue.

Steps to reproduce

Probably not always reproducible, because the apparent problem is with non-determined priorities between the two nftables chains below:

# nft list chain qubes forward; nft list chain qubes-firewall forward
table ip qubes {
	chain forward {
		type filter hook forward priority filter; policy accept;
		jump custom-forward
		ct state invalid counter packets 0 bytes 0 drop
		ct state established,related accept
		oifgroup 2 counter packets 0 bytes 0 drop
	}
}
table ip qubes-firewall {
	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept
		iifname != "vif*" accept
	}
}

The two priorities filter of these chains are equal, so that the order of execution as far as I know is undetermined. A packet with ct state = new and coming from interface vif59.0 will be rejected by the forward chain in the qubes-firewall table.

I experienced not being able to ping between two AppVMs, and tcpdump -i any, running on a persistent firewall between them, logs lines showing echo requests coming into the firewall, but not going out, i.e. apparently not being forwarded:

21:59:18.097468 vif60.0 In  IP 10.137.0.47 > 10.137.0.56: ICMP echo request, id 5, seq 1, length 64

, even though the custom-forward chain was set up to allow this:

# nft list chain qubes custom-forward | grep 47
		ip saddr 10.137.0.56 ip daddr 10.137.0.47 ct state established,related,new counter packets 0 bytes 0 accept
		ip saddr 10.137.0.47 ip daddr 10.137.0.56 ct state established,related,new counter packets 0 bytes 0 accept

Expected behavior

Ping forwarding is expected to work when following said instructions.

Actual behavior

Ping doesn't work, but can be made to work by executing

nft delete chain ip qubes-firewall forward 
nft add chain ip qubes-firewall forward "{ type filter hook forward priority filter ; policy accept ; }"

Hint at root cause

Since the original two forward chains in tables qubes and qubes-firewall have equal priority, their execution order is indeterminate and the qubes-firewall forward chain may run before the qubes forward chain, dropping new packets. This is evidenced by that modification of the qubes-firewall forward chain mentioned above

[marmarek]

Is that all you have in the qubes-firewall->forward chain? There should be rules for each connected appvm, with just "accept" rule by default (unless you configure something else with qvm-firewall tool). If you don't have them, check if qubes-firewall service is running - it's responsible for managing those chains.

[yourcase]

Is that all you have in the qubes-firewall->forward chain?

Yes, that's all I had, and no I never used qvm-firewall. I did however create the firewall VM as a new AppVM with "Provides network access to other qubes" selected in qubes manager's qube creation dialog, and fedora-41 as template. Looks like maybe that breaks the expected behavior you mention. Indeed the qubes-firewall service on that AppVM seems to have involuntarily exited:

[user@sys-firewall-persist config]$ sudo su -
[root@sys-firewall-persist ~]# systemctl status qubes-firewall
× qubes-firewall.service - Qubes firewall updater
     Loaded: loaded (/usr/lib/systemd/system/qubes-firewall.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf, 50-keep-warm.conf
     Active: failed (Result: exit-code) since Tue 2025-01-14 15:52:55 CET; 1min 22s ago
 Invocation: 849a8edd43884dddad4bc279dd5bf65f
    Process: 560 ExecStart=/usr/bin/qubes-firewall (code=exited, status=1/FAILURE)
   Main PID: 560 (code=exited, status=1/FAILURE)
   Mem peak: 15.3M
        CPU: 123ms

Jan 14 15:52:55 sys-firewall-persist qubes-firewall[560]:                       ^^^^^^^^^^^^^^^^^^
Jan 14 15:52:55 sys-firewall-persist qubes-firewall[560]:                       errread, errwrite)
Jan 14 15:52:55 sys-firewall-persist qubes-firewall[560]:                       ^^^^^^^^^^^^^^^^^^
Jan 14 15:52:55 sys-firewall-persist qubes-firewall[560]:   File "/usr/lib64/python3.13/subprocess.py", line 1795, in _posix_spawn
Jan 14 15:52:55 sys-firewall-persist qubes-firewall[560]:     self.pid = os.posix_spawn(executable, args, env, **kwargs)
Jan 14 15:52:55 sys-firewall-persist qubes-firewall[560]:                ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 14 15:52:55 sys-firewall-persist qubes-firewall[560]: OSError: [Errno 8] Exec format error: '/rw/config/qubes-firewall-user-script'
Jan 14 15:52:55 sys-firewall-persist systemd[1]: qubes-firewall.service: Main process exited, code=exited, status=1/FAILURE
Jan 14 15:52:55 sys-firewall-persist systemd[1]: qubes-firewall.service: Failed with result 'exit-code'.
Jan 14 15:52:55 sys-firewall-persist systemd[1]: Failed to start qubes-firewall.service - Qubes firewall updater.
[root@sys-firewall-persist ~]# cd /rw/config/
[root@sys-firewall-persist config]# ls -l qubes-firewall-user-script*
-rwxr-xr-x. 1 root root  150 Jan 13 13:15 qubes-firewall-user-script
-rwxr-xr-x. 1 root root 2875 Jan 13 18:19 qubes-firewall-user-script2
-rwxr-xr-x. 1 root root 4072 Jan 13 18:23 qubes-firewall-user-script3
[root@sys-firewall-persist config]# ./qubes-firewall-user-script
[root@sys-firewall-persist config]# cat ./qubes-firewall-user-script
vmname=$(qubesdb-read /name)
filtername=sys-firewall-persist

if [ "${vmname}" == "${filtername}" ]
then
   /rw/config/qubes-firewall-user-script3
fi
[root@sys-firewall-persist config]# 

[yourcase]

Ah OK here's the cause for the qubes-firewall failure: My '/rw/config/qubes-firewall-user-script' should have started with #!/bin/sh . Adding that line and rebooting the firewall fixes my issue. I will create a documentation issue, noting that the /rw/config/qubes-firewall-user-script script should not only be executable but also start with that hashbang line.

[yourcase]

I created a documentation improvement suggestion . Closing this issue.