port forwarding does not work for sudo nerdctl (i.e. rootful CNI portmap)

[AkihiroSuda]

$ limactl --version
limactl version 0.5.2-19-g2180618

$ cat default.yaml 
images:
  - location: "https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-amd64.img"
    arch: "x86_64"
ssh:
  localPort: 60022
containerd:
  system: true
  user: true

$ limactl start default.yaml

$ lima nerdctl run -d -p 127.0.0.1:8080:80 nginx:alpine

$ curl localhost:8080
...
<title>Welcome to nginx!</title>
...

$ lima sudo nerdctl run -d -p 127.0.0.1:8081:80 nginx:alpine

$ curl localhost:8081
curl: (7) Failed to connect to localhost port 8081: Connection refused

Port forwarding does not seem working for sudo nerdctl, i.e. rootful CNI portmap, with iptables.
This is because iptables ports do not appear in /proc/net/tcp, /proc/net/tcp6.

Note that rootless ports are not affected by this issue because RootlessKit (or slirp4netns, depending on the configuration) manages the ports in userspace.

[AkihiroSuda]

@jandubois

Thoughts? What would be the most robust way to detect iptables ports?

Output of sudo iptables -t nat -n seems hard to parse (and requires root)

`sudo iptables -t nat -n`

$ sudo iptables -L -t nat
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
CNI-HOSTPORT-DNAT  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
CNI-HOSTPORT-DNAT  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
CNI-HOSTPORT-MASQ  all  --  anywhere             anywhere             /* CNI portfwd requiring masquerade */
CNI-46b9886802b6bb8440b5ff43  all  --  10.4.0.2             anywhere             /* name: "bridge" id: "default-54485df8369bd33b8aa281955836c282534ffb9f5fca4f5922bb5528237ea82a" */

Chain CNI-46b9886802b6bb8440b5ff43 (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             10.4.0.0/24          /* name: "bridge" id: "default-54485df8369bd33b8aa281955836c282534ffb9f5fca4f5922bb5528237ea82a" */
MASQUERADE  all  --  anywhere            !base-address.mcast.net/4  /* name: "bridge" id: "default-54485df8369bd33b8aa281955836c282534ffb9f5fca4f5922bb5528237ea82a" */

Chain CNI-DN-46b9886802b6bb8440b5f (1 references)
target     prot opt source               destination         
CNI-HOSTPORT-SETMARK  tcp  --  10.4.0.0/24          localhost            tcp dpt:tproxy
CNI-HOSTPORT-SETMARK  tcp  --  localhost            localhost            tcp dpt:tproxy
DNAT       tcp  --  anywhere             localhost            tcp dpt:tproxy to:10.4.0.2:80

Chain CNI-HOSTPORT-DNAT (2 references)
target     prot opt source               destination         
CNI-DN-46b9886802b6bb8440b5f  tcp  --  anywhere             anywhere             /* dnat name: "bridge" id: "default-54485df8369bd33b8aa281955836c282534ffb9f5fca4f5922bb5528237ea82a" */ multiport dports tproxy

Chain CNI-HOSTPORT-MASQ (1 references)
target     prot opt source               destination         
MASQUERADE  all  --  anywhere             anywhere             mark match 0x2000/0x2000

Chain CNI-HOSTPORT-SETMARK (2 references)
target     prot opt source               destination         
MARK       all  --  anywhere             anywhere             /* CNI portfwd masquerade mark */ MARK or 0x2000

[jandubois]

Thoughts? What would be the most robust way to detect iptables ports?

Sorry, coming up empty too. I was wondering if conntrack could help, but I think it only tracks connections, not bind/listen calls. Also thought about adding -j LOG rules to iptables, but again they won't trigger until a connection has been made, and would probably be brittle anyways.

Would eBPF provide a mechanism?

[AkihiroSuda]

Maybe we can use this, idk

https://github.com/ti-mo/conntrack

Listen for create/update/destroy events

https://pkg.go.dev/github.com/ti-mo/conntrack#Conn.Listen

[jandubois]

Maybe we can use this, idk

I was simply trying sudo conntrack -E. It did not show any events when the port was opened, only when a connection was made.

I don't know if there is a way to get it to show listen events. I suspect that netfilter does not generate such events; it may just bind the port once the rules match and there may not be such a thing as a listening port in that case. But I don't really know how this works.

[AkihiroSuda]

nft monitor chains could be probably used

$ sudo nerdctl run -it --rm -p 8080:80 nginx:alpine

$ sudo nft -j monitor chains
{"add": {"chain": {"family": "ip", "table": "nat", "name": "CNI-46beb695d3d07935c2465aef", "handle": 160}}}
{"add": {"chain": {"family": "ip", "table": "nat", "name": "CNI-DN-46beb695d3d07935c2465", "handle": 164}}}

$ sudo nft list chain nat CNI-46beb695d3d07935c2465aef
table ip nat {
        chain CNI-46beb695d3d07935c2465aef {
                @nh,128,24 656384  counter packets 0 bytes 0 accept
                ip daddr != 224.0.0.0/4  counter packets 0 bytes 0 masquerade 
        }
}

$ sudo nft list chain nat CNI-DN-46beb695d3d07935c2465
table ip nat {
        chain CNI-DN-46beb695d3d07935c2465 {
                meta l4proto tcp @nh,96,24 656384 tcp dport 8080 counter packets 0 bytes 0 jump CNI-HOSTPORT-SETMARK
                meta l4proto tcp ip saddr 127.0.0.1 tcp dport 8080 counter packets 0 bytes 0 jump CNI-HOSTPORT-SETMARK
                meta l4proto tcp tcp dport 8080 counter packets 0 bytes 0 dnat to 10.4.0.16:80
        }
}

[mattfarina]

I noticed that the port isn't available inside the VM. For example, I ran the default config and commands in the original issue. Looking at the ports in the VM, I don't see 8081.

mfarina@lima-default:~$ ss -lntu
Netid              State               Recv-Q              Send-Q                                Local Address:Port                              Peer Address:Port              Process
udp                UNCONN              0                   0                                     127.0.0.53%lo:53                                     0.0.0.0:*
udp                UNCONN              0                   0                                 192.168.5.15%eth0:68                                     0.0.0.0:*
tcp                LISTEN              0                   4096                                      127.0.0.1:8080                                   0.0.0.0:*
tcp                LISTEN              0                   4096                                  127.0.0.53%lo:53                                     0.0.0.0:*
tcp                LISTEN              0                   128                                         0.0.0.0:22                                     0.0.0.0:*
tcp                LISTEN              0                   4096                                      127.0.0.1:42235                                  0.0.0.0:*
tcp                LISTEN              0                   128                                            [::]:22                                        [::]:*

It looks like the port isn't available in the VM to forward. Was it not setup by nerdctl/containerd?

[mattfarina]

First, let me say that networking is NOT my space. I am trying to track down the problem.

I've found that when I run lima sudo ctr run -d --net-host docker.io/library/nginx:alpine asdf the container port is available internally. I realize this is not using CNI or port mapping.

I wonder if something happens in nerdctl differently, when run as root, to setup the CNI networking.

[mattfarina]

I don't think I was clear before, inside the VM I try to connect to port 8081 and it is unable to connect.

[AkihiroSuda]

Rootful mode uses iptables for port forwarding, which does not create entries in /proc/net/tcp, so lima port forwarding does not work automatically.

We will have to embed an equivalent of nft monitor in the guest agent.

[mattfarina]

@AkihiroSuda if I'm inside the VM why can't I use curl to connect to port 8081 the same way I can to 8080?

update: nevermind. Connecting inside the VM is working for me now. I must have broken my environment yesterday. Sorry about that.

[mattfarina]

It seems that a dependency of lima is using a coreos iptables package that portmap is also using. We may be able to use that to list the current iptables and get the port forwards. No new dependencies in doing that. Investigating that now.

[mattfarina]

My update, for anyone following along...

I've discovered the iptables rule that handles the forwarding. It is created for the port forward and removed when the container is removed. Note, it remains when the container is stopped. I'm not sure if this is intentional.

A single iptables query and parser should be possible to get a port out of one of these iptables rules. Working on that now. Once that is done I'll create a PR to lima. Using existing lima dependencies so nothing but new lima code in the agent to handle this.

[AkihiroSuda]

Thank you @mattfarina !

In future we will want to cover non-iptables nftables too, but for now we can just periodically scan iptables and call it a day.

[mattfarina]

@AkihiroSuda I just realized the agent does not run as root. To access iptables (and nftables) it will need to run as root. Is that ok?

[AkihiroSuda]

Yes