nerdctl port forwarding issue after Reset Kubernetes

[gunamata]

Steps to reproduce:

I use the sample at https://github.com/rancher-sandbox/rancher-desktop/tree/e2e-nginx-sample/e2e/assets/python-flask-app-sample

• I start by switching to kubernetes v1.20.0

• Reset Kubernetes to start fresh

• nerdctl --namespace k8s.io build -t simple-flask-app:v1.0 .

• Check whether the image appears in the image list in the RD UI

• nerdctl --namespace k8s.io run -p 5000:5000 -it simple-flask-app:v1.0

• Try localhost:5000 in browser

• Upgrade kubernetes to v1.21.0

• nerdctl --namespace k8s.io run -p 5000:5000 -it simple-flask-app:v1.0

• Try localhost:5000 in browser

[mook-as]

From debugging, it looks like the issue is, roughly:

• Doing a port forwarding adds an iptables entry (well, multiple, but whatever) (let's call this to container A)
• The CNI plugin doesn't seem to remove stale entries
• So when the container goes away (because of a cluster restart), the entries are left over
• New entries are added when we run a new container (call this container B)
• The RD agent sees this and adds a forwarder
• When traffic comes in, the agent forwards to 127.0.0.1 (on the VM), and goes to iptables
• The first entry (for A) get hit, and the packet is directed to a container that no longer exists.

[mook-as]

Matt was correct — the first container was never removed (just no longer running) after the restart:

> nerdctl -n k8s.io ps -a | grep flask
b18669a3b087    docker.io/library/simple-flask-app:v1.0                                                          "/bin/sh -c python f…"    46 minutes ago    Created    0.0.0.0:5000->5000/tcp

I haven't yet figured out how to restart that container, though.

[gaktive]

Once we have settings in place, we can look at this again.

[gunamata]

A better way to repro this issue as described here, would be to:

• Run nerdctl run -d -p 85:80 --restart=always nginx

• Reset Kubernetes with Images 1

• Run nerdctl run -d -p 85:80 --restart=always nginx