Possible netlink leak on 3.29.1

[imbstack]

We recently updated calico to 3.29.1 on one of our staging clusters and found that after a few hours there was a clear upward trend in the number of file descriptors held by calico-node pods.

Checking on a running instance after a couple days, we found that the calico-node -felix process had nearly 6000 file descriptors according to lsof, nearly all of which were like the following:

# lsof -p 1517383 | tail
calico-no 1517383 root 5778u  netlink                 0t0  689245146 ROUTE
calico-no 1517383 root 5779u  netlink                 0t0  688913733 ROUTE
calico-no 1517383 root 5780u  netlink                 0t0  689385180 ROUTE
calico-no 1517383 root 5781u  netlink                 0t0  689391746 ROUTE
calico-no 1517383 root 5782u  netlink                 0t0  689402663 ROUTE
calico-no 1517383 root 5783u  netlink                 0t0  689407738 ROUTE
calico-no 1517383 root 5784u  netlink                 0t0  689296536 ROUTE
calico-no 1517383 root 5785u  netlink                 0t0  689301559 ROUTE
calico-no 1517383 root 5790u  netlink                 0t0  689395559 ROUTE
calico-no 1517383 root 5791u  netlink                 0t0  689400833 ROUTE

Deleting that pod made dropped the fds although the new pod is starting the trend all over again.

Let me know if there is any other debugging data I can provide.

Expected Behavior

A relatively steady state of file descriptors for a calico-node pod.

Current Behavior

A steady increase in open file descriptors.

Possible Solution

Steps to Reproduce (for bugs)

1. Just deploy calico 3.29.1 afaict

Context

This is ok for now in our staging environment but are worried about going to production this way. It is entirely possible this is due to some weird config on our side but nothing is jumping out at me so far.

Your Environment

• Calico version: 3.29.1
• Calico dataplane (iptables, windows etc.): iptables
• Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes
• Operating System and version: Linux ip-10-213-23-129 6.8.0-1018-aws #19~22.04.1-Ubuntu SMP Wed Oct 9 16:48:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• Link to your project (optional):

[fasaxc]

Uh oh, likely to be one of my PRs to rework route programming:

• Unify all the RouteTables that point at the main kernel table #8418
• Switch to netlink.RouteListFilteredIter() API #9139
• Rev netlink to pick up RouteListIter #8979

Any interesting logs in calico-node from route_table.go? I'd expect re-opening netlink sockets after a failure, might help to know which failure you're hitting (if any).

[fasaxc]

Actually, think it might be this one: #9135 some of the calls to netlink.XXX got converted to use a Handle (which is a socket under the covers) and looks like we might have forgotten to close the Handle when done.

[imbstack]

Well that was quick. Just responding to confirm I'm not seeing any logs from route_table.go just from poking around a bit.

[sridhartigera]

@imbstack Is it possible for you to test an image with the fix? Image: docker.io/calico/node:v3.29.1-9-g95d2ad73b69e

[fasaxc]

Manifests here if that's more convenient; this is a nightly build of v3.29.next https://2024-12-19-v3-29-viper.docs.eng.tigera.net/

[imbstack]

I've only had it deployed for an hour or so but it definitely looks better to me. At this point before the patch a pod would've had over 400 open fd but this time around they are sitting at ~160 and its looking flat!

I'll report back here if I see something different in the next couple days but this looks fixed to me. Thanks for the quick turnaround!

[sridhartigera]

Thanks for testing this.

[sridhartigera]

@imbstack Hope you did not see any FD leak.

[imbstack]

Just checked and it's still flat! Looks fixed to me

[sridhartigera]

Thank you.