Pod networking is terminated while the pod is still in the grace time

[imriz]

What happened:

Attach logs

What you expected to happen:
When pod deletion is initiated, respect the grace time, and don't remove routes etc before the grace time.
Instead what I am seeing is that the routes are being removed as soon as the docker stop process begins.
This cuts off (and causes timeouts) for clients.

How to reproduce it (as minimally and precisely as possible):
Create a pod which ignores SIGTERM, enter into it using docker exec or whatever, and ping the outside (1.1.1.1? any reachable IP).
Start pod replacement (e.g deployment rollout).
Observe as the ping stops as soon as the pod removal begins, before the end of the grace period.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.5-eks-bc4871b", GitCommit:"5236faf39f1b7a7dabea8df12726f25608131aa9", GitTreeState:"clean", BuildDate:"2021-10-29T23:32:16Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
• CNI Version: 602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni-init:v1.10.1-eksbuild.1
• OS (e.g: cat /etc/os-release): Amazon Linux 2
• Kernel (e.g. uname -a): Linux ip-10-10-40-143.ec2.internal 5.4.181-99.354.amzn2.x86_64 Initial commit of amazon-vpc-cni-k8s #1 SMP Wed Mar 2 18:50:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[achevuru]

@imriz I'm not sure I followed the description completely. You can attempt to control the behavior via preStop hook. But in general, CNI has no visibility in to Pod specific settings and it will just act on the kubelet invocations (i.e.,) CNI will delete the routes if kubelet invokes CNI with del command for a particular sandbox. So, it is up to kubelet to honor the pod specific lifecycle hooks..

[imriz]

Hi, after a deeper examination, this seems to be a duplicate of #1531
Removing Calico fixes the issue.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.