Calico VXLAN: No ClusterIP networking on the same node

[xmudrii]

What happened?

Consider the following scenario:

• Pod is running on Node A
• Pod is exposed via a ClusterIP service
• SSH to the Node A
• Try curl-ing the ClusterIP service

The cluster is running Calico VXLAN deployed using the addon provided by KubeOne (see the provided KubeOneCluster manifest).

Expected behavior

curl returns the proper/expected response.

Actual behavior

curl is stuck waiting for a response forever. I believe this causes the Kubernetes conformance tests to fail.

*   Trying 10.106.181.168:80...
* TCP_NODELAY set
* connect to 10.106.181.168 port 80 failed: Connection timed out
* Failed to connect to 10.106.181.168 port 80: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to 10.106.181.168 port 80: Connection timed out

What KubeOne version are you using?

$ kubeone version
{
  "kubeone": {
    "major": "1",
    "minor": "4",
    "gitVersion": "v1.4.0-162-g3b7397f6",
    "gitCommit": "3b7397f62ed2852292d18bda68adaf053d776a45",
    "gitTreeState": "",
    "buildDate": "2022-07-26T07:51:23+02:00",
    "goVersion": "go1.18.3",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "machine_controller": {
    "major": "1",
    "minor": "52",
    "gitVersion": "v1.52.0",
    "gitCommit": "",
    "gitTreeState": "",
    "buildDate": "",
    "goVersion": "",
    "compiler": "",
    "platform": "linux/amd64"
  }
}

Provide your KubeOneCluster manifest here (if applicable)

apiVersion: kubeone.k8c.io/v1beta2
kind: KubeOneCluster

versions:
  kubernetes: "1.23.9"

containerRuntime:
  containerd: {}

clusterNetwork:
  cni:
    external: {}

addons:
  enable: true
  addons:
  - name: default-storage-class
  - name: calico-vxlan
    params:
      mtu: "8951" # same happens without a custom MTU

What cloud provider are you running on?

AWS

What operating system are you running in your cluster?

Ubuntu 20.04.4 LTS

Additional information

If a pod is running on a different node, there are no issues. In other words, this scenario works as expected:

• Pod is running on Node A
• Pod is exposed via a ClusterIP service
• SSH to the Node B
• Try curl-ing the ClusterIP service
• Response is returned

[kubermatic-bot]

Issues go stale after 90d of inactivity.
After a furter 30 days, they will turn rotten.
Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[xmudrii]

/remove-lifecycle stale