Calico v3.19.1 Upgrade Fails

[cmukai-cisco]

Describe the bug
Upgrading to Calico v3.19.1 using stable/aws-calico chart does not launch the CNI.

Steps to reproduce
We are currently running v3.15.1, so I cleaned it up as follows:

1. helm -n kube-system uninstall aws-calico

Then install Calico v3.19.1 as follows:

1. git clone https://github.com/aws/eks-charts.git
2. cd eks-charts/stable
3. create custom values.yaml with the following content:

calico:
  tag: v3.19.1

4. kubectl apply -k aws-calico/crds
5. helm -n kube-system install -f values.yaml aws-calico ./aws-calico
6. kubectl -n kube-system describe pod <calico-typha-horizontal-autoscaler-pod-name>
7. note the following warning in the events:

Warning  FailedCreatePodSandBox  34s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "72944aaaa37cd176e948865cd20b5c11a3900bdd9e1d644680ad82ba520b2c42" network for pod "calico-typha-horizontal-autoscaler-5bfc7d6966-h7c8z": networkPlugin cni failed to set up pod "calico-typha-horizontal-autoscaler-5bfc7d6966-h7c8z_kube-system" network: error getting ClusterInformation: connection is unauthorized: Unauthorized, failed to clean up sandbox container "72944aaaa37cd176e948865cd20b5c11a3900bdd9e1d644680ad82ba520b2c42" network for pod "calico-typha-horizontal-autoscaler-5bfc7d6966-h7c8z": networkPlugin cni failed to teardown pod "calico-typha-horizontal-autoscaler-5bfc7d6966-h7c8z_kube-system" network: error getting ClusterInformation: connection is unauthorized: Unauthorized]

The service account for this pod is calico-typha-cpha. This is bound to the role of the same name. The helm chart defining this role is:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: "{{ include "aws-calico.fullname" . }}-typha-cpha"
  labels:
{{ include "aws-calico.labels" . | indent 4 }}
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get"]
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments/scale"]
    verbs: ["get", "update"]

It does not allow access to ClusterInformation resources.

Also calico-node pods are constantly restarting.

Expected outcome
All Calico pods using v3.19.1 should come up cleanly.

Environment
AWS

• Chart name: stable/aws-calico
• Chart version: 0.3.5
• Kubernetes version: 1.18
• Using EKS (yes/no), if so version? yes, eks.3

Additional Context:

[jayanthvn]

Hi,

Can you please try 3.19.2 or later? You might be hitting projectcalico/calico#4518.

Thanks.

[haouc]

@cmukai-cisco
If you still have concerns, please be aware of our latest announcement. Since we are deprecating Calico charts in this repo, we recommend you to check our important announcement and suggested installation instruction for required or latest Calico version. You can also refer to Calico configuration reference.
If you need migrate Calico from non-operator based installation to operator based installation, we recommend you to check this Tigera document.
If you need upgrade you calico operator but you didn't install Calico via helm, you can check this Tigera document. Please being aware that any customized installation may encounter compatibility issue during upgrading.
If you have any issues or questions during Calico resources installation/upgrading/deletion, please refer to the README and send request to Calico repositories.
I am closing this ticket due to unsupported charts upgrade. Please feel free to send questions to our triage email (k8s-awscni-triage@amazon.com).