1.16.9: managed-premium, slow volume mounting/attaching: Unable to attach or mount volumes: timed out waiting for the condition

[bitsofinfo]

I constantly see 3-5 minute waits where pods are stuck initializing w/ this kind of warning shown on describe. After 3-5 minutes the warnings on pods like the below example go away then things function as normal.

  Warning  FailedMount       110s       kubelet, xyz-27390584-0  Unable to attach or mount volumes: unmounted volumes=[ws-xddtn], unattached volumes=[tekton-internal-workspace tekton-internal-home tekton-internal-secret-volume-yyapps-cicd-x-ssh-co-fxrfd tekton-internal-scripts tekton-creds-init-home tekton-internal-secret-volume-docker-x-2wx65 ws-xddtn tekton-internal-results yy-builder-token-8dhtc tekton-internal-tools tekton-internal-downward]: timed out waiting for the condition

My PVC looks like this:

              spec:
                storageClassName: managed-premium
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 5Gi

The PV + PVC both look fine upon describe. But pods referencing the PVC constantly hang up and take forever to initialize.

2 nodes: (Standard D2s v3)

#884

[bitsofinfo]

maybe related, #1464 the --node-status-update-frequency=5m on a cluster created

[bitsofinfo]

this definitely was the issue, reducing --node-status-update-frequency=10s massively improved the situation...

i.e. shell into each of your nodes and check the value in /etc/default/kubelet adjust and reboot

[andyzhangx]

We fixed this issue(node-status-update-frequency=5m) on early March 2020, so if your cluster was created before that date, pls add a new node pool, another workaround is using a daemonset to modify that kubelet flag and restart kubelet, here is an example:

https://raw.githubusercontent.com/andyzhangx/demo/master/dev/sysctl-node-status-update-frequency.yaml