helm install is getting "toomanyrequests: You have reached your pull rate limit" error

[kdombros]

My helm install failed with:

Error: failed pre-install: timed out waiting for the condition
helm.go:81: [debug] failed pre-install: timed out waiting for the condition

I used the following to get details on the pod: kubectl describe pod releasename-apps-create-ltpa-keys-xrdp5 - note the "toomanyrequests: You have reached your pull rate limit" error:

Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  3m46s               default-scheduler  Successfully assigned ocp/releasename-apps-create-ltpa-keys-fnkq7 to crc-ctj2r-master-0
  Normal   Pulled     3m6s                kubelet            Container image "ibmcom/websphere-liberty:kernel-java8-ibmjava-ubi" already present on machine
  Normal   Created    2m58s               kubelet            Created container create-ltpa-keys
  Normal   Started    2m48s               kubelet            Started container create-ltpa-keys
  Warning  Failed     60s                 kubelet            Failed to pull image "bitnami/kubectl:1.19": rpc error: code = Unknown desc = Error reading manifest 1.19 in docker.io/bitnami/kubectl: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Warning  Failed     60s                 kubelet            Error: ErrImagePull
  Normal   BackOff    59s                 kubelet            Back-off pulling image "bitnami/kubectl:1.19"
  Warning  Failed     59s                 kubelet            Error: ImagePullBackOff 
  Normal   Pulling    46s (x2 over 2m9s)  kubelet            Pulling image "bitnami/kubectl:1.19"

To try to get around that, I authenticated to Docker Hub, pulled and tagged "kubectl" image as just "ocp/kubectl:latest" and did:
docker push default-route-openshift-image-registry.apps-crc.testing/ocp/kubectl:latest

I then modified Helm Charts to look for "kubectl:latest" instead of "bitnami/kubectl:1.19", issued
helm push apps + helm repo update
and then tried it all again, but that also failed - note the "unauthorized: authentication required":

Events:
  Type     Reason          Age                    From               Message
  ----     ------          ----                   ----               -------
  Normal   Scheduled       10m                    default-scheduler  Successfully assigned ocp/releasename-apps-create-ltpa-keys-rfq2j to crc-ctj2r-master-0
  Normal   AddedInterface  9m56s                  multus             Add eth0 [10.217.0.189/23]
  Normal   Pulled          9m41s                  kubelet            Container image "ibmcom/websphere-liberty:kernel-java8-ibmjava-ubi" already present on machine
  Normal   Created         9m32s                  kubelet            Created container create-ltpa-keys
  Normal   Started         9m19s                  kubelet            Started container create-ltpa-keys
  Normal   Pulling         7m3s (x4 over 8m45s)   kubelet            Pulling image "kubectl:latest"
  Warning  Failed          6m58s (x4 over 8m40s)  kubelet            Failed to pull image "kubectl:latest": rpc error: code = Unknown desc = Error reading manifest latest in docker.io/library/kubectl: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
  Warning  Failed   6m58s (x4 over 8m40s)   kubelet  Error: ErrImagePull
  Normal   BackOff  6m44s (x5 over 8m14s)   kubelet  Back-off pulling image "kubectl:latest"
  Warning  Failed   4m32s (x14 over 8m14s)  kubelet  Error: ImagePullBackOff

Question: it seems that one way or another, I need to authenticate to a registry so that the docker pull's can succeed. Please advise - thanks!

[kdombros]

I reverted my Helm chart back to requesting bitnami/kubectl from Docker Hub, and I got lucky today I guess and made it through :-). We don't need to work on/around this issue / close it up you prefer.

[kdombros]

Is there any guidance on how to create and use a "secret" for a private registry where the bitnami/kubectl image can be stored? Fyre VM's consistently experience/fail with this error due to shared IP addresses. Any guidance on how to tweak the Helm charts to authenticate and pull from a private registry for that image? Please let us know! Tagging @podgek - we were attempting to assist Kieran Hippsley yesterday.

[khippsley]

This should probably be added to the known issues along with some instructions on how and what to update to use an alternative registry.

[andreyzher]

Hi,

Sorry we missed this.

The correct way to fix this would be to specify an imagePullSecret for DockerHub as part of the Helm chart definition.

In the meantime, you could update the OpenShift global pull-secret, similar to the workaround posted in the runbook for Minikube.

[khippsley]

I followed the Minikube workaround to get past this. I'll try it a few more times to make sure it wasn't a fluke of timing, but thanks for the pointer!

[andreyzher]

Hi @kdombros, @khippsley

A new property has been added to the values file in 21.5.0 to handle the Docker Hub credentials.

global:
  hubPullSecret:
    registry: 'https://index.docker.io/v1/'
    secretName: ''
    username: ''
    password: ''

You can use your Docker Hub password or access token in the global.hubPullSecret.password field.

If you have a DockerHub secret already created and would like to reuse it, set the global.hubPullSecret.secretName to the name of your secret (it must be in the same namespace).

Thank you for your support. If you have any further issues on this topic, feel free to reopen this issue or create a new one.

Regards

andreyzher

[kdombros]

Very helpful - thanks!