ECR Published Image Cannot be Fetched for Custom Cluster

[jinglejengel]

What happened: When trying to run make cluster-up deployments get stuck in an image pull backoff., this causes the latter parts of the make cluster-up to fail in that there are no CSRs. When trying to pull the image locally, the ECR repository is not able to be pulled from publicly.

e.g.

$ docker pull 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pod-identity-webhook:latest
Error response from daemon: Get https://602401143452.dkr.ecr.us-west-2.amazonaws.com/v2/eks/pod-identity-webhook/manifests/latest: no basic auth credentials

Relevant k8s events from the pod:

Events:
  Type     Reason     Age                  From                     Message
  ----     ------     ----                 ----                     -------
  Normal   Scheduled  2m45s                default-scheduler        Successfully assigned default/pod-identity-webhook-56c67dff46-c6h9h to docker-desktop
  Normal   Pulling    78s (x4 over 2m44s)  kubelet, docker-desktop  Pulling image "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pod-identity-webhook"
  Warning  Failed     78s (x4 over 2m44s)  kubelet, docker-desktop  Failed to pull image "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pod-identity-webhook": rpc error: code = Unknown desc = Error response from daemon: Get https://602401143452.dkr.ecr.us-west-2.amazonaws.com/v2/eks/pod-identity-webhook/manifests/latest: no basic auth credentials
  Warning  Failed     78s (x4 over 2m44s)  kubelet, docker-desktop  Error: ErrImagePull
  Warning  Failed     65s (x6 over 2m43s)  kubelet, docker-desktop  Error: ImagePullBackOff
  Normal   BackOff    51s (x7 over 2m43s)  kubelet, docker-desktop  Back-off pulling image "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pod-identity-webhook"

What you expected to happen: The pod-identity-webhook image to be publicly pullable.

How to reproduce it (as minimally and precisely as possible):

• Be on a machine with no authentication to the 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pod-identity-webhook repo and attempt to run:

docker pull 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pod-identity-webhook:latest

Anything else we need to know?: Obviously this can be fixed by pushing an image the cluster can be reached from and passing in the relevant env vars to make, just following the current README.md instructions does not produce an expected result.

Environment:

(N/A here since this is unrelated to k8s/EKS specific bits)

• AWS Region: N/A
• EKS Platform version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.platformVersion): N/A
• Kubernetes version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.version): N/A
• Webhook Version: N/A

[DZDomi]

Running into the same exact issue, building the image locally and pushing into the ecr registry in my account solved the issue for now. Would be good if this image would be publicly available

[siwyd]

Yeah, other repo's such as aws-iam-authenticator in the same account seem to be pullable, 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-iam-authenticator:v0.4.0-scratch for instance.

[micahhausler]

Thanks for this report, we're working on getting the image publishing process automated for this repo

[serialx]

I've also experienced this problem. Built the image myself to solve it. Please make the images public.

[Isan-Rivkin]

I build and pushed it to Docker hub.
Feel free to use it.

Seems like the there is a certificate issue getting error log form pod:
2019/09/23 14:04:07 http: TLS handshake error from 10.110.242.147:60022: remote error: tls: bad certificate

[FutureSharks]

Any update on official docker images being available?

[max-rocket-internet]

I think @nckturner is going to fix this 🎉

[mohsen0]

Hi @micahhausler Can this move forward?

[povils]

What is the status? :)

[max-rocket-internet]

@micahhausler @nckturner any update? It's been MONTHS 😅

[max-rocket-internet]

Ahhh, it seems I have misunderstood what is required to use IAM roles with pods on EKS. We don't need to have this docker image or run this ourselves, it's built into the EKS service. We only need to create an OIDC provider, annotate the ServiceAccount and create an IAM role with the correct trust policy.

This will teach me for skim reading READMEs and AWS announcements 😅

[btalbot]

I'm hesitant to actually use this as I'm afraid that it's abandon-ware.

Why are no public images yet available? Is there a general refusal to make them public? But why then leave links to private images in the documentation? Is the current state too unstable to be used so AWS won't make them public? Why does making an image public take so many months after being reported?

No releases, few commits ... doesn't look good to me.

Edit: I missed the releases somehow

[micahhausler]

Hi @btalbot this is not at all abandoned and the reason there have only been few commits is the project has not needed many updates. :)

In regard to a public image, it has not been a high priority for the EKS team, not because it isn't an issue, but because we have a lot on our plate! In EKS, the webhook is part of the managed service and not required to be run by customers. We very much do intend to make a public image for this, but have not yet built an automated infrastructure for this project to cut public release images.

[jinglejengel]

In EKS, the webhook is part of the managed service and not required to be run by customers

I can definitely honour that Micah, however if AWS and the EKS team are making a commitment to open source projects like this, it should hopefully encompass the original commitment in the announcement post that this is also intended for use outside of EKS managed clusters. Especially those of us building k8s clusters in EC2 outside of EKS, it doesn't drive a bunch of trust with the OS community around k8s to not have something like a public signed image from AWS available after so many months of waiting.

[micahhausler]

I definitely agree and this is something that we want to do better at, and this is one where your feedback helps us prioritize.

On the issue of signed images, that is not something ECR supports today, but you can follow along on image signing on the aws/containers-roadmap#43 issue.

[btalbot]

I'm happy to hear that it's not abandoned, and will be maintained publicly going forward. Maybe push a public image to docker hub where Amazon already has at least 22 image repos (https://hub.docker.com/u/amazon) until you can get things sorted out on ECR?

[tabern]

Hi - dropping into to provide an update.

The reason this is not available on ECR is as Micah said, because the automation and publishing pipeline setup is not yet complete. While this image will not be not public in the sense of being able to pull from any computer anywhere, when published it will be accessible from any EC2 host.

We are working on the timeline for making this build pipeline live and will update here when it's available.

[jinglejengel]

@tabern I appreciate the update! However, is there a reason not to circumvent the manual process for now and have it published to dockerhub like @btalbot mentioned above? Automation is obviously preferred, then engineers don't have to be involved to ship the latest changes ;P but for now literally having just ANY docker image available from the EKS team in a public fashion would immediately satisfy this use case.

Additionally, why would we not be able to pull this from any computer anywhere? What about customers who would be using on-prem to tie into IAM fo this authentication, or customers who use cross cloud functionality with this? It seems reductive to only have it be able to be pulled from EC2 hosts.

[tabern]

@Joeskyyy of course. It's really a matter of validation and automation either way.

The reason you can't pull this from any computer anywhere is because ECR does not yet support fully public repositories. (I realize this bolsters the argument for DockerHub even more... 😊)

[jinglejengel]

@tabern any word? An issue to just publish a docker image has been open for over 6 months now...

[jinglejengel]

First anniversary of this ticket being opened has past, any updates here?

[josselin-c]

Docker images are now available on https://hub.docker.com/r/amazon/amazon-eks-pod-identity-webhook