Manifest unknown for docker pull docker:*

[max-sternitzke]

Hey there,

since your recent docker image push today 14th Dec 6:05 UTC i cannot pull any docker image from the official docker registry.
I tried a docker pull:

• docker:27.4.0-dind
• docker:27.4.0
• docker:latest
• docker:dind

All these result in a manifest for docker:dind not found: manifest unknown: manifest unknown

I have not modified my docker/daemon.json. My local Docker version is Docker version 27.4.0, build bde2b89 on the latest macOS. This problem also occurs on remote Ubuntu VMs.

I'm sorry if its my mistake, but it seems suspicious that the problem occured just after your image push and has not yet been resolved. (We're running a ci pipeline every 5 minutes and it first failed at 6:13am UTC this morning.)
If you need anything, just let me know.

Thanks for any hint or solution. Cheers
Max

[azlkiniue]

for now i'm downgrading to version 26 temporarily
pulling docker:26-dind or docker:26 is working for me

[oxilor]

This error started to occur only 4 hours ago since the version 27.4.0 was released. No need to downgrade to docker:26-dind because the previous version docker:27.3.1-dind is also working.

So just replace docker:dind with docker:27.3.1-dind temporarily.

[nalingovind]

This worked for me docker:26-dind

[tanc]

docker:27.3-dind works fine but latest fails

[oleksandrmikhnenko]

dind and latest tags are also affected

[nfacha]

Also experiencing this with the dind tag

[muocod]

The same shit here

[thaJeztah]

This looks to be a duplicate of #519

👋 I work at docker, but not on the official images; I'm currently trying our internal slack chatting with folks on call to see if someone working on official images is available to look.

It looks like the "manifest-index" for the image exists, but the image that's referenced from it is either missing, or resulting in the "not found".

As a workaround, it looks like the previous versions of the image are functional, so if you pin to the previous version, you can use that as a workaround, e.g.;

docker pull docker:27.3.1-dind / docker pull docker:27.3.1-cli both work

[max-sternitzke]

This looks to be a duplicate of #519

It sure is @thaJeztah . As i was starting to file the issue the was none, after submitting there were two :)

You better reach someone soon - many pipelines are failing just now because of this.

[arthurlm44]

The missing tag has completely collapsed our CI system.

[0xCiBeR]

Same behavior here, we manged to work arround the issue pointing to this image: docker pull public.ecr.aws/docker/library/docker:dind

EDIT: docker:27.3.1-dind also works perfectly, thanks @oxilor

[marcschroeder]

I think we should keep in mind that most of us are using Docker and Docker Official Images without paying for it... Keep in mind that the teams around such tools are putting a lot of effort in to make such great OSS software available to us. I understand that is might be frustrating to see some CIs failing - but no need for strong words & demands such as in the comments above IMHO...
PS: Thanks to everyone working on resolving this for the community!!

[thaJeztah]

I can give an intermediate status update from the Slack thread.

Team is still looking. They managed to narrow down the cause of the issue to be related to cross-repo links. The image was pushed successfully (data / blobs are there), but some links were not created after the push, causing it to be producing a 404 for those references. A fix was deployed that should prevent this situation from happening, but won't automatically apply to content that's already there, and those parts are really locked down, so no direct access to (manually) correct, so they're currently checking logs to identify images that are affected (beyond the ones mentioned in this ticket), and to see if there's ways to correct it. Re-pushing the images (same image, but trigger a push again) may work, but may also be locked down (currently being looked into).

[thaJeztah]

OK, team was able to get the list of images affected, and may have found a solution; some manual work is involved, so fixes may roll out over time. I'll post updates once available.

[max-sternitzke]

docker:dind is working for me again. Thanks for the work @thaJeztah & team! 🎉

Nevertheless I have to disagree with @marcschroeder on the comments here part quite a bit. After the weekend the Docker team will recap that for sure and draw the necessary conclusions so such a situation will most likely not happen again and/or will be monitored more closely by themselves.

[MrCskncn]

"the one should not release a version on friday evenings."

[thaJeztah]

x-posting from the other ticket;

I think they managed to go through most of the list by now, but some may still be pending. Currently verifying all tags and all variants of those images (which takes a while to run).

There may be some images in the platform-specific orgs that are used as intermediate image that may not yet be addressed, and (given much lower / corner-case use), e.g. images in the platrform-specific namespace (e.g. https://hub.docker.com/u/arm64v8/, and other similar namespaces listed in; https://github.com/docker-library/official-images#architectures-other-than-amd64. Those namespaces are effectively intermediate stages for the multi-platform images.

OK; colleague confirmed that the list of official images should be all processed. The verification script will still be running for some hours, but we're updating the statuspage to "monitoring"

There may be some images in the platform-specific orgs that are used as intermediate image that may not yet be addressed, and (given much lower / corner-case use), e.g. images in the platrform-specific namespace (e.g. https://hub.docker.com/u/arm64v8/, and other similar namespaces listed in; https://github.com/docker-library/official-images#architectures-other-than-amd64. Those namespaces are effectively intermediate stages for the multi-platform images.

These images will likely be processed later, which may be done after the weekend.

[LaurentGoderre]

Closing in favor of #519. It should be mostly resolve now though