Part of manifests were not pushed when multi-platform build

[YaSuenag]

I couldn't pull linux/amd64 image built by build-push-action from GitHub Packages (ghcr.io). I guess it is caused by manifests for actual container images have not been uploaded. Do you know something to resolve this?

I'm trying to build both linux/amd64 and linux/arm64 images from build-push-action. Workflow file is here, but I paste that action here:

      - name: Build and push Docker image
        uses: docker/build-push-action@v4.0.0
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          provenance: false

I can see the result from package page, it shows two images both linux/amd64 and linux/arm64 what I expect. However I haven't pulled them as following:

$ podman pull ghcr.io/yasuenag/hsdis-builder:ghatest
Trying to pull ghcr.io/yasuenag/hsdis-builder:ghatest...
Error: determining manifest MIME type for docker://ghcr.io/yasuenag/hsdis-builder:ghatest: reading manifest sha256:128e1cfe4e5f76af75df1121fccaf6da0cba412deec293ad988d7ade34499d09 in ghcr.io/yasuenag/hsdis-builder: manifest unknown

Manifest list has been uploaded:

$ podman manifest inspect ghcr.io/yasuenag/hsdis-builder:ghatest
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 1091,
            "digest": "sha256:128e1cfe4e5f76af75df1121fccaf6da0cba412deec293ad988d7ade34499d09",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 1091,
            "digest": "sha256:e369c6544bd65fc0dc0f95c1b1d7102f4e5df73a38b17b795765869c25b45719",
            "platform": {
                "architecture": "arm64",
                "os": "linux"
            }
        }
    ]
}

However the manifest for linux/amd64 has not been uploaded - I got HTTP 404 when I requested the manifest directly:

$ curl -vsL -H "Authorization: Bearer ..." "https://ghcr.io/v2/yasuenag/hsdis-builder/manifests/sha256:128e1cfe4e5f76af75df1121fccaf6da0cba412deec293ad988d7ade34499d09"
*   Trying 20.27.177.117:443...
* Connected to ghcr.io (20.27.177.117) port 443 (#0)

  <snip>

< HTTP/2 404
< content-type: application/json
< docker-distribution-api-version: registry/2.0
< date: Sun, 12 Mar 2023 06:31:22 GMT
< content-length: 70
< x-github-request-id: E1A2:2A3F:54872:EBD30:640D71BA
<
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown"}]}
* Connection #0 to host ghcr.io left intact

I can see GHA log of build-push-action as following, it seems to be uploaded manifest list only, any manifests for actual images seems not to be uploaded.

#15 exporting to image
#15 exporting layers
#15 exporting layers 31.8s done
#15 exporting manifest sha256:128e1cfe4e5f76af75df1121fccaf6da0cba412deec293ad988d7ade34499d09 0.0s done
#15 exporting config sha256:673105e697ae54a0630f8179d7abb98e80b01e904a6587a8584b53542ccfd310 done
#15 exporting manifest sha256:e369c6544bd65fc0dc0f95c1b1d7102f4e5df73a38b17b795765869c25b45719 done
#15 exporting config sha256:612d76aca9aa03b3bd6e3b9188886094b9af2ddcc4d337d944030406eeaa0b33 done
#15 exporting manifest list sha256:14caeadc94ed4cf989eec67a003a89493238f9b920b1a1b4d2907ab3ea2a6ca6 done
#15 ...

#16 [auth] yasuenag/hsdis-builder:pull,push token for ghcr.io
#16 DONE 0.0s

#15 exporting to image
#15 pushing layers
#15 pushing layers 5.7s done
#15 pushing manifest for ghcr.io/yasuenag/hsdis-builder:ghatest@sha256:14caeadc94ed4cf989eec67a003a89493238f9b920b1a1b4d2907ab3ea2a6ca6
#15 pushing manifest for ghcr.io/yasuenag/hsdis-builder:ghatest@sha256:14caeadc94ed4cf989eec67a003a89493238f9b920b1a1b4d2907ab3ea2a6ca6 1.9s done
#15 DONE 39.3s

[jedevc]

Hm, this is really odd 🤔 The logs you see in the exporting to image section look about right to me though - we only print the digest of the top-level object, which is the manifest list (though, it does look confusing in the output, it might be worth improving that on the buildkit side).

I just used docker buildx imagetools - and got:

$ docker buildx imagetools inspect ghcr.io/yasuenag/hsdis-builder:ghatest --format "{{ json . }}"
ERROR: failed to copy: httpReadSeeker: failed open: content at https://ghcr.io/v2/yasuenag/hsdis-builder/manifests/sha256:128e1cfe4e5f76af75df1121fccaf6da0cba412deec293ad988d7ade34499d09 not found: not found

Somehow the image index is incorrect, pointing to content that doesn't exist 😢 This is a bit weird though - I think the registry should probably prevent content from being pushed that's invalid - but it's also very odd that we get here at all in BuildKit. This maybe seems similar to something like docker/buildx#1653?

Could you also re-run but enabling debug mode? https://docs.docker.com/build/ci/github-actions/configure-builder/#buildkit-container-logs

[YaSuenag]

Thank you for checking this problem!

I enabled debug log in this commit, but any failure log like docker/buildx#1653 does not seem to be there.

You can see all logs here: https://github.com/YaSuenag/hsdis-builder/actions/runs/4397040284/jobs/7699896898

[YaSuenag]

Sorry, it's my mistake.

I use https://github.com/actions/delete-package-versions to remove older images. I expected to remove all of untagged images. However it is incorrect - actual containers (I don't know what to say - I mean they are actual container for linux/amd64 and linux/arm64) are untagged, so I can't pull them.

I tweaked min-versions-to-keep in delete-package-versions action, then it works well. Buildx and build-push-action are not bad.

So I close this issue. Thanks for your advice!