Error mirroring git-init image in private registry. Unsupported MIME type for compression

[jfuerneisen]

Expected Behavior

git-init image can be mirrored from gcr.io to private registry.

Actual Behavior

Mirroring fails with error: unsupported MIME type for compression: application/vnd.docker.image.rootfs.diff.tar.gzip

Steps to Reproduce the Problem

skopeo copy docker://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.37.2 docker://example.example.com/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.37.2

FATA[0001] Uploading manifest failed, attempted the following formats: application/vnd.oci.image.manifest.v1+json(writing manifest "{\"schemaVersion\":2,\"mediaType\":\"application/vnd.oci.image.manifest.v1+json\",\"config\":{\"mediaType\":\"application/vnd.oci.image.config.v1+json\",\"size\":1132,\"digest\":\"sha256:92846c512aba07f16a37baac3ac2650c4af38d6cc986010941568d174a4fb7a1\"},\"layers\":[{\"mediaType\":\"application/vnd.oci.image.layer.v1.tar+gzip\",\"size\":16196893,\"digest\":\"sha256:fa4f96b22123a850bc0727f32c52099e618ac215d9a06832562a41af6110c644\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar.gzip\",\"size\":8914910,\"digest\":\"sha256:1cd258f7c0bb909f632b3dd5faedf3e3f73191608364cc6c24c1a41519e2fa09\"},{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar.gzip\",\"size\":15465791,\"digest\":\"sha256:199f56e7ac1acca896b89ead8ba89d60039b2cabf68b45c5c74c85571d999aaa\"}]}": uploading manifest v0.37.2 to quay.example.com/okd/catalog/tekton-releases/github.com/tektoncd/pipeline/cmd/susi/git-init: manifest invalid: manifest invalid), application/vnd.docker.distribution.manifest.v2+json(creating an updated image manifest: Unknown media type during manifest conversion: "application/vnd.docker.image.rootfs.diff.tar.gzip"), application/vnd.docker.distribution.manifest.v1+prettyjws(creating an updated image manifest: Unknown media type during manifest conversion: "application/vnd.docker.image.rootfs.diff.tar.gzip"), application/vnd.oci.image.index.v1+json(creating an updated image manifest: Unsupported conversion type: application/vnd.oci.image.index.v1+json), application/vnd.docker.distribution.manifest.list.v2+json(creating an updated image manifest: Unsupported conversion type: application/vnd.docker.distribution.manifest.list.v2+json), application/vnd.docker.distribution.manifest.v1+json(creating an updated image manifest: Unknown media type during manifest conversion: "application/vnd.docker.image.rootfs.diff.tar.gzip")

podman pull gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.37.2
podman tag  gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.37.2 registry.example.com/cm/git-init-tekton:v0.37.2
podman push registry.example.com/cm/git-init-tekton:v0.37.2

Error: creating an updated image manifest: preparing updated manifest, layer "sha256:aefddf89d81647c74d761e2b4f77827afb785280cde0886509a333442711ef4d": unsupported MIME type for compression: application/vnd.docker.image.rootfs.diff.tar.gzip

Additional Info

Mirroring worked fine with this image until Release v0.36.0. The error occurs since v0.37.0.
All other tekton release images are working.

• Kubernetes version:

  **Output of kubectl version:

  not applicable
  

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

  not applicable
  

[imjasonh]

What registry implementation are you trying to mirror into? (Quay, Harbor, etc)

This might help narrow down the issue.

Recently we switched the git-init image to be based on distroless.dev/git, which is built with apko. This may be related. We've also switched our base image for everything else to distroless.dev/static so other images may also start exhibiting the same behavior when that's released.

[imjasonh]

Could you try mirroring our nightly release images to see if this will be a problem when those changes are in a full release?

Apko also switched to producing OCI-typed images by default since that git-init image was released, it's possible that this confluence of events fixes the issue in your registry, but it would be good to be sure.

[jfuerneisen]

I tried with quay and also gitlab registry.
Quay v3.7.3
Gitlab 15.1.2-ee

I also ran into an error, when i tried to extend this image in a buildah build.

[jfuerneisen]

Same error with gcr.io/tekton-nightly/github.com/tektoncd/pipeline/cmd/git-init:latest

[disposab1e]

Same problem here.... I think this is NOT a problem specific to an registry... it looks like apko is not producing oci conform images?

However... if you're not able to mirror tekton images in private registries this will be a show stopper for all operating tekton in an disconnected environment e.g. air gapped OpenShift.

[disposab1e]

For me it's a little bit like this Kaniko problem with mixed layers..... see Kaniko#1836.

When you inspect the actual tekton image you only see oci layers in the json output... but in reality it might be a docker layer....

[imjasonh]

The fix for this is in ko-build/ko#776 -- I expect a ko release soon, that Tekton should start using shortly after, to produce consistently-typed image layers so this isn't an issue.

Sorry for the disruption, and thanks for reporting this!

[disposab1e]

@imjasonh Cool + Respect!

[chaospuppy]

@imjasonh This issue has persisted in Tekton v0.41.1 (gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/controller@sha256:02aef94088e2f5487b224c6a9d380cac7fa72e8fec883c9e2833569f302cb53c is the corresponding image sha), and presents itself when attempting to run skopeo copy --remove-signatures docker://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/controller@sha256:02aef94088e2f5487b224c6a9d380cac7fa72e8fec883c9e2833569f302cb53c docker-archive:gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/controller:v0.40.1

This will result in the same error as posted above:

FATA[0013] creating an updated image manifest: Unknown media type during manifest conversion: "application/vnd.docker.image.rootfs.diff.tar.gzip"

[vdemeester]

Ah 😅 this didn't apply to 0.41.1 because I did not cherry-pick #5539 to the release branch (and thus.. it used ko 0.11 I guess).
I'll cherry-pick today and prepare a 0.41.2 tomorrow 😉