Kaniko 1.9.1 generates invalid OCI image, Quay 3.8.x rejects it

[sumkincpp]

Actual behavior

• Multi-stage images with multiple registries is used
• Multiple registries - Gitlab (main repository), Quay and JFROG are used
• Cache is used

When I build an image with Kaniko 1.9.1 in Gitlab(used as cache and base repo), it's layers become mixed with different manifests - the problem seen for example with Quay 3.8.0/3.8.1, that rejects pushed image, that expects valid oci image oder docker image -

error pushing image: failed to push to destination QUAY_HOST: PUT  .../latest: MANIFEST_INVALID: manifest invalid; map[message:failed to parse manifest: manifest data does not match schema: 'application/vnd.docker.image.rootfs.diff.tar.gzip' is not one of ['application/vnd.oci.image.layer.v1.tar', 'application/vnd.oci.image.layer.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+zstd', 'application/vnd.oci.image.layer.nondistributable.v1.tar', 'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip', 'application/vnd.dev.cosign.simplesigning.v1+json', 'application/vnd.dsse.envelope.v1+json', 'text/spdx', 'text/spdx+xml', 'text/spdx+json', 'application/vnd.syft+json', 'application/vnd.cyclonedx', 'application/vnd.cyclonedx+xml', 'application/vnd.cyclonedx+json', 'application/vnd.in-toto+json', 'application/tar+gzip', 'application/vnd.cncf.helm.chart.content.v1.tar+gzip', 'application/vnd.oci.image.layer.v1.tar+gzip']
Failed validating 'enum' in schema['properties']['layers']['items']['properties']['mediaType']:
    {'description': 'The MIME type of the referenced manifest',
     'enum': ['application/vnd.oci.image.layer.v1.tar',
              'application/vnd.oci.image.layer.v1.tar+gzip',
              'application/vnd.oci.image.layer.v1.tar+zstd',
              'application/vnd.oci.image.layer.nondistributable.v1.tar',
              'application/vnd.oci.image.layer.nondistributable.v1.tar+gzip',
              'application/vnd.dev.cosign.simplesigning.v1+json',
              'application/vnd.dsse.envelope.v1+json',
              'text/spdx',
              'text/spdx+xml',
              'text/spdx+json',
              'application/vnd.syft+json',
              'application/vnd.cyclonedx',
              'application/vnd.cyclonedx+xml',
              'application/vnd.cyclonedx+json',
              'application/vnd.in-toto+json',
              'application/tar+gzip',
              'application/vnd.cncf.helm.chart.content.v1.tar+gzip',
              'application/vnd.oci.image.layer.v1.tar+gzip'],
     'type': 'string'}
On instance['layers'][1]['mediaType']:
    'application/vnd.docker.image.rootfs.diff.tar.gzip']

Inspecting temporary image in Gitlab shows that image was generated by Kaniko with Invalid Specification(OCI should have no docker diff layers), though Gitlab had allowed saving this image :

[vagrant@localhost images]$ skopeo inspect --raw docker://same-image-in-gitlab/  | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "size": 3764,
    "digest": "sha256:b9c7883aa34d88051b1e6cfe7d064207f6b818f8d99c2a5703832e588cfb604c"
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": 29528717,
      "digest": "sha256:677076032cca0a2362d25cf3660072e738d1b96fe860409a33ce901d695d7ee8"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 46992646,
      "digest": "sha256:ad8b2a5db75cae1fb3962be244aa89999930dfd6f309fd65c433a652323fdfe9"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1015,
      "digest": "sha256:491ee3009c9026f93c8e51f735de7eff2b321c7369f41ade9db0600b3798f2a0"
    }
  ]
}

Expected behavior

Correct media-type of image - all layers should be conform to each other. They should OCI (application/vnd.oci.image.config.v1+json) or Docker (application/vnd.docker.distribution.manifest.v2+json) conform.

Correct image generated with Kaniko 1.9.0 :

[vagrant@localhost images]$ skopeo inspect --raw docker://our-image/  | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 5395,
    "digest": "sha256:21b0d07fa4cf7db7e252006c11f0a386d9d0f6670db29b530d90a10a756acce7"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 30429004,
      "digest": "sha256:10ac4908093d4325f2c94b2c9a571fa1071a17a72dd9c21c1ffb2c86f68ca028"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 393098,
      "digest": "sha256:6fb956ad156c221705b55842e7350dd46063386585c4b75cd37a9a4e8cf9e3e3"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 81403231,
      "digest": "sha256:34561caa4681b6924aa65c61c4809c6c8ddd16cb99020a2827f7f6277ba36492"
    }
  ]
}

Triage Notes for the Maintainers

Bug seems to be introduced in Kaniko 1.9.1 with a change how first layer is handled - it is being not unpacked and probably saved with initial media type.

[sumkincpp]

duplicate #1836