make -garbage-collect aware of manifest lists (aka "fat" manifests)

[listx]

Currently the garbage collection logic just checks if a digest has no associated tags, and if so, deletes them. This doesn't account for manifest lists.

Thankfully, the gcloud call we use currently in version 234.0.0 crashes if we attempt to delete a digest that is referenced by a manifest list, so nothing happens (and we can use -delete-extra-tags safely).

The solution is to just use the Docker Registry API for this. Let's use the example of gcr.io/cip-demo-prod/coredns. The first call we can use is:

 $ curl -H "Authorization: Bearer $(gcloud auth print-access-token)" https://gcr.io/v2/cip-demo-prod/coredns/tags/list | jq
{
  "child": [],
  "manifest": {
    "sha256:02382353821b12c21b062c59184e227e001079bb13ebd01f9d3270ba0fcbf1e4": {
      "imageSizeBytes": "0",
      "layerId": "",
      "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
      "tag": [
        "1.3.1"
      ],
      "timeCreatedMs": "0",
      "timeUploadedMs": "1550296103137"
    },
    "sha256:613916bb25abd7c28d807e0695eed8c18b1b91555c9071bd742f5d3ba4b05dbe": {
      "imageSizeBytes": "11932168",
      "layerId": "",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "tag": [],
      "timeCreatedMs": "1547391218295",
      "timeUploadedMs": "1550296102546"
    },
    "sha256:638adb0319813f2479ba3642bbe37136db8cf363b48fb3eb7dc8db634d8d5a5b": {
      "imageSizeBytes": "12300583",
      "layerId": "",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "tag": [],
      "timeCreatedMs": "1547391212560",
      "timeUploadedMs": "1550296077349"
    },
    "sha256:7ce7f621749df5cd5f32b6daef6c192c66b067d4fde863f1d7004eeb8e148cae": {
      "imageSizeBytes": "10932753",
      "layerId": "",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "tag": [],
      "timeCreatedMs": "1547391216855",
      "timeUploadedMs": "1550296096170"
    },
    "sha256:d8d1f57070089244091c79002cf486b5108393062a0d7e6b43c4abfb77da7adb": {
      "imageSizeBytes": "11514698",
      "layerId": "",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "tag": [],
      "timeCreatedMs": "1547391214033",
      "timeUploadedMs": "1550296083921"
    },
    "sha256:eec40e97de27c11bf6840ece1a9a829e37e7a37ecae535e1907ec490aa829539": {
      "imageSizeBytes": "11136918",
      "layerId": "",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "tag": [],
      "timeCreatedMs": "1547391215381",
      "timeUploadedMs": "1550296090208"
    }
  },
  "name": "cip-demo-prod/coredns",
  "tags": [
    "1.3.1"
  ]
}

to get all digests/tag information for an image path. Then we can identify if a digest is actually a manifest list (and not a regular digest) if it has "imageSizeBytes": "0". Then for each manifest list, we can figure out the child manifests that it references by doing a 2nd call:

 $ curl -H "Authorization: Bearer $(gcloud auth print-access-token)" https://gcr.io/v2/cip-demo-prod/coredns/manifests/sha256:02382353821b12c21b062c59184e227e001079bb13ebd01f9d3270ba0fcbf1e4    
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:638adb0319813f2479ba3642bbe37136db8cf363b48fb3eb7dc8db634d8d5a5b",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:d8d1f57070089244091c79002cf486b5108393062a0d7e6b43c4abfb77da7adb",
         "platform": {
            "architecture": "arm",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:eec40e97de27c11bf6840ece1a9a829e37e7a37ecae535e1907ec490aa829539",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:7ce7f621749df5cd5f32b6daef6c192c66b067d4fde863f1d7004eeb8e148cae",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:613916bb25abd7c28d807e0695eed8c18b1b91555c9071bd742f5d3ba4b05dbe",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

The annoying/ugly part is that we need to migrate away from our simple dependence of gcloud, but that is probably inevitable anyway. AFAIK gcloud does not support querying about manifest lists (to figure out what images a manifest list references). The docker CLI does support this operation, but depending on the docker CLI is not desirable (when a library will do). Thankfully it appears that there is already a Go library that implements the Docker registry API here https://github.com/google/go-containerregistry/blob/52aa2729ea7597c8d5f7d6a977fa5cab9fc3a77a/pkg/v1/google/list.go#L69, so we could use that instead (hmm, but it doesn't appear to support distinguishing between a plain manifest and manifest list (see https://github.com/google/go-containerregistry/blob/52aa2729ea7597c8d5f7d6a977fa5cab9fc3a77a/pkg/gcrane/copy.go#L196 for the only usage of a DockerManifestList type in that repo).

We could just curl out instead --- it's the dumbest thing that will work.

[listx]

I think this is a high-priority item; labeling it as a bug for now because I'm too lazy to create P0/P1/P2/P3 priority labels.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[listx]

/remove-lifecycle stale

[listx]

Closing this issue for now because we don't plan on using garbage collection for the promoter any more (it will be a separate tool). As this issue won't be worked on for the foreseeable future, I'm closing this.