bundle references generated tag of invocation image instead of just the digest

[jasmdk]

Is your feature request related to a problem? Please describe.
Porter v1 no longer pushes the invocation image as a separate repo, but instead it adds a tag with a random value. The tag is obsolete if porter knows the digest of the image already and the tag prevents standard garbage collection routines (like in Harbor) to implement a strategy for removing intermediate bundles using a tag pattern.

Describe the solution you'd like
I expect that porter uses the published manifest to determine the image that it must use as invocation image, however the invocationImages[].image points to the image using both a tag and a digest which means that the digest takes precedence and the tag is actually ignored.
By not tagging the invocation image, then garbage collection routines that uses tags to determine which artifacts to retain or delete can consistently remove all images in a bundle including the invocation image - otherwise removed bundles will produce dangling invocation images no longer referenced from a bundle, but unable to be garbage collected as they are tagged.
If a tag is needed, could it then be the same tag as the bundle, which again would allow garbage collection routines to identify bundles to delete based on tags.

Describe alternatives you've considered

1. Create our own cron-job to identify all CNAB bundles and their invocation images and compare this with all docker images in the same repo, which is tagged. Remove the tag from the images that is not currently referenced by a CNAB manifest and then let Harbor do the actual garbage collection.
2. Accept that storage grows at an even faster rate with unused dangling images

Additional context
A docker file on the form <registry>/<repo>:<tag>@<digest> which is what is stored in the manifest can also be pulled using <registry>/<repo>@<digest> because the digest takes precence over a tag, so the tag is effectively ignored.

[carolynvs]

The short answer is that it is impossible to calculate the repository digest of the invocation image without pushing it to the repository first. So we push it to the bundle repository, then retrieve the repository digest and rewrite the bundle to reference it by digest and not the tag.

@VinozzZ I thought that you looked into removing the tag afterwards but I don't remember the details on why we don't do that. I think it was because removing a tag isn't universally supported or something like that?

[jasmdk]

If it can not be removed, is there a reason why the tag seems to be a random value and not the bundle tag?

[carolynvs]

The tag is a hash of the bundle reference, so not really random. We do not push the image to a location that could be used by other versions of the bundle to avoid race conditions that arise when publishing multiple versions of the bundle to the same registry. Reusing a constant tag value for the image is not an option, sorry.

[carolynvs]

Here's one workaround we can try for registries that doesn't support removing a tag:

https://stackoverflow.com/a/71580038

tldr: push an empty manifest to the tag, and then issue a delete.

I just realized that the behavior you are seeing is partly caused by how we are referencing the generated tag and not just the digest by itself. I'll get that fixed, thank you for bringing it up!

[carolynvs]

I've put this into the 1.0 milestone since I feel like it would be much better to not cement our current behavior in the 1.0 release. At the very least we'll stop referencing the generated tag, but stretch goal is to delete the tag at the end of publish using that trick I linked to.

[carolynvs]

As a side note, thank you for all the detailed feedback and bug reports on the beta! It is incredibly helpful.

[jasmdk]

As the tag is needed, could it be named something like "porter-" which would make it possible to exclude them from immutability rules. If it is considered immutable, then deleting the tag will be rejected.
At least Harbor can not use regular expressions, so we make all tags immutable and then exclude certain patterns (https://goharbor.io/docs/2.6.0/working-with-projects/working-with-images/create-tag-immutability-rules/)

[carolynvs]

• Do not reference the tag for the invocation image, only the digest (Fix in Discard the temporary invocation image tag during publish #2328)
• Add a porter- prefix to the generated tag for invocation images and referenced images so that people can a) tell what it is and b) write scripts and rules to match the tags. Update the docs if we reference this tag value anywhere.
• (stretch goal, will happen in a later PR possibly post v1) Try the suggested workaround to remove the tag once we are done with it at the end of publish. (Tracking issue is "untag" temporary tagged images after pushing #2329)

[VinozzZ]

The short answer is that it is impossible to calculate the repository digest of the invocation image without pushing it to the repository first. So we push it to the bundle repository, then retrieve the repository digest and rewrite the bundle to reference it by digest and not the tag.

@VinozzZ I thought that you looked into removing the tag afterwards but I don't remember the details on why we don't do that. I think it was because removing a tag isn't universally supported or something like that?

Hey, sorry for the late response. When I did the research, it looks like the proposal for tag deletion was added into the spec last year: opencontainers/distribution-spec#114. At the time we decided that we will wait to implement the deletion once registry has implemented this new API. It seems like currently not many registry has implemented this new spec. I found one project did the workaround you mentioned here: https://github.com/regclient/regclient/blob/0532f69020b0e59c497523c2b016f018480ce27f/scheme/reg/tag.go#L35

[carolynvs]

Thank you for confirming that the workaround is our best bet for now (to come after v1). 👍

[github-actions]

Closed by #2336.