Initial proposal for changes to the OCI specification

[mnm678]

I added a file with some comments about changes to the OCI specification that will be needed to support this design. I'd appreciate reviews from those with more OCI experience to make sure this captures all the required changes.

cc @samuelkarp @sudo-bmitch

[sudo-bmitch]

Are we planning to only have one set of these files per repository? I ask because I'm thinking about race conditions when there's more than one signer. One nice feature of Steve's proposal is that each signature is a separate blob with it's own hash to avoid these race conditions.

[mnm678]

Are we planning to only have one set of these files per repository? I ask because I'm thinking about race conditions when there's more than one signer. One nice feature of Steve's proposal is that each signature is a separate blob with it's own hash to avoid these race conditions.

There would be one set of root, timestamp, and snapshot metadata for each root (currently per registry, but possibly per organization). There will also be one top-level targets metadata file, that can delegate to many other targets metadata files.

Timestamp and snapshot can likely be signed by a single key, and root won't be updated too often, so I think the race condition is most likely to come up for targets metadata. One option would be to limit signers of the same file to people that can coordinate off of the registry, then upload the metadata once. They could use delegations to require signatures from folks who can't coordinate offline (ie, one targets file for each team/individual, then delegate to a threshold of those targets files).

[SteveLasker]

We're trying, really hard, to avoid making changes to a registry for a specific scenario, including signing.
We're trying to design generic solutions for storing artifacts in a way that would support signatures of various artifact types. This recent Artifacts PR starts down that path.
Taking the OCI Artifacts approach will allow copying of content within and across registries, including the signatures/signing elements. This is important as we want/need to make it easy for Artifact clients (like container images) to easily incorporate signing.

An OCI Artifact can be placed at any level of a registry, so it may serve the needs of TUF.
Can I suggest some research is done to identify how TUF can be saved as an artifact, similar to the Notary v2 prototype-1 effort?
Each signature has a pointer to the thing it signs. This enables the easy movement from a registry to a client, and copying within/across registries. We'd want the same for a TUF implementation as well.

[mnm678]

We're trying, really hard, to avoid making changes to a registry for a specific scenario, including signing.
We're trying to design generic solutions for storing artifacts in a way that would support signatures of various artifact types.

For this reason, I think the more generic metadata type described in this document might be a better fit. All it needs is the ability to store arbitrary data about an image or other metadata file and signature support.

[sudo-bmitch]

When thinking about how to use the registry and artifact interfaces, one important change from how TUF is used to working is that there's currently only one thing we pull from a registry by name, and that's tags that point to manifests. Everything else is pushing an immutable digest to a blob of some kind (currently json or a tgz file, but we can add more media types).

Some of that may change with the artifact work, but the complication I'm seeing with the TUF logic is you wouldn't push and pull something called "snapshot" in a repo. We would push an artifact that has a signature media type, and maybe an annotation indicating it is a pointer to a "snapshot" blob, along with another pointer to indicate what manifests it is associated with.

The nice part about the registry API is there are no concurrency collisions (unless you manage to get sha256 hash collision within a repo). Multiple user jobs can be running to push/pull images, and all the data caches nicely with a CDN. So as we consider how to move TUF into the registry design, concepts of a mutable object that we reference by name need to be rethought.

One hacky solution I can think of is an artifact pointer with a timestamp that allows us to GC the stale pointers so that a query of all the artifacts only returns 1-2 snapshot artifacts, and the annotation could be checked for the most recent. I'm not a big fan of this, so I feel like we might end up pushing towards another type of "tag" functionality for artifacts, where the registry stores this mutable pointer. I don't think we can extend the manifest tags because TUF "snapshot" objects span multiple manifests.

Before I get too far down the thought process of how this could be implemented, I wanted to raise this concern in case others had an easy answer I was overlooking.

[mnm678]

Before I get too far down the thought process of how this could be implemented, I wanted to raise this concern in case others had an easy answer I was overlooking.

TUF has a concept of 'consistent snapshots' that may help here. Basically, every metadata file can be prepended with a version number (1.root.json, etc), so that if one user is downloading 4.snapshot.json while another user is uploading 5.snapshot.json, both of those requests would work. The old versions could be GC'd after the new version is uploaded.

[dlorenc]

Whoa! Somehow I missed this proposal earlier. I'm sorry about that! I should have referenced it in opencontainers/image-spec#828

@mnm678 do you think that reference proposal meets these requirements? At first glance it appears to.

[mnm678]

@mnm678 do you think that reference proposal meets these requirements? At first glance it appears to.

The reference field would allow for the linking between metadata and artifacts, which is a big part of what this proposal needs. The only additional feature needed would be a way to query directly for the metadata. This is one of the big differences between TUF's key management and just using signatures. With TUF the user first downloads some top-level metadata (root, targets, etc), then uses this to determine which keys should have signed the artifact. It would be possible to do this using an API that downloads all metadata associated with an object, then only verifying the subset of signatures you trust, but it would be more efficient if clients could request specific signatures/metadata objects.

[dlorenc]

What type of fields would we want to query on? I think @SteveLasker's proposal included a query parameter for media type.

I'm not sure we'll be able to get much further than that with the current OCI spec unfortunately, at least without maintaining a separate index.

Cc @jonjohnsonjr who has been working on formalizing basic list support for awhile now.

[mnm678]

A query on media type would work, though it might be nice if there was something more specific, especially because we've discussed having multiple roots of trust on the same registry. In that case a registry like DockerHub might have the Docker root metadata as well as root metadata for any organization that wants to manage their own root keys. Even worse, a query for targets metadata would include every targets metadata file for every image. This could be mitigated by querying on both the media type and the reference, but that doesn't solve the problem for root metadata.

Something like tags might work better, where the metadata is tagged as 'docker-root' or 'wabbit-root' by the uploader and just the single metadata object with that tag is downloaded. These tags could be specified in TUF delegations so that the user would know which tag to use for the next piece of metadata they need to download. TUF can work on unsecured networks, so relying on the registry for tag resolution shouldn't be an issues for TUF metadata.

I'd appreciate any feedback from registry operators about what would be easy/feasible here.

[sudo-bmitch]

What type of fields would we want to query on?

The ideal to me would support querying by media type and annotation. Annotation support allows the client to specify a search on one key that the client trusts/requires. Without that, we have to pull back all artifacts and filter them at the client, which could get expensive with lots of registry get requests and round trips.

[mnm678]

Closing this here. See related issues in the OCI (opencontainers/image-spec#828, opencontainers/artifacts#29, etc)