Support Docker (OCI) repositories

[dzikoysk]

Request details

To support OCI repositories for e.g. Docker, we need to figure out how to do that. Specification:

• https://github.com/opencontainers/distribution-spec/blob/main/spec.md

[dzikoysk]

If someone has some experience with it, it'd be nice to prepare list of routes with short description per each endpoint, so we could create a minimal working implementation on top of that.

[TOTHT0MI]

For future investigation I've collected resources, that can be useful. It seems pretty straight forward to implement, however the flow of uploading is quite different from Maven.

Official spec
OCI Distribution spec: https://github.com/opencontainers/distribution-spec/blob/main/spec.md
OCI Image spec: https://github.com/opencontainers/image-spec/blob/main/spec.md

Third party
A summary of the specification: https://medium.com/@zbase000/oci-image-and-registry-specifications-a59db599991f

[dzikoysk]

Thanks, Docker support is something I'd really like to see in 4.x (at some point) :) I'll probably try to bootstrap some sort of proof of concept, or browse GitHub to check existing projects, to get a brief overview on how the minimal working impl would look like 🙏

[zrdzn]

I did some research and created descriptions for the endpoints, which should help visualize how we should approach the implementation of this specification, at least to some extent.

Notes

Definitions

• <name> - refers to the namespace of the repository
• <blob> - binary form of stored content
• <digest> - unique identifier of a blob
• <reference> - either a manifest's digest OR a tag (custom readable pointer to a manifest)
• <location> - upload URL or session ID that was generated in 1st part of 2-step method
• <manifest> - a configuration and set of layers for a single container image for a specific architecture and operating system. Read more here.

Blob Upload Strategies

We have the option to support two blob upload strategies. We can either support one or both at the same time, but this will certainly affect the implementation time.

1-STEP

The simplest option is to support a single request via POST, which uploads the blob to the registry. This is done by making a POST /v2/<name>/blobs/uploads/?digest=<digest> request, where we include the byte stream in the body and receive a location for the newly created blob in response.

Advantages

• Easy and quick implementation
• A good choice for small blobs

Disadvantages

• No option to split the blob into parts, which can be problematic for large blobs
• No option to resume the upload, which is disadvantageous for poor connections

2-STEP

This approach is slightly more complex. First, we make a POST /v2/<name>/blobs/uploads/ request, which returns a path or session ID that we use in the next PUT /v2/<name>/blobs/uploads/<location>?digest=<digest> request. The second request (PUT) is very similar to the request in the 1-STEP strategy.

Advantages

• Allows splitting the blob into chunks
• Allows resuming an existing upload
• Suitable for uploads with poor connections
• Suitable for large blobs

Disadvantages

• Longer implementation time

Conclusions

I think we should implement both approaches due to the different needs they address. The specification mentions that some implementations do not support the 1-STEP option, so we could treat it as an optional feature. If we were to support only one option, it would likely be better to use the 2-STEP approach – it has greater long-term benefits.

Endpoints (Incompleted)

GET /v2/

Checks if registry implements specification.

Codes:

• 200
• 404
• 401

GET/HEAD /v2/<name>/blobs/<digest>

Finds a blob by providing a digest.

Codes:

• 200
• 404

DELETE /v2/<name>/blobs/<digest>

Deletes a blob by providing its digest.

Codes:

• 202
• 404
• 400
• 405

POST /v2/<name>/blobs/uploads/?mount=<digest>&from=<other_name>

Mounting a blob from another repository in the same registry. It will mount blob with provided digest to <name> repository, from repository <other_name>.

Response:

• Headers
  • Location - location of a newly mounted blob

Codes:

• 201
• 202
• 404

GET/HEAD /v2/<name>/manifests/<reference>

Finds a manifest by its digest or a tag.

Codes:

• 200
• 404

POST /v2/<name>/blobs/uploads/ 2-STEP METHOD (1/2)

It is used in 2-step method which basically initiates an upload session and generates an upload URL or session ID, which is used for the subsequent upload or chunks.

Response:

• Headers
  • Location - upload URL/session ID
  • (Optional) OCI-Chunk-Min-Length - minimum size of a chunk in bytes

Codes:

• 202

PUT /v2/<name>/blobs/uploads/<location>?digest=<digest> 2-STEP METHOD (2/2)

Uploads the blob (file or data) to the registry using the upload URL obtained in the 1st part of 2-step method.

Request:

• Headers
  • Content-Length - size of blob in bytes
  • Content-Type - application/octet-stream
• Body
  • Byte stream

Response:

• Headers
  • Location - location of a newly created blob

Codes:

• 201
• 404
• 400

POST /v2/<name>/blobs/uploads/?digest=<digest> 1-STEP METHOD

Simply push blobs by a single request.

Request:

• Headers
  • Content-Length - size of blob in bytes
  • Content-Type - application/octet-stream
• Body
  • Byte stream

Response:

• Headers
  • Location - location of a newly created blob

Codes:

• 201
• 202
• 404
• 400

PATCH /v2/<name>/blobs/uploads/<location>

Allows to upload blob by chunks instead of the single big one. They must however, be uploaded in correct order - it means that the <min> in chunk must be the last chunk's <max> + 1. Location must be generated from 1st part of 2-step method above.

Request:

• Headers
  • Content-Type - application/octet-stream
  • Content-Range - inclusive range of bytes in format <min>-<max> (First chunk must start from 0)
  • Content-Length - size of chunk in bytes
• Body
  • Byte stream of a chunk

Codes:

• 202
• 404
• 416 - chunks are uploaded out of order (ranges do not not connect to each other)

PUT /v2/<name>/manifests/<reference>

Uploads a manifest with a specified digest/tag.

Request:

• Headers
  • Content-Type - type of manifest being pushed e.g. application/vnd.oci.image.manifest.v1+json (Registry MUST ignore parameters key=value)
• Body
  • mediaType in body MUST match the media type provided in Content-Type

{
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  <rest of the content>
}

Response:

• Headers
  • Location - location of a newly created manifest

Codes:

• 201
• 404
• 413 - maximum size of manifest achieved

DELETE /v2/<name>/manifests/<reference>

Codes:

• 202
• 404
• 400
• 405

GET /v2/<name>/tags/list

Returns a list of existing tags for a given namespace. Response can contain empty list of tags. Tags MUST be in case-insensitive alphanumeric order.

Response:

• Body

{
  "name": "<name>",
  "tags": [
    "<tag1>",
    "<tag2>",
    "<tag3>"
  ]
}

Codes:

• 200
• 404

GET /v2/<name>/tags/list?n=<amount>&last=<tagname>

Returns a subset of tags with amount equal or less to provided in <amount>. When given <amount> is 0, response MUST contain empty list of tags. Additionally there is <tagname> parameter which defines from which non-exclusive tag should the tags be returned for <amount> tags above - provided tag in <tagname> will not be included in the response. When the <tagname> is provided, then the <amount> is optional.

Response:

• Body

{
  "name": "<name>",
  "tags": [
    "<tag1>",
    "<tag2>",
    "<tag3>"
  ]
}

Codes:

• 200
• 404

GET /v2/<name>/referrers/<digest>?artifactType=<artifactType>

Fetches a list of referrers by a manifest's digest. Descriptors MUST include artifactType field where the value matches to the manifest's artifact type - if there is none, just set artifactType to the mediaType value. Annotations MUST be taken from manifest. If there are no referrers, manifests list MUST be empty. Optionally client may want to filter referrers by artifactType, if the filter is present - the header OCI-Filters-Applied: artifactType is required.

Response:

• Headers
  • Content-Type - application/vnd.oci.image.index.v1+json
  • (When artifactType query present) OCI-Filters-Applied - artifactType
• Body

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "size": <size of manifest>,
      "digest": <digest of manifest>,
      "artifactType": <artifact type>,
      "annotations": {
        "org.opencontainers.image.created": "2022-01-01T14:42:55Z"
      }
    }
  ]
}

Codes:

• 200
• 400

GET /v2/<name>/blobs/uploads/<location>

Used when there was an upload of chunks, but they were out of correct order. It is required to get status of an active upload to resume it.

Response:

• Headers
  • Location - location provided in the url
  • Range - 0-<position of last uploaded byte>

Codes:

• 204
• 404

[dzikoysk]

Thanks to @zrdzn, we were able to prepare a working prototype of Docker repository:

• https://github.com/reposilite-playground/oci-repository-prototype

Considering our current resources & value of this feature, I'm including basic Docker repository support in the scope of 4.x:

• Reposilite 4.x #2224