Simplifying the ORAS client experience

[SteveLasker]

Simplifying the ORAS Client Experience

When pushing artifacts to an OCI compliant registry, a manifest.config.mediaType is required to identify the artifact, and layer.config.mediaTypes are required for each layer.

As we look to improve on the current experiences, the current ORAS client requires either too little, defaulting in the wrong defaults, or too much, creating a less than productive experience, easy to make mistakes.

To provide a progressive disclosure experience, I'm proposing two changes:

1. Provide for an orasConfig file, enabling the user to provide the repetitive values to the oras client.
2. ORAS provides for default unkown mediaTypes for the artifact and layers. ORAS mediaTypes should default to unknown #115

Current Basic Example

Walking through a simple example, the user specifies the artifact path and the files.

oras login demo42.azurecr.io -u $USR -p $PWD 
oras push demo42.azurecr.io/samples/docs-in-markdown:v1 \
    ./readme.md \
    ./detail.md

This generates a manifest that defaults to application/vnd.oci.image.config.v1+json for the artifact and vnd.oci.image.layer.v1.tar for the layers.

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
    "size": 2
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:532eaabd9574880dbf76b9b8cc00832c20a6ec113d682299550d7a6e0f345e25",
      "size": 4,
      "annotations": {
        "org.opencontainers.image.title": "orasTest.txt"
      }
    }
  ]
}

This is incorrect, as application/vnd.oci.image.config.v1+json and vnd.oci.image.layer.v1.tar are reserved for container images that would be instanced with the docker or containerD client. For the initial experience to be correct, the user must specify their type, which they may not be ready to think about.

This is the context behind the two-stage proposal.

Providing mediaTypes

To properly set the mediaTypes with the current version of the ORAS client, the user would have to provide a much more complex cli call:

oras push demo42.azurecr.io/samples/docs-in-markdown:1 \
  --manifest-config ./config.json:application/vnd.contoso.docs-in-markdown.config.v1+json \
  ./readme.md:application/vnd.contoso.docs-in-markdown.layer.v1+md \
  ./detail.tar:application/vnd.contoso.docs-in-markdown.layer.v1+tar

This isn't something the user would want to type each time. Considering these values don't change very often, requiring them to be entered each time leads to frustration and errors.

Optimizing for Usage

There are a few small enhancements we could make, such as a config file for defining the artifact mediaType.

artifactType: application/vnd.contoso.docsinmarkdown.config.v1+json

However, what do we do about the layer mediaTypes?

I considered a mapping of extensions to mediaTypes

artifactType: application/vnd.contoso.docs-in-markdown.config.v1+json
layerTypes:
    - md: application/vnd.contoso.docs-in-markdown.layer.v1+md
    - tar: application/vnd.contoso.docs-in-markdown.content.v1.tar

However, this doesn't scale well as an artifact could likely use the same .tar or even .md extension for several layer types.

Even in the simplest case, the user would still need to pass in the collection of files, each time.

oras push demo42.azurecr.io/samples/docs-in-markdown:v1 \
    ./readme.md \
    ./detail.md

orasConfig

Artifact authors define these types once for an artifact. They continue to iterate, possibly adding new files. To simplify ORAS usage, either as a client or go library, an orasConfig is proposed.

The elements to be stored in the mapping file include:

example: demo42.azurecr.io

• artifactName: The content addressable path to the artifact.

  example: demo42.azurecr.io/samples/docs-in-markdown:v1

• artifactType: the single mediaType used to define the artifact.

  example: application/vnd.contoso.docs-in-markdown.config.v1+json

• layerTypes: a collection of layer types used for the artifact.

  example:

  • application/vnd.oci.image.layer.v1.tar
  • application/vnd.oci.image.layer.v1.tar+gzip

• files: a collection of files to upload, mapped to a specific type, with wildcard support to minimize having to specify every file.

  example:

  • ./readme.md
  • ./description.md
  • ./docs/*.md

Smart Defaults

Using a config file, the user could minimize the content they would type with the ORAS client:

oras push -c orasConfig.yaml

or, we could make orasConfig.yaml a default, if found:

oras push

If the orasConfig.yaml isn't found, the ORAS client generates an error.

orasConfig Format

The usual yaml vs. json format debate is the obvious, but lets consider this file is mostly user editable. There's likely not a lot of content. For this reason, even with the normal space vs. tab yaml issues, I'm proposing the following yaml format.

version: "0.1-preview"
name: demo42.azurecr.io/samples/docs-in-markdown:v1
artifactType: application/vnd.contoso.docs-in-markdown.config.v1+json
layers:
  - readme:
    - type: application/vnd.contoso.docs-in-markdown.readme.layer.v1+md 
    - files: 
      - "./readme.md"
  - content:
    - type: application/vnd.contoso.docs-in-markdown.content.layer.v1+md 
    - files:
      - "./index.md"
      - "./docs/*.md"

Why the Nested Layer Collections?

The types of artifacts likely group into 3 different scenarios:

1. There are 1 or more layers. The order may, or may not be important, but all the types are the same.
2. There are multiple layer types, however the ordering isn't important.
3. There are multiple layer types, and the ordering is important.

Some artifact types, like the docker/OCI image format, depend on ordinal layers to build the filesystem layout. In these cases, it's critical the layers can be specified in a specific order, with their respective types.

In the above example, the first type readme.layer could be repeated after content.layer. This example is a bit contrived, but it future-proofs the design to support the three scenarios, without too much complexity.

orasConfig Tokens

To support dynamic environments, environment variables would be supported.

version: "0.1-preview"
name: {{$registry}}/samples/docs-in-markdown:{{$tag}}
artifactType: application/vnd.contoso.docs-in-markdown.config.v1+json
layers:
  - readme:
    - type: application/vnd.contoso.docs-in-markdown.readme.layer.v1+md 
    - files: 
      - "./readme.md"
  - content:
    - type: application/vnd.contoso.docs-in-markdown.content.layer.v1+md 
    - files:
      - "./index.md"
      - "./docs/*.md"

Once the orasConfig.yaml file is defined, the user simply add/edits files in the collection. Or, iterates on the types. They can then simply type oras push from the directory of the orasConfig.yaml file.

Correlation to Artifact Mapping Schema

The above proposal is not meant to be a replacement of the Artifact Mapping Definition

There are two use cases for development and publishing

1. Making the artifact well known to registry operators. This is the purpose of the Artifact Mapping. Artifact Authors will define their types, with content related to the name of their type and reference information on all instances of the type.
2. Pushing specific artifacts to registries. For each artifact pushed, there's a collection of files, mapped to artifact and layer types. This is the purpose of this configuration.

What's in a Name?

The thing I'm most debating, and looking for suggestions, is how to name the 3 types of "config" files.

Assuming we agree to the following scenarios:

1. Defining the content of a specific artifact - the purpose of this proposal
2. Defining the artifact type, for consumption by registry operators. See Artifact Mapping Definition for more info.
3. Defining the schema of manifest.config.json, used by artifact tooling. See Config Schema Validation for more info.

Working reference of names:

fileName	Scenario
orasConfig.yaml	1 - the collection of files and types, used by the ORAS client/library when pushing to an OCI compliant registry. The focus of this proposal.
artifactMapping.json	2 - defining the type, used by registry operators.
artifactMappingSchema.json	2 - defining the schema used by artifactMapping.json
config.json	3 the optional config.json file used in the oci-manifest to define configuration values of the specific artifact pushed to a registry.
artifactConfigSchema.json	3 provides schema validation for a specific artifact type, when the artifact optionally uses a config file.

Conversation

• What do people think about simplifying the ORAS client experience?
• Does orasConfig make sense when ORAS is used as a library?
• What's the config filename (propose orasConfig.yaml) ?
• The schema of the config file?

[shizhMSFT]

What do people think about simplifying the ORAS client experience?

Initially, oras has simple usage as

oras push localhost:5000/hello:latest hi.txt

with everything by default. It was easy and simple until we shfit lots of configs from defaults to CLI. Therefore, simplifying the ORAS client experience indeed follows the orignal philosophy of this tool.

Does orasConfig make sense when ORAS is used as a library?

The answer is NO. When ORAS is used as a library, the developer can do as many options as they want. They can even wrap the ORAS library with their defaults to be a new library. The problem for the CLI is that the CLI interface is limited to be just one line.

What's the config filename (propose orasConfig.yaml) ?

I am guessing oras-defaults.yaml (.yml should also be accepted) will be better? The reason is simple that we store default values in it. The name oras-defaults.yaml will also not confuse users with other config files.
An alternative name is oras-artifact.yaml.

The schema of the config file?

Will it be better if we do

version: "0.1-preview"
name: ${registry}/samples/docs-in-markdown:${tag}
artifact:
  type: application/vnd.contoso.docs-in-markdown.config.v1+json
  layers:
    - name: readme # or description. for human only.
      type: application/vnd.contoso.docs-in-markdown.readme.layer.v1+md 
      files:
        - "./readme.md"
    - name: content
      type: application/vnd.contoso.docs-in-markdown.content.layer.v1+md 
      files: # should we support regex instead of simple wildcard?
        - "./index.md"
        - "./docs/*.md"

[SteveLasker]

Thanks @shizhMSFT,
One thing that has evolved is the realization that application/vnd.oci.image.* should be reserved for actual container images, not a generic artifact bucket.
With #115 fixed, we at least will default away from this overlap, but it does require the user to pass in their mediaTypez each time, thus the proposal on a config file. I view this similar to the simplicity that users get with docker-compose.yaml where they save their defaults, that rarely change.
The proposed changes to the config file look cleaner. I am confused on the ${ vs. {{}} syntax used in tasks.yaml scenarios. To avoid contextual confusion, I'd like to see these aligned as I thought the {{}} was the standard for go templating.

For the file name, I was hoping to see some other thoughts before commenting.

[shizhMSFT]

@SteveLasker Please re-open this issue if it is still valid and up to date.