podman load/save: support multi-image docker archive

[vrothberg]

Fixes: #2669
Signed-off-by: Valentin Rothberg rothberg@redhat.com

[vrothberg]

Need to get containers/image#975 in before.

[mtrmac]

Just a few notes while primarily reading the called c/image implementation.

[mtrmac]

The changes to libpod/image/pull.go were dropped — is that intentional?

In a sense they really don’t belong into this PR, OTOH fixing the existing fairly broken podman pull docker-archive:… code would be nice.

[vrothberg]

The changes to libpod/image/pull.go were dropped — is that intentional?

In a sense they really don’t belong into this PR, OTOH fixing the existing fairly broken podman pull docker-archive:… code would be nice.

That was intentional as I want us to focus on the multi-image part first. Once that's in, we can tackle the next issue - unless you feel strongly to tackle both at once.

[mtrmac]

I’m fine with having the PR narrowly scoped; what is/isn’t necessary to keep the Podman repo maintainable is up to the judgment of Podman maintainers.

[mtrmac]

More skeletons in the closet…

[vrothberg]

Back to c/storage v1.23.0

[vrothberg]

@edsantiago are https://github.com/containers/podman/pull/6811/checks?check_run_id=1007722626 flakes?

@mtrmac, I am running out of time. If you find some cycles, feel free to take over this PR and push it over the finish line.

[edsantiago]

@vrothberg those show every sign of being #7195 but I've never seen so many in one run.

[vrothberg]

Restarted the last flaky job. The flakes are hitting this PR hard.

[rhatdan]

Your getting hit by the flake I can not get by.

[vrothberg]

It's green, it's green!

@rhatdan, let's merge before the CI gods stop shining upon me head.

[mtrmac]

ACK (full review now), just some maintainability suggestions.

[mtrmac]

Also, this vendors an unreleased version of c/image. Is that OK?

[mheon]

OK for now, but we'll need to get a c/image release out in the next several weeks for Podman 2.1.0 so we don't cut a full release with a prerelease c/image

[vrothberg]

Final polish based on @mtrmac's comments pushed.

[rhatdan]

Isn't this a WIP until this is released? I think we should release containers/image so that @mheon can release pdoman 2.1.0

[vrothberg]

@mheon acked above (see #6811 (comment)). We just need to release before v2.1.0 which is still a bit in the future.

[mheon]

@rhatdan We're still waiting on a few more PRs in Podman land (I don't want to cut a 2.1.0 without #7452 because that's blocking some folks from moving off 1.9) so we have a bit of time.

[mtrmac]

LGTM.

[rhatdan]

/lgtm
But this Podman can not be released until containers/image is updated.

[maybe-sybr]

Is there any follow up work planned to bring the other formats into line with this? In particular I'm interested in oci-dir.

[vrothberg]

Is there any follow up work planned to bring the other formats into line with this? In particular I'm interested in oci-dir.

The oci transports already supports multiple image for storing images. However, there's no easy way to use it as a source without knowing the exact images in the directory: Error initializing source oci:/home/vrothberg/foo:: more than one image in oci, choose an image

@nalind @mtrmac I haven't checked the code but assume that the oci transport may need some wiring w.r.t. to image indexes to make it work?

[mtrmac]

A similar list/get-reference API seems natural; the thing I’m not currently sure about is multi-arch support; that somehow uses OCI indices, and I’m not quite sure how. IIRC it’s not a nested-index design, so the top-level index might be a multi-arch image set, not a multi-image archive. @nalind probably knows more.

[nalind]

When we merged manifest list support, the oci transports stopped assuming that a manifest being added to them wasn't a list, so entries they contain can now themselves be either lists, or images that would be included in a list, and tag ("org.opencontainers.image.ref.name") annotations stored in the top-level index can be attached to either. They no longer attempt to auto-generate Platform information for their contents, either.

Copying using skopeo copy --all from a source that is a manifest list to an oci location produces this:

- index.json (1 entry which may have an "org.opencontainers.image.ref.name" annotation)
- blobs/
  - sha256/
    - manifest list as a blob (only entry in index.json)
    - manifests referred to by the manifest list, as blobs
    - blobs referred to by those manifests

Copying from another location into the same oci location adds another entry to the top-level index.json, and either the single manifest or the manifest list and some number of the manifests it refers to, along with other blobs, to the blobs/sha256 subdirectory, so you get either:

- index.json (2 entries which may have "org.opencontainers.image.ref.name" annotations)
- blobs/
  - sha256/
    - two manifest lists as blobs (first and second entries in index.json)
    - manifests referred to by either manifest list, as blobs
    - blobs referred to by those manifests

or

- index.json (2 entries which may have "org.opencontainers.image.ref.name" annotations)
- blobs/
  - sha256/
    - a manifest list as a blob (first entry in index.json)
    - manifests referred to by the manifest list, as blobs
    - a manifest as a blob (second entry in index.json)
    - blobs referred to by those manifests

When using an oci location as a source, if there's exactly one item (whether it's a list or not is irrelevant at this point) listed in the oci location's index.json, not specifying a tag when referring to the layout as a source will return that one item as a starting point, and if that's a manifest list, the library's copying logic either copies everything in the list or some subset, just as it would if it were reading from a registry and starting with the same item.

If there's more than one item in the oci location, then they really need to have "org.opencontainers.image.ref.name" annotations on them, because otherwise we don't know how to specify which of them to start copying from, list or not, since references for those transports currently don't have a form that incorporates digests, and we don't have a way to list their contents.