Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tools: need a tool to update/create manifests #59

Open
cyphar opened this issue Oct 26, 2016 · 5 comments
Open

tools: need a tool to update/create manifests #59

cyphar opened this issue Oct 26, 2016 · 5 comments

Comments

@cyphar
Copy link
Member

cyphar commented Oct 26, 2016

I've been playing around with skopeo+image-tools+#5+#8 and I've found that almost all of the necessary tooling we need is available. You can pull base images, create layers, validate layers, unpack layers and then run them. That's awesome.

However, the one missing piece is updating image manifests. This is the same as creating manifests (which is also missing). What we need is the ability to take an existing image manifest (or from scratch) change the different properties. While you could do this with some jq foo, since it's a spec thing it should live in the spec repository.

If no-one else is working on this, I'd be glad to help out. I'm thinking of a UX like

% oci-manifest new --cwd /new_cwd --layer baselayer --layer layer2 --layer layer3 >the_manifest.json
% oci-manifest update --cwd /new_cwd --layer layer4 <the_manifest.json >new_manifest.json

Management of history could be quite a pain but we could just implement it as an append operation, with the option to change the base. So something like this would allow you to change the base layer with oci-manifest update:

% oci-manifest update --base-layer baselayer

I'm not too sure, we can hash that out once I have an implementation. It might be nice to also extend this tool to handle manifest-lists. But that's for a future date, maybe we could implement it as part of oci-refs where we add features to deref a manifest list. Not sure.

@runcom
Copy link
Member

runcom commented Oct 26, 2016

I'm not too sure, we can hash that out once I have an implementation. It might be nice to also extend this tool to handle manifest-lists. But that's for a future date, maybe we could implement it as part of oci-refs where we add features to deref a manifest list. Not sure.

see https://github.com/estesp/manifest-tool (which could maybe be extended to work with non-list manifests) (also moby/moby#27455)

@cyphar
Copy link
Member Author

cyphar commented Oct 26, 2016

Yeah, we could probably implement it as a config-based thing. It's probably not going to be that complicated to implement.

That reminds me, we need to have a way of handling descriptors for images because currently images are two-links-removed from the reference'd JSON.

@cyphar
Copy link
Member Author

cyphar commented Oct 26, 2016

We also need something to handle descriptors. Descriptors appear very simple but are not. There's probably a case to be made for oci-gc as well, since oci-manifest will generate more than one blob.

@wking
Copy link
Contributor

wking commented Oct 26, 2016

I agree that we'll want oci-gc eventually, but it seems orthogonal to this issue. And the only reason descriptors are complicated is remembering the associated media type (which you presumably knew when you pushed the blob into CAS). For folks building manifests in scripts, remembering those media types isn't complicated (just create descriptors at push time and insert them into the structure you're building (e.g. pushing into the manifest's layers). For folks building manifests on the command line, you can do something similar with jq:

$ MANIFEST='{}'
$ DIGEST=$(echo foo-bar | oci-cas put image)
$ SIZE=$(oci-cas get image ${DIGEST} | wc -c)
$ MANIFEST="$(echo "${MANIFEST}" | jq ".layers |= [{\"mediaType\": \"application/vnd.oci.image.layer.v1.tar+gzip\", \"size\": \"${SIZE}\", \"digest\": \"${DIGEST}\"}]")"
$ echo "${MANIFEST}"
{
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "size": "8",
      "digest": "sha256:5442d1bef41adc4dc3992b4d5736950092a96dea65b842813924033d5d930022"
    }
  ]
}

And if you had a tarball file available (vs. my echo foo-bar) you could get the file size with stat instead of fetching the data back out of CAS to pipe into wc. If we collect those three lines into a script, you could have:

$ MANIFEST="$(oci-manifest-add-layer.sh --media-type application/vnd.oci.image.layer.v1.tar+gzip image <(echo foo-bar))"

but I expect folks building images with image-tools commands will be comfortable without the wrapping shell, and that most folks will be using some higher-level tooling like Dockerfiles and such.

@cyphar
Copy link
Member Author

cyphar commented Oct 26, 2016

@wking Yeah, my above review is the things I reckon is necessary for the OCI integration we're planning into KIWI (and by extension oboci which I'll implement in spare cycles). The idea is that I want to integrate the minimum amount of information about the spec into KIWI, so that the maintainers of KIWI don't have to handle all of the interplay between components. So I will end up writing a wrapper, I would just prefer that the wrapper be smaller rather than larger and that the majority of the non-problem-specific components are put inside image-tools as general purpose utilities.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants