Exporter: support nydus image export #2045
Conversation
Nydus image is a container accelerated image format provided by the Dragonfly image-service project, which offers the ability to pull image data on demand, without waiting for the entire image pull to complete and then start the container. It has been put in production usage and shown vast improvements over the old OCI image format in terms of container launching speed, image space, and network bandwidth efficiency, as well as data integrity. The Nydus image can also serve as an example and reference implementation for the on-going OCI image spec v2 discussion. Nydus has provided a conversion tool Nydusify for converting OCI image to Nydus image, which assumes that the OCI image is already available in the registry, but a better way would be to build the Nydus images directly from the build system instead of using the conversion tool, which would increase the speed of the image export, so we experimentally integrated the Nydus exporter in Buildkit. Nydusify provides a package to do the actual image conversion. We use the same package to implement the buildkit nydus exporter. The exporter mounts all layers and calls the package to build to a Nydus image and push it to the remote registry. The current implementation relies on the Nydus builder, which is used to build a layer to a RAFS layer, and in the future, we can explore simpler, less dependent integrations. Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
Pack the nydus-image binary into integration image, export Nydus image with Nydus exporter, and simply check. Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
| ExporterTar = "tar" | ||
| ExporterOCI = "oci" | ||
| ExporterDocker = "docker" | ||
| ExporterNydusImage = "nydus" |
There was a problem hiding this comment.
Do we really need a new exporter for nydus? Can we add "nydus" as a "compression format" for existing exporters? Please see #2057.
There was a problem hiding this comment.
Hi, here is an example of Nydus image. Nydus image is not currently a standard OCI image format, it is composed of a bootstrap layer and some blob layers, so we can't just treat the format of these layers as another compression type.
There was a problem hiding this comment.
We can rename "compression type" to another word as you like, e.g., "blob type" , but having another exporter is fine to me as well
@ktock WDYT?
There was a problem hiding this comment.
Based on #2057, the manifest index converter is configurable so nydus image should be able to generated without introducing a new exporter.
There was a problem hiding this comment.
Thanks, that's a good point if using a configurable converter, we will take a look.
|
There's a concern of future maintenance cost of this code should it be merged. Would you be interested on taking a look at the problem of providing out-of-tree exporters(or blob converters per #2045 (review)) support instead that would run in containers like buildkit frontends do and what capabilities they would need. |
@tonistiigi Thanks, will take a look at blob converter first, this PR can be closed for now. |
Related issue: #2046
Nydus image is a container accelerated image format provided by the Dragonfly image-service project, which offers the ability to pull image data on demand, without waiting for the entire image pull to complete and then start the container. It has been put in production usage and shown vast improvements over the old OCI image format in terms of container launching speed, image space, and network bandwidth efficiency, as well as data integrity. The Nydus image can also serve as an example and reference implementation for the on-going OCI image spec v2 discussion.
Nydus has provided a conversion tool Nydusify for converting OCI image to Nydus image, which assumes that the OCI image is already available in the registry, but a better way would be to build the Nydus images directly from the build system instead of using the conversion tool, which would increase the speed of the image export, so we experimentally integrated the Nydus exporter in Buildkit.
Nydusify provides a package to do the actual image conversion. We use the same package to implement the buildkit nydus exporter. The exporter mounts all layers and calls the package to build to a Nydus image and push it to the remote registry. The current implementation relies on the Nydus builder, which is used to build a layer to a RAFS layer, and in the future, we can explore simpler, less dependent integrations.