Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add create-layer sub command to create filesystem changeset #8

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
/oci-create-runtime-bundle
/oci-unpack
/oci-image-validate
/oci-create-layer
3 changes: 2 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ EPOCH_TEST_COMMIT ?= v0.2.0
TOOLS := \
oci-create-runtime-bundle \
oci-image-validate \
oci-unpack
oci-unpack \
oci-create-layer

default: help

Expand Down
80 changes: 80 additions & 0 deletions cmd/oci-create-layer/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
// Copyright 2016 The Linux Foundation
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package main

import (
"log"
"os"

"github.com/opencontainers/image-tools/image"
"github.com/spf13/cobra"
)

type layerCmd struct {
stdout *log.Logger
stderr *log.Logger
dest string
}

func main() {
stdout := log.New(os.Stdout, "", 0)
stderr := log.New(os.Stderr, "", 0)

cmd := newLayerCmd(stdout, stderr)
if err := cmd.Execute(); err != nil {
stderr.Println(err)
os.Exit(1)
}
}

func newLayerCmd(stdout, stderr *log.Logger) *cobra.Command {
v := &layerCmd{
stdout: stdout,
stderr: stderr,
}

cmd := &cobra.Command{
Use: "oci-create-layer [child] [parent]",
Short: "Create an OCI layer",
Long: `Create an OCI layer based on the changeset between filesystems.`,
Run: v.Run,
}
cmd.Flags().StringVar(
&v.dest, "dest", "",
`The dest specify a particular filename where the layer write to`,
)
return cmd
}

func (v *layerCmd) Run(cmd *cobra.Command, args []string) {
if len(args) != 1 && len(args) != 2 {
v.stderr.Print("One or two filesystems are required")
if err := cmd.Usage(); err != nil {
v.stderr.Println(err)
}
os.Exit(1)
}
var err error
if len(args) == 1 {
err = image.CreateLayer(args[0], "", v.dest)
} else {
err = image.CreateLayer(args[0], args[1], v.dest)
}
if err != nil {
v.stderr.Printf("create layer failed: %v", err)
os.Exit(1)
}
os.Exit(0)
}
29 changes: 29 additions & 0 deletions cmd/oci-create-layer/oci-create-layer.1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
% OCI(1) OCI-CREATE-LAYER User Manuals
% OCI Community
% October 2016
# NAME
oci-create-layer \- Create filesystem changeset

# SYNOPSIS
**oci-create-layer** [child] [parent] [flags]

# DESCRIPTION
`oci-create-layer` creates a filesystem changeset from two layers. It compares child with parent and generates a filsystem diff, pack the diff into a uncompressed tar archive. The output tar archive name is the child name with .tar suffix by default, use `--dest` to specify a custom one.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we write to stdout by default? Then with #5 you could:

$ oci-create-layer child-dir parent-dir | gzip | oci-cas put image-layout-dir

to push a layer into CAS without landing the uncompressed layer tarball on the disk.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wking How about write to the stdout by default and use --dest to specify a custom destination?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that sounds good to me. Although I think --output is a more popular name for this sort of thing than --dest.


# OPTIONS
**--help**
Print usage statement

**--dest**
The dest specify a particular filename where the layer write to

# EXAMPLES
```
$ oci-create-layer rootfs-1-s rootfs-1
$ ls
rootfs-1 rootfs-1-s rootfs-1-s.tar

```

# HISTORY
Oct 2016, Originally compiled by Lei Jitang (coolljt0725 at huawei dot com)
8 changes: 5 additions & 3 deletions glide.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions glide.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,4 @@ import:
- package: github.com/pkg/errors
version: ~0.7.1
- package: github.com/spf13/cobra
- package: github.com/Sirupsen/logrus
58 changes: 58 additions & 0 deletions image/layer.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
// Copyright 2016 The Linux Foundation
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package image

import (
"fmt"
"io"
"os"
"path/filepath"

"github.com/opencontainers/image-tools/utils"
)

// CreateLayer cretes filesystem changset from child and parent
func CreateLayer(child, parent, dest string) error {
arch, err := Diff(child, parent)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would somewhat like to use @vbatts' go-mtree here. To be fair, I've been slacking on the required PRs to implement it, but would you be open to switching to go-mtree once it's up to snuff? The main benefit is that it has far more explicit design constraints about reproducibility and manifest verifiability than pkg/archive.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cyphar The main purpose to use pkg/archive is it's well tested and have been used for a long time on docker. I'm personally ok to try to switch to go-mtree once it's up to snuff, but others may have different thoughts.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another benefit of go-mtree that I forgot to mention is that you could make it possible to create layers without needing to keep two copies of the tree -- since mtree generates a manifest that you can compare against. So you could do something like:

% go-mtree -c -K sha256sum -p rootfs/ > before.mtree
% # modify the rootfs
% oci-image-tool --base before.mtree rootfs > layer.tar

if err != nil {
return err
}
defer arch.Close()
filename := fmt.Sprintf("%s.tar", filepath.Clean(child))
if dest != "" {
filename = filepath.Clean(dest)
}
f, err := os.Create(filename)
if err != nil {
return err
}
defer f.Close()
_, err = io.Copy(f, arch)
return err
}

// Diff produces an archive of the changes between the specified
// layer and its parent layer which may be "".
func Diff(child, parent string) (rc io.ReadCloser, err error) {
changes, err := utils.ChangesDirs(child, parent)
if err != nil {
return nil, err
}
archive, err := utils.ExportChanges(child, changes)
if err != nil {
return nil, err
}
return archive, nil
}
Loading