[WIP] fuse-overlayfs: add plugin system

[giuseppe]

support loading layers on demand.

Add a simple plugin mechanism that will help to expand fuse-overlayfs
functionalities, in particular it allows to load data from a layer on
demand.

A plugin is loaded into fuse-overlayfs using the option:

-o plugins=path/to/plugin.so:path/to/another/plugin.so

A layer can use a plugin with the syntax:

-o lowerdir=//plugin-name/DATA-FOR-THE-PLUGIN/path

Each time a file/directory is looked up, if a plugin is registered for
a layer, the plugin is first notified about the request.

After the callback is invoked, fuse-overlayfs still expects the data
to be accessible at the specified directory.

Signed-off-by: Giuseppe Scrivano gscrivan@redhat.com

[giuseppe]

to give some context, I am using it to play with it: https://github.com/giuseppe/crfs/tree/fuse-overlay-plugin.

I've prepared a stargz fedora image and pushed it to quay.io/giuseppe/fedora.

## using the fuse-overlay-plugin branch
$ (cd $GOPATH/src/github.com/giuseppe/crfs; make -C plugin/ && go build)
$ export PATH=$GOPATH/src/github.com/giuseppe/crfs:$PATH
$ mkdir lower upperdir work merged
$ fuse-overlayfs -oplugins=$GOPATH/src/github.com/giuseppe/crfs/plugin/crfs.so,lowerdir=//crfs/$(echo -n https://quay.io/v2/giuseppe/fedora/blobs/sha256:376969999a023ffbd0de97eddc3b87b2f868b3ae71a43cc7c4de39438383b608 | base64 -w0)/lower,upperdir=upper,workdir=work merged
$ time podman run --rm -ti --rootfs $(pwd)/merged /usr/bin/sh -c 'echo hello world'
hello world

real    0m3.684s
user    0m0.106s
sys     0m0.037s

[rhatdan]

Very cool, could we make this a feature of Podman? IE Push in Star.tgz format?

[giuseppe]

yes, it is not difficult to generate the tarballs in the star.tgz format.

What is difficult is to add the concept of read-on-the-fly layers in our tools. In any case, before that, we should investigate better if it has any significant improvement. Using the fedora image, I don't see any clear advantage so far.

[rhatdan]

Do we need something far larger, like one of the jboss container images

[bradfitz]

@giuseppe, I haven't done a number of HTTP optimizations in the crfs code yet. Its access patterns are still pretty poor, no readahead, no caching of requests already done, etc. I was waiting to get it working end-to-end before optimizing those.

[giuseppe]

@giuseppe, I haven't done a number of HTTP optimizations in the crfs code yet. Its access patterns are still pretty poor, no readahead, no caching of requests already done, etc. I was waiting to get it working end-to-end before optimizing those.

@bradfitz yes I agree there is still much margin for improvements and I am optimist this can be helpful for different use cases :-)

The plugin mechanism I've added in this PR was the simplest implementation I could think of to allow me to play directly with CRFS. It works fine, I've tested a few images and could not see any issue. The ugliest part is that a readdir(x) requires to pull also every child of x. We could delegate each FUSE request to the plugin/driver, but that requires a much more significant refactoring, and also introduces some other problems (inode numbers and passing FDs so that read/write operations can use splice(2).

From a quick profiling, most of the CPU time is spent in the gzip decompression. We are already using an alternative version in our libraries (klauspost/pgzip) that significantly improves the decompression time on the single stream, but in this case I've seen it performing worse than compress/gzip. I had no time yet to try it out, but I'd like to see how facebook/zstd performs with container images. Unfortunately this is a breaking change for existing clients, while stargz is not.

In the last version I am working on, I've added a readahead so that the file is fetched also while there are not active requests: https://github.com/giuseppe/crfs/blob/537cd10cae39662cdbf8dbcd688894bc53aa124c/crfs.go#L284-L301
(For our container tools it could also make sense in future to have a mode to store the image once the layers are fully downloaded).

The generic use case I was trying, yum upgrade on a Fedora image, is probably an extreme test case, as yum needs to load many tiny python files. Golang binaries will surely perform better.

@rhatdan I can try a jboss-like image and see how it performs

[giuseppe]

@bradfitz I've noticed a huge performance improvement if I use zstd instead of gzip. I've opened two PRs so that our containers tools understand the zstd compression:

containers/image#639
containers/storage#363

[AkihiroSuda]

@giuseppe Using zstd requires defining new OCI media type for distribution?

[giuseppe]

@giuseppe Using zstd requires defining new OCI media type for distribution?

from containers/image#639 (comment) it seems it is enough to add the +zstd suffix to the mediaType. It would be nice to have the support for zstd also in Docker.

From some quick tests, there are significant improvements also when not using something like stargz (starzstd? :) ): containers/image#639 (comment)

[AkihiroSuda]

any plan toward merging this? ^^

[lgtm-com]

This pull request introduces 3 alerts when merging b7777f5 into 4dc60f0 - view on LGTM.com

new alerts:

• 1 for Missing return statement
• 1 for Declaration hides parameter
• 1 for Too few arguments to formatting function

[giuseppe]

any plan toward merging this? ^^

I had no time to look at this recently. I've still some doubts about it as the current implementation for crfs using gzip is quite slow, that is why I have started looking into zstd (PR still blocked on review for containers/image) as a way to speed it up.

[rhatdan]

@giuseppe I just merged your zstd PR in containers/image

[giuseppe]

@giuseppe I just merged your zstd PR in containers/image

thanks, I've opened a PR for opencontainers/image-spec to add the zstd compression: opencontainers/image-spec#788

[giuseppe]

there are other use cases I'd like to play with in fuse-overlayfs, so I've worked on an alternative version for plugins that offer more flexibility: #119

[giuseppe]

I am closing this PR in favour of the new implementation.

Unfortunately I'll need to adapt the CRFS PoC I had worked with as it is broken now with the new API