image-layout: addition of companion data

[philips]

@stevvooe introduced a new term in #94 (comment) called "companion data" which @wking and I don't know. Instead of forking conversation over this new concept lets discuss the idea in a new issue.

[philips]

@stevvooe can you fill in details on what you mean by companion data?

[stevvooe]

The concept is described here: #94 (comment).

The directory layout from #94 will require suffixes at the top-level as a format makes additions for unverified, blob associated data:

./blobs/sha256-abdef
./blobs/sha256-abdef.references
./blobs/sha256-abdef.conversion
./blobs/sha256-abdef.mirrors

If we use ./blobs/sha256-abdef/data as the primary data storage location, these just become extra files:

./blobs/sha256-abdef/data
./blobs/sha256-abdef/references
./blobs/sha256-abdef/conversion
./blobs/sha256-abdef/mirrors

This may be interesting in regards to #59.

[philips]

@stevvooe why postfix vs prefix? For example:

./mirrors/sha256-abdef
./conversion/sha256-abdef

[stevvooe]

mv ./blobs/sha256-abdef ./somewhere vs mv {mirrors,conversion}/sha256-abdef ./somewhere

Depends on how we want to support merging.

[wking]

On Wed, Jun 08, 2016 at 10:07:29PM -0700, Stephen Day wrote:

The directory layout from #94 will require suffixes at the top-level
as a format makes additions for unverified, blob associated data:

./blobs/sha256-abdef
./blobs/sha256-abdef.references
./blobs/sha256-abdef.conversion
./blobs/sha256-abdef.mirrors

I agree that that may be useful information, but I'd rather keep
non-CAS information outside the CAS blobs/. How about a separate
notes/{alg}-{digest}/ for folks to dump this unverified,
blob-associated data? Then when folks get around to putting engines
behind this, they are less likely to forget which data is CAS and
which is not.

[philips]

Another advantage of keeping the CAS as a pure CAS and not co-mingling the hierarchy is that you could make the whole tree a RO mount. Where the expectation for this other data might be that it is writeable by a local system.

[vbatts]

Not sure where we've left off here.

[wking]

On Tue, Jul 05, 2016 at 10:50:45AM -0700, Vincent Batts wrote:

Not sure where we've left off here.

I think:

• There are use-cases for mutable data that's associated with CAS
  blobs 1.
• That data is not content-addressable, so it doesn't belong in CAS
  [2,3].
• There's no image-spec specified location for storing this
  information, although you could use refs.
• Not having an image-spec specified location is probably fine,
  because we also have no image-spec specified companion content.

We should probably table this until image-spec has something to say
about references, conversions, mirrors, or some other functionality
that would use companion data. Until then, external tooling can
handle this however it likes.

[vbatts]

it seems like this is a duplicate of #59

[wking]

On Thu, Oct 06, 2016 at 10:23:19AM -0700, Vincent Batts wrote:

it seems like this is a duplicate of #59

I think this is the generic issue and #59 is a particular use case.
But yeah, they're pretty close.

[stevvooe]

@vbatts Do the recent changes to manifest list cover this? Things like appstream definitely fit in here.

[vbatts]

@stevvooe i think so too. closing