wnfs usage in Capyloon

[fabricedesre]

In Capyloon we use a virtual file system to store user data in a broad sense. It can be usual objects like pictures, downloaded documents etc. but also contacts, navigation history entries, media playback sessions. These are organized as a hierarchy, but leafs are not plain files: they are resources that can be represented by different variants: for instance, an image will have a default variant with the full size version, but also a thumbnail variant and a exif-metadata variant if applicable. Media playback sessions store the page url, title and track info in the default variant, but also a poster image for the video as a variant, etc.

Resources also have custom metadata: we maintain a "visit score" state that helps with surfacing the most relevant results in search queries (similar to what Firefox uses in the awesomebar).

We want to persist these resources in wnfs, so we need some solution to map resources with variants to the more simple model of wnfs. I can think of a couple of options:

1. Turn each resource into a directory, named eg. $RESOURCE_NAME.capyloon and store variants in that directory.

• Pros: simple and direct mapping with the Capyloon vfs data model.
• Cons: quite ugly since it's adding a whole additional depth layer. Could be mitigated by teaching wnfs browsers about the special directory structure.

2. Store the default variant as a regular file, and only the non-default ones in a sub-direcetory.

• Pros: provides direct access to the default variant in wnfs tools.
• Cons: Easy to get out of sync with the vfs model: what if the default variant is deleted but not the sub-directory?

3. Use mime multipart to store all variants in a single file.

• Pros: Keep the overall hierarchy shape clean, and easily syncable.
• Cons: Implementation overhead, and potential storage overhead too (updating a single variant would cause the whole multipart encrypted blob to change). Need special support in wnfs browsing tools.

4. Teach wnfs that "files" could be more than just a unixfs file.

• Pros: suits Capyloon well!
• Cons: adds overall complexity?

Does that make any sense?

[bmann]

This might be a good live discussion to have at the next meeting. 1 and 4 is my blink reaction.

I have been thinking about supporting various other things that are first class supported like image variant sizes. So ... evolving WNFS to support arbitrary metadata might be a thing.

I had previously strayed away from the whole .... but that leads to you the whole suite of "Operating System" primitives like contacts, calendars, and so on.... but of course, you are maintaining an operating system and perhaps that's as well how we might think about WNFS.

[fabricedesre]

This might be a good live discussion to have at the next meeting. 1 and 4 is my blink reaction.

Sure, when & where are meetings happening?

[bmann]

@fabricedesre hilariously I actually got my wires crossed and was thinking UCAN ;)

We’ll follow the pattern there and do a FIRST community meeting, post agenda in the Discussions here, and have you on the first one ;)

Stay tuned!

[matheus23]

@fabricedesre Thanks for raising this!

I answered some stuff on the call, but it's probably good to have this persisted on here as well:

Technically there's nothing preventing us from teaching WNFS that files can be something else. Files can link to a CID which may be a UnixFS-encoded file or it may just be something entirely different. Other examples of what CIDs can link to are git commits or hash-addressed blocks in a blockchain. This mechanism is also extensible in IPLD, which is why we can create something like WNFS in the first place, so it's totally possible to do number (4), the only remaining question is what interoperability story do we want to have? E.g. it's not possible to load any kind of CID as an image on gateways, for that you'd need a unixfs-encoded file.

Another option would be to store the CID of thumbnails or other related data as UnixFS-encoded files in the metadata of an "original" image.
However, we'd need some support API-wise in e.g. rs-wnfs to do that.

Which brings me to my last point: As I mentioned on the community call, I'd love to know more about the virtual file system's API you're currently using in capyloon @fabricedesre! If you have a link and maybe even some extra words about it, like what APIs you're using quite a bit and which aren't useful, as well as what you'd like an API to ideally look like. I'm personally not the biggest fan of file systems as an abstraction for app data, too, but it's the "simplest" and most easily explainable thing to people we have right now and I've been thinking a lot about what a more "database" oriented WNFS would look like.

[fabricedesre]

Hi @matheus23 !

A good entry point to the API as it is exposed to web pages is https://github.com/capyloon/api-daemon/blob/main/services/contentmanager/src/contentmanager.sidl#L74 - it's written in a custom IDL but fairly self explanatory. This repo wraps the underlying implementation that you can find in https://github.com/capyloon/costaeres . This api evolves as we have needs coming from the UI side; I think we'll need some better ways to slice data but since the querying is handled by a local sqlite db that should not impact too much the WNFS side.

The persistence is plugable, implementations need to implement this trait. We build a sqlite db for queries & full text indexing that can be re-hydrated once we get the root node out of the persistence provider.

Currently there is a FileStore implementation that simply saves to the local filesystem. From a high level point of view, we would need to "just" implement a WnfsStore, and decide how to represent resources and their metadata. It would be very nice to at least expose the default variant in a way that "just works" with a standard WNFS browser.

[fabricedesre]

Something else I didn't talk about and that is not implemented: there is currently a single root to the resource tree, but that is in tension with the same origin policy that is key property of the web security model: in general siteA.com and siteB.com data are fully isolated from each other. However there are use cases that require cross-origin data linking.

I envision scenarii such as:

• I use https://geovelo.fr to record a bike ride gps points.
• At the same time, the system was recording which songs were played by spotify.
  Back home I want to know which songs was playing when I was at some waypoint. None of the apps can do it by themselves (which is fine), but I can create an app that combines both data sets: this new app could be granted just enough access to fullfill its needs using some UCANs (expressing "today's location from geovelo and today's medias from the system").

We could either have one root per origin or a single root and the first sub-level being per-origin folders - the difference will mostly be how efficient and easy to implement cross-origin queries will end up I guess.

[RangerMauve]

@fabricedesre

Are you thinking of using a single wnfs root in capyloon or managing a bunch of them? I was thinking it'd be cool to integrate it with IPNS based roots. e.g. when you create an IPNS key, specify the structure you wish to use, then when you use PUT ipns://MyWnfsPubKeyHere/example.txt, it could fetch your encryption keys and all that and update the IPNS record after the wnds dir updates
Or are you thinking of having new DOM APIs for this stuff?
Or maybe even just the FileSystem DOM API but having a path to WNFS FUSE mounts or something?

[fabricedesre]

@RangerMauve the one thing I'm sure about is to not go the FUSE or DOM FileSystem route. Way too many footguns here.

You ipns://pubkey approach looks neat. From a user point of view I'm not yet sure if it's better to have a single root and let people choose what is public vs. private vs. shared, or if having explicit roots for each is better. I tend to think multiple roots could be safer to avoid user mistakes. I know personally I would likely organize my data with /private and /public top level containers.

Where can I read more about the ipns key protocol you describe?

[RangerMauve]

The IPNS thing is what we're doing in Agregore. https://github.com/AgregoreWeb/agregore-ipfs-daemon/blob/main/spec.md#ipns-key-api

Currently when you create an IPNS key it's just assuming it's going to publish UnixFS, but I think it'd be cool to have a way to publish WNFS or IPLD too.