[docs-only] Dev docs about collaborative storage #9317

dragotin · 2024-06-04T14:23:49Z

This documentation should explain technical background, design options and opportunities and show how to set up at this stage.

Later, the setup section might be moved to other area of doc.

@aduffeck @mmattel @hodyroff

update-docs · 2024-06-04T14:23:54Z

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

docs/architecture/posixfs.md

phil-davis · 2024-06-05T09:07:59Z

docs/architecture/posixfs.md

+
+Posix FS is a backend component that manages files on the server utilizing a "real" file tree that represents the data with folders and files in the file system as users are used to it. That is the big difference compared to Decomposed FS which is the default storage driver in Infinite Scale.
+
+This does not mean that Infinte Scale is trading any of it's benefits to this new feature: It still implements simplicity by running without a database, it continues to store metadata in the file system and adds them transparently to chaches and search index, and it also features the full spaces concept as before, just to name a few example.


Suggested change

This does not mean that Infinte Scale is trading any of it's benefits to this new feature: It still implements simplicity by running without a database, it continues to store metadata in the file system and adds them transparently to chaches and search index, and it also features the full spaces concept as before, just to name a few example.

This does not mean that Infinite Scale is trading any of it's benefits to this new feature: It still implements simplicity by running without a database, it continues to store metadata in the file system and adds them transparently to caches and search indexes, and it also features the full spaces concept as before, just to name a few examples.

phil-davis · 2024-06-05T09:08:27Z

docs/architecture/posixfs.md

+
+This does not mean that Infinte Scale is trading any of it's benefits to this new feature: It still implements simplicity by running without a database, it continues to store metadata in the file system and adds them transparently to chaches and search index, and it also features the full spaces concept as before, just to name a few example.
+
+The architecture of Infinite Scale allows to configure different storage drivers for specific storage types and purposes on a space granularity. The Posix FS storage driver is an alternative to the default driver called Decomposed FS.


Suggested change

The architecture of Infinite Scale allows to configure different storage drivers for specific storage types and purposes on a space granularity. The Posix FS storage driver is an alternative to the default driver called Decomposed FS.

The architecture of Infinite Scale allows configuring different storage drivers for specific storage types and purposes on a space granularity. The Posix FS storage driver is an alternative to the default driver called Decomposed FS.

phil-davis · 2024-06-05T09:09:31Z

docs/architecture/posixfs.md

+
+The architecture of Infinite Scale allows to configure different storage drivers for specific storage types and purposes on a space granularity. The Posix FS storage driver is an alternative to the default driver called Decomposed FS.
+
+However, the clearance of the file structure in the underlying file system is not the only benefit of the Posix FS. This new technology allows users to manipulate the data directly in the file system, and any changes to files made aside of Infinite Scale are monitored and directly reflected in Infinite Scale.


Suggested change

However, the clearance of the file structure in the underlying file system is not the only benefit of the Posix FS. This new technology allows users to manipulate the data directly in the file system, and any changes to files made aside of Infinite Scale are monitored and directly reflected in Infinite Scale.

However, the clarity of the file structure in the underlying file system is not the only benefit of the Posix FS. This new technology allows users to manipulate the data directly in the file system, and any changes to files made aside of Infinite Scale are monitored and directly reflected in Infinite Scale.

Maybe you mean something like "clarity"?

phil-davis · 2024-06-05T09:09:56Z

docs/architecture/posixfs.md

+
+However, the clearance of the file structure in the underlying file system is not the only benefit of the Posix FS. This new technology allows users to manipulate the data directly in the file system, and any changes to files made aside of Infinite Scale are monitored and directly reflected in Infinite Scale.
+
+The first time ever with feature rich open source file synce & share systems, users can either choose to work with their data through the clients of the system, it's API's or even directly in the underlying file system on the server.


Suggested change

The first time ever with feature rich open source file synce & share systems, users can either choose to work with their data through the clients of the system, it's API's or even directly in the underlying file system on the server.

For the first time ever with feature rich open source file sync & share systems, users can either choose to work with their data through the clients of the system, it's API's or even directly in the underlying file system on the server.

phil-davis · 2024-06-05T09:18:10Z

docs/architecture/posixfs.md

+
+### Monitoring
+
+To get information about changes such as new files added, files edited or removed, Infinte Sale uses a monitoring system to directly watch the file system. This starts with the Linux inotify system and ranges to much more sophisticated services as for example in Spectrum Scale (see [GPFS Specifics](#gpfs-specifics) for more details on GPFS file systems).


Suggested change

To get information about changes such as new files added, files edited or removed, Infinte Sale uses a monitoring system to directly watch the file system. This starts with the Linux inotify system and ranges to much more sophisticated services as for example in Spectrum Scale (see [GPFS Specifics](#gpfs-specifics) for more details on GPFS file systems).

To get information about changes such as new files added, files edited or removed, Infinite Scale uses a monitoring system to directly watch the file system. This starts with the Linux inotify system and ranges to much more sophisticated services as, for example, in Spectrum Scale (see [GPFS Specifics](#gpfs-specifics) for more details on GPFS file systems).

There's another typo here Infinite Sale

phil-davis · 2024-06-05T09:18:57Z

docs/architecture/posixfs.md

+
+Based on the information transmitted by the watching service, Infinite Scale is able to "register" new or changed files into its own caches and internal management structures. This enables Infinite Scale to deliver resource changes through the "traditional" channels such as APIs and clients.
+
+Since the most important metadata is the file tree structure itself, the "split brain" situation between data and metadata is impossible to cause trouble.


Suggested change

Since the most important metadata is the file tree structure itself, the "split brain" situation between data and metadata is impossible to cause trouble.

Since the most important metadata is the file tree structure itself, it is impossible for the "split brain" situation between data and metadata to cause trouble.

phil-davis · 2024-06-05T09:20:29Z

docs/architecture/posixfs.md

+
+### File Id Resolution
+
+Infinite Scale uses an Id based approach to work with resources, rather than a file path based mechanism. The reason for that is that Id based lookups can be done way more efficient compared to tree traversals, just to name one reason.


Suggested change

Infinite Scale uses an Id based approach to work with resources, rather than a file path based mechanism. The reason for that is that Id based lookups can be done way more efficient compared to tree traversals, just to name one reason.

Infinite Scale uses an Id based approach to work with resources, rather than a file path based mechanism. The reason for that is that Id based lookups can be done way more efficiently compared to tree traversals, just to name one reason.

phil-davis · 2024-06-05T09:22:56Z

docs/architecture/posixfs.md

+
+The tech preview comes with the following limitations:
+
+1. User Management: Manipulations in the file system have to be done by the same user that runs Infinte Scale


Suggested change

1. User Management: Manipulations in the file system have to be done by the same user that runs Infinte Scale

1. User Management: Manipulations in the file system have to be done by the same user that runs Infinite Scale

phil-davis · 2024-06-05T09:23:44Z

docs/architecture/posixfs.md

+1. There must be storage available to store meta data and blobs, available under a root path
+2. When using inotify, the storage must be local on the same machine. Network mounts do not work with Inotify
+3. The storage root path must be writeable and executable by the same user Infinite Scale is running under
+4. An appropiate version of Infinte Scale is installed, version number 5.0.5 and later


Suggested change

4. An appropiate version of Infinte Scale is installed, version number 5.0.5 and later

4. An appropiate version of Infinite Scale is installed, version number 5.0.5 and later

kobergj

Looks good 👍 Just some nagging as always...

kobergj · 2024-06-05T09:07:52Z

docs/architecture/posixfs.md

+
+Posix FS is a backend component that manages files on the server utilizing a "real" file tree that represents the data with folders and files in the file system as users are used to it. That is the big difference compared to Decomposed FS which is the default storage driver in Infinite Scale.
+
+This does not mean that Infinte Scale is trading any of it's benefits to this new feature: It still implements simplicity by running without a database, it continues to store metadata in the file system and adds them transparently to chaches and search index, and it also features the full spaces concept as before, just to name a few example.


Suggested change

This does not mean that Infinte Scale is trading any of it's benefits to this new feature: It still implements simplicity by running without a database, it continues to store metadata in the file system and adds them transparently to chaches and search index, and it also features the full spaces concept as before, just to name a few example.

This does not mean that Infinte Scale is trading any of its benefits to this new feature: It still implements simplicity by running without a database, it continues to store metadata in the file system and adds them transparently to chaches and search index, and it also features the full spaces concept as before, just to name a few example.

kobergj · 2024-06-05T09:08:54Z

docs/architecture/posixfs.md

+
+However, the clearance of the file structure in the underlying file system is not the only benefit of the Posix FS. This new technology allows users to manipulate the data directly in the file system, and any changes to files made aside of Infinite Scale are monitored and directly reflected in Infinite Scale.
+
+The first time ever with feature rich open source file synce & share systems, users can either choose to work with their data through the clients of the system, it's API's or even directly in the underlying file system on the server.


Suggested change

The first time ever with feature rich open source file synce & share systems, users can either choose to work with their data through the clients of the system, it's API's or even directly in the underlying file system on the server.

The first time ever with feature rich open source file synce & share systems, users can either choose to work with their data through the clients of the system, its APIs or even directly in the underlying file system on the server.

kobergj · 2024-06-05T09:11:00Z

docs/architecture/posixfs.md

+
+The first time ever with feature rich open source file synce & share systems, users can either choose to work with their data through the clients of the system, it's API's or even directly in the underlying file system on the server.
+
+That is another powerful vector for integration and enables a new spectrum of use cases across all domains. Just imagine how many software can write files, and can now directly make them accessible real time in a convenient, secure and efficient way.


Just imagine how many software can write files, and can now directly make them accessible real time in a convenient, secure and efficient way.

This I don't understand. Are you talking about other competitors who can't do this?

kobergj · 2024-06-05T09:41:33Z

docs/architecture/posixfs.md

+
+## Advanced Features
+
+Depending on the capabilities of the underlying file system, the Infinite Scale PosixFS can benefit from more advanced funcitonality described here.


Suggested change

Depending on the capabilities of the underlying file system, the Infinite Scale PosixFS can benefit from more advanced funcitonality described here.

Depending on the capabilities of the underlying file system, the Infinite Scale PosixFS can benefit from more advanced functionality described here.

I knew I could find a typo if I only looked hard enough!

kobergj · 2024-06-05T09:45:08Z

docs/architecture/posixfs.md

+2. When using inotify, the storage must be local on the same machine. Network mounts do not work with Inotify
+3. The storage root path must be writeable and executable by the same user Infinite Scale is running under
+4. An appropiate version of Infinte Scale is installed, version number 5.0.5 and later
+5. Either redis or nats-js-kv cache service


Please don't mention redis. It is not working properly. We want to use nats-js-kv as only suggested distributed cache solution.

kobergj · 2024-06-05T09:45:45Z

docs/helpers/env_vars.yaml

Why did you commit this?

kobergj · 2024-06-05T09:46:55Z

docs/services/app-registry/_index.md

Same question: Was this commited intentionally?

mmattel · 2024-06-05T10:01:19Z

docs/architecture/posixfs.md

+
+{{< toc >}}
+
+Posix FS is the working name for the collaborative storage driver for Infinite Scale.


As a personal opinion, I am not happy with the name Posix FS.
This is imho misleading as we also have POSIX for metadata and this will cause confusion.
I know that we have a new service recently added named collaboration but when thinking about, we could use Collaborative FS as alternative for Posix FS to make things more clear and distinctive.

Note that the envvar name should then also be corrected (hopefully I catched them all):
STORAGE_USERS_POSIX_ROOT --> STORAGE_USERS_COLLABORATIVE_FS_ROOT
STORAGE_USERS_DRIVER --> collaborative as value
STORAGE_USERS_POSIX_WATCH_TYPE --> STORAGE_USERS_COLLABORATIVE_FS_WATCH_TYPE
STORAGE_USERS_POSIX_WATCH_PATH --> STORAGE_USERS_COLLABORATIVE_FS_WATCH_PATH

POSIX is IMHO the correct technical term. It makes the difference between Decomposed vs. POSIX

mmattel · 2024-06-05T10:34:17Z

docs/architecture/posixfs.md

+
+## Introduction
+
+Posix FS is a backend component that manages files on the server utilizing a "real" file tree that represents the data with folders and files in the file system as users are used to it. That is the big difference compared to Decomposed FS which is the default storage driver in Infinite Scale.


Looking into the dev docs and searching for decomposed, I only find Decomposed FS on NFS. We should add a new documentation about Decomposed FS, what it is about and link that document here for reference so one can compare. This is especially true as Decomposed FS uses POSIX (eg NFS) for metadata stored via messagepack and this new FS driver uses XATTRs as part of the files stored.

To clarify: Decomposed FS is "decomposed".

A POSIX compatible filesystem has some requirements. https://www.quobyte.com/storage-explained/posix-filesystem/

POSIX is not NFS.

mmattel · 2024-06-05T11:01:17Z

docs/architecture/posixfs.md

+
+The architecture of Infinite Scale allows to configure different storage drivers for specific storage types and purposes on a space granularity. The Posix FS storage driver is an alternative to the default driver called Decomposed FS.
+
+However, the clearance of the file structure in the underlying file system is not the only benefit of the Posix FS. This new technology allows users to manipulate the data directly in the file system, and any changes to files made aside of Infinite Scale are monitored and directly reflected in Infinite Scale.


Maybe to add an example usecase to make things clear:
This new technology allows users to manipulate the data directly in the file system like when using a scanner storing the output on a filesystem.
I know that there are many other usecases, but such an example makes the idea clear right at the beginning without counteracting the Decomposed FS.

added a sentence with a real life example (scanner)

mmattel · 2024-06-05T11:21:40Z

docs/architecture/posixfs.md

+
+The Posix FS technology uses a few features of the underlying file system, which are mandatory and directly contributing to the performance of the system.
+
+While the simplest form of Posix FS runs with default file systems of every modern Linux system, the full power of this unfolds with more capable file systems such as IBM Storage Scale or Ceph. These are recommended as reliable foundations for big installations of Infinite Scale.


Do "default filesystems" include mounted shared file systems like NFS or SMB ?
If yes, this should be added.
I remember, that NFS needs a minimum version (v4.1 or 4.2?) to support xattrs which was some time ago one important reason to move to messagepack. If a minimum version of a mounted FS is still valid, we should add a note about that. Else one thinks NFSv3 is ok to use. No need to go into a detail, but a note about a min versions would be necessary - also for us...

Yes, thanks. It only works with local mounted fs - made that more clear in the text.

mmattel · 2024-06-05T11:23:23Z

docs/architecture/posixfs.md

+
+### Automatic ETag Propagation
+
+The ETag of a resource can be understood as a content fingerprint of any file- or folder resource in Infinite Scale. It is mainly used by clients to detect changes of resources. The rule is that if the content of a file changed the ETag has to change as well, as well as the ETag of all parent folders up to the root of the space.


Suggested change

The ETag of a resource can be understood as a content fingerprint of any file- or folder resource in Infinite Scale. It is mainly used by clients to detect changes of resources. The rule is that if the content of a file changed the ETag has to change as well, as well as the ETag of all parent folders up to the root of the space.

The ETag of a resource can be understood as a content fingerprint of any file- or folder resource in Infinite Scale. It is mainly used by clients to detect changes of resources. The rule is, that if the content of a file changed, the ETag has to change as well, as well as the ETag of all parent folders up to the root of the space.

mmattel · 2024-06-05T11:23:52Z

docs/architecture/posixfs.md

+
+Infinite Scale uses a built in mechanism to maintain the ETag for each resource in the file meta data, and also propagates it automatically.
+
+In the future a sophisticated underlying file system could provide an attribute that fulfills this requirement and changes whenever content or metadata of a resource changes, and - which is most important - also changes the attribute of the parent resource and the parent of the parent etc.


Suggested change

In the future a sophisticated underlying file system could provide an attribute that fulfills this requirement and changes whenever content or metadata of a resource changes, and - which is most important - also changes the attribute of the parent resource and the parent of the parent etc.

In the future, a sophisticated underlying file system could provide an attribute that fulfills this requirement and changes whenever content or metadata of a resource changes, and - which is most important - also changes the attribute of the parent resource and the parent of the parent etc.

mmattel · 2024-06-05T11:24:20Z

docs/architecture/posixfs.md

+
+Similar to the ETag propagation described before, Infinite Scale also tracks the accumulated tree size in all nodes of the file tree. A change to any file requires a re-calculation of the size attribute in all parent folders.
+
+In the future Infinite Scale could benefit from file systems with native tree size propagation.


Suggested change

In the future Infinite Scale could benefit from file systems with native tree size propagation.

In the future, Infinite Scale could benefit from file systems with native tree size propagation.

mmattel · 2024-06-05T11:25:45Z

docs/architecture/posixfs.md

+
+Other systems store quota data in the metadata storage and implement propagation of used quota similar to the tree size propagation.
+
+### File Id Resolution


Not sure, but should we write File ID Resolution (ID in capital letters, consequently in all occurrences).

mmattel · 2024-06-05T11:27:27Z

docs/architecture/posixfs.md

+
+### User Management
+
+With the requirement that data can be manipulated either through the filesystem or the Infinite Scale system, the question under which uid the manipulation happens is an important question.


Suggested change

With the requirement that data can be manipulated either through the filesystem or the Infinite Scale system, the question under which uid the manipulation happens is an important question.

With the requirement that data can be manipulated either through the filesystem or the Infinite Scale system, the question under which uid the manipulation happens is important.

mmattel · 2024-06-05T11:28:05Z

docs/architecture/posixfs.md

+With the requirement that data can be manipulated either through the filesystem or the Infinite Scale system, the question under which uid the manipulation happens is an important question.
+
+There are a few possible ways for user management:
+1. Changes can either be only accepted by the same  user that Infinite Scale is running under, for example the user `ocis`. All manipulations in the filesystem have to be done by, and only by, this user.


Suggested change

1. Changes can either be only accepted by the same user that Infinite Scale is running under, for example the user `ocis`. All manipulations in the filesystem have to be done by, and only by, this user.

1. Changes can either be only accepted by the same user that Infinite Scale is running under, for example the user `ocis`. All manipulations in the filesystem have to be done by, and only by this user.

mmattel · 2024-06-05T11:29:11Z

docs/architecture/posixfs.md

+
+There are a few possible ways for user management:
+1. Changes can either be only accepted by the same  user that Infinite Scale is running under, for example the user `ocis`. All manipulations in the filesystem have to be done by, and only by, this user.
+2. Group based: All users who should be able to manipulate files have to be in a unix group. The Infinite Scale user has also to be in there. The default umask in the directory used has to allow group writing all over the place.


Suggested change

2. Group based: All users who should be able to manipulate files have to be in a unix group. The Infinite Scale user has also to be in there. The default umask in the directory used has to allow group writing all over the place.

2. Group based: All users who should be able to manipulate files have to be in a unix group. The Infinite Scale user has also to be member of that group. The default umask in the directory used has to allow group writing all over the place.

mmattel · 2024-06-05T11:32:11Z

docs/architecture/posixfs.md

+If the underlying file system is able to create versions of single resources (imagine a git based file system) this functionality could directly be used by Infinite Scale.
+
+In the current state of the PosixFS, versioning is not supported.


Suggested change

If the underlying file system is able to create versions of single resources (imagine a git based file system) this functionality could directly be used by Infinite Scale.

In the current state of the PosixFS, versioning is not supported.

In the current state of the PosixFS, versioning is not supported.

If the underlying file system is able to create versions of single resources (imagine a git based file system) this functionality could directly be used by Infinite Scale provided in a later version.

Switching the order. imho no candy and then currently not possible...

hm, I have the same sequence in other locations, so I'd like to stay with it.

mmattel · 2024-06-05T11:36:48Z

docs/architecture/posixfs.md

+If the underlying file system handles deleted files in a trash bin that allows restoring of previously removed files, this functionality could directly be used by Infinite Scale.
+
+If not available it will follow the [the Free Desktop Trash specificaton](https://specifications.freedesktop.org/trash-spec/trashspec-latest.html).
+
+In the current state of the PosixFS, trash bin is not supported.


Suggested change

If the underlying file system handles deleted files in a trash bin that allows restoring of previously removed files, this functionality could directly be used by Infinite Scale.

If not available it will follow the [the Free Desktop Trash specificaton](https://specifications.freedesktop.org/trash-spec/trashspec-latest.html).

In the current state of the PosixFS, trash bin is not supported.

In the current state of the PosixFS, trash bin is not supported.

Infinite Scale could, if provided by a later version:

- If the underlying file system handles deleted files in a trash bin that allows restoring of previously removed files, this functionality could directly be used by Infinite Scale.

- If not available it will follow the [Free Desktop Trash specificaton](https://specifications.freedesktop.org/trash-spec/trashspec-latest.html).

mmattel · 2024-06-05T11:37:39Z

docs/architecture/posixfs.md

+
+## Limitations
+
+As of Q2/2024 the PosixFS is in technical preview state which means that it is not officially supported.


Suggested change

As of Q2/2024 the PosixFS is in technical preview state which means that it is not officially supported.

As of Q2/2024 the PosixFS is not officially supported and in technical preview state.

tbsbdr · 2024-06-12T10:09:50Z

Naming proposal for this feature:

Joint Access Storage Driver

mmattel · 2024-06-13T13:32:37Z

@dragotin pls exclude changes in env_vars.yaml and enxtended_vars.yaml.
I need to update the files in a separate PR which needs some care, also see our documentation for this process.

Note that the files get always updated when using make docs generate manually...

mmattel · 2024-06-14T06:28:59Z