Add high level filesystem documention

From nextcloud/server#26982

Adds some documentation for the filesystem layer, both a high level overview of how the various pieces interact and a high level guide for apps interacting with the filesystem.

Hopefully this will be useful to anyone trying to either use the filesystem or work on the filesystem.

Signed-off-by: Louis Chemineau <louis@chmn.me>

developer_manual/basics/storage/filesystem.rst

@@ -1,37 +1,81 @@
-==========
-Filesystem
-==========
+========================
+Nextcloud filesystem API
+========================
+
 
 .. sectionauthor:: Bernhard Posselt <dev@bernhard-posselt.com>
 
-Because users can choose their storage backend, the filesystem should be accessed by using the appropriate filesystem classes.
+High level guide to using the Nextcloud filesystem API.
+
+Because users can choose their storage backend, the filesystem should be accessed by using the appropriate filesystem classes. For a simplified filesystem for app specific data see `IAppData <appdata.html>`_
+
+Node API
+^^^^^^^^
+
+The "Node API" is the primary api for apps to access the Nextcloud filesystem, each item in the filesystem is
+represented as either a File or Folder node with each node providing access to the relevant filesystem information
+and actions for the node.
+
+
+Getting access
+--------------
+
+Access to the filesystem is provided by the ``IRootFolder`` which can be injected into your class.
+From the root folder you can either access a user's home folder or access a file or folder by its absolute path.
+
+.. code-block:: php
+    <?php
+
+    use OCP\Files\IRootFolder;
+    use OCP\IUserSession;
+
+    class FileSystemAccessExample {
+        private IUserSession $userSession;
+        private IRootFolder $rootFolder;
+
+        public function __constructor(IUserSession $userSession, IRootFolder $rootFolder) {
+            $this->userSession = $userSession;
+            $this->rootFolder = $rootFolder;
+        }
+
+        /**
+        * Create a new file with specified content in the home folder of the current user
+        * returning the size of the resulting file.
+        */
+        public function getCurrentUserFolder(string $path, string $content): int {
+            $user = $this->userSession->getUser();
+
+            if ($user === null) {
+                return null;
+            }
+
+            // the "user folder" corresponds to the root of the user visible files
+            return $this->rootFolder->getUserFolder($user->getUID());
+        }
+    }
+
+For more details on the specific methods provided by file and folder nodes see the method documentation from the ``OCP\Files\File`` and ``OCP\Files\Folder`` interfaces.
 
-Filesystem classes can be injected automatically with dependency injection. This is the user filesystem.
-For a simplified filestystem for app specific data see `IAppData <appdata.html>`_
 
 Writing to a file
 -----------------
 
-
 All methods return a Folder object on which files and folders can be accessed, or filesystem operations can be performed relatively to their root. For instance for writing to file:`nextcloud/data/myfile.txt` you should get the root folder and use:
 
 .. code-block:: php
-
     <?php
-    namespace OCA\MyApp\Storage;
 
     use OCP\Files\IRootFolder;
 
-    class AuthorStorage {
+    class FileWritingExample {
 
-        /** @var IRootStorage */
-        private $storage;
+        private IRootStorage $storage;
 
         public function __construct(IRootFolder $storage){
             $this->storage = $storage;
         }
 
-        public function writeTxt($content) {
+        public function writeContentToFile($content) {
 
             $userFolder = $this->storage->getUserFolder('myUser');
 
@@ -54,35 +98,33 @@ All methods return a Folder object on which files and folders can be accessed, o
         }
     }
 
+
 Reading from a file
 -------------------
 
 Files and folders can also be accessed by id, by calling the **getById** method on the folder.
 
 .. code-block:: php
-
     <?php
-    namespace OCA\MyApp\Storage;
 
     use OCP\Files\IRootFolder;
 
-    class AuthorStorage {
+    class FileReadingExample {
 
-        /** @var IRootFolder */
-        private $storage;
+        private IRootFolder $storage;
 
         public function __construct(IRootFolder $storage){
             $this->storage = $storage;
         }
 
-        public function getContent($id) {
+        public function getFileContent($id) {
 
             $userFolder = $this->storage->getUserFolder('myUser');
 
             // check if file exists and read from it if possible
             try {
                 $file = $userFolder->getById($id);
-                if($file instanceof \OCP\Files\File) {
+                if ($file instanceof \OCP\Files\File) {
                     return $file->getContent();
                 } else {
                     throw new StorageException('Can not read from folder');
@@ -92,3 +134,36 @@ Files and folders can also be accessed by id, by calling the **getById** method
             }
         }
     }
+
+
+Direct storage access
+---------------------
+
+While it should be generally avoided in favor of the higher level apis,
+sometimes an app needs to talk directly to the storage implementation of it's metadata cache.
+
+You can get access to the underlying storage of a file or folder by calling ``getStorage`` on the node or first getting
+the mountpoint by calling ``getMountPoint`` and getting the storage from there.
+
+Once you have the storage instance you can use the storage api from ``OCP\Files\Storage\IStorage``, note however that
+all paths used in the storage api are internal to the storage, the ``IMountPoint`` returned from ``getMountPoint`` provides
+methods for translating between absolute filesystem paths and internal storage paths.
+
+If you need to query the cached metadata directory you can get the ``OCP\Files\Cache\ICache`` from the storage by calling ``getCache``.
+
+Implementing a storage
+----------------------
+
+The recommended way for implementing a storage backend is by sub-classing ``OC\Files\Storage\Common`` which provides
+fallback implementations for various methods, reducing the amount of work required to implement the full storage api.
+Note however that various of these fallback implementations are likely to be significantly less efficient than an
+implementation of the method optimized for the abilities of the storage backend.
+
+Adding mounts to the filesystem
+-------------------------------
+
+The recommended way of adding your own mounts to the filesystem from an app is implementing ``OCP\Files\Config\IMountProvider``
+and registering the provider using ``OCP\Files\Config\IMountProviderCollection::registerProvider``.
+
+Once registered, your provider will be called every time the filesystem is being setup for a user and your mount provider
+can return a list of mounts to add for that user.

developer_manual/core/architecture/files.rst

@@ -0,0 +1,148 @@
+========================
+Nextcloud filesystem API
+========================
+
+High level overview
+-------------------
+
+The Nextcloud filesystem is roughly based on the unix filesystem, consisting of multiple storages
+mounted at various locations.
+
+.. code-block:: text
+
+         ┌──────────────────────────────────┐
+         │Code wanting to use the filesystem│
+         └─────────┬─────────────────────┬──┘
+                   │                     │
+                   │                     │
+    ┌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐
+    ╎Filesystem    │                     │         ╎
+    ╎layer         │new                  │legacy   ╎
+    ╎              │                     │         ╎
+    ╎              ▼                     ▼         ╎
+    ╎      ┌────────┐ Partly build on  ┌─┴──────┐  ╎
+    ╎      │Node API├─────────────────►│View API│  ╎
+    ╎      └───────┬┘                  └─┬──────┘  ╎
+    ╎              │                     │         ╎
+    └╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘
+                   │                     │
+    ┌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┐
+    ╎Storage layer │                     │         ╎
+    ╎              ├─────────────────────┤         ╎
+    ╎              │                     │         ╎
+    ╎              ▼                     ▼         ╎
+    ╎        ┌───────┐    ┌───────┐    ┌──────┐    ╎
+    ╎        │Storage│═══>│Scanner│═══>│Cache │    ╎
+    ╎        └───────┘    └───────┘    └──────┘    ╎
+    ╎                                              ╎
+    ╎                                              ╎
+    └╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┘
+
+Filesystem layer
+^^^^^^^^^^^^^^^^
+
+Any code that wants to use the filesystem has two API options to use, the new ``Node`` api and the old ``View`` api.
+New code should preferably use the ``Node`` api as it allows building systems with less overhead than the old api.
+
+Besides the filesystem apis, this layer also manages the available mounts, containing the logic to allow apps
+to setup their mounts and translating filesystem paths into a mountpoint + "internal" path.
+
+Storage layer
+^^^^^^^^^^^^^
+
+The storage implementation handles the details of communicating with the filesystem or remote storage api
+and provide a uniform api for Nextcloud to use the storage.
+
+For each storage a metadata cache/index is maintained to allow reading metadata of the storage without having
+to talk to the (potentially) slow storage backend. The scanner is responsible for updating the cache with
+information from the storage backend.
+
+Storage/Cache wrappers
+----------------------
+
+To allow apps to customize the behavior of a storage without requiring the app to implement this for every
+possible storage backend, a ``Wrapper`` system is used.
+
+A ``Wrapper`` encapsulates an inner storage and allows overwriting any method to customize its behavior, with
+all other methods being passed through to the inner storage.
+
+Generally search storage wrapper has an equivalent cache wrapper encapsulating the cache of the inner storage
+to provide the same behavior modifications when reading metadata from the cache.
+
+Wrappers can be layered to stack the behavior of the wrappers, for example the ``groupfolders`` app works by
+stacking a wrapper to provide access to a single folder on the root storage with a wrapper to limit the permissions
+of the storage.
+
+.. code-block:: text
+
+    ┌───────────────┐      ┌────────────────────┐
+    │PermissionsMask├─────►│CachePermissionsMask│  PermissionsMask applies a mask to the permissions of a storage
+    └───────┬───────┘      └─────────┬──────────┘  to provide less-privilaged access to a storage
+            │                        │
+            ▼                        ▼
+    ┌───────────────┐      ┌────────────────────┐
+    │Jail           ├─────►│CacheJail           │  Jail restricts access to a file or folder of a storage providing
+    └───────┬───────┘      └─────────┬──────────┘  a limited view into the storage (think unix chroot or bind mount)
+            │                        │
+            ▼                        ▼
+    ┌───────────────┐      ┌────────────────────┐
+    │Base Storage   ├─────►│Base Cache          │
+    └───────────────┘      └────────────────────┘
+
+Code Map
+--------
+
+Approximate overview of the significant filesystem code
+
+AppData
+^^^^^^^
+
+High level api for accessing "appdata" folders, based on the ``Node`` API
+
+Cache
+^^^^^
+
+- ``Cache`` implementation
+- Cache wrappers
+- Scanner and cache update logic
+- Search infrastructure
+
+Mount
+^^^^^
+
+Mountpoint management and setup
+
+Node
+^^^^
+
+``Node`` filesystem api implementation
+
+ObjectStorage
+^^^^^^^^^^^^^
+
+Implementation of the various supported object store storage backends
+
+SimpleFS
+^^^^^^^^
+
+Simplified version of the Node api, for providing a more limited api for some filesystem bits
+
+Storage
+^^^^^^^
+
+Implementation of various storage backends and wrappers
+
+Streams
+^^^^^^^
+
+Various low-level php stream wrapper used in storage implementations
+
+Type
+^^^^
+
+Mimetype management and detection
+
+View.php
+^^^^^^^^
+
+Legacy View api

developer_manual/core/architecture/index.rst

@@ -0,0 +1,9 @@
+.. _architectureindex:
+
+======================
+Nextcloud architecture
+======================
+
+.. toctree::
+
+   files

developer_manual/core/index.rst

@@ -11,4 +11,5 @@ Please make sure you have set up a :ref:`devenv`.
 
    unit-testing
    externalapi
+   architecture/index
    ../how_to/index