diff --git a/src/docs/guides/arc_WorkingWithLargeDataFiles.md b/src/docs/guides/arc_WorkingWithLargeDataFiles.md index f029ab689..f8890c9c1 100644 --- a/src/docs/guides/arc_WorkingWithLargeDataFiles.md +++ b/src/docs/guides/arc_WorkingWithLargeDataFiles.md @@ -32,19 +32,33 @@ In this guide we show you how you can actively handle large data files in your A ## Large File Storage (LFS) -ARCs and the DataHUB come with a mechanism to sync and store large files called *Large File Storage (LFS)*. LFS is an efficient way to store your large data files. These files are called "LFS objects". Rather than checking every file during every `arc sync`, the ARCitect or ARC Commander first checks *wether there was a change at all*. And only if this is the case, it scans *what* was changed. This way it saves time and computing power compared to always scanning all large files for possible changes. +ARCs and the DataHUB come with a mechanism to sync and store large files called *Large File Storage (LFS)*. LFS is an efficient way to store your large data files. These files are called "LFS objects". Rather than checking every file during every `arc sync` (ARC Commander) or DataHUB Sync (ARCitect), the tools first check *whether there was a change at all*. And only if this is the case, it scans *what* was changed. This way it saves time and computing power compared to always scanning all large files for possible changes. ### ARCitect -- The ARCitect offers to activate or deactivate the use of LFS via checkboxes in the "Download" as well as the "DataHUB sync" menus, respectively. -- In addition you can set a threshold for what you consider a large file in the "Commit" menu. -- Finally, you can individually download large files via right-click -> "Download LFS File" +The ARCitect offers to activate or deactivate the use of LFS: +- in the "Download ARC" (1) menu via the "LFS" checkbox (2) + + + +- as well as in the "DataHUB Sync" menu (1) via the "Use Large File Storage" checkbox (2), which are available once an ARC has been open in ARCitect. + + + +In addition you can set a threshold (2) in megabytes (MB) for what you consider a large file in the "Commit" menu (1). + + + +Finally, you can individually download large files via right-click -> "Download LFS File" (1) + + ### ARC Commander By default, the ARC Commander tracks the following files via LFS: + 1. All files stored in an assay's `dataset` folder, and - 2. All files with a size larger than 150 MB. + 2. All files with a size larger than 150 MB. The threshold of 150 MB can easily be adjusted using the ARC Commander. For instance, if you want to decrease it to 5 MB (i.e. 5000000 bytes), run @@ -68,7 +82,12 @@ git add .gitattributes ``` 4. Sync your ARC to the DataHUB via `arc sync` -5. Open your ARC in the DataHUB and navigate to the folder with LFS objects and see them flagged as "LFS". +5. Open your ARC in the DataHUB and navigate to the folder with LFS objects and see them flagged as "LFS" (1). + + + +:warning: Please avoid uploading large files without [git LFS](https://git-lfs.github.com/) (i.e. accidentally with pure git, when git-lfs is not available). + #### Downloading an ARC without large data files @@ -105,3 +124,8 @@ If at some point you wish to download **all** LFS files of your ARC, you can use ```bash git lfs pull --include "*" ``` + +#### Checking usage quota of LFS + +If at some point you would like to check how much free storage you have for your ARC, you can easily do so by navigating to your ARC in the DataHUB and clicking on "Project Storage" in the right sidebar (1). + \ No newline at end of file diff --git a/src/docs/img/ARCitect-datahub-sync-lfs.drawio.svg b/src/docs/img/ARCitect-datahub-sync-lfs.drawio.svg new file mode 100644 index 000000000..6e508d861 --- /dev/null +++ b/src/docs/img/ARCitect-datahub-sync-lfs.drawio.svg @@ -0,0 +1,56 @@ + + + + + + + + +
+
+
+ + + 1 + + +
+
+
+ + + 1 + + + + + + + +
+
+
+ + + 2 + + +
+
+
+ + + 2 + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/src/docs/img/ARCitect-download-lfs-file-right-click.drawio.svg b/src/docs/img/ARCitect-download-lfs-file-right-click.drawio.svg new file mode 100644 index 000000000..2fc22268c --- /dev/null +++ b/src/docs/img/ARCitect-download-lfs-file-right-click.drawio.svg @@ -0,0 +1,35 @@ + + + + + + + + +
+
+
+ + + 1 + + +
+
+
+ + + 1 + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/src/docs/img/ARCitect-download-lfs.drawio.svg b/src/docs/img/ARCitect-download-lfs.drawio.svg new file mode 100644 index 000000000..0bc882783 --- /dev/null +++ b/src/docs/img/ARCitect-download-lfs.drawio.svg @@ -0,0 +1,56 @@ + + + + + + + + + + + + +
+
+
+ 2 +
+
+
+ + + 2 + + + + + + + +
+
+
+ + + 1 + + +
+
+
+ + + 1 + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/src/docs/img/ARCitect-lfs-flagged.drawio.svg b/src/docs/img/ARCitect-lfs-flagged.drawio.svg new file mode 100644 index 000000000..7b62da9b2 --- /dev/null +++ b/src/docs/img/ARCitect-lfs-flagged.drawio.svg @@ -0,0 +1,35 @@ + + + + + + + + +
+
+
+ + + 1 + + +
+
+
+ + + 1 + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/src/docs/img/ARCitect-lfs-project-storage.drawio.svg b/src/docs/img/ARCitect-lfs-project-storage.drawio.svg new file mode 100644 index 000000000..b4db91645 --- /dev/null +++ b/src/docs/img/ARCitect-lfs-project-storage.drawio.svg @@ -0,0 +1,35 @@ + + + + + + + + +
+
+
+ + + 1 + + +
+
+
+ + + 1 + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/src/docs/img/ARCitect-lfs-project-storage.png b/src/docs/img/ARCitect-lfs-project-storage.png new file mode 100644 index 000000000..bcc27226d Binary files /dev/null and b/src/docs/img/ARCitect-lfs-project-storage.png differ diff --git a/src/docs/img/ARCitect-lfs-threshold.drawio.svg b/src/docs/img/ARCitect-lfs-threshold.drawio.svg new file mode 100644 index 000000000..f9a1cfa39 --- /dev/null +++ b/src/docs/img/ARCitect-lfs-threshold.drawio.svg @@ -0,0 +1,77 @@ + + + + + + + + +
+
+
+ + + 1 + + +
+
+
+ + + 1 + + + + + + + +
+
+
+ + + 2 + + +
+
+
+ + + 2 + + + + + + + +
+
+
+ + + demoUser@gmail.com + + +
+
+
+ + + demoUser@gmail.com + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file