tutorials/06-Data-sync.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d6dbf987",
   "metadata": {},
   "source": [
    "# iRODS sync tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfb329bc",
   "metadata": {},
   "source": [
    "`sync` synchronizes the data between a local copy (local file system) and the copy stored in iRODS. It compares the checksum of local and remote files to determine whether they have changed and should be synchronized. It creates files or overwrites older copies, but does not delete files from the target location when they have been deleted from the source.\n",
    "\n",
    "The command can be in one of the two modes: synchronization of data from the client's local file system to iRODS, or from iRODS to the local file system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de207d54",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "from pprint import pprint\n",
    "from ibridges.interactive import interactive_auth\n",
    "from ibridges.path import IrodsPath\n",
    "from ibridges import sync"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bd887f2",
   "metadata": {},
   "source": [
    "### Create a session\n",
    "\n",
    "Set up a session to an iRODS server. In this example we assume you have a valid locally cached iRODS password from a previous session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "648f8c49",
   "metadata": {},
   "outputs": [],
   "source": [
    "session = interactive_auth()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "123e9829",
   "metadata": {},
   "source": [
    "### Uploading/downloading\n",
    "Upload or download mode is determined by the type of `source` and `target` (`IrodsPath` or `str`/`Path`).\n",
    "\n",
    "When uploading, `source` must be an existing local folder, and `target` an existing iRODS collection, and vice versa when downloading. An exception will be raised if either doesn't exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39de2a38-2c35-48e8-9de4-cf426eaecefd",
   "metadata": {},
   "outputs": [],
   "source": [
    "source = IrodsPath(session, \"~\", \"Demo\")\n",
    "target = Path.home().joinpath(\"Downloads/Sync\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b180cee-3753-4a1e-a10f-5cfc50fcb8b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(target, target.is_dir())\n",
    "print(source, source.collection_exists())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d20dd782",
   "metadata": {},
   "source": [
    "### Setting sync options\n",
    "`sync` takes various options:\n",
    "\n",
    "- The `max_level` option controls the depth up to which the file tree will be synchronized. With `max_level` set to None (default), there is no limit (full recursive synchronization). A max level of 1 synchronizes only the source's root, max level 2 also includes the first set of subfolders/subcollections and their contents, etc.\n",
    "- The `copy_empty_folders` (default False) option controls whether folders/collections that contain no files or subfolders/subcollections will be synchronized.\n",
    "- The `dry_run` option lists all the source files and folders that need to be synchronized without actually performing the synchronization.\n",
    "\n",
    "By default, checksums of all transferred files will be calculated and verified after up- or downloading. A checksum mismatch will generate an error, aborting the synchronization process. Should this happen, it is possible some hiccup occurred during the transfer process. Check both copies of the offending file, and retain the correct one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edc53ead",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_level=None\n",
    "copy_empty_folders=True\n",
    "dry_run=True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9abc0ae",
   "metadata": {},
   "source": [
    "### Dry run\n",
    "Setting `dry_run` to True will list what will be synchronized without any actual transfers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd8b1155",
   "metadata": {},
   "outputs": [],
   "source": [
    "ops = sync(\n",
    "    session=session,\n",
    "    source=source,\n",
    "    target=target,\n",
    "    max_level=max_level,      \n",
    "    dry_run=dry_run,\n",
    "    copy_empty_folders=copy_empty_folders\n",
    ")\n",
    "ops.print_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec01f5ca",
   "metadata": {},
   "source": [
    "To perform the actual synchronization, set `dry_run` to False, and run again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20785359",
   "metadata": {},
   "outputs": [],
   "source": [
    "dry_run=False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed24c7bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "ops = sync(\n",
    "    session=session,\n",
    "    source=source,\n",
    "    target=target,\n",
    "    max_level=max_level,      \n",
    "    dry_run=dry_run,\n",
    "    copy_empty_folders=copy_empty_folders)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91174060-3f3f-4460-aaae-8da9618a8933",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}