Add basic documentation & readthedocs configuration (#24)

.gitignore

@@ -14,3 +14,4 @@ __pycache__
 .eggs
 .idea
 out
+.DS_Store

.readthedocs.yml

@@ -0,0 +1,24 @@
+# .readthedocs.yml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+sphinx:
+  configuration: doc/source/conf.py
+  fail_on_warning: true
+
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.10"
+  apt_packages:
+    - graphviz
+
+python:
+  install:
+    - method: pip
+      path: .
+      extra_requirements:
+          - docs

doc/source/api.rst

@@ -0,0 +1,27 @@
+API Documentation
+=================
+
+.. autosummary::
+    :toctree: generated
+
+    blue_cwl
+    blue_cwl.staging
+    blue_cwl.registering
+    blue_cwl.core
+    blue_cwl.core.command
+    blue_cwl.core.common
+    blue_cwl.core.config
+    blue_cwl.core.cwl
+    blue_cwl.core.cwl_types
+    blue_cwl.core.environment
+    blue_cwl.core.executor
+    blue_cwl.core.parser
+    blue_cwl.core.resolve
+    blue_cwl.connectome
+    blue_cwl.nexus
+    blue_cwl.population_utils
+    blue_cwl.recipes
+    blue_cwl.statistics
+    blue_cwl.density_manipulation
+    blue_cwl.model
+    blue_cwl.variant

doc/source/concepts/commandlinetool.rst

@@ -0,0 +1,110 @@
+CommandLineTool
+===============
+
+A CommandLineTool definition is a yaml file that follows the `Common Workflow Language specification <https://www.commonwl.org/v1.2/CommandLineTool.html>`_
+
+A full example of a generator task defined as a CWL file is as follows:
+
+.. code:: yaml
+
+   cwlVersion: v1.2
+    class: CommandLineTool
+
+    id: me_type_property
+    label: Morph-Electric type property generator
+
+    baseCommand: ['blue-cwl', 'execute', 'neurons-cell-position']
+
+    environment:
+      env_type: MODULE
+      modules:
+        - unstable
+        - brainbuilder
+        - py-blue-cwl
+      enable_internet: true
+
+    executor:
+      type: slurm
+      slurm_config:
+        partition: prod
+        nodes: 1
+        exclusive: true
+        time: '8:00:00'
+        account: proj134
+      remote_config:
+        host: bbpv1.epfl.ch
+      env_vars:
+        FOO: foo
+        BAR: bar
+
+    inputs:
+
+        - id: region
+          type: string
+          inputBinding:
+            prefix: --region
+
+        - id: cell_composition
+          type: NexusType
+          inputBinding:
+            prefix: --cell-composition
+
+        - id: variant_config
+          type: NexusType
+          inputBinding:
+            prefix: --variant-config
+
+        - id: output_dir
+          type: Directory
+          inputBinding:
+            prefix: --output-dir
+
+    outputs:
+
+        - id: partial_circuit
+          type: NexusType
+          doc: Circuit bundle with me-types and soma positions.
+          outputBinding:
+            glob: "partial-circuit.json"
+
+NexusType
+---------
+
+A NexusType is a special type that is treated as a string or as a file.
+
+.. attention::
+    NexusType is an extension to the CWL standard and only a valid type in blue-cwl definitions.
+
+A `NexusType`, when specified as an input, it's treated as a string, representing the nexus id of a resource, whereas as an output it is a file with the json-ld metadata of the registered resource at the end of the task.
+
+.. note::
+    An output NexusType resource file, being registered, always has an id.
+
+NexusType outputs can be coerced to NexusType inputs when used in a workflow, by extracting the id from the json-ld metadata and passing it as a string input.
+
+baseCommand
+-----------
+
+A `baseCommand` is a list of the CLI command that will be executed in the tool definition. Fixed arguments can be added there. CWL arguments section is not yet supported.
+
+Inputs
+------
+
+A task's inputs in workflows can either be a NEXUS resource (NexusType) or regular CWL primitive types.
+
+Given that task's may be executed in remote filesystems, local file paths are not preferred. However, if the filesystem is mounted it is also possible to work with files.
+
+Outputs
+-------
+
+Workflow generator tasks should define nexus resources as outputs. These are picked by the workflow generators and are passed downstream as registred NEXUS resources for each step in the workflow.
+
+Environment
+-----------
+
+For CWL environments please read :ref:`cwl-environments`.
+
+Executor
+--------
+
+For CWL executors please read :ref:`cwl-executors`.

doc/source/concepts/concepts.rst

@@ -0,0 +1,22 @@
+Concepts
+========
+
+The Generator is at the epicenter of the blue-cwl library.
+
+It is a distinct group of variants, i.e. tools or workflows, designed to perform a specific task within a workflow.
+
+.. tip::
+    An example of a Generator is "cell placement". A Variant, which is a specific realization of cell placement, would be "neuron cell placement". A version would define a particular iteration of the tool or workflow that implements the neuron cell placement algorithm.
+
+
+How to read the following sections
+----------------------------------
+
+When working with a Generator's Variant, three key aspects need to be considered: the API for interacting with the Variant, defining the Variant using the Common Workflow Language (CWL), and creating CLI wrappers to handle intermediate tasks such as staging and registration.
+
+.. toctree::
+    :maxdepth: 2
+
+    variant
+    definitions
+    wrappers

doc/source/concepts/definitions.rst

@@ -0,0 +1,14 @@
+CWL Definitions
+===============
+
+A Common Workflow Language (CWL) definition describes how to run command line tools and connect them to create workflows.
+
+
+.. toctree::
+    :maxdepth: 1
+
+    commandlinetool
+    workflow
+    environments
+    executors
+    supported_features

doc/source/concepts/environments.rst

@@ -0,0 +1,102 @@
+.. _cwl-environments:
+
+Environments
+============
+
+Environments allow specifying how to make the command line tool available as an executable.
+
+.. attention::
+    Environment definitions is an extension to the CWL standard and only a valid section in blue-cwl definitions.
+
+Specifying a custom module
+--------------------------
+
+To use a custom module, you can add an ``environments.yaml`` file with the following content:
+
+.. code-block:: yaml
+
+    version: 1
+    env_config:
+      brainbuilder:
+        env_type: MODULE
+        modules:
+        - archive/2022-03
+        - brainbuilder/0.17.0
+
+If needed, you can specify also the ``modulepath`` as in the following example:
+
+.. code-block:: yaml
+
+    version: 1
+    env_config:
+      brainbuilder:
+        env_type: MODULE
+        modulepath: /gpfs/bbp.cscs.ch/ssd/apps/bsd/modules/_meta
+        modules:
+        - archive/2022-03
+        - brainbuilder/0.17.0
+
+
+Specifying a custom virtual environment
+---------------------------------------
+
+To use an existing custom virtual environment, you can add an ``environments.yaml`` file with the following content:
+
+.. code-block:: yaml
+
+    version: 1
+    env_config:
+      brainbuilder:
+        env_type: VENV
+        path: /absolute/path/to/existing/venv/
+
+The configuration key ``path`` should be set to the directory containing the existing python virtual environment.
+Alternatively, it may be set to any existing file, that will be sourced before executing the commands in the rule.
+
+If needed, it's possible to specify some optional keys as in the following example:
+
+.. code-block:: yaml
+
+    version: 1
+    env_config:
+      brainbuilder:
+        env_type: VENV
+        path: /absolute/path/to/existing/venv/
+        modulepath: /path/to/spack/modules
+        modules:
+        - archive/2023-02
+        - hpe-mpi/2.25.hmpt
+
+.. warning::
+
+    In most cases, you shouldn't load modules that modify PYTHONPATH to avoid issues with conflicting versions of the libraries.
+
+
+Specyfing an Apptainer/Singularity image
+----------------------------------------
+
+To use a custom Apptainer/Singularity image, you can add an ``environments.yaml`` file with the following content:
+
+.. code-block:: yaml
+
+    version: 1
+    env_config:
+      brainbuilder:
+        env_type: APPTAINER
+        image: /path/to/apptainer/image.sif
+
+If needed, it's possible to specify some optional keys as in the following example:
+
+.. code-block:: yaml
+
+    version: 1
+    env_config:
+      brainbuilder:
+        env_type: APPTAINER
+        image: /path/to/apptainer/image.sif
+        executable: singularity
+        options: "--cleanenv --containall --bind $TMPDIR:/tmp,/gpfs/bbp.cscs.ch/project"
+        modulepath: /path/to/spack/modules
+        modules:
+        - archive/2022-06
+        - singularityce

doc/source/concepts/executors.rst

@@ -0,0 +1,54 @@
+.. _cwl-executors:
+
+CWL Executors
+=============
+
+CWL executors allow specifying the local or remote executor that will launch the command.
+
+.. attention::
+    Executors definitions is an extension to the CWL standard and only a valid section in blue-cwl definitions.
+
+Local vs. Remote executors
+--------------------------
+
+A LocalExecutor is the default executor in a CWL specification and will be assigned if no `executor` section is present in the definition. To explicitly spefify one:
+
+.. code:: yaml
+
+    executor:
+      type: local
+      env_vars:
+        FOO: foo
+        BAR: bar
+
+A RemoteExecutor requires a host to be executed via ssh. A `remote_config` section needs to be added in the executor configuration as follows:
+
+.. code:: yaml
+
+    executor:
+      type: slurm
+      remote_config:
+        host: bbpv1.epfl.ch
+
+SallocExecutor
+~~~~~~~~~~~~~~
+
+A SallocExecutor is a slurm executor that builds a command using a blocking salloc directive.
+
+.. code:: yaml
+
+    executor:
+      type: slurm
+      slurm_config:
+        partition: prod
+        nodes: 1
+        exclusive: true
+        time: '8:00:00'
+        account: proj134
+      remote_config:
+        host: bbpv1.epfl.ch
+      env_vars:
+        FOO: foo
+        BAR: bar
+
+For the full slurm config parameters see :class:`blue_cwl.core.config.SlurmConfig`