-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add basic documentation & readthedocs configuration (#24)
- Loading branch information
1 parent
8572d0c
commit d82e003
Showing
22 changed files
with
1,217 additions
and
74 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -14,3 +14,4 @@ __pycache__ | |
.eggs | ||
.idea | ||
out | ||
.DS_Store |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
# .readthedocs.yml | ||
# Read the Docs configuration file | ||
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details | ||
|
||
# Required | ||
version: 2 | ||
|
||
sphinx: | ||
configuration: doc/source/conf.py | ||
fail_on_warning: true | ||
|
||
build: | ||
os: ubuntu-22.04 | ||
tools: | ||
python: "3.10" | ||
apt_packages: | ||
- graphviz | ||
|
||
python: | ||
install: | ||
- method: pip | ||
path: . | ||
extra_requirements: | ||
- docs |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
API Documentation | ||
================= | ||
|
||
.. autosummary:: | ||
:toctree: generated | ||
|
||
blue_cwl | ||
blue_cwl.staging | ||
blue_cwl.registering | ||
blue_cwl.core | ||
blue_cwl.core.command | ||
blue_cwl.core.common | ||
blue_cwl.core.config | ||
blue_cwl.core.cwl | ||
blue_cwl.core.cwl_types | ||
blue_cwl.core.environment | ||
blue_cwl.core.executor | ||
blue_cwl.core.parser | ||
blue_cwl.core.resolve | ||
blue_cwl.connectome | ||
blue_cwl.nexus | ||
blue_cwl.population_utils | ||
blue_cwl.recipes | ||
blue_cwl.statistics | ||
blue_cwl.density_manipulation | ||
blue_cwl.model | ||
blue_cwl.variant |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,110 @@ | ||
CommandLineTool | ||
=============== | ||
|
||
A CommandLineTool definition is a yaml file that follows the `Common Workflow Language specification <https://www.commonwl.org/v1.2/CommandLineTool.html>`_ | ||
|
||
A full example of a generator task defined as a CWL file is as follows: | ||
|
||
.. code:: yaml | ||
cwlVersion: v1.2 | ||
class: CommandLineTool | ||
id: me_type_property | ||
label: Morph-Electric type property generator | ||
baseCommand: ['blue-cwl', 'execute', 'neurons-cell-position'] | ||
environment: | ||
env_type: MODULE | ||
modules: | ||
- unstable | ||
- brainbuilder | ||
- py-blue-cwl | ||
enable_internet: true | ||
executor: | ||
type: slurm | ||
slurm_config: | ||
partition: prod | ||
nodes: 1 | ||
exclusive: true | ||
time: '8:00:00' | ||
account: proj134 | ||
remote_config: | ||
host: bbpv1.epfl.ch | ||
env_vars: | ||
FOO: foo | ||
BAR: bar | ||
inputs: | ||
- id: region | ||
type: string | ||
inputBinding: | ||
prefix: --region | ||
- id: cell_composition | ||
type: NexusType | ||
inputBinding: | ||
prefix: --cell-composition | ||
- id: variant_config | ||
type: NexusType | ||
inputBinding: | ||
prefix: --variant-config | ||
- id: output_dir | ||
type: Directory | ||
inputBinding: | ||
prefix: --output-dir | ||
outputs: | ||
- id: partial_circuit | ||
type: NexusType | ||
doc: Circuit bundle with me-types and soma positions. | ||
outputBinding: | ||
glob: "partial-circuit.json" | ||
NexusType | ||
--------- | ||
|
||
A NexusType is a special type that is treated as a string or as a file. | ||
|
||
.. attention:: | ||
NexusType is an extension to the CWL standard and only a valid type in blue-cwl definitions. | ||
|
||
A `NexusType`, when specified as an input, it's treated as a string, representing the nexus id of a resource, whereas as an output it is a file with the json-ld metadata of the registered resource at the end of the task. | ||
|
||
.. note:: | ||
An output NexusType resource file, being registered, always has an id. | ||
|
||
NexusType outputs can be coerced to NexusType inputs when used in a workflow, by extracting the id from the json-ld metadata and passing it as a string input. | ||
|
||
baseCommand | ||
----------- | ||
|
||
A `baseCommand` is a list of the CLI command that will be executed in the tool definition. Fixed arguments can be added there. CWL arguments section is not yet supported. | ||
|
||
Inputs | ||
------ | ||
|
||
A task's inputs in workflows can either be a NEXUS resource (NexusType) or regular CWL primitive types. | ||
|
||
Given that task's may be executed in remote filesystems, local file paths are not preferred. However, if the filesystem is mounted it is also possible to work with files. | ||
|
||
Outputs | ||
------- | ||
|
||
Workflow generator tasks should define nexus resources as outputs. These are picked by the workflow generators and are passed downstream as registred NEXUS resources for each step in the workflow. | ||
|
||
Environment | ||
----------- | ||
|
||
For CWL environments please read :ref:`cwl-environments`. | ||
|
||
Executor | ||
-------- | ||
|
||
For CWL executors please read :ref:`cwl-executors`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
Concepts | ||
======== | ||
|
||
The Generator is at the epicenter of the blue-cwl library. | ||
|
||
It is a distinct group of variants, i.e. tools or workflows, designed to perform a specific task within a workflow. | ||
|
||
.. tip:: | ||
An example of a Generator is "cell placement". A Variant, which is a specific realization of cell placement, would be "neuron cell placement". A version would define a particular iteration of the tool or workflow that implements the neuron cell placement algorithm. | ||
|
||
|
||
How to read the following sections | ||
---------------------------------- | ||
|
||
When working with a Generator's Variant, three key aspects need to be considered: the API for interacting with the Variant, defining the Variant using the Common Workflow Language (CWL), and creating CLI wrappers to handle intermediate tasks such as staging and registration. | ||
|
||
.. toctree:: | ||
:maxdepth: 2 | ||
|
||
variant | ||
definitions | ||
wrappers |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
CWL Definitions | ||
=============== | ||
|
||
A Common Workflow Language (CWL) definition describes how to run command line tools and connect them to create workflows. | ||
|
||
|
||
.. toctree:: | ||
:maxdepth: 1 | ||
|
||
commandlinetool | ||
workflow | ||
environments | ||
executors | ||
supported_features |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,102 @@ | ||
.. _cwl-environments: | ||
|
||
Environments | ||
============ | ||
|
||
Environments allow specifying how to make the command line tool available as an executable. | ||
|
||
.. attention:: | ||
Environment definitions is an extension to the CWL standard and only a valid section in blue-cwl definitions. | ||
|
||
Specifying a custom module | ||
-------------------------- | ||
|
||
To use a custom module, you can add an ``environments.yaml`` file with the following content: | ||
|
||
.. code-block:: yaml | ||
version: 1 | ||
env_config: | ||
brainbuilder: | ||
env_type: MODULE | ||
modules: | ||
- archive/2022-03 | ||
- brainbuilder/0.17.0 | ||
If needed, you can specify also the ``modulepath`` as in the following example: | ||
|
||
.. code-block:: yaml | ||
version: 1 | ||
env_config: | ||
brainbuilder: | ||
env_type: MODULE | ||
modulepath: /gpfs/bbp.cscs.ch/ssd/apps/bsd/modules/_meta | ||
modules: | ||
- archive/2022-03 | ||
- brainbuilder/0.17.0 | ||
Specifying a custom virtual environment | ||
--------------------------------------- | ||
|
||
To use an existing custom virtual environment, you can add an ``environments.yaml`` file with the following content: | ||
|
||
.. code-block:: yaml | ||
version: 1 | ||
env_config: | ||
brainbuilder: | ||
env_type: VENV | ||
path: /absolute/path/to/existing/venv/ | ||
The configuration key ``path`` should be set to the directory containing the existing python virtual environment. | ||
Alternatively, it may be set to any existing file, that will be sourced before executing the commands in the rule. | ||
|
||
If needed, it's possible to specify some optional keys as in the following example: | ||
|
||
.. code-block:: yaml | ||
version: 1 | ||
env_config: | ||
brainbuilder: | ||
env_type: VENV | ||
path: /absolute/path/to/existing/venv/ | ||
modulepath: /path/to/spack/modules | ||
modules: | ||
- archive/2023-02 | ||
- hpe-mpi/2.25.hmpt | ||
.. warning:: | ||
|
||
In most cases, you shouldn't load modules that modify PYTHONPATH to avoid issues with conflicting versions of the libraries. | ||
|
||
|
||
Specyfing an Apptainer/Singularity image | ||
---------------------------------------- | ||
|
||
To use a custom Apptainer/Singularity image, you can add an ``environments.yaml`` file with the following content: | ||
|
||
.. code-block:: yaml | ||
version: 1 | ||
env_config: | ||
brainbuilder: | ||
env_type: APPTAINER | ||
image: /path/to/apptainer/image.sif | ||
If needed, it's possible to specify some optional keys as in the following example: | ||
|
||
.. code-block:: yaml | ||
version: 1 | ||
env_config: | ||
brainbuilder: | ||
env_type: APPTAINER | ||
image: /path/to/apptainer/image.sif | ||
executable: singularity | ||
options: "--cleanenv --containall --bind $TMPDIR:/tmp,/gpfs/bbp.cscs.ch/project" | ||
modulepath: /path/to/spack/modules | ||
modules: | ||
- archive/2022-06 | ||
- singularityce |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
.. _cwl-executors: | ||
|
||
CWL Executors | ||
============= | ||
|
||
CWL executors allow specifying the local or remote executor that will launch the command. | ||
|
||
.. attention:: | ||
Executors definitions is an extension to the CWL standard and only a valid section in blue-cwl definitions. | ||
|
||
Local vs. Remote executors | ||
-------------------------- | ||
|
||
A LocalExecutor is the default executor in a CWL specification and will be assigned if no `executor` section is present in the definition. To explicitly spefify one: | ||
|
||
.. code:: yaml | ||
executor: | ||
type: local | ||
env_vars: | ||
FOO: foo | ||
BAR: bar | ||
A RemoteExecutor requires a host to be executed via ssh. A `remote_config` section needs to be added in the executor configuration as follows: | ||
|
||
.. code:: yaml | ||
executor: | ||
type: slurm | ||
remote_config: | ||
host: bbpv1.epfl.ch | ||
SallocExecutor | ||
~~~~~~~~~~~~~~ | ||
|
||
A SallocExecutor is a slurm executor that builds a command using a blocking salloc directive. | ||
|
||
.. code:: yaml | ||
executor: | ||
type: slurm | ||
slurm_config: | ||
partition: prod | ||
nodes: 1 | ||
exclusive: true | ||
time: '8:00:00' | ||
account: proj134 | ||
remote_config: | ||
host: bbpv1.epfl.ch | ||
env_vars: | ||
FOO: foo | ||
BAR: bar | ||
For the full slurm config parameters see :class:`blue_cwl.core.config.SlurmConfig` |
Oops, something went wrong.