Add functionality to run single diagnostic task to notebook API

doc/api/esmvalcore.api.config.rst

@@ -3,13 +3,13 @@
 Configuration
 =============
 
-This section describes the config submodule of the API.
+This section describes the :py:class:`~esmvalcore.experimental.config` submodule of the API (:py:mod:`esmvalcore.experimental`).
 
 Config
 ******
 
-Configuration of ESMValCore/Tool is done via the ``Config`` object.
-The global configuration can be imported from the ``esmvalcore.experimental`` module as ``CFG``:
+Configuration of ESMValCore/Tool is done via the :py:class:`~esmvalcore.experimental.config.Config` object.
+The global configuration can be imported from the :py:mod:`esmvalcore.experimental` module as :py:data:`~esmvalcore.experimental.CFG`:
 
 .. code-block:: python
 
@@ -36,7 +36,7 @@ The global configuration can be imported from the ``esmvalcore.experimental`` mo
 
 The parameters for the user configuration file are listed `here <https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#user-configuration-file>`__.
 
-``CFG`` is essentially a python dictionary with a few extra functions, similar to ``matplotlib.rcParams``.
+:py:data:`~esmvalcore.experimental.CFG` is essentially a python dictionary with a few extra functions, similar to :py:mod:`matplotlib.rcParams`.
 This means that values can be updated like this:
 
 .. code-block:: python
@@ -45,27 +45,27 @@ This means that values can be updated like this:
     >>> CFG['output_dir']
     PosixPath('/home/user/esmvaltool_output')
 
-Notice that ``CFG`` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
+Notice that :py:data:`~esmvalcore.experimental.CFG` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
 All values entered into the config are validated to prevent mistakes, for example, it will warn you if you make a typo in the key:
 
 .. code-block:: python
 
-    >>> CFG['otoptu_dri'] = '~/esmvaltool_output'
-    InvalidConfigParameter: `otoptu_dri` is not a valid config parameter.
+    >>> CFG['output_directory'] = '~/esmvaltool_output'
+    InvalidConfigParameter: `output_directory` is not a valid config parameter.
 
 Or, if the value entered cannot be converted to the expected type:
 
 .. code-block:: python
 
-    >>> CFG['max_years'] = '🐜'
-    InvalidConfigParameter: Key `max_years`: Could not convert '🐜' to int
+    >>> CFG['max_parallel_tasks'] = '🐜'
+    InvalidConfigParameter: Key `max_parallel_tasks`: Could not convert '🐜' to int
 
-``Config`` is also flexible, so it tries to correct the type of your input if possible:
+:py:class:`~esmvalcore.experimental.config.Config` is also flexible, so it tries to correct the type of your input if possible:
 
 .. code-block:: python
 
-    >>> CFG['max_years'] = '123'  # str
-    >>> type(CFG['max_years'])
+    >>> CFG['max_parallel_tasks'] = '8'  # str
+    >>> type(CFG['max_parallel_tasks'])
     int
 
 By default, the config is loaded from the default location (``/home/user/.esmvaltool/config-user.yml``).
@@ -87,16 +87,16 @@ Session
 *******
 
 Recipes and diagnostics will be run in their own directories.
-This behaviour can be controlled via the ``Session`` object.
-A ``Session`` can be initiated from the global ``Config``.
+This behaviour can be controlled via the :py:data:`~esmvalcore.experimental.config.Session` object.
+A :py:data:`~esmvalcore.experimental.config.Session` can be initiated from the global :py:class:`~esmvalcore.experimental.config.Config`.
 
 .. code-block:: python
 
     >>> session = CFG.start_session(name='my_session')
 
-A ``Session`` is very similar to the config.
-It is also a dictionary, and copies all the keys from the ``Config``.
-At this moment, ``session`` is essentially a copy of ``CFG``:
+A :py:data:`~esmvalcore.experimental.config.Session` is very similar to the config.
+It is also a dictionary, and copies all the keys from the :py:class:`~esmvalcore.experimental.config.Config`.
+At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.experimental.CFG`:
 
 .. code-block:: python
 
@@ -106,7 +106,7 @@ At this moment, ``session`` is essentially a copy of ``CFG``:
     >>> print(session == CFG)  # False
     False
 
-A ``Session`` also knows about the directories where the data will stored.
+A :py:data:`~esmvalcore.experimental.config.Session` also knows about the directories where the data will stored.
 The session name is used to prefix the directories.
 
 .. code-block:: python
@@ -122,7 +122,7 @@ The session name is used to prefix the directories.
     >>> session.plot_dir
     /home/user/my_output_dir/my_session_20201203_155821/plots
 
-Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from the ``Config``.
+Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from :py:class:`~esmvalcore.experimental.config.Config`.
 
 
 API reference

doc/api/esmvalcore.api.recipe.rst

@@ -3,12 +3,12 @@
 Recipes
 =======
 
-This section describes the :py:mod:`esmvalcore.experimental.recipe` submodule of the API.
+This section describes the :py:mod:`~esmvalcore.experimental.recipe` submodule of the API (:py:mod:`esmvalcore.experimental`).
 
 Recipe metadata
 ***************
 
-:py:class:`esmvalcore.experimental.recipe.Recipe` is a class that holds metadata from a recipe.
+:py:class:`~esmvalcore.experimental.recipe.Recipe` is a class that holds metadata from a recipe.
 
 .. code-block:: python
 
@@ -41,14 +41,14 @@ Printing the recipe will give a nice overview of the recipe:
 Running a recipe
 ****************
 
-To run the recipe, call the :py:meth:`esmvalcore.experimental.Recipe.run` method.
+To run the recipe, call the :py:meth:`~esmvalcore.experimental.recipe.Recipe.run` method.
 
 .. code-block:: python
 
     >>> output = recipe.run()
     <log messages>
 
-By default, a new :py:class:`esmvalcore.experimental.config.Session` is automatically created, so that data are never overwritten.
+By default, a new :py:class:`~esmvalcore.experimental.config.Session` is automatically created, so that data are never overwritten.
 Data are stored in the ``esmvaltool_output`` directory specified in the config.
 Sessions can also be explicitly specified.
 
@@ -59,12 +59,52 @@ Sessions can also be explicitly specified.
     >>> output = recipe.run(session)
     <log messages>
 
-:py:meth:`esmvalcore.experimental.Recipe.run` returns an dictionary of objects that can be used to inspect
-the output of the recipe. The output is an instance of :py:class:`esmvalcore.experimental.recipe_output.ImageFile` or
-:py:class:`esmvalcore.experimental.recipe_output.ImageFile` depending on its type.
+:py:meth:`~esmvalcore.experimental.recipe.Recipe.run` returns an dictionary of objects that can be used to inspect
+the output of the recipe. The output is an instance of :py:class:`~esmvalcore.experimental.recipe_output.ImageFile` or
+:py:class:`~esmvalcore.experimental.recipe_output.DataFile` depending on its type.
 
 For working with recipe output, see: :ref:`api_recipe_output`.
 
+Running a single diagnostic or preprocessor task
+************************************************
+
+The python example recipe contains 5 preprocessors:
+
+Preprocessors:
+
+- ``timeseries/tas_amsterdam``
+- ``timeseries/script1``
+- ``map/tas``
+
+Diagnostics:
+
+- ``timeseries/tas_global``
+- ``map/script1``
+
+To run a single diagnostic or preprocessor, the name of the task can be passed
+as an argument to :py:meth:`~esmvalcore.experimental.recipe.Recipe.run`. If a diagnostic
+is passed, all ancestors will automatically be run too.
+
+.. code-block:: python
+
+    >>> output = recipe.run('map/script1')
+    >>> output
+    map/script1:
+      DataFile('CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.nc')
+      DataFile('CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.nc')
+      ImageFile('CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.png')
+      ImageFile('CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.png')
+
+It is also possible to run a single preprocessor task:
+
+.. code-block:: python
+
+    >>> output = recipe.run('map/tas')
+    >>> output
+    map/tas:
+      DataFile('CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.nc')
+      DataFile('CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.nc')
+
 
 API reference
 *************

doc/api/esmvalcore.api.recipe_metadata.rst

@@ -3,7 +3,7 @@
 Recipe Metadata
 ===============
 
-This section describes the :py:mod:`esmvalcore.experimental.recipe_metadata` submodule of the API.
+This section describes the :py:mod:`~esmvalcore.experimental.recipe_metadata` submodule of the API (:py:mod:`esmvalcore.experimental`).
 
 API reference
 *************

doc/api/esmvalcore.api.recipe_output.rst

@@ -3,9 +3,9 @@
 Recipe output
 =============
 
-This section describes the :py:mod:`esmvalcore.experimental.recipe_output` submodule of the API.
+This section describes the :py:mod:`~esmvalcore.experimental.recipe_output` submodule of the API (:py:mod:`esmvalcore.experimental`).
 
-After running a recipe, output is returned by the :py:meth:`esmvalcore.experimental.recipe.Recipe.run` method. Alternatively, it can be retrieved using the :py:meth:`esmvalcore.experimental.recipe.Recipe.get_output` method.
+After running a recipe, output is returned by the :py:meth:`~esmvalcore.experimental.recipe.Recipe.run` method. Alternatively, it can be retrieved using the :py:meth:`~esmvalcore.experimental.recipe.Recipe.get_output` method.
 
 .. code:: python
 
@@ -53,7 +53,7 @@ a dictionary.
 
 The task output has a list of files associated with them, usually image
 (``.png``) or data files (``.nc``). To get a list of all files, use
-``.files``:
+`:py:meth:`~esmvalcore.experimental.recipe_output.TaskOutput.files`.
 
 .. code:: python
 
@@ -62,7 +62,7 @@ The task output has a list of files associated with them, usually image
     ..., ImageFile('CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.png'))
 
 
-It is also possible to select the image files or data files only:
+It is also possible to select the image (`:py:meth:`~esmvalcore.experimental.recipe_output.TaskOutput.image_files`) files or data files (`:py:meth:`~esmvalcore.experimental.recipe_output.TaskOutput.data_files`) only.
 
 .. code:: python
 
@@ -80,12 +80,12 @@ It is also possible to select the image files or data files only:
 Working with output files
 *************************
 
-Output comes in two kinds, :py:class:`esmvalcore.experimental.recipe_output.DataFile` corresponds to data
-files in ``.nc`` format and :py:class:`esmvalcore.experimental.recipe_output.ImageFile` corresponds to plots
+Output comes in two kinds, :py:class:`~esmvalcore.experimental.recipe_output.DataFile` corresponds to data
+files in ``.nc`` format and :py:class:`~esmvalcore.experimental.recipe_output.ImageFile` corresponds to plots
 in ``.png`` format (see below). Both object are derived from the same base class
-(:py:class:`esmvalcore.experimental.recipe_output.OutputFile`) and therefore share most of the functionality.
+(:py:class:`~esmvalcore.experimental.recipe_output.OutputFile`) and therefore share most of the functionality.
 
-For example, author information can be accessed as instances of :py:class:`esmvalcore.experimental.recipe_metadata.Contributor`  via
+For example, author information can be accessed as instances of :py:class:`~esmvalcore.experimental.recipe_metadata.Contributor`  via
 
 .. code:: python
 
@@ -94,14 +94,14 @@ For example, author information can be accessed as instances of :py:class:`esmva
     (Contributor('Andela, Bouwe', institute='NLeSC, Netherlands', orcid='https://orcid.org/0000-0001-9005-8940'),
      Contributor('Righi, Mattia', institute='DLR, Germany', orcid='https://orcid.org/0000-0003-3827-5950'))
 
-And associated references as instances of :py:class:`esmvalcore.experimental.recipe_metadata.Reference` via
+And associated references as instances of :py:class:`~esmvalcore.experimental.recipe_metadata.Reference` via
 
 .. code:: python
 
     >>> output_file.references
     (Reference('acknow_project'),)
 
-:py:class:`esmvalcore.experimental.recipe_output.OutputFile` also knows about associated files
+:py:class:`~esmvalcore.experimental.recipe_output.OutputFile` also knows about associated files
 
 .. code:: python
 

doc/api/esmvalcore.api.utils.rst

@@ -5,22 +5,23 @@
 Utils
 =====
 
-This section describes the utilities submodule of the API.
+This section describes the :py:class:`~esmvalcore.experimental.utils` submodule of the API (:py:mod:`esmvalcore.experimental`).
+
 
 Finding recipes
 ***************
 
 One of the first thing we may want to do, is to simply get one of the recipes available in ``ESMValTool``
 
-If you already know which recipe you want to load, call :py:func:`esmvalcore.experimental.utils.get_recipe`.
+If you already know which recipe you want to load, call :py:func:`~esmvalcore.experimental.utils.get_recipe`.
 
 .. code-block:: python
 
     from esmvalcore.experimental import get_recipe
     >>> get_recipe('examples/recipe_python')
     Recipe('Recipe python')
 
-Call the :py:func:`esmvalcore.experimental.utils.get_all_recipes` function to get a list of all available recipes.
+Call the :py:func:`~esmvalcore.experimental.utils.get_all_recipes` function to get a list of all available recipes.
 
 .. code-block:: python
 
@@ -33,14 +34,14 @@ Call the :py:func:`esmvalcore.experimental.utils.get_all_recipes` function to ge
      Recipe('Recipe wflow'),
      Recipe('Recipe pcrglobwb')]
 
-To search for a specific recipe, you can use the :py:meth:`esmvalcore.experimental.utils.RecipeList.find` method. This takes a search query that looks through the recipe metadata and returns any matches. The query can be a regex pattern, so you can make it as complex as you like.
+To search for a specific recipe, you can use the :py:meth:`~esmvalcore.experimental.utils.RecipeList.find` method. This takes a search query that looks through the recipe metadata and returns any matches. The query can be a regex pattern, so you can make it as complex as you like.
 
 .. code-block:: python
 
     >>> results = recipes.find('climwip')
     [Recipe('Recipe climwip')]
 
-The recipes are loaded in a :py:class:`esmvalcore.experimental.recipe.Recipe` object, which knows about the documentation, authors, project, and related references of the recipe. It resolves all the tags, so that it knows which institute an author belongs to and which references are associated with the recipe.
+The recipes are loaded in a :py:class:`~esmvalcore.experimental.recipe.Recipe` object, which knows about the documentation, authors, project, and related references of the recipe. It resolves all the tags, so that it knows which institute an author belongs to and which references are associated with the recipe.
 
 This means you can search for something like this:
 

doc/conf.py

@@ -418,7 +418,7 @@
     (f'https://docs.esmvaltool.org/projects/esmvalcore/en/{rtd_version}/',
      None),
     'esmvaltool': (f'https://docs.esmvaltool.org/en/{rtd_version}/', None),
-    'iris': ('https://scitools.org.uk/iris/docs/latest/', None),
+    'iris': ('https://scitools-iris.readthedocs.io/en/latest/', None),
     'matplotlib': ('https://matplotlib.org/', None),
     'numpy': ('https://numpy.org/doc/stable/', None),
     'python': ('https://docs.python.org/3/', None),

esmvalcore/_recipe.py

@@ -1261,6 +1261,9 @@ def initialize_tasks(self):
         logger.info("Creating tasks from recipe")
         tasks = set()
 
+        run_diagnostic = self._cfg.get('run_diagnostic', True)
+        tasknames_to_run = self._cfg.get('diagnostics')
+
         priority = 0
         for diagnostic_name, diagnostic in self.diagnostics.items():
             logger.info("Creating tasks for diagnostic %s", diagnostic_name)
@@ -1302,12 +1305,12 @@ def initialize_tasks(self):
 
         # Select only requested tasks
         tasks = get_flattened_tasks(tasks)
-        if not self._cfg.get('run_diagnostic', True):
+        if not run_diagnostic:
             tasks = {t for t in tasks if isinstance(t, PreprocessingTask)}
-        if self._cfg.get('diagnostics'):
+        if tasknames_to_run:
             names = {t.name for t in tasks}
             selection = set()
-            for pattern in self._cfg.get('diagnostics'):
+            for pattern in tasknames_to_run:
                 selection |= set(fnmatch.filter(names, pattern))
             tasks = {t for t in tasks if t.name in selection}
 
@@ -1330,6 +1333,9 @@ def __str__(self):
 
     def run(self):
         """Run all tasks in the recipe."""
+        if not self.tasks:
+            raise RecipeError('No tasks to run!')
+
         run_tasks(self.tasks,
                   max_parallel_tasks=self._cfg['max_parallel_tasks'])
 
@@ -1344,9 +1350,6 @@ def get_product_output(self) -> dict:
         product_filenames = {}
 
         for task in self.tasks:
-            product_filenames[task.name] = {
-                product.filename: product.attributes
-                for product in task.products
-            }
+            product_filenames[task.name] = task.get_product_attributes()
 
         return product_filenames

esmvalcore/_task.py

@@ -257,6 +257,13 @@ def run(self, input_files=None):
     def _run(self, input_files):
         """Run task."""
 
+    def get_product_attributes(self) -> dict:
+        """Return a mapping of product attributes."""
+        return {
+            product.filename: product.attributes
+            for product in self.products
+        }
+
     def str(self):
         """Return a nicely formatted description."""
         def _indent(txt):
@@ -627,15 +634,14 @@ def _collect_provenance(self):
                      self.name,
                      time.time() - start)
 
-    def __str__(self):
+    def __repr__(self):
         """Get human readable description."""
-        settings_string = pprint.pformat(self.settings, indent=2)
-        txt = (f"{self.__class__.__name__}:\n"
-               f"script: {self.script}\n"
-               f"settings:\n{settings_string}\n"
-               f"{super(DiagnosticTask, self)}\n")
+        settings_string = pprint.pformat(self.settings)
+        string = (f"{self.__class__.__name__}: {self.name}\n"
+                  f"script: {self.script}\n"
+                  f"settings:\n{settings_string}\n")
 
-        return txt
+        return string
 
 
 def get_flattened_tasks(tasks):