Documentation: add detailed section on work chains (#2848)

A section is added for the detailed discussion on work chain
implementation rules and best practices.

docs/source/concepts/workflows.rst

@@ -75,7 +75,7 @@ In addition to the 'orchestration' role that the work function can fullfill, it
 Imagine that you want to write a process function that takes a set of input integer nodes and returns the one with the highest value.
 We cannot employ the ``calcfunction`` for this, because it would have to return one of its input nodes, which is explicitly forbidden.
 However, for the ``workfunction``, returning existing nodes, even one of its inputs, is perfectly fine.
-An example implementation might look like the following: 
+An example implementation might look like the following:
 
 .. include:: include/snippets/workflows/workfunctions/add_multiply_workfunction_select.py
     :code: python
@@ -88,7 +88,7 @@ The provenance of the execution of this select work function will look like the
 
     The provenance generated by the work function that selects one of its input nodes
 
-.. warning:: 
+.. warning::
     It is important to realize once again, that in the work function examples given above, all the nodes returned by the work functions are *already stored*.
     That is to say, they were either created by a calculation function called by the work function or were passed in as one of the inputs.
     This is no accident, as the work function **can** only return stored nodes.
@@ -120,54 +120,76 @@ Implementation
 
 If we were to reimplement our work function solution of the simple example problem of the previous section, but this time using a work chain, it would look something like the following:
 
-.. include:: include/snippets/workflows/workchains/add_multiply_workchain_outline_computation.py
+.. include:: include/snippets/workflows/workchains/add_multiply_workchain_external_computation.py
     :code: python
 
 Don't be intimidated by all the code in this snippet.
 The point of this example is not to explain the exact syntax, which will be done in greater detail in the :ref:`advanced workflows<working_workchains>` section, but to merely introduce the concept of the work chain.
-The core attributes of a work chain are defined by its :ref:`process specification<working_process_spec>` which is setup in the :py:meth:`~aiida.engine.processes.process.Process.define` method.
+The core attributes of a work chain are defined by its :ref:`process specification<working_processes_spec>` which is setup in the :py:meth:`~aiida.engine.processes.process.Process.define` method.
 The only thing you need to notice here is that it defines the *inputs* that the work chain takes, its logical *outline* and the *outputs* that it will produce.
 The steps of the outline are implemented as class methods of the work chain.
-The ``add`` step will add the first two integers and store the sum temporarily in the :ref:`context<working_workchain_context>`.
-The next step in the outline, ``multiply``, will take the sum from the first step and multiply it with the third input integer.
+The ``add`` step will add the first two integers by calling the ``add`` calculation function, and store the sum temporarily in the :ref:`context<working_workchains_context>`.
+The next step in the outline, ``multiply``, will take the sum stored in the context that was computed in the first outline step and call the ``multiply`` calculation function with the third input integer.
 Finally, the ``result`` step will take the product produced by the previous step and record it as an output of the work chain.
 The resulting provenance when we run this work chain looks like the following:
 
-.. _fig_work_chains_provenance_add_multiply_workchain_logical:
-.. figure:: include/images/add_multiply_workchain_logical.png
+.. _fig_work_chains_provenance_add_multiply_workchain_full:
+.. figure:: include/images/add_multiply_workchain_full.png
 
-    The provenance generated by the work chain example
+    The provenance generated by the work chain example calling calculation functions to perform the addition and multiplication.
 
-Note how, in contrast with the provenance of the work function solution from :numref:`fig_work_functions_provenance_add_multiply_full`, the individual steps of the outline are all represented by the single workflow node that represents the execution of the work chain.
-The logic inside of those outline steps are then 'hidden' in the provenance graph.
-Although, a more accurate description would be to say that they are 'encapsulated' in a single workflow node.
+As you can see, the produced provenance graph is identical to that of :numref:`fig_work_functions_provenance_add_multiply_full` that was produced by the work function solution, except that the workflow node is a work chain instead of a work function node.
+Full data provenance is kept as the calculation of the sum and the product through the work chain are represented explicitly by the calculation nodes of the ``add`` and ``multiply`` calculation functions that it called.
 
-Additionally, the output has a ``return`` link, even though it was 'created' by the work chain.
-This is because workflow processes do not have the capacity to create new nodes, and in a sense in this example, the provenance is lost.
-If we want to restore the provenance we have to make sure that any new data is created by actual calculations, so we can replace the inline computation of the sum and the product within the outline steps, with proper calculation functions, as done in the work function example.
-To do this, we would have to change the original snippet slightly into:
+.. warning::
 
-.. include:: include/snippets/workflows/workchains/add_multiply_workchain_external_computation.py
+    The usage of calculation functions for the computation of the sum and the product is not an accident but a concious design choice.
+    Since work chains are 'workflow'-like process and as such cannot 'create' data, performing the calculations directly in the work chain outline steps itself would lose data provenance.
+
+To illustrate what it means for worklow processes not being able to 'create' new data and how doing so causes a loss of data provenance, let's change the previous implementation to perform the sum and product in the work chain outline steps itself, instead of calling the calculation functions.
+
+.. include:: include/snippets/workflows/workchains/add_multiply_workchain_outline_computation.py
     :code: python
 
-All that was changed is that the inline python calculations in the ``add`` and ``multiply`` steps, were replaced with the ``calcfunction`` that were also used in the work function example.
-If we now run this work chain, the provenance graph would look very different:
+The resulting provenance would look like the following:
 
-.. _fig_work_chains_provenance_add_multiply_workchain_full:
-.. figure:: include/images/add_multiply_workchain_full.png
+.. _fig_work_chains_provenance_add_multiply_workchain_logical:
+.. figure:: include/images/add_multiply_workchain_logical.png
 
-    The provenance generated by the work chain example
+    The provenance generated by the work chain example that computers the sum and product directly in its outline steps instead of delegating it to calculation functions.
 
-As you can see, now the full provenance is restored as the calculation of the sum and the product by the work chain are represented explicitly by the calculation nodes of the ``add`` and ``multiply`` calculation functions that it called.
-It is also exactly the same as the provenance graph of :numref:`fig_work_functions_provenance_add_multiply_full` that was produced by the work function solution.
-An important thing to remember is that *any computation* that happens in the body of outline steps of a work chain, will not be explicitly represented but will be encapsulated by a single node in the graph that represents that work chain execution.
-This example demonstrates that AiiDA does not force any particular method but allows the user to choose exactly what level of granularity they would like to maintain in the provenance.
-However, the rule of thumb is that if you want to reduce a loss , or 'hiding' of provenance, one should keep real computation within the body of work functions and work chains to a minimum and delegate that to calculations.
-Any real computational work that is relevant to the provenance is better to implement in explicit processes, usually a separate calculation function.
+Note how, in contrast with the provenance of the previous correct solution from :numref:`fig_work_chains_provenance_add_multiply_workchain_full`, there are no explicit calculation nodes representing the computation of the sum and the product.
+Instead, all that computation is abstracted and represented by the single workflow node that represents the execution of the work chain.
+The logic inside of those outline steps is then 'hidden' or 'encapsulated' in the provenance graph by a single workflow node.
+Additionally, the output node representing the final product, only has a ``return`` link, even though it was 'created' by the work chain.
+This is because :ref:`workflow processes do not have the capacity to create new nodes<working_workfunctions_returning_data>`, and therefore in a sense in this example, the data provenance is lost.
 
-This was a very quick overview of how a work chain works but of course it has a lot more features.
+An important thing to remember is that *any computation* that happens in the body of outline steps of a work chain, will not be explicitly represented but will be encapsulated by a single node in the graph that represents that work chain execution.
+Whether that loss of data provenance is relevant depends on the use case and is left to the developer of the workflow.
+These two examples demonstrate that AiiDA does not force any particular method but allows the user to choose exactly what level of granularity they would like to maintain in the provenance.
+However, the rule of thumb is that if you want to reduce the loss, or 'hiding' of provenance to a minimum, one should keep real computation within the body of work functions and work chains to a minimum and delegate that to calculations.
+For any real computational work that is relevant to the data provenance, it is better to implement it in explicit calculation processes, usually a separate calculation function.
+
+Advantages
+----------
+The work chain solution to the add-multiply problem, requires significantly more code compared to the work function solution presented in the beginning of this section.
+Why should one bother using the work chain then?
+The advantages for this trivial example may be difficult to see, but imagine that the logic of the workflow becomes more complicated and the calculations become more intensive.
+The process specification of the work chain provides a central way of defining the inputs and outputs, making it easy to see at a glance how the work chain operates.
+In addition, the ``outline`` can give a succinct summary of the logical steps that the work chain will perform, all of which a work function does not have.
+The outline in this example was trivially simple, but the :ref:`advanced work chain development section<working_workchains>` will show how complex logic can be implemented directly in the process specification.
+The process specification also makes it easy to 'wrap' existing work chains into more complex work chains through the :ref:`expose functionality<working_workchains_expose_inputs_outputs>`.
+
+Finally, as mentioned before, the work chain provides the possibility of checkpoints, i.e. to save progress at certain points from which the computation can be continued after it had been interrupted.
+The state of the work chain is saved after each outline step.
+If expensive calculation jobs are performed in an individual outline step, they will be saved as soon as they finish.
+This is impossible for work functions and if it were to be interrupted before *all* the computations had been completed, all intermediate progress would be lost.
+The rule of thumb therefore is, as soon as the worfklow becomes only slightly complex or computationally intensive, preference should be given to :ref:`work chains<concepts_workchains>` and :ref:`calculation jobs<concepts_calcjobs>`.
+
+This was a very quick overview of the intended use is of work chain works and how they work, but of course it has a lot more features.
 To learn how to write work chains for real life problems, continue reading at the :ref:`work chain development<working_workchains>` section, but before you do, read the following part on when to use a work function and when it is better to use a work chain.
 
+
 When to use which
 =================
 Now that we know how the two workflow components, workflows and work chains, work in AiiDA, you might wonder: when should I use which one?

docs/source/nitpick-exceptions

@@ -227,6 +227,10 @@ py:class  collections.abc.Sized
 # backend-dependent implementation
 py:class  WorkChainSpec
 py:class  aiida.orm.nodes.Node
+py:meth   aiida.engine.processes.process_spec.ProcessSpec.input
+py:meth   aiida.engine.processes.process_spec.ProcessSpec.output
+py:meth   aiida.engine.processes.process_spec.ProcessSpec.outline
+
 # This comes from ABCMeta
 py:meth   aiida.orm.groups.Group.get_from_string
 

docs/source/working/calculations.rst

@@ -45,7 +45,9 @@ Because the returned node is already stored, the engine will raise the following
 
 .. code:: bash
 
-	ValueError: trying to return an already stored Data node from a @calcfunction, however, @calcfunctions cannot return data. If you stored the node yourself, simply do not call `store()` yourself. If you want to return an input node, use a @workfunction instead.
+    ValueError: trying to return an already stored Data node from a @calcfunction, however, @calcfunctions cannot return data.
+    If you stored the node yourself, simply do not call `store()` yourself.
+    If you want to return an input node, use a @workfunction instead.
 
 The reason for this strictness is that a node that was stored after being created in the function body, is indistinguishable from a node that was already stored and had simply been loaded in the function body and returned, e.g.:
 

docs/source/working/functions.rst

@@ -19,7 +19,7 @@ Since the calculation function and work function are both process functions and
     It is therefore crucial that, before you continue, you have read and understood the basic concept of :ref:`calculation functions<concepts_calcfunctions>` and :ref:`work functions<concepts_workfunctions>`.
 
 The simple example in the :ref:`introductory section on calculation functions<concepts_calcfunctions>` showed how a simple python function can be turned into a calculation function simply by adorning it with the :py:func:`~aiida.engine.processes.functions.calcfunction` decorator.
-When the function is run, AiiDA will dynamically generate a :py:class:`~aiida.engine.processes.functions.FunctionProcess` and build its :ref:`process specification<working_process_spec>` based on the function signature.
+When the function is run, AiiDA will dynamically generate a :py:class:`~aiida.engine.processes.functions.FunctionProcess` and build its :ref:`process specification<working_processes_spec>` based on the function signature.
 Here we will explain how this is accomplished and what features of the python function signature standard are supported.
 
 Function signatures