Process functions: Restore support for dynamic nested input namespaces

[sphuber]

Fixes #5806

This behavior was supported before v2.0.4 but was accidentally removed by
putting more strict type validation on inputs passed to process
functions. In d2323d4, a bug was fixed
where it was possible to pass non-storable inputs to dynamic port
namespaces. For example, the following was allowed:

@calcfunction
def sum(**kwargs):
    return sum(kwargs.values())

sum(**{'a': 1, 'b': 2})

However, since the values in the kwargs are not Data nodes (and this
was before automatic input serialization was added for process functions)
and so they could not be stored in the provenance graph. This would
therefore cause accidental provenance loss.

This bug was fixed in the aforementioned commit by explicitly setting
valid_type=Data on the inputs namespace of the Process base class.

The fix, however, inadvertently also removed support for arbitrarily
nested keyword arguments in process functions. While this would work
before, it would now raise a ValidationError saying that the dynamic
namespace received a dict instead of a Data node. The cause is that
the PortNamespace.validate_dynamic_ports did not recurse down nested
namespaces, but only validated top-level values.

This behavior is fixed in plumpy==0.21.3 where now the validation
method will recursively go down any nested namespaces for a dynamic port
namespace and validate as long as all leaf-values match the valid_type
of the PortNamespace.

[sphuber]

@unkcpz as a quick test, I simply overrode the PortNamespace.validate_dynamic_ports method to add the correct behavior. But I think this fix is not specific to process functions but all processes and so really should probably be in plumpy.

Note that essentially this bug also exists for normal processes like WorkChains. If you create a dynamic namespace, you can pass any inputs, even if they are not storable and the process will validate. Example:

from aiida.engine import WorkChain, run_get_node

class NestedWorkChain(WorkChain):

    @classmethod
    def define(cls, spec):
        super().define(spec)
        spec.input_namespace('nested', dynamic=True)
        spec.outline(cls.do_run)

    def do_run(self):
        pas

_, node = run_get_node(NestedWorkChain, **{'nested': {'namespace': {'int': 1}}})

assert node.get_incoming().nested()

The following fails, because non of the inputs are storable, but the process validates and so the inputs of the node are empty. I am thinking that we should not allow this behavior and require that leaf inputs in dynamic namespaces are storable, i.e. Data instances, unless the namespace is explicitly marked as non_db=True. This might break existing implementations out there though. But then again, maybe this would be good, since they are losing (potentially unintentionally) provenance.

[sphuber]

This is the alternative PR in plumpy if we decide that the change should be implemented there instead: aiidateam/plumpy#255

Decision was made to put this in plumpy and the change was released with plumpy==0.21.3.

[unkcpz]

Thanks! looks all good to me. The install test failed since the conda forge of plumpy is not ready I guess?

[sphuber]

Thanks! looks all good to me. The install test failed since the conda forge of plumpy is not ready I guess?

Exactly, I will take care of that now before merging