Adding to dynamic output_namespace results in ValidationError

[ltalirz]

In essence I'm just trying to follow the example of aiida-quantumespresso:

Define a dynamic output namespace
https://github.com/aiidateam/aiida-quantumespresso/blob/95a7d94230c37c9a3f3e0cec4226bd2eed7a1939/src/aiida_quantumespresso/calculations/pp.py#L89
and then output a dictionary to that dynamic output namespace:
https://github.com/aiidateam/aiida-quantumespresso/blob/95a7d94230c37c9a3f3e0cec4226bd2eed7a1939/src/aiida_quantumespresso/parsers/pp.py#L155

In my case, the output namespace is called json and one of its dynamic ports is called scf.
However, when I do this, I get an unexpected validation error from plumpy

ValueError: Error validating output 'uuid: 4dcdcc1f-1a4a-4484-828f-81fdb1949c27 (pk: 789)' for port 'outputs': Unexpected ports {'json__scf': <Dict: uuid: 4dcdcc1f-1a4a-4484-828f-81fdb1949c27 (pk: 789)>}, for a non dynamic namespace

The exception is raised inside the call to validate_dynamic_ports here:
https://github.com/aiidateam/plumpy/blob/8c7640b6443410424bf3cfdd5b201ed24bf0dae1/src/plumpy/processes.py#L1269-L1276

Stepping through the code, I realize that

• port_name is json__scf
• the KeyError is raised
• On line 1274, port is set to the top-level 'outputs' namespace.

This leads to the validation error because port.validate_dynamic_ports checks whether port is dynamic (and the top-level 'outputs' port is not, only the 'json' port namespace is).
https://github.com/aiidateam/plumpy/blob/8c7640b6443410424bf3cfdd5b201ed24bf0dae1/src/plumpy/ports.py#L735-L737

It's not exactly clear to me what the intended logic is here but it seems to me that there is an issue.

It's a bit weird because I don't see any obvious recent changes here and I don't understand how the code in aiida-qe can work but not mine.

@sphuber any ideas what might be going on here?

[ltalirz]

After looking further into this, it seems to me this issue is somehow connected to caching.

In the case above caching was enabled.
I'm a little confused - do the parsers still run when caching is enabled?
The documentation makes it sound like the result nodes are simply copied...

After disabling caching for the calculation class in question I'm no longer able to reproduce the validation error.
I'll keep this open for a bit in case I'm able to reproduce it in the future when I re-enable caching

[sphuber]

Seems possible that caching is involved. The validation is supposed to happen on the nested namespace structure in memory so the path would be outputs['json']['scf']. The link labels in the database don't support nesting however and labels have to be flat strings. To represent the nesting in the string each nesting is represented by a double underscore. It seems here that the validation is being done after that link label conversion has already been done. I think you may be right that when caching is enabled, the outputs of the source node are copied to the current node and the links are retrieved from the database but the labels are probably not "re-nested" before being passed to the out call.

[ltalirz]

Great, that makes sense.

I guess this means the issue is not in plumpy but rather in aiida-core; let me know in case you're planning to have a look into this - if not I can also look later today

I guess it will be somewhere around here

aiida-core/aiida/orm/nodes/node.py

Lines 529 to 538 in 5c1eb3f

	self._store(with_transaction=with_transaction, clean=False)
	self._add_outputs_from_cache(cache_node)
	self.base.extras.set('_aiida_cached_from', cache_node.uuid)
	def _add_outputs_from_cache(self, cache_node: 'Node') -> None:
	"""Replicate the output links and nodes from the cached node onto this node."""
	for entry in cache_node.base.links.get_outgoing(link_type=LinkType.CREATE):
	new_node = entry.node.clone()
	new_node.base.links.add_incoming(self, link_type=LinkType.CREATE, link_label=entry.link_label)
	new_node.store()

[sphuber]

I am not sure it is, because that is on the Node and the exception is raised on the Process. I can have a quick look yeah. Let's anyway move this issue to aiida-core.