Defect: Improve the caching policy when the RemoteData is cleaned up

[unkcpz]

Motivation

I ran my calculation on Eiger, which has a limited remote files handler quota of 50K files per user. Due to the nature of my calculation, it quickly generates numerous small files, and with just five work chains, the file handler quota is exhausted. To submit more work chains, I need to use the clearworkdir function provided by AiiDA to dynamically clear the remote folder once a calculation is finished.

Initially, this approach works well without the caching feature. However, when caching is enabled, unexpected workflow failures occur.

Desired Outcome

Users can turn on the caching and clean the remote work folder of calculations as they want without sophistically changing there workflow logic to work around the issue.

Impact

The users who are cleaning remote work directory in their workflow with the caching feature turned on will benefit from this improvement.

Complexity

The following issues need to be solved:

• Shallow copy on RemoteData.clone()
• Invalidate cache after RemoteData.clean()
• Hash calculation of RemoteData
• Prospective workchain caching

We don't allow the caching of the workflow but only the calcjobs are cached in the design. What we want to have here is actually allow at some point to cache the work chain. If the design needs to be changed then one month of work is required to properly tackle the problem.

Progress

The AEP is made on aiidateam/AEP#35, which is a draft since more details are to be discussed.

In the aiida-core 2.x, it supported control the caching for the single process by setting the attribute Node.is_valid_cache which introduced in aiidateam/aiida-core#5207