Make sure to only use provenance copies when using product.wasderivedfrom()

[schlunma]

Description

This PR makes sure to only use provenance copies (product.copy_provenance()) as input for product.wasderivedfrom(). This prevents recursively adding large amounts of ancestors which blows up memory usage and run time.

Closes #1981

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🧪 and 🛠 Documentation is available
• 🛠 Unit tests have been added
• 🛠 Changes are backward compatible
• 🛠 Any changed dependencies have been added or removed correctly
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

Merging #1984 (491f7cb) into main (9406b7e) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1984   +/-   ##
=======================================
  Coverage   92.80%   92.80%           
=======================================
  Files         236      236           
  Lines       12445    12446    +1     
=======================================
+ Hits        11549    11550    +1     
  Misses        896      896           

Impacted Files	Coverage Δ
esmvalcore/preprocessor/_mask.py	86.40% <100.00%> (+0.05%)	⬆️

[valeriupredoi]

@schlunma cheers, Manu! I have just tested this with your example recipe in #1981 - varying the number of models leads to a very tiny increase in total max memory, this time round barely linear, if at all, with data, not to Powah 4 😁 ie 0.9G for 14 models that were otherwise needing 16G 🍺

[bouweandela]

This could be implemented more efficiently by making fewer copies. For example, this implementation seems to copy all input file provenance for all output files, leading to n x (n-1) copies for the multi dataset mask (assuming all datasets are masked), where n is the number of datasets. However, only the input file provenance needs to be copied, which would be just n copies.

[valeriupredoi]

@bouweandela would you be able to suggest a code snippet for @schlunma to use and expand on pls 🍺 I am not 100% sure of the severity of memory issue ie how many particular cases are affected, so I'd feel more comfortable if this was plugged in, even if not totally efficient, definitely waay more effeicient in terms of memory consumption compared to what we have now 😁

[schlunma]

This could be implemented more efficiently by making fewer copies. For example, this implementation seems to copy all input file provenance for all output files, leading to n x (n-1) copies for the multi dataset mask (assuming all datasets are masked), where n is the number of datasets. However, only the input file provenance needs to be copied, which would be just n copies.

Makes sense, though the performance gain is probably not noticeable for common recipes. Implemented in 341c650.

[valeriupredoi]

good work, gents! @bouweandela could you maybe pls check the new commits, review, and merge this then 🍺

[valeriupredoi]

@bouweandela frenly yodeling ping

[valeriupredoi]

@bouweandela one for you here, bud - quick one, get you back to work after the Easter break 😁

[bouweandela]

Looking good now, thanks @schlunma!