Using the output from a diagnostic that ran earlier

[malininae]

Hi all,

I know it is possible, but I'm not sure what the best way is. I am creating a diagnostic where I will calculate certain statistics and weight them by the number of the realizations, so I don't think I'll be able to do those calculations in the preprocessor. Since the calculations are memory heavy, I was hoping to calculate the stats in the first diagnostic, save that in a .nc-file and then re-use it in the next diagnostic, So far, I was organizing my diagnostics by variable_group and I want my stats .nc-files also be saved using that attribute (something like skewness_prw_annual_1980-2022.nc). Do I put this data somehow in .yml file? Or any other elegant ways I can re-use the files from the previous diagnostic?

Any advice appreciated, thanks!

[lukruh]

Hi @malininae,

In the esmvaltool diagnostics that rely on each other (i.e. one using the output of another one) are considered as ancestors. If you specify them in the recipe as such, the tool will take care of running them in the correct order, waiting for tasks to finish when run parallel and so on. It also passes the output data folder from the ancestors as input data for the next diagnostic. You can find more on how to use ancestors in the documentation.

Afaik, it's possible to create a metadata.yml file from within an ancestor with the same structure as the one the preprocessor creates. This file will be read automatically and passed to the next diagnostic, but only contains metadata per data file. I can provide more details/example if needed.

However, you can always write and save your own yaml file, containing any data you want in one diagnostic and load it from another one, if the first is set as ancestor in the recipe. For saving data files you can use get_diagnostic_filename() (read more) to make sure the right paths are used across different installations of the tool.

[bouweandela]

We were just discussing the usefulness of weighting by a statistic in ESMValGroup/ESMValCore#2189 (comment). Maybe that could also be a solution to your issue, once it is available?

[malininae]

@bouweandela thanks for the comment. I think it's a bit different thing. So what I am interested in let's say we want to a multi-model mean, but we need not just apply the mean over all the realisations of the model but rather create weighted mean with weights being 1/n_real_model*n_models. I've done it in the multiple diagnostics including IPCC ones, however, I've done it only in the diagnostic, not in the preprocessor. Furthermore, the means are not the problem, the weighted percentiles become more complicated. I'll join this months meeting, I can try to explain it there.