Refactor esmvalcore.Recipe

[stefsmeets]

Is your feature request related to a problem? Please describe.
esmvalcore.Recipe contains a lot of functionality related to parsing / setting up a recipe, but it is somewhat difficult to use.

ESMValCore/esmvalcore/_recipe.py

Line 901 in 9b6f095

class Recipe:

For #907 we want to re-use much of the code and data in Recipe (e.g. accessing diagnostics / settings data), but the design of the class in its current state prohibits this.

At present, Recipe is only used once in the entire code base.

ESMValCore/esmvalcore/_recipe.py

Line 63 in 9b6f095

return Recipe(raw_recipe,

I propose we refactor Recipe to make the functionality and data more accessible to other objects, so that we can subclass or delegate to it more easily. This will also help in part to address #639

Would you be able to help out?
👍

[stefsmeets]

One of the key challenges with esmvalcore.Recipe is that it is not designed to be run multiple times in the same process. In its current state, Recipe must be initialized with a user config.

A simple use case that we would want in the API (in #962): obtain a list of diagnostic or preprocessor tasks that can be run. At the moment, this required initializing Recipe.tasks , which is only possible with a config_user dict. In other words, to obtain a list of tasks, we must create the entire work directory. This seems a little bit excessive.

Ideally, we would decouple config_user entirely from the creation of tasks, and only update the paths / user parameters at the last possible moment. But this is hardly possible, because config_user is passed down all the way to the data finder.

I had a good discussion with @Peter9192 about this and our conclusion was that esmvalcore.Recipe should be initialized without config_user, and delay passing config_user to the latest moment. We came to the conclusion that a good first step would be to pass config_user only when running the recipe (i.e. recipe.run(cfg='config_user'). This means that a recipe can be parsed and initialized for the most part (provenance, datasets, scripts, etc), but the tasks would only be derived at the last moment.

[stefsmeets]

Had a good discussion with @bouweandela @nielsdrost and @Peter9192 about what the design of Recipe should be. Conclusion was that it is too much work to do right now, and to go a different way for #962