Add warning about usability of run command line flags

[bouweandela]

The command line flags of the esmvaltool run command are quite unreliable as they were implemented to be able to quickly run a recipe for debugging and are not intended for use when creating figures for publications.

The following command line options are quite unreliable (i.e. it depends on the recipe and availability of data whether they will work as expected or not):

--max_datasets
--max_years
--skip_nonexistent
--synda_download
--diagnostics
--check_level

It would be good to add a warning about this, and possibly even move all of these options away from the esmvaltool run command to e.g. an esmvaltool debug command.

[stefsmeets]

So how should we treat the corresponding variables in the code?

Several parts of the code depend on these variables existing, even crashing if they do not exist. So for #907 and #868 I am treating them as first class citizens. Since the API adds a different entry point into the code, I think we should at least try to define them in a central place (i.e. not during CLI initialization).

[bouweandela]

So how should we treat the corresponding variables in the code?

The easiest would probably be by providing a good default value, as is done now. This ensures that the value is present where needed in the code.

The problem with

--max_datasets
--max_years
--skip_nonexistent

is that you can never be sure that you're not messing up the input that the diagnostic script expects.

Changing

--check_level

to a lower strictness level is unsafe and should never be done for publication figures.

--synda_download

does not download anything if part of the expected input data is available. This should probably be fixed, but may be better done as part of #31 or at the very least using esgf-python instead of synda, because synda is hard to install and configure for users.

--diagnostics

has the problem that it still expects all input data to be there. Note that this was done on purpose, to avoid broken recipes from getting merged because the people testing those recipes might only test some of the diagnostics in it if this is possible. We may want a smarter solution for this though, but it will also be quite a bit of work to implement.

[bouweandela]

Note that the --synda_download option has been removed in favour of the --offline=False option in #1130 and that the problem with --diagnostics should be fixed by #1264.

[SarahAlidoost]

Note that the --synda_download option has been removed in favour of the --offline=False option in #1130 and that the problem with --diagnostics should be fixed by #1264.

I noticed that the documentation has not been updated, I added it to this issue.