Add support for downloading CMIP6 data with Synda

[bouweandela]

Downloading CMIP6 data with Synda did not work because the query is different from CMIP5. This pull request adds the correct query.

Tasks

• This pull request has a descriptive title that can be used in a changelog
• Add unit tests
• Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
• Codacy code quality checks pass. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.

[stefsmeets]

Hi, just wanted to chime in that I used this branch to successfully download some CMIP6 data with Synda for a recipe yesterday! 👍

[Peter9192]

Not a full review, but I tried to use this branch to download the data required for ESMValGroup/ESMValTool#1391, and found the following.

It works for most datasets, but I also get time-out errors and dataset not found problems, e.g.:

• No data found for GFDL-CM4 (dataset entry: dataset: GFDL-CM4, project: CMIP6, exp: historical, ensemble: r1i1p1f1, grid: gr2, start_year: 2000, end_year: 2005, with mip=Amon or day)
• HTTP query failed (url=http://esgf-data.dkrz.de/esg-search/search?project=CMIP5&fields=*&cmor_table=day&experiment=historical&limit=100&variable=vas&model=MPI-ESM-LR&type=File&ensemble=r1i1p1&format=application%2Fsolr%2Bjson&offset=0,exception=timed out,timeout=300)

Not sure if it's related to this PR, but I thought a real use case test could be relevant.

[bouweandela]

@valeriupredoi Would you have time to review this pull request, so it can still be included in v2.1?

It works for most datasets, but I also get time-out errors and dataset not found problems

@Peter9192 I'm afraid that's not something we can fix in the ESMValTool.

[jvegreg]

I am testing it with a cmip6-only recipe_ecs.yml. It will take a while...

By the way, I have some issues with my synda installation and the error messages were not helpful, as the stdout and stderr of the command are not shown. Do you want me to push this changes here or did I create another one to keep this focused?

Forget about this, sometimes I get the output, sometimes not. It will need a bit more investiggation to ahve something that looks well in all cases

[jvegreg]

It is working (almost) well, with the almost being fault of synda I think.

I find a case in which synda search finds a file but then can not download it. I think we already face this kind of things with CMIP5, but I am nor sure

[jvegreg]

Nice addition! I will create an issue to remind us that we may want to take a look in adding support for obs4mips and ana4mips