Add institute name used on ESGF for CMIP5 CanAM4, CanCM4, and CanESM2

[bouweandela]

Description

This adds the institute name CCCMA used on ESGF for the CMIP5 datasets CanAM4, CanCM4, and CanESM2. Without this institute name, the tool is not able to find the files for these datasets on ESGF.

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Changes are backward compatible
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[schlunma]

Thanks for opening this PR @bouweandela! I am currently testing #1609 again and I am getting lots of errors due to this. Can you make this ready for review? I tested it and it worked fine 👍

Question on this: Why did this only occur now? Could this also happen for other CMIP5 datasets or are we sure that these models are the only case?

[bouweandela]

Why did this only occur now?

Prior to #1920 the value for institute was ignored when searching ESGF. When I originally implemented ESGF search, I ran into issues like this (i.e. files with wrong facet values), so I only selected a rather minimal set of facets to use. However, that doesn't play nice with the automatic finding of datasets.

Could this also happen for other CMIP5 datasets or are we sure that these models are the only case?

Yes, there are quite a few files on ESGF that have wrong facet values. However, this was the most obvious case. A simple way to check this is to use Dataset.from_files() with an * for as many facets as possible (e.g. only define project, mip, and short_name as not wildcard) and then checking if the tool can find the files for each of the datasets that come out.

[bouweandela]

I am currently testing #1609 again and I am getting lots of errors due to this. Can you make this ready for review? I tested it and it worked fine 👍

There are several tests failing because they use this dataset as an example. Those tests may have been changed in #1609, so I was planning to fix them only once #1609 is merged to avoid fixing outdated stuff. Would it work for you to just merge this into your testing branch for #1609 for now?

[schlunma]

I see, that makes sense. How exactly does the tool handle lists as additional facets? Is it simply checking for all possible combinations and is happy when it finds at least one file?

Would it work for you to just merge this into your testing branch for #1609 for now?

Yeah, that's what I already did.

[bouweandela]

How exactly does the tool handle lists as additional facets?

It will try to concatenate the data. In this case the institute CCCMA is used on ESGF and the institute CCCma is used on the local file system, so this shouldn't be an issue. For finding files, using a list will match any of the specified options, as described here and here.

[valeriupredoi]

thanks lots @bouweandela - as directly affected via Tool's #2994 and scratching my head interminably before you told me it was the blithering CanESM2, I fully embrace this arms open. Can I fix the tests for you so we get this ready-ready?

[bouweandela]

That's a great offer, thanks! Could you do if after #1609 is merged, to avoid fixing outdated things and creating merge conflicts?

[valeriupredoi]

That's a great offer, thanks! Could you do if after #1609 is merged, to avoid fixing outdated things and creating merge conflicts?

sounds like a plan Batman 🦇

[codecov]

Codecov Report

Merging #1937 (d0b9065) into main (aa4a8c5) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1937   +/-   ##
=======================================
  Coverage   92.60%   92.60%           
=======================================
  Files         236      236           
  Lines       12361    12361           
=======================================
  Hits        11447    11447           
  Misses        914      914           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[bouweandela]

Can I fix the tests for you so we get this ready-ready?

Just fixed it myself, there was only one test remaining that failed.

[valeriupredoi]

Can I fix the tests for you so we get this ready-ready?

Just fixed it myself, there was only one test remaining that failed.

there goes my chance for fame and glory then 😁

[valeriupredoi]

cheers @bouweandela - but do we still have tests with CCCma and CCCMA?

[bouweandela]

No, and I don't think we need those. We have tests that adding extra facets works correctly. The only reason that CanESM2 featured in many examples is that it's early in the alphabet. Hopefully, we can just get this issue corrected on ESGF (did not receive a reply to my email asking how to go about that so far) and then we no longer need the extra value in all uppercase.

[valeriupredoi]

sounds good to me, man! OK - I can merge this now if you all good 👍 Let me go do make some noise myself towards ESGF after a few more days have passed

[schlunma]

Looks good to me! Please don't forget to add a lable and tick/remove all boxes in the PR description.