Fix masking of surface climatologies

[xylar]

What should be fill values were coming through as zeros. A temporary fix is to use validMask to assign fill values.

closes #860

[pep8speaks]

Hello @xylar! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-02-08 00:59:18 UTC

[xylar]

Testing

I'm still running the full suite on Anvil and will update here. For now, the results look right in a smaller test.

Old stats and masking:

New stats and masking:

Update:
I ran the full analysis suite on Anvil, and things look good:
https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.xylar/analysis_testing/anvil/fix_sst_masking/
The main vs. control (model vs. model) run hasn't completed yet but a few sample plots look fine.

[xylar]

I realize looking at the observational stats that the fix is only a partial one. We need to mask the obs. in the same places as the MPAS mesh in order for this the stats to make sense.

[xylar]

With @milenaveneziani's blessing, I'm computing obs. stats masked by the MPAS-Ocean land mask.

[xylar]

Starting my testing over...

[xylar]

Okay, I think we're good now. Here's the results:

I just need to make a new release of pyreamp that brings in the necessary additional flag.

[milenaveneziani]

Looks really nice @xylar.

[xylar]

Hmm, it's still may not be working as expected in some cases:

Here's what's going on. The MPAS mesh is valid in a lot of places where the obs are not. Thus, the mean on the MPAS mesh is really different than the mean on the MPAS mesh where both the obs and the MPAS data are valid.

@milenaveneziani and @golaz, what do we actually want in such cases? Should we restrict the stats on the MPAS data to where both MPAS and obs are valid? Or live with the discrepancy?

[xylar]

@golaz, this evening, I'll make a new E3SM-Unified test environment (1.6.0rc8) on Cori-Haswell for you to try out. If that works on the runs that were giving you trouble, I'll release MPAS-Analysis 1.6.1 and then E3SM-Unified 1.6.0.

[xylar]

@golaz, I have set up a (final?!?!) release candidate of E3SM-Unified on Cori-Haswell:

source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.6.0rc8_cori-haswell.sh

Would it be easy for you to retest one of your problem runs where you saw the results in #860 sometime tomorrow? As soon as I get your go-ahead, I'll merge this, release MPAS-Analysis 1.6.1 and deploy E3SM-Unified 1.6.0.

[milenaveneziani]

@xylar : I am less worried about the SSH stats because, to be honest, I don't look at them much.. I mostly look at the patterns of the model and obs SSH plots, and at the sign and pattern of the bias.
I can take a look at a full climo analysis from this change and comment more. Hopefully the issue is just with SSH.

[xylar]

Hopefully the issue is just with SSH.

Thanks @milenaveneziani. The other ones I noticed were mixed layer depth plots because there are just a lot of "holes" in the observations, particularly in the polar regions. Here's the link to the test suite results:
https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.xylar/analysis_testing/anvil/fix_sst_masking/

[golaz]

Thanks for identifying the problem so quickly. Please go ahead. Merge and incorporate in the next release. I'll start using it at that point.

[xylar]

@golaz said he will rerun his analysis once this is in the new E3SM-Unified.