Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix severe bug in mask_fillvalues #1823

Merged
merged 5 commits into from
Nov 25, 2022
Merged

Fix severe bug in mask_fillvalues #1823

merged 5 commits into from
Nov 25, 2022

Conversation

schlunma
Copy link
Contributor

@schlunma schlunma commented Nov 24, 2022

Description

This PR fixes a severe bug which lead to a wrong masking of data with mask_fillvalues if the data contains zeros. This has already been discussed 2 years ago and has recently lead to problems.

I also fixed some prospector issues along the way.

Related to #487
Closes #1827


Before you get started

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.


To help with the number pull requests:

@schlunma schlunma added the bug Something isn't working label Nov 24, 2022
@schlunma schlunma added this to the v2.8.0 milestone Nov 24, 2022
@schlunma schlunma self-assigned this Nov 24, 2022
@@ -377,7 +377,7 @@ def count_spells(data, threshold, axis, spell_length):
axis += data.ndim
# Threshold the data to find the 'significant' points.
if not threshold:
data_hits = data
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line lead to the buggy behavior: Since data_hits is interpreted as bool later in the code a 0 in data leads to (undesirable) False in the data and thus to wrong masking.

@codecov
Copy link

codecov bot commented Nov 24, 2022

Codecov Report

Merging #1823 (cd347cb) into main (63267b6) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1823   +/-   ##
=======================================
  Coverage   91.51%   91.52%           
=======================================
  Files         203      203           
  Lines       10937    10937           
=======================================
+ Hits        10009    10010    +1     
+ Misses        928      927    -1     
Impacted Files Coverage Δ
esmvalcore/preprocessor/_mask.py 85.89% <100.00%> (+0.42%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Copy link
Contributor

@axel-lauer axel-lauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this bug fix and it works! Without the fix, values that are zero (0.0) are wrongly masked as missing values, see example of observed precipitation here (note e.g. parts of Antarctica):

main

With the fix, the zeros are kept as valid points, just as it should be:

fix

Here is the test recipe used: recipe_masking_test.yml.txt

Thanks @schlunma for fixing this long-standing bug!

Copy link
Contributor

@valeriupredoi valeriupredoi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what an ancient bug! Cheers for spotting this @schlunma and cheers for fixing it 🍺

@schlunma
Copy link
Contributor Author

@ESMValGroup/esmvaltool-coreteam anyone time to merge this? Thanks!

@bouweandela
Copy link
Member

@schlunma Thanks for fixing this! A severe bug deserves an issue, could you please open it? That will make it much easier to find this problem and that it has been solved for anyone who encounters it. I will merge this now, so you can immediately close it again with a reference to this pull request.

@bouweandela bouweandela merged commit 44505c5 into main Nov 25, 2022
@bouweandela bouweandela deleted the fix_mask_fillvalues branch November 25, 2022 16:04
@valeriupredoi
Copy link
Contributor

@schlunma Thanks for fixing this! A severe bug deserves an issue, could you please open it? That will make it much easier to find this problem and that it has been solved for anyone who encounters it. I will merge this now, so you can immediately close it again with a reference to this pull request.

Good point, Bouwe! I wouldn't call it a severe bug myself, but heyho. More of an old, annoying horsefly-like bug 😁

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

mask_fillvalues may wrongly mask values which are 0.0.
4 participants