Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Empty cubes after volume_statistics for certain datasets #611

Closed
sloosvel opened this issue Apr 8, 2020 · 4 comments · Fixed by #618
Closed

Empty cubes after volume_statistics for certain datasets #611

sloosvel opened this issue Apr 8, 2020 · 4 comments · Fixed by #618
Assignees
Labels
bug Something isn't working preprocessor Related to the preprocessor

Comments

@sloosvel
Copy link
Contributor

sloosvel commented Apr 8, 2020

Describe the bug
CMIP6 EC-Earth ocean 3D data has the whole bottom depth layer set to masked values. Both the variables and the volcello. The volume_statistics preproc computes the weighted mean per layer just fine, and for the bottom layer the results are masked as well. But when it comes to computing the weighted mean over the water volume, the presence of masked values sets the results to NaNs and the data comes out of the preprocessing empty:

result.append(np.average(column, weights=depth_volume))

It looks like the function np.average is not handling well the presence of layers with only masked values. Would it be correct check and remove this sort of layers before calling np.average ?

Please attach
I'm working with EC-Earth local data and can't seem to find Ecearth data with volcello on /badc, sorry.

@sloosvel sloosvel added the preprocessor Related to the preprocessor label Apr 8, 2020
@valeriupredoi
Copy link
Contributor

this is more of a scientific question (well, statistical) - @ledm I believe is the person to ask - I would definitely replace it with numpy masked mean though 🍺

@ledm
Copy link
Contributor

ledm commented Apr 14, 2020

Replacing

result.append(np.average(column, weights=depth_volume))

with

result.append(np.ma.average(column, weights=depth_volume))

should fix this (Numpy.ma.average ). I don't believe that there's an option to add weights to numpy masked mean.

Although there might be another error if the volume includes masked values. I don't know whether numpy.ma.average is able to receive a masked array in the weights argument. Worth checking, it's not explicitly mentioned in the documentation.

@valeriupredoi
Copy link
Contributor

will you PR that @ledm ? 🍺

@valeriupredoi
Copy link
Contributor

#618 guys 🍺

@bouweandela bouweandela added the bug Something isn't working label May 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working preprocessor Related to the preprocessor
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants