Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preserve metadata during anomalies computation when using iris cubes difference #652

Merged
merged 8 commits into from
Jun 5, 2020

Conversation

valeriupredoi
Copy link
Contributor

@valeriupredoi valeriupredoi commented May 26, 2020

Before you start, please read CONTRIBUTING.md.

Tasks

  • Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
  • This pull request has a descriptive title that can be used in a changelog
  • Add unit tests
  • Public functions should have a numpy-style docstring so they appear properly in the API documentation. For all other functions a one line docstring is sufficient.
  • If writing a new/modified preprocessor function, please update the documentation
  • Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
  • Codacy code quality checks pass. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
  • Please use yamllint to check that your YAML files do not contain mistakes
  • If you make backward incompatible changes to the recipe format, make a new pull request in the ESMValTool repository and add the link below

If you need help with any of the tasks above, please do not hesitate to ask by commenting in the issue or pull request.


Closes #651

@valeriupredoi valeriupredoi added bug Something isn't working preprocessor Related to the preprocessor labels May 26, 2020
@valeriupredoi
Copy link
Contributor Author

Note on implementation: cube.core_data = cube.core_data() - reference.core_data(), albeit preferred to the proletarian copy(), is not practical because the iris difference actually performs a nice cube broadcasting, so that would have to be emulated here -> too much fuss and useless code 🍺

@Peter9192
Copy link
Contributor

Wow, that was fast, thanks! I'll try to it today or tomorrow.

Copy link
Contributor

@Peter9192 Peter9192 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can confirm that this fix solves my problem. Thanks a lot for the quick reply!

Comment on lines 1031 to 1032
assert result.var_name == cube.var_name
assert result.units == cube.units
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we can use

Suggested change
assert result.var_name == cube.var_name
assert result.units == cube.units
assert result.xml() == cube.xml()

to verify that all metadata matches. That seems to pass for my use case, and it's more complete. For example, one of the subsequent problems I got with this bugs is that the time bounds were removed. These are included in the xml.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that'll not work, there are differences in the xml's that got nothing to do with the actual metadata; I have enhanced the test to look for identical bounds and identical coord objects

valeriupredoi and others added 2 commits May 27, 2020 17:06
Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>
Comment on lines +1032 to +1035
for coord_cube, coord_res in zip(cube.coords(), result.coords()):
if coord_cube.has_bounds() and coord_res.has_bounds():
assert_array_equal(coord_cube.bounds, coord_res.bounds)
assert coord_cube == coord_res
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this is needed, because you're testing that the metadata was preserved, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the metadata object does not contain information on the coords, I was quite surprised by that too

cube.units = "m"
result = anomalies(cube, period, reference, standardize=standardize)
assert result.metadata == cube.metadata
for coord_cube, coord_res in zip(cube.coords(), result.coords()):
Copy link
Member

@bouweandela bouweandela May 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that if one array is longer than the other, zip will only iterate over the number of elements in the shortest iterable, so this does not test that the cubes have the same coordinates

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point actually! but the cubes should have the same number of coords otherwise the iris cube difference would fail beforehand

valeriupredoi and others added 2 commits May 28, 2020 12:15
Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>
Copy link
Member

@bouweandela bouweandela left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattiarighi Could you please test?

@mattiarighi mattiarighi merged commit 5016e84 into master Jun 5, 2020
@mattiarighi mattiarighi deleted the preserve_metadata_anomalies branch June 5, 2020 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working preprocessor Related to the preprocessor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Anomalies preprocessor with period 'full' removes important metadata from cube
4 participants