Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add percentiles to multi-model stats #679

Merged
merged 11 commits into from
Jul 1, 2020
Merged

Conversation

Peter9192
Copy link
Contributor

@Peter9192 Peter9192 commented Jun 17, 2020

Add the option to calculate percentiles in the multi-model preprocessor, using scipy.stats.mstats.mquantiles. This is the same function as used in Iris.

It works by adding e.g. p10 or p99 to the list of operators in the recipe. p99.5 will be interpreted as the 99.5 percentile, i.e. the 0.995 quantile, but the decimal point will be changed to a dash - in the output filename.

Unfortunately, the scipy function doesn't work directly on multidimensional arrays. Thus, I added a wrapper that flattens the extra dimensions before passing the array to scipy, similar to what's done in Iris.

Tasks

  • Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
  • This pull request has a descriptive title that can be used in a changelog
  • Add unit tests
  • Public functions should have a numpy-style docstring so they appear properly in the API documentation. For all other functions a one line docstring is sufficient.
  • If writing a new/modified preprocessor function, please update the documentation
  • Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
  • Codacy code quality checks pass. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.

If you need help with any of the tasks above, please do not hesitate to ask by commenting in the issue or pull request.


Closes #563

@Peter9192 Peter9192 changed the title Add quantiles to multi-model statistics preprocessor Add percentiles to multi-model statistics preprocessor Jun 17, 2020
@Peter9192 Peter9192 changed the title Add percentiles to multi-model statistics preprocessor Add percentiles to multi-model stats Jun 17, 2020
@Peter9192 Peter9192 marked this pull request as ready for review June 17, 2020 15:05
Copy link
Contributor

@valeriupredoi valeriupredoi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wee comment there, mate. All good! Cheers

Copy link
Member

@bouweandela bouweandela left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great addition @Peter9192! Just a few suggestions for improvement

@mattiarighi mattiarighi added the preprocessor Related to the preprocessor label Jun 22, 2020
@Peter9192
Copy link
Contributor Author

Thanks for the suggestions @bouweandela! I think I got them all

@bouweandela bouweandela merged commit c4aaae2 into master Jul 1, 2020
@bouweandela bouweandela deleted the mmstats_add_percentiles branch July 1, 2020 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
preprocessor Related to the preprocessor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

multi model statistics -- more statistics
4 participants