Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add warning for log scale default in compare/loo/waic functions #1150

Merged
merged 7 commits into from
Apr 16, 2020

Conversation

AlexAndorra
Copy link
Contributor

Description

This PR adds a reminder about log scale default and how it should be interpreted (higher is better) for az.waic and az.loo. This merely mirrors what we already say in the doc.

For az.compare I added a UserWarning following the change to log scale in 0.7.0. This adresses this issue, and the warning will be temporary, as discussed in the issue.

In passing, I trailed all the white spaces before colons in the docstrings of stats.py. This messes up the formatting on the website.

I'm here for any change, and thanks in advance for the review 🖖

Copy link
Member

@OriolAbril OriolAbril left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally like the space before the colon, and it looks like it is common formatting (see pandas.read_csv), but I would not oppose to changing the format

arviz/stats/stats_utils.py Outdated Show resolved Hide resolved
@AlexAndorra
Copy link
Contributor Author

Yeah, but weirdly it also take the argument's type when double-clicking it. When you remove the white space, it just selects the parameter, as it should -- don't know why 🤷

Thanks for the review Oriol, I updated in light of your comments.

@OriolAbril
Copy link
Member

Yeah, but weirdly it also take the argument's type when double-clicking it. When you remove the white space, it just selects the parameter, as it should -- don't know why shrug

Ohh, true, I hadn't noticed, and it looks like it happens with any documentation with space before the colon 🤔

@AlexAndorra
Copy link
Contributor Author

Yeah that's weird 😅

Do we still need assert loo_data == loo_pointwise[: len(loo_data)] in test_loo_print? It's this line that messes up the tests, because it's no longer the case.

@ahartikainen
Copy link
Contributor

I think we don't need it. What is it now?

@AlexAndorra
Copy link
Contributor Author

Now it includes the warning I introduced in this PR:

Computed from 2000 by 8 log-likelihood matrix
         Estimate       SE
elpd_loo   -30.81     1.43
p_loo        0.95        -
The scale is now log by default. Use 'scale' argument if you rely on a specific value. 
A higher log-score (or a lower deviance) indicates a model with better predictive 
accuracy.

when default,

Computed from 2000 by 8 log-likelihood matrix
         Estimate       SE
elpd_loo   -30.81     1.43
p_loo        0.95        -
------
Pareto k diagnostic values:
                         Count   Pct.
(-Inf, 0.5]   (good)        6   75.0%
 (0.5, 0.7]   (ok)          2   25.0%
   (0.7, 1]   (bad)         0    0.0%
   (1, Inf)   (very bad)    0    0.0%
The scale is now log by default. Use 'scale' argument if you rely on a specific value. 
A higher log-score (or a lower deviance) indicates a model with better predictive 
accuracy.

when pointwise.
Which explains why the test is failing.

@OriolAbril
Copy link
Member

It can be removed, black is also complaining, make sure to run black right before commiting

@AlexAndorra
Copy link
Contributor Author

Done! Thanks guys 👌

@codecov
Copy link

codecov bot commented Apr 14, 2020

Codecov Report

Merging #1150 into master will increase coverage by 0.13%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1150      +/-   ##
==========================================
+ Coverage   92.99%   93.12%   +0.13%     
==========================================
  Files          94       94              
  Lines        9242     9286      +44     
==========================================
+ Hits         8595     8648      +53     
+ Misses        647      638       -9     
Impacted Files Coverage Δ
arviz/stats/stats.py 96.33% <100.00%> (+<0.01%) ⬆️
arviz/stats/stats_utils.py 91.72% <100.00%> (+0.05%) ⬆️
arviz/data/io_numpyro.py 95.23% <0.00%> (+3.64%) ⬆️
arviz/data/io_dict.py 89.89% <0.00%> (+7.24%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1e1a2a...1272225. Read the comment docs.

Copy link
Member

@OriolAbril OriolAbril left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very minor comments. it should also be added to the changelog and will be ready to merge

arviz/stats/stats.py Outdated Show resolved Hide resolved
arviz/stats/stats_utils.py Outdated Show resolved Hide resolved
@AlexAndorra
Copy link
Contributor Author

Thanks Oriol! It's now updated -- and I had already added it to the changelog 👌

@OriolAbril OriolAbril merged commit 5e3dcc3 into arviz-devs:master Apr 16, 2020
nitishp25 added a commit to nitishp25/arviz that referenced this pull request Apr 27, 2020
* added groups

* update tests and changelog

* lint changes

* update from_dict and tests

* update changelog

* modify io_dict

* minor fixes

* update changelog again

Add warning for log scale default in compare/loo/waic functions (arviz-devs#1150)

* Added scale warning to ELPDData

* Added scale warning to compare function

* Added changes to Changelog

* Moved warning to end of string in loo function

* Removed last test for loo_print

* Ran Black

* Integrated Oriol's comments

add local to docstring, and use lowercase for ess (arviz-devs#1152)

hardcode show=False in plot_posterior subplots (arviz-devs#1151)

* hardcode show=False in plot_posterior subplots

* update changelog

Fix documentation and deprecation warning in pair plot (arviz-devs#1156)

The "kind" argument is not described correctly.

* Fix pairplot warning.

Previously, because of incorrect argument checking, pairplot would
mistakenly warn the caller not to use the "contour" argument when that
argument was NOT supplied.  Changed default value to None and did the
defaulting by hand to fix this issue.

Also added some type declarations.

* Clarified docstring for contour argument.

Co-authored-by: Robert P. Goldman <rpgoldman@sift.net>

add viridis as default cmap (arviz-devs#1160)

add examples to customize 2D KDE (arviz-devs#1158)

* add examples to customize 2D KDE

* blackify gallery example

* update changelog

* briefly explain contour_kwargs and contourf_kwargs
@AlexAndorra AlexAndorra deleted the add-scale-warning branch May 13, 2020 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants