Improve loo functionality

[ahartikainen]

loo functionality should be more verbose and return more informative data. (see Model Selection lecture in StanCon Helsinki by @avehtari)

Also docstrings needs extensive examples for loo-usage.

[avehtari]

I'm in progress of adding more explanations to loo package about interpreting loo output: approx SE of elpd_diff, p_loo, k stan-dev/loo#81. I'll comment here when ready with those.

[avehtari]

Because of some confusing questions in discourse.mc-stan.org I found out that loo() help says

Calculates leave-one-out (LOO) cross-validation for out of sample predictive model fit, following Vehtari et al. (2015).

The year should be 2017 (http://link.springer.com/article/10.1007/s11222-016-9696-4), but more importantly in that paper log score is used. However the code computes

loo_lppd_i = -2 * _logsumexp(log_weights, axis=0)

that is, log score multiplied by -2.

As the lppd has a specific definition in Vehtari et al (2017) which doesn't include -2, I recommend to remove that -2. If you want to print out a value multiplied by .2, call it something else and make it clear that it's lppd multiplied by -2. This will make it much easier to answer questions whether one model is better than the other.

[avehtari]

I also noticed that waic() help refers to DIC paper by Spiegelhalter et al, and should refer to one of the Watanabe's papers. The code has again -2, but Watanabe didn't multiply by -2, so again if you want to keep -2, you have to be explicit that the function is returning something different than what is in the paper.

[ahartikainen]

Hmm, where does that -2 come from?

@aloctavodia @junpenglao

pymc-devs/pymc@90c7286#diff-e41ef58a3a4077bc00fca799f90d12a4

And this

pymc-devs/pymc@5d8a8c5#diff-e41ef58a3a4077bc00fca799f90d12a4

[ahartikainen]

So it is the deviance scale. I think we could let user to decide the scale or return all scales (deviance, log, neglog?)

[avehtari]

R loo package stores and shows elpd_loo and looic. elpd_loo is clear as it has the specific definition in Vehtari et al (2017). I'm not happy with looic as it's still not indicate whether it's -1 or -2, but it's been there for a while. (DIC is the only information criterion indicating the deviance in the name. Unfortunately the original definition of deviance makes really sense only in case of point estimates, and thus deviance scale is also a bit misleading name for -2 times the log score)

[avehtari]

There is additional documentation now in loo package. See the pull request stan-dev/loo#98 (review). The glossary will appear in loo web pages also after the next release in spring.