Refactor approximation methods

[bvdmitri]

Approximation methods section is a bit outdated in comparison to other parts of the package and has been implemented long ago. It need revision, especially methods like GaussHermite, GaussLaguerre, approximate_meancov and others. We need to come up with a similar interface for all approximation methods or maybe just exploit another package?

Approximation methods and their code is located under src/approximations/ folder:

We use it here:
https://github.com/biaslab/ReactiveMP.jl/blob/c8fab6b4d78f704ab3ba1311005ce6204efdda15/src/nodes/probit.jl#L34
https://github.com/biaslab/ReactiveMP.jl/blob/c7e60f907393bd732933ffb1503af88a38e2e0b0/src/distributions/exp_linear_quadratic.jl#L18
https://github.com/biaslab/ReactiveMP.jl/blob/c7e60f907393bd732933ffb1503af88a38e2e0b0/src/nodes/kernel_gcv.jl#L26
and maybe somewhere else..

[bvdmitri]

We can create a separate package for this purpose, but we need to come up with convenient user-friendly API. Some notes from our private discussion in Slack:

We have quite some approximation methods in both ForneyLab.jl and ReactiveMP.jl (approximations folder) which are unstructured. Users tend just to copy-paste them in their notebooks and make slight modifications for their particular use-cases. It might be improved, but the API should be flexible to support as many as possible use-cases.

Our suggestion is that the top-level API looks smth like:

approximate(...) = approximate_as(Gaussian, ...)

approximate_as(::Type{T}, ::ApproximationMethod, ::Distribution, ::Function) where { T } = ...

• where ::ApproximationMethod is, for example, UnscentedTransform() or GaussLaguerre().
• where ::Distribution is any object that we can sample from or evaluate logpdf
• where ::Function is any callable object
• the resulting approximation is of type T

Most probably by default the result of any approximation is a Gaussian distribution. This can be controlled by a user with e.g. approximate_as(Gamma, ...).

What ::Functions are possible?

Should it return values from the domain of the input ::Distribution? Or should it return just numbers (aka. weights) even in a multivariate case? Or maybe both are possible? What range of functions do we expect here?

Exp vs Log

Ideally for robustness we should perform approximations routines in log-domain. However it is difficult to say now if all approximation methods are straightforward to implement in log-domain. Should we give a user an option? Or should it be automatic? E.g. if method supports log-domain approximation then use log-domain.

Side feature

We may want to control the return values as well. E.g. user might be not interested in cov of the resulting approximation and only in mean (in case of a Gaussian). We may want to support smth like MeanOnly() or MeanCov(). Or approximate_as(GaussianMeanOnly). This needs to be discussed.

Third-party solutions:

Are there any packages that do the same or at least close to what we want?

[albertpod]

Let's close this one as refactoring has happened - a separate package is created. This issue doesn't make much sense to me any longer.