Futher Algorithms - CVB

[andrewcz]

Hi @VietTran86 !

Really like the repo! would like to use it for further university work.
Was just wondering if other examples are planned to be added.

Many thanks,
Best,
Andrew

[VietTran86]

Yes definitely. Thank you so much for your interest!

As mentioned in Readme above, there are hundreds/thousands of models that we
can apply CVB and make breakthrough algorithms.

For examples:

1. Currently Gaussian Markov Model (GMM) or HMM is trained via EM algorithm. We can use CVB instead, since CVB is far superior to EM.

2. Overfitting: currently people use AIC or cross-validation to avoid overfitting, but it is only useful asymptotically. By using CVB, we can estimate the unknown range/variance of possible values of groundtruth and avoid overfitting very quickly with each coming data (c.f. my latest paper Bayesian PCA on my Github)

3. dropout in Neural Net is simply an application of VB algorithm. If we use CVB, it will be much better

etc... Each application will be a breakthrough paper (I can bet my whole career on this sentence, because it is indeed my career : )

This paper is an intersection of three big fields (Bayesian, information theory, copula in financial statistics). My hunch is that it may create a new field of research (e.g. to use copula for generating neural networks like GAN). With copula/conditional structures, we can build a true AI with optimally adaptive structure, not heuristic like current GAN.

My current focus is to replace the Stochastic Gradient Decent (SGD) in Neural Net with CVB. Some people already did it via EM algorithm, but EM is not good enough to surpass SGD. It’s likely that CVB will work better than SGD and/or message-passing, since CVB is a guaranteed optimal method, not heuristic.

Bayesian Neuralnet will replace current Neuralnet in a few years. CVB is just as big as NeuralNet, because currently CVB is the only way that we can use to produce a closed-form inference in linear time for Bayesian machine learning systems, particularly for very fast signal processing systems like:

• online sensor/cloud networks etc.
• or offline smartphones/offline smart speakers, etc.
• or very high dimensional data like Healthcare systems etc.
  (MCMC is truly great but it’s too slow).

Please feel free to come back for discussion any time. I really appreciate it xD

Best,
Viet Hung

p.s: I don’t mean CVB will replace Neural Net or anything. Neural Net is really useful, hands down. I merely want to absorb NeuralNet and make it better, that’s all about it xD

[andrewcz]

Amazing!! super interesting!
so more examples will be developed over time.
I'm going to try and have a look at the code and convert it to python and try and build some examples.
Best,
Andrew

[VietTran86]

Cool! I wish that I have time to convert it to python also, but I am currently occupied with Bayesian neural networks : (

Please feel free to comment here. I will assist your work as best as I can!

Best,
Viet