Train a simple neural network using non-linear least squares algorithms (Levenberg-Marquardt), and test them to approximate certain non-linear functions.
A neural network is widely used parametric model to approximate non-linear functions. These networks are parameterized by a set of weights. Neural networks are often referred to as “universal approximators", which means that they can approximate a large class of non-linear functions by suitably tuning the weights. Here I programmed a simple neural network to approximate different types of non-linear functions using “non-linear" least squares algorithms.
A neural network is a model of the form
I used sigmoid as our psi function.
The non-linear function to approximate is
and
I used Levenberg-Marquardt algorithm to determine the weighhts for approximating different non-linear functions.
The loss function is defined as
where r is simply the difference between our prediction fw, and the ground-truth values y.