Optimisers for Neural Nework

---

PyGAD

• PyGAD is a genetic algorithm Python 3 library for solving optimization problems. What makes this package interesting with respect to the others is that they cretaed a nice wrapper around both Keras and PyTorch.
• Train PyTorch Models Using Genetic Algorithm With PyGAD
• Genetic Algorithm: Tutorial of PyGAD - Kaggle

---

Limitation of stachastic gradient methods

• The "Terpret Problem" and the limits of SGD
• sklearn.neural_network.MLPRegressor¶, allows you to use both quasi-Newton and stochastic gradient descent

---

Available tutorials

• Scipy optimize wrapper for Keras
• Example where a SGD would fail to find the optimal solution

---