By Michal Rolínek, Georg Martius.
Autonomous Learning Group, Max Planck Institute for Intelligent Systems.
This repository contains TensorFlow implementation code for the paper "L4: Practical loss-based stepsize adaptation for deep learning". This work proposes an explicit rule for stepsize adaptation on top of existing optimizers such as Adam or momentum SGD.
Disclaimer: This code is a PROTOTYPE and most likely contains bugs. It should work with most Tensorflow models but most likely it doesn't comply with TensorFlow production code standards. Use at your own risk.
TensorFlow >= 1.4.
For a PyTorch implementation of L4, see l4-pytorch (also linked here as a submodule).
Either use one of the following python pip commands,
python -m pip install git+https://github.com/martius-lab/l4-optimizer
python3 -m pip install git+https://github.com/martius-lab/l4-optimizer
or simply drop the L4/L4.py file to your project directory
Exaclty as you would expect from a TensorFlow optimizer. Empirically, good values for the 'fraction' parameter are 0.1 < fraction < 0.3, where 0.15 is set default and should work well enough in most cases. Decreasing 'fraction' is typically a good idea in case of a small batch size or more generally for very little signal in the gradients. Too high values of 'fraction' behave similarly to too high learning rates (i.e. divergence or very early plateauing).
import L4
...
opt = L4.L4Adam(fraction=0.20)
opt.minimize(loss)
...or
import L4
...
opt = L4.L4Mom() # default value fraction=0.15 is used
grads_and_vars = opt.compute_gradients(loss)
...
# Gradient manipulation
...
opt.apply_gradients(grads_and_vars) # (!) Passing the loss is no longer needed (!)
...Contribute: If you spot a bug or some incompatibility, contribute via a pull request! Thank you!