Add documentation for Accelerators #411
Conversation
|
Hi @sydneyvernon to get the docs to work you will need to add a link accelerator page into |
There was a problem hiding this comment.
Hi Sydney - this looks great!
Recently we have found that people much prefer seeing the API, and how-to-use, and graph performance above the math, so reordering is the main comment! Otherwise I have a couple of extra references and phrasing of the math introduction
docs/src/accelerators.md
Outdated
|
|
||
| ## Implementation in Ensemble Kalman Inversion Algorithm | ||
|
|
||
| EKI can be understood as an approximation of gradient descent ([Kovachki and Stuart 2019](https://iopscience.iop.org/article/10.1088/1361-6420/ab1c3a)). This fact inspired the implementation of Nesterov momentum-inspired accelerators in the EKP package. For an overview on the algorithm without modification, please see the documentation for Ensemble Kalman Inversion. |
There was a problem hiding this comment.
As a better story line for this paragraph, maybe say that
(1) Nesterov can be used to accelerate Gradient flows (e.g. Su et al. 2016, https://arxiv.org/abs/1503.01243)
(2) EKI approximates "a form of gradient flow". use reference https://arxiv.org/abs/2209.11371 to talk about this rather than Kovachki stuart.
(3) Kovachki stuart 2019 used a particle-based modification of the Nesterov acceleration and saw acceleration.
(4) We use this update to accelerate a variety of our processes
There was a problem hiding this comment.
Hi! I just pushed a modified version with this story line, but I am not confident that the section flows well. Please let me know if there's anything else I can do to make it clearer. Thanks!
|
|
||
| The `NesterovAccelerator` (shown in blue) has been found to produce the most consistent acceleration on this problem, as seen below. The `FirstOrderNesterovAccelerator` (shown in red) uses a momentum coefficient very similar to that of the `NesterovAccelerator`, and enjoys similar performance. The `ConstantNesterovAccelerator` (shown in green) is effective in this test case, but can be very unstable. These methods differ only in their momentum coefficient values, which are plotted on the right. Vanilla EKI is shown in black. The experiment is repeated 50 times; ribbons denote one standard error from the mean. | ||
|
|
||
| <img src="assets/momentumcoeffs.png" alt="EKI convergence for different momentum coefficients" width="300"/> <img src="assets/momentumcoeffs_values.png" alt="Coefficient values" width="300"/> |
There was a problem hiding this comment.
these do not display for me offline but I think they are linked correctly
No description provided.