Step-size methods and preconditoners for GD

[MargaretDuff]

Currently a draft for discussion purposes

TODO:

• testing the preconditioners
• Unit tests for the preconditioners
• Documentation for the preconditioners
• Unit tests for the step-size methods
• Think about efficiency for the Armijo rule
• Think about efficiency for safe division in preconditioners
• Documentation for the step-size rules
• Testing for ISTA and FISTA with step sizes and preconditioners
• Documentation for ISTA and FISTA
• Think of a new name for the preconditioner class - it does more than precondition, it can modify the gradient (or anything else) in any way it wants

Changes

Added step-size methods and preconditioners for GD

Testing you performed

Please add any demo scripts to https://github.com/TomographicImaging/CIL-Demos/tree/main/misc

Related issues/links

• Closes Add options for step-size rules and pre-conditioners to the GD algorithm  #1762,
• closes StepSize Class #1592,
• closes Preconditioner #1588
• closes Add preconditioning in GD #1359
• closes Lipschitz constant for Constant Functions #1817

Checklist

• I have performed a self-review of my code
• I have added docstrings in line with the guidance in the developer guide
• I have updated the relevant documentation
• I have implemented unit tests that cover any new or modified functionality
• CHANGELOG.md has been updated with any functionality change
• Request review from all relevant developers
• Change pull request label to 'Waiting for review'

Contribution Notes

Please read and adhere to the developer guide and local patterns and conventions.

• The content of this Pull Request (the Contribution) is intentionally submitted for inclusion in CIL (the Work) under the terms and conditions of the Apache-2.0 License
• I confirm that the contribution does not violate any intellectual property rights of third parties

--->

[paskino]

I suggest to remove a couple of else statements, but other than that it looks good to me|!

[MargaretDuff]

Closes #1817

[epapoutsellis]

Approved, we only need to test #1818 for CIL+SIRF (precond+step_sizes) but I am almost certain that it will work. Once merged, please remind me again and we can add this test in test_SIRF

[MargaretDuff]

Jenkins is happy

[KrisThielemans]

Thanks a lot all! Yet another major milestone