BPCG with direct solve extension

[pokutta]

adds a new option to BPCG to directly solve local optimality system as LP if the objective function is of the form f(x) = || x - x_0||^2, i.e., we are solving projection problems. Important for e.g., Sébastien etc. Also, using the same methodology added a sparsify_iterate function that sparsifies the active_set and iterate by solving the representation LP and taking a basis.

Update: added a generic direct_solve method for any quadratic function exploiting chain rule + affine-linearity + convex combinations to build optimality system with black-box access to gradients only. However, slower than the one specific for \ell-2projections; see plots below.

TODO:

• adjust documentation
• add unit tests
• solve the stack overflow issue (@matbesancon this seems to be a julia issue or recursion issue. fixed with loops for now)
• Parametrize the direct solve intervals
• Add reporting of step type
• Add same methodology to sparsify last iterate
• Move functions direct_solve and sparsify_iterate out of blended_pairwise.jl into e.g., utils.jl
• Decide whether to propagate, e.g., sparsify to PP for all active set based methods

Speed-ups are very substantial in some cases. See plots below (LP solver was HiGHS):

General Quadratics:

[pokutta]

@matbesancon there is a weird issue with a stack overflow though that seems to come from julia, e.g., in the example if you set n = 1e4. I guess it happens in the direct_solve function when doing the hcat. maybe there is a better way to do it.

[pokutta]

@sebastiendesignolle might be also relevant or useful for you to test for bellpolytopes.jl. This is an advanced version of what I explained this morning

[pokutta]

@matbesancon there is a weird issue with a stack overflow though that seems to come from julia, e.g., in the example if you set n = 1e4. I guess it happens in the direct_solve function when doing the hcat. maybe there is a better way to do it.

seems to be resolved now. I unwrapped the loops

[matbesancon]

Preview will be at:
https://zib-iol.github.io/FrankWolfe.jl/previews/PR507/

[pokutta]

@matbesancon ready for review now. feel free to add unit tests and please check the documentation; didn't show up in the preview.