Calculate Q matrix using Chinese Remainder Theorem, FLINT and BLAS libraries

[vasdommes]

TODO:

(NB: Do not merge to the 2.6 release!)

• Unit tests
• Benchmark for realistic problems
• Fix block timing estimates (see write_timing.cxx). Currently we cannot use run.step.initializeSchurComplementSolver.Q.syrk_XXX time, because all cores on the node are working together on their blocks.
• Better load balancing for BLAS jobs (simple round-robin algorithm works bad if procs_per_node greatly exceeds num_primes)
• bigint-syrk-blas: account for MPI shared memory limits #207: use smaller shared memory windows for P and Q (split P horizontally and split Q into blocks).
• Mark --procGranularity option as obsolete #210
• Add FLINT and BLAS to installation instructions #213.

Desciption

We calculate Q matrix using Chinese Remainder Theorem (implemented by FLINT library) and BLAS routines. The same idea is used in FLINT libarary for bigint matrices, see mul_blas.c. We reuse some FLINT code for our purposes.

The details and implementation are discussed in https://github.com/davidsd/sdpb/blob/master/src/sdp_solve/SDP_Solver/run/bigint_syrk/Readme.md

Brief description:

Suppose that we want to calculate Q = P^T P, where P is a BigFloat matrix. Instead of (slow) BigFloat arithmetic, we can proceed as follows:

• Convert P to integer matrix: normalize columns of P, multiply all by 2^N (N is BigFloat precision).
• Calculate residues of P modulo a set of (small enough) primes.
• Calculate squares of these residue matrices, Q_i = P_i^T P_i, using BLAS, namely cblas_dsyrk() function.
• Restore Q from residues Q_i using Chinese Remainder Theorem, remove normalization.

Since BLAS is highly optimized, this method proves to be much faster than BigFloat arithmetics: according to our preliminary (and rather imprecise) benchmarks, bigint_syrk_blas() is faster than El::Syrk() by a factor of ~10-20x.
It also allows to reduce RAM usage (only one copy of Q is stored on each node) and remove unnecessary --procGranularity option from SDPB.

[vasdommes]

I performed timing runs for both old and new algorithm on Expanse cluster, for GNY model.
Discussion in Slack: https://bootstrapcollab.slack.com/archives/CG450R9QX/p1699498297583039

First two plots show duration of one SDPB solver step for old and new algorithm.

• First plot: nmax=8, 1 node, 128 cores. With the new algorithm, SDPB is 1.5x faster.
• Second plot: nmax=14. With the new algorithm, SDPB is 1.8x faster for 3 nodes (384 cores). Old algorithm runs out of memory on 2 nodes, while the new one works and makes SDPB 2x faster than the old SDPB running on 3 nodes.

Note that these numbers reflect overall speedup for the program.
If we compare matrix multiplication itself, we can see that BLAS calls are up to 10x faster than El::Syrk from the old algorithm (e.g. nmax=14 on 3 nodes: El::Syrk takes 62s, BLAS calls - 6.5s).

Different parts of solver step shown in the plot:
Compute local Q (old algorithm): each process calculates its local Q_i.
Compute Q on node (new algorithm): all processes on a node work together to calculate their contribution to Q.
New algorithm turns out to be faster, thanks to BLAS optimizations.
Reduce-scatter Q: accumulate contributions to Q from all processes/all nodes.
Here new algorithm is faster too, since it has only on copy of Q per node instead of one copy of Q per process.
For a single node, it doesn't need reduce-scatter is required at all, since we already have Q calculated from shared memory window!
For several nodes, synchronization is still needed, but becomes faster as well.

• The third plot shows different steps of Compute Q on node: normalize, compute residues, call BLAS etc., for nmax=8 nodes=1, nmax=14 nodes=2, nmax=14 nodes=3.

For nmax=8, "normalize P" takes more time than BLAS. But this time is mostly spent not on calculating column norms, but on waiting for other MPI processes (note that this step requires global synchronization). This is not surprising, since we cannot provide perfect load balancing for a small problem (190 SDP blocks) on a large number of cores (128).
For larger problems (nmax=14), BLAS is the longest step. This is a good news for us: we are spending most of the time running very effective and heavily optimized code.