Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add binary SDP format to skydiving_release branch #145

Closed
wants to merge 29 commits into from
Closed

Add binary SDP format to skydiving_release branch #145

wants to merge 29 commits into from

Conversation

vasdommes
Copy link
Collaborator

@vasdommes vasdommes commented Nov 10, 2023 •
edited
Loading

Cherry-picked commits from PR #114 implementing binary SDP.
Also cherry-picked some minor fixes and improvements for sdp2input-related code.

NB: The code is incompatible with old SDP's, so you'll have to regenerate them.

UPD other relevant fixes that are (or should be) cherry-picked:

@vasdommes
Add .gitignore
7d6020c
@vasdommes
sdp2input: print matrix sizes and RAM estimates (debug mode)
4712258 
TODO: same for pvm2sdp

Related: #76 Print out info about the sdp at startup
(cherry picked from commit 40ec237)
@vasdommes
Fix #86 Error writing profile in "sdp2input --debug=true"
5d371f6 
(cherry picked from commit 46e0aa7)
@vasdommes
Fix #80 Add total RAM usage to debug output
f752daf 
Printed for rank=0 at timer start, example:

0 run.step.frobenius_product_symmetric  MemAvailable, GB: 216.604
0 run.step.frobenius_product_symmetric  MemUsed, GB: 34.8657

(cherry picked from commit 4e1ab5d)
@vasdommes
Add boost::filesystem::create_directories() for parent dir before wri…
3f46198 
…ting to file

(cherry picked from commit 31f1a1c)
@vasdommes
write_free_var_matrix(): use matrix.Width() instead of dual_objective…
1d1aa29 
…_b_size

also assert that they are equal

(cherry picked from commit 6d09479)
@vasdommes
sdp2input: Throw exception if input file not found
5cb0d7e 
(cherry picked from commit 7d1f137)
@vasdommes
rename read_file_list -> read_nsv_file_list
a3e7ba3 
(cherry picked from commit c7a5bf8)
@vasdommes
When reading .nsv files, treat relative paths as relative to .nsv dir…
b5525b8 
…ectory.

Previously, the current working directory was used. This could lead to less predictable behaviour.

(cherry picked from commit 156fc7e)
@vasdommes
sdp2input and pvm2sdp: fail if input file has unexpected extension.
52a5cb9 
(cherry picked from commit 412df53)
@vasdommes
Dual_Constraint_Group: replace degree with num_points = (degree - 1)
fd2280c 
In all other places in our code we actually need num_points, not degree;
in particular, when writing sdp.zip

(cherry picked from commit a467030)
@vasdommes
Extract serialize_json() function for Dual_Constraint_Group
db4c230 
(cherry picked from commit ca91712)
@vasdommes
Fix #79 Binary input for sdpb
77c2f8d 
sdp2input: add --outputFormat option (json or bin)
by default, --outputFormat=bin

TODO test

(cherry picked from commit 407169c)
@vasdommes
Some refactoring for write_sdpb_input_files and related functions
fe96c72 
Ideally we should use rapidjson for serializing, but according to my benchmarks it is ~2x slower when writing arrays of BigFloats, because with rapidjson we have to write BigFloat to stringstream, setting precision each time.

(cherry picked from commit 3c62abd)
@vasdommes
Archive_Reader: throw exception if read result is not OK and not EOF.
969f92b 
For example, after introducing a bug I caught ARCHIVE_WARN with "Inconsistent uncompressed size: 34 in central directory, 134031 in local header".
In the old version, reader just pretended that there are no more files.

(cherry picked from commit 41e5a69)
@vasdommes
SDP.zip: split block_XXX file to (small) block_info_XXX.json and (lar…
b01ef9b 
…ge) block_data_XXX

Fixes #110 Optimize reading block info (dim and num_points) from sdp.zip

(cherry picked from commit 8f127d9)
@vasdommes
Update Usage.md for commit 412df53 sdp2input and pvm2sdp: fail if inp…
014ae84 
…ut file has unexpected extension.

(cherry picked from commit 94317df)
@vasdommes
read_blocks.cxx: use map instead of vector in lookup table, to improv…
b329a99 
…e readability

(cherry picked from commit c798f7c)
@vasdommes
(cosmetics) El::RuntimeError in read_block_info.cxx, El::Int in read_…
50ded48 
…blocks

Replaced size_t with El::Int to avoid compiler warnings

(cherry picked from commit 65f2beb)
@vasdommes
Fix #113 cxx14 vs cxx17 in build config files
ee8b5af 
Use cxx17 always.

(cherry picked from commit fb646c8)
@vasdommes
Optimize read_blocks() and read_block_info(): stop after reading num_…
03016b4 
…blocks blocks.

(cherry picked from commit edd3640)
@vasdommes
Fix #124 sdpb fails to parse block_data with empty bilinear_bases_odd
dbf0ab7 
(cherry picked from commit 3420367)
@vasdommes
replace 'git describe --dirty' with 'git describe --always --dirty'
fd8f7ee 
git will return current commit, if no tags found (e.g. in forked repo)

(cherry picked from commit f0d29ea)
@vasdommes
Fix #123 --outputFormat option for pvm2sdp
9e4efd6 
Usage: pvm2sdp [FORMAT] PRECISION INPUT... OUTPUT

FORMAT is 'bin' or 'json'
If FORMAT is missing, bin is used by default

(cherry picked from commit dfd37ba)
@vasdommes
Fix #127 sdp2input and pmp2sdp: use binary output format by default
8b31ba9 
(pmp2sdp was fixed by previous commit)

(cherry picked from commit b83564b)
@vasdommes
Better error messages if SDP is missing in read_objectives() or read_…
3b56917 
…blocks()

Previously, SDP constructor could fail with obscure rapidjson error "Assertion `IsObject ()' failed."
@vasdommes
Copy link
Collaborator Author

vasdommes commented Nov 15, 2023 •
edited
Loading

TODO: cherry-pick #149 to fix binary sdp performance for blocks with many zeros

@vasdommes
Fix #148 Binary SDP format is larger than json format if blocks conta…
45de9db 
…in many zeros

If BigFloat is zero, then we write only boolean flag indicating it.
If not, we write the flag(=false) and then serialized bytes.

This makes zeros ~100x more compact (for precision=768, BigFloat needs ~100B).

TODO: maybe we can use the same trick for MPI communication
@vasdommes
Fix #159 Memory leak in binary SDP deserialization
4b4d563 
El::Matrix::Control() does not take ownership correctly, which caused theleak
https://gitlab.com/bootstrapcollaboration/elemental/-/issues/3
@vasdommes
boost::serialization: disable object tracking, which may cause increa…
012dea9 
…sed RAM consumption

We don't need any tracking, we're just writing arrays.
@vasdommes
Copy link
Collaborator Author

Closing this PR as obsolete, new skydiving-master branch includes all updates from master.

@vasdommes vasdommes closed this Jan 20, 2024
@vasdommes vasdommes deleted the skydiving_binary-sdp branch January 20, 2024 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
enhancement io performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@vasdommes