Improve performance when writing lp and reading sol files #9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mbiggio
force-pushed
the
wip/mbiggio/improve-io-perf
branch
from
f9c60eb to
606552d
Compare
|
Lemme know if it might be worth adding some specific benchmark, maybe with some non-trivial .sol file to parse. Anyway, just to provide some numbers, on my machine (Intel i5 processor, 10 cores, 32 GB RAM), solving a problem with:
takes:
so a considerable amount of time was spent in IO, and this IO time is probably going to be proportional to the problem size |
mbiggio
force-pushed
the
wip/mbiggio/improve-io-perf
branch
from
606552d to
7bd14f8
Compare
lovasoa
requested changes
Mar 19, 2024
We improve the performance of the solution file parser in the case of cplex, by making use of the more performant quick-xml crate and buffering the reads through `BufReader`. quick-xml indeed requires the reader to be buffered, probably to avoid making too many read syscalls. Buffering is also applied to writing the lp file. Manual benchmarks are showing significant improvements over non-buffered read/writes for problem sizes of several thousands of variables
mbiggio
force-pushed
the
wip/mbiggio/improve-io-perf
branch
from
7bd14f8 to
2b1e5af
Compare
lovasoa
approved these changes
Mar 19, 2024
|
Merged; thank you! |
You're welcome! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We improve the performance of the solution file parser in the case of cplex, by making use of the more performant
quick-xmlcrate and buffering the reads throughBufReader.quick-xmlindeed requires the reader to be buffered, probably to avoid making too many read syscalls.Buffering is also applied to writing the lp file. Manual benchmarks are showing significant improvements over non-buffered read/writes for problem sizes of several thousands of variables