Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

An option to write sdp to directory instead of zip archive #177

Closed
vasdommes opened this issue Jan 23, 2024 · 2 comments
Closed

An option to write sdp to directory instead of zip archive #177

vasdommes opened this issue Jan 23, 2024 · 2 comments
Assignees
Milestone

Comments

@vasdommes
Copy link
Collaborator

For large problems, writing block files to sdp.zip is a bottleneck in sdp2input/pvm2sdp, see plots in #176.
It is done by a single process and cannot be parallelized.

The purpose of writing to zip archive instead of plain directory is to prevent running out of inodes error on some filesystems.

If this limit is not a problem and if time of writing sdp.zip is important (e.g. in Skydive, where one may call sdp2input after each several iterations of SDPB solver), it will be better to write to a directory. We can add e.g. an option --zip=false.
(NB: don't forget to validate all file sizes after writing!)

@vasdommes vasdommes added this to the Backlog milestone Jan 23, 2024
@vasdommes vasdommes self-assigned this Jan 23, 2024
@davidsd
Copy link
Owner

davidsd commented Jan 23, 2024

Might there be some other kind of archive format besides zip that can be written to in parallel?

@vasdommes
Copy link
Collaborator Author

Sorry, something went wrong.

bharathr98 pushed a commit to bharathr98/sdpb that referenced this issue Mar 1, 2024
…chive

pmp2sdp now has an option --zip (true|false)

(obsolete) pvm2sdp and sdp2input executables now write to directory only
bharathr98 pushed a commit to bharathr98/sdpb that referenced this issue Mar 1, 2024
bharathr98 pushed a commit to bharathr98/sdpb that referenced this issue Mar 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants