Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add xz_decompress_with_options API #110

Open
jtmoon79 opened this issue May 5, 2024 · 0 comments
Open

add xz_decompress_with_options API #110

jtmoon79 opened this issue May 5, 2024 · 0 comments
Labels
enhancement good first issue help wanted

Comments

@jtmoon79
Copy link

jtmoon79 commented May 5, 2024 •
edited
Loading

Problem

xz_decompress reads the entire .xz file in one call. The entire file is stored in memory. Decompressing large files can lead to Out Of Memory errors.

Solution

Provide xz_decompress_with_options, similar to other _with_options API functions. Notably, this would allow passing an Options with a set memlimit. Then the user need not store the entire file in memory.

Similar to #49
@jtmoon79 jtmoon79 mentioned this issue May 5, 2024
Open
jtmoon79 added a commit to jtmoon79/super-speedy-syslog-searcher that referenced this issue May 6, 2024
@jtmoon79
blockreader.rs attempt to parse more of the XZ header
5750cf3 
Attempt to parse more of the XZ header and block #0 header.
Unfortunately, I couldn't figure get this working entirely. Leaving
the code in place as it does function.
The intent was to compensate for lzma-rs reading the entire file
during xz_decompress. However, that's a larger problem, see
gendx/lzma-rs#110

Issue #12
Issue 283
@jtmoon79 jtmoon79 mentioned this issue Jun 11, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement good first issue help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jtmoon79 @gendx