Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support processing LZMA and LZMA2 format .lz files #128

Open
jtmoon79 opened this issue May 21, 2023 · 2 comments
Open

Support processing LZMA and LZMA2 format .lz files #128

jtmoon79 opened this issue May 21, 2023 · 2 comments
Labels
enhancement New feature or request file parser P2 less important

Comments

@jtmoon79
Copy link
Owner

jtmoon79 commented May 21, 2023

Summary

File of type .lz are not processed.
An .lz file is:

A file with .lzma extension is a compressed archive file created using the LZMA (Lempel-Ziv-Markov chain Algorithm) compression method.

Current behavior

A file log.lz would attempt to be parsed as a syslog file. Seeing no text, processing would stop (except in case of errant parsing, Issue #122).

Suggested behavior

Process .lz files as compressed files. The crate lzma-rs is already used to process .xz files.

Ideally, this would be a fair amount of work but should be straight-forward.

Other considerations

LZMA is considered a legacy format so this is low priority. However, there is also LZMA2 format. I have never encountered either, so I suspect they are very rare, if not totally deprecated in favor of XZ format.

Note that .lz named files may be many varying types of formats; LZMA, LZMA2, LZMA4, LZMA within XZ container. This adds more work to determing the precise type.

@jtmoon79 jtmoon79 added enhancement New feature or request good first issue Good for newcomers labels May 21, 2023
@jtmoon79
Copy link
Owner Author

Relates to #12

@jtmoon79 jtmoon79 changed the title Handle LZMA and LZMA2 format .lzma files Support processing LZMA and LZMA2 format .lzma files May 21, 2023
@jtmoon79 jtmoon79 added P1 important P2 less important and removed P1 important labels Aug 13, 2023
@jtmoon79 jtmoon79 added file parser and removed good first issue Good for newcomers labels Nov 5, 2023
@jtmoon79 jtmoon79 changed the title Support processing LZMA and LZMA2 format .lzma files Support processing LZMA and LZMA2 format .lz files May 6, 2024
@jtmoon79
Copy link
Owner Author

jtmoon79 commented May 6, 2024

Make sure to use lzma program, part of XZ Utils. There are many lz* programs that will compress files. And there are also many many LZMA standardized formats.

Only program lzma appears to write LZMA format 1, the oldest format.

$ lzma -z9kv -c file > file.lz

jtmoon79 added a commit that referenced this issue May 6, 2024
support parsing lz4 compressed files `.lz4`

refactor processing sequential-only syslogs (compressed logs that can only
be read linearly; no binary searching)

add logs for LZ4 `.lz4`, and old LZMA `.lz`, and variations of those

add tests in compare-current-and-expected

add lz4 flamegraph in flamegraphs.sh

Fix Issue #201 with tests that had unknown panics for blockreader
processing gzip files.

Issue PSeitz/lz4_flex#159
Issue #201
Issue #128
Issue #291
Issue #283
jtmoon79 added a commit that referenced this issue May 6, 2024
support parsing lz4 compressed files `.lz4`

refactor processing sequential-only syslogs (compressed logs that can only
be read linearly; no binary searching)

add logs for LZ4 `.lz4`, and old LZMA `.lz`, and variations of those

add tests in compare-current-and-expected

add lz4 flamegraph in flamegraphs.sh

Fix Issue #201 with tests that had unknown panics for blockreader
processing gzip files.

Issue PSeitz/lz4_flex#159
Issue #201
Issue #128
Issue #291
Issue #283
jtmoon79 added a commit that referenced this issue May 7, 2024
support parsing lz4 compressed files `.lz4`

refactor processing sequential-only syslogs (compressed logs that can only
be read linearly; no binary searching)

add logs for LZ4 `.lz4`, and old LZMA `.lz`, and variations of those

add tests in compare-current-and-expected

add lz4 flamegraph in flamegraphs.sh

Fix Issue #201 with tests that had unknown panics for blockreader
processing gzip files.

Issue PSeitz/lz4_flex#159
Issue #201
Issue #128
Issue #291
Issue #283
jtmoon79 added a commit that referenced this issue May 7, 2024
support parsing lz4 compressed files `.lz4`

refactor processing sequential-only syslogs (compressed logs that can only
be read linearly; no binary searching)

add logs for LZ4 `.lz4`, and old LZMA `.lz`, and variations of those

add tests in compare-current-and-expected

add lz4 flamegraph in flamegraphs.sh

Fix Issue #201 with tests that had unknown panics for blockreader
processing gzip files.

Issue PSeitz/lz4_flex#159
Issue #201
Issue #128
Issue #291
Issue #283
jtmoon79 added a commit that referenced this issue May 7, 2024
support parsing lz4 compressed files `.lz4`

refactor processing sequential-only syslogs (compressed logs that can only
be read linearly; no binary searching)

add logs for LZ4 `.lz4`, and old LZMA `.lz`, and variations of those

add tests in compare-current-and-expected

add lz4 flamegraph in flamegraphs.sh

Fix Issue #201 with tests that had unknown panics for blockreader
processing gzip files.

README.md mention lz4 and update comparison table

Issue PSeitz/lz4_flex#159
Issue #201
Issue #128
Issue #291
Issue #283
jtmoon79 added a commit that referenced this issue May 7, 2024
support parsing lz4 compressed files `.lz4`

refactor processing sequential-only syslogs (compressed logs that can only
be read linearly; no binary searching)

add logs for LZ4 `.lz4`, and old LZMA `.lz`, and variations of those

add tests in compare-current-and-expected

add lz4 flamegraph in flamegraphs.sh

Fix Issue #201 with tests that had unknown panics for blockreader
processing gzip files.

README.md mention lz4 and update comparison table

Issue PSeitz/lz4_flex#159
Issue #201
Issue #128
Issue #291
Issue #283
jtmoon79 added a commit that referenced this issue May 7, 2024
support parsing lz4 compressed files `.lz4`

refactor processing sequential-only syslogs (compressed logs that can only
be read linearly; no binary searching)

add logs for LZ4 `.lz4`, and old LZMA `.lz`, and variations of those

add tests in compare-current-and-expected

add lz4 flamegraph in flamegraphs.sh

Fix Issue #201 with tests that had unknown panics for blockreader
processing gzip files.

README.md mention lz4 and update comparison table

Issue PSeitz/lz4_flex#159
Issue #201
Issue #128
Issue #291
Issue #283
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request file parser P2 less important
Projects
None yet
Development

No branches or pull requests

1 participant