Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experiment with alternative compression #63

Open
kermitt2 opened this issue Sep 6, 2021 · 0 comments
Open

Experiment with alternative compression #63

kermitt2 opened this issue Sep 6, 2021 · 0 comments

Comments

@kermitt2
Copy link
Owner

kermitt2 commented Sep 6, 2021

We use snappy right now for LMDB stored records. There are other compression methods that might be relevant to small objects to get higher compression ratio and faster decompression (in principle we are less interested in compression speed).

See for instance https://morotti.github.io/lzbench-web as a benchmark study. zstd (which has a training mode) and lz4 might be more interesting than snappy for our usage.

@kermitt2 kermitt2 changed the title Experiement with alternative compression Experiment with alternative compression Apr 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant