Compression benchmarks

[Wumpf]

After noticing that by far the slowest dependency to build on my machine was zstd (see rerun-io/rerun#1316), I set out in search for finding an alternative compression library that is pure Rust and has similar compression ratio & speed as zstd.

Not very successful so far! Seems the C written zstd is a lot faster than everything tried here!

Measurements on my M1 Max so far:

zstd encode            27.214 ms
zstd decode            8.7848 ms

gzip (flate2) encode   34.933 ms
gzip (flate2) decode   20.368 ms

lzma (lzma-rs) encode  157.91 ms
lzma (lzma-rs) decode  369.42 ms

brotli encode          67.711 ms
brotli decode          36.595 ms

slightly tweaked for roughly similar compression ratios for the generated test data:

zstd:   3.16
gzip:   2.98
lzma:   2.16
brotli: 3.46

Checklist

• I have read the Contributor Guide
• I have read and agree to the Code of Conduct
• I have added a description of my changes and why I'd like them included in the section below

Description of Changes

Describe your changes here

Related Issues

List related issues here

[emilk]

I added https://github.com/pseitz/lz4_flex which is VERY fast:

zstd encode             27   ms
zstd decode              9.3 ms

gzip (flate2) encode    35   ms
gzip (flate2) decode    20   ms

lzma (lzma-rs) encode  157   ms
lzma (lzma-rs) decode  403   ms

brotli encode           68   ms
brotli decode           37   ms

lz4_flex encode         10.6 ms
lz4_flex decode          2.9 ms

but doesn't compress as well:

zstd:     3.16
gzip:     2.98
lzma:     2.16
brotli:   3.46
lz4_flex: 1.97

[emilk]

I think it would be worth to refactor puffin to support multiple compression schemes/libraries (zstd and lz4_flex, initially). The compression scheme would need to be encoded in the data stream. The puffin_viewer binary would support all of them for maximal compatibility, but we would let users of puffin to opt-in to different compression libraries so the users can chose between fast compression and compilation, or high compression ratio.

[repi]

can close this now as we do support lz4 as well