Code

Compression support#79

Code

Merged

KillingSpark merged 136 commits intomasterKillingSpark/zstd-rs:masterfrom huffman_encodingKillingSpark/zstd-rs:huffman_encodingCopy head branch name to clipboardDec 13, 2024

+2,830-358

Commits

Commits on Nov 28, 2024

• start a huffman encder

  KillingSpark

  committed
  42c046e

• build huffman table according to spec

  KillingSpark

  committed
  a0d2031

• calc max num bits from weights

  KillingSpark

  committed
  246e609

• create valid weight distributions

  KillingSpark

  committed
  35833ba

• implement redistributing weights to fit a given maximum weight

  KillingSpark

  committed
  7f5ac83

• fix chosen weight to redistribute

  KillingSpark

  committed
  05788ac

• fix huffman code distribution

  KillingSpark

  committed
  48efed9

• build huffman table for counts

  KillingSpark

  committed
  bad020d

• allow building from a slice of data

  KillingSpark

  committed
  0e29503

• start actual huffman encoder

  KillingSpark

  committed
  39e612d

• test for prefix free codes, shows it's still broken

  KillingSpark

  committed
  190f08c

• fix broken prefix allocation

  KillingSpark

  committed
  a7cd816

• test more weight distributions

  KillingSpark

  committed
  e799da7

• add roundtrip test for huffman coding that's fit to also be fuzzed

  KillingSpark

  committed
  f7d32c7

• fix roundtrip test

  KillingSpark

  committed
  cd794a8

• huffman streams are encoded backwards

  KillingSpark

  committed
  1980dbf

• rewrite bit writer to fill bytes starting at the lower bits

  KillingSpark

  committed
  90ede9e

• start fuzzing huffman encoder

  KillingSpark

  committed
  bfa1c9e

• test that max num bits doesnt get too big

  KillingSpark

  committed
  289e664

• only execute fuzz targets when present

  KillingSpark

  committed
  df60a7a

• Fix weight redistribution to limit to a specific number of bits

  KillingSpark

  committed
  4c05304

• add two crashes that are already fixed

  KillingSpark

  committed
  6d4920d

• expand test to cover a weight distribution of all sizes

  KillingSpark

  committed
  2726eaa

• cargo fmt

  KillingSpark

  committed
  9d07056

• no need to gate encoding behind std

  KillingSpark

  committed
  10ee6e1

• mak clippy happy

  KillingSpark

  committed
  f7a5ad2

• more std import shenanigans

  KillingSpark

  committed
  ed219cc

• cargo fmt

  KillingSpark

  committed
  0e2435b

• start fse encoder

  KillingSpark

  committed
  02c4789

• need different table representation for encoding

  KillingSpark

  committed
  d5e2543

• build fse table for encoding

  KillingSpark

  committed
  96925de

• start test for fse table creation, fix edgecase for 0 probability symbols in decoder

  KillingSpark

  committed
  22bfdb3

• table generation now matches

  KillingSpark

  committed
  9039fd9

• first happy fse roundtrip

  KillingSpark

  committed
  4ca250d

• cargo fmt

  KillingSpark

  committed
  b97f962

• unnecessary import

  KillingSpark

  committed
  55b2918

• more roundtrips

  KillingSpark

  committed
  fd9c968

• fuzz fse encoder

  KillingSpark

  committed
  3221f28

• minimum acc_log is 5

  KillingSpark

  committed
  2f4e5d0

• implement fse table encoding

  KillingSpark

  committed
  78ce10a

• misaligned should never report 8

  KillingSpark

  committed
  8ebd0db

• encode huffman tables according to spec

  KillingSpark

  committed
  b7b7d0b

• start compression mode based solely on compressing literals

  KillingSpark

  committed
  561d7d5

• add test for a simple frame encode/decode cycle

  KillingSpark

  committed
  91d46cc

• fix blocksize for compressed blocks, a block may only contain 1<<18 - 1 uncompressed literals

  KillingSpark

  committed
  9c3e610

• add support for RLE blocks

  KillingSpark

  committed
  fc48aa5

• start fuzzing compression interop

  KillingSpark

  committed
  e13886f

• added support for literals less than 5

  KillingSpark

  committed
  cb8d4b7

• fix single stream encoding

  KillingSpark

  committed
  402d180

• add compression and decoding to encoding fuzzer

  KillingSpark

  committed
  454ad7b

• add compression and decoding to encoding fuzzer

  KillingSpark

  committed
  fc91d6e

• huffman weights may only use a fse compression with 6 bits

  KillingSpark

  committed
  1106501

• implement decreasing the fse probabilities to fit max_log

  KillingSpark

  committed
  eeb940f

• remove table modes from empty sequence sections

  KillingSpark

  committed
  61d5fe3

• check in decoder if a sequence section with 0 sequences contains nothing else

  KillingSpark

  committed
  0176e0f

• not using single segment frames and a window size allows inefficient use of the compressed blocks

  KillingSpark

  committed
  3d259ea

• switch back to huffman encoding literals

  KillingSpark

  committed
  45d5fba

• improve interop fuzz

  KillingSpark

  committed
  1a19804

• make window size big enough to fit whole 128kb block into it

  KillingSpark

  committed
  facb5d5

• extend encodecorpus to test compression

  KillingSpark

  committed
  932a12a

• if compressing a block makes it larger just encode it as a raw block and move on

  KillingSpark

  committed
  3df96dd

• raise the first fse probability to avoid 0 num bit states not the last

  KillingSpark

  committed
  79e1c99

• cargo fmt

  KillingSpark

  committed
  5ca4eaa

• make clippy happy

  KillingSpark

  committed
  443cac8

• bitwriter can now work on a &mut Vec as output

  KillingSpark

  committed
  aaad836

• 4x huffman encoding doesnt use 4 vecs anymore

  KillingSpark

  committed
  e95fd06

• add compression support to zstd binary

  KillingSpark

  committed
  a8ab25e

• optimize bitwriter to collect bits in a u64 before flushing to the output

  KillingSpark

  committed
  79c2491

• make source a generic read impl

  KillingSpark

  committed
  0f96d67

• make the target a generic write impl

  KillingSpark

  committed
  39aac8e

• todo

  KillingSpark

  committed
  c83918b

• remove unnecessary code from hot function

  KillingSpark

  committed
  eeb4e08

• remove one += from hot function

  KillingSpark

  committed
  aea6a90

• more efficient check for hot path

  KillingSpark

  committed
  a355e30

• dry common code

  KillingSpark

  committed
  00e61d6

• employ bufreader in zstd binary

  KillingSpark

  committed
  6b26b2d

• start match finding algorithm

  KillingSpark

  committed
  8493d58

• start implementing a skip list for use in the match generator

  KillingSpark

  committed
  262c034

• use hashmap to do find indexes of matches of a minimum length

  KillingSpark

  committed
  e175cd9

• only return None if the whole last data slice has been processed

  KillingSpark

  committed
  a6024dc

• matcher can now also accept data that doesnt need matches

  KillingSpark

  committed
  219dbe2

• start working on encoding the sequences

  KillingSpark

  committed
  dc8068a

• implement translation tables for ll/ml/of tables to codes and additional bits for the bitstream

  KillingSpark

  committed
  14bc05d

• implement FSE encoding sequence codes

  KillingSpark

  committed
  c9f4f03

• default tables

  KillingSpark

  committed
  8457c25

• offset in last window slice is calculated differently

  KillingSpark

  committed
  44fcb62

• properly encode number of sequences

  KillingSpark

  committed
  845008d

• cargo fmt

  KillingSpark

  committed
  3f34638

• matcher now correctly returns a last literals sequence for the last unmatched bytes in a block

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  5b0a142

• add debug assertions that check that matches are correctly generated

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  e4fcb81

• simplify offset calc, doesn't need a special case for the last slice

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  37ec1e8

• offsets are encoded as +3 to allow 1,2,3 to represent the last 3 offsets

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  fc2a5cb

• fix error in default table impl

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  90e0e62

• fix edgecases around small data slices or few sequences

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  a5774de

• remove warning

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  25315c9

• fix doc test

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  7d32f52

• fix fuzzing

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  70e1501

• make clippy happy

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  677ddf4

• if literals get bigger by encoding them with a huffman tree, just write the literals raw

  KillingSpark

  committed
  741a3f9

• expand nostd_io module to include write_all, read_to_end and impl Write for Vec<u8>

  KillingSpark

  committed
  fd0ae1f

• use slices instead of fixed size arrays as key for match generator

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  b3f20f6

• use more efficient method of windowing slices

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  247d870

• cargo fmt

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  0530eb6

• disable clippy lint for collapsible if. I think it's clearer this way in this instance

  

  Moritz Borcherding

  authored and

  KillingSpark

  committed
  54939d4

• encoding now actually streams the data instead of reading the source until the end before processing

  KillingSpark

  committed
  f9c2484

• delete residual eprintln

  KillingSpark

  committed
  1690bd9

• make clippy happy

  KillingSpark

  committed
  48fac5a

• remove decoding check from zstd binary

  KillingSpark

  committed
  afc8cc7

• unused import

  KillingSpark

  committed
  cb07e1c

• zstd binary reports compression progress now

  KillingSpark

  committed
  d81c76b

• this definitely still needs work...

  KillingSpark

  committed
  7cbeeac

Commits on Nov 29, 2024

• No need for a hashtable actually

  KillingSpark

  committed
  421d1c1

• make clippy happy

  KillingSpark

  committed
  8b42cb6

• better key function

  

  Moritz Borcherding

  committed
  699c075

• cargo fmt

  

  Moritz Borcherding

  committed
  899410d

• even better key function

  

  Moritz Borcherding

  committed
  79e45c4

• fix multiply overflow

  KillingSpark

  committed
  f98a837

• reenable test

  KillingSpark

  committed
  7d23961

• better poly in key function

  KillingSpark

  committed
  869a7bf

Commits on Dec 2, 2024

• improve match length finding

  

  Moritz Borcherding

  committed
  d17af02

• inline makes a significant difference

  

  Moritz Borcherding

  committed
  feb6f5f

• document the matcher trait and make it slightly easier to use

  

  Moritz Borcherding

  committed
  228353b

• cargo fmt

  

  Moritz Borcherding

  committed
  9fbe7dd

• fix hash collision in test

  

  Moritz Borcherding

  committed
  8988bc3

• also reuse suffix store allocations

  

  Moritz Borcherding

  committed
  b0c65fa

• fix test and enable niche optimization for Option<NonZeroUsize>

  

  Moritz Borcherding

  committed
  e99c64f

• std -> core import

  

  Moritz Borcherding

  committed
  124793c

Commits on Dec 4, 2024

• small optimization around fse encoding

  

  Moritz Borcherding

  committed
  7819c14

Commits on Dec 12, 2024

• doc and comments in fse encoder

  

  Moritz Borcherding

  committed
  28c3084

Commits on Dec 13, 2024

• doc and comments in huff0 encoder

  

  Moritz Borcherding

  committed
  61829d7

• more doc comments

  

  Moritz Borcherding

  committed
  b9029f0

• doc and comments in match generator

  

  Moritz Borcherding

  committed
  a56a40b

• more doc and slightly better Matcher api

  

  Moritz Borcherding

  committed
  4499aac

• add convenience functions for compressing data

  

  Moritz Borcherding

  committed
  1237041

• update readme and changelog

  

  Moritz Borcherding

  committed
  99dee17

• cargo fmt

  

  Moritz Borcherding

  committed
  6c2bcd7