feat(handler): add geom_uzip handler

[rxpha3l]

geom_uzip is a FreeBSD feature for creating compressed disk images (usually containing UFS). The compression is done in blocks, and the resulting .uzip file can be mounted via the GEOM framework on FreeBSD.

The mkuzip header includes a table with block counts and sizes. The header declares the block size (size of decompressed blocks) and total number of blocks. Block size must be a multiple of 512 and defaults to 16384 in mkuzip.
It has the following structure:

Magic, which is a shebang that is stored on 10 bytes.
Version, which can change and is stored on 13 bytes.
Command, which can change and is stored on 105 bytes.
Block size, stored on 4 bytes.
Block count, stored on 4 bytes.
Table of content (TOC), which depends on the file lentgh.
The TOC is a list of uint64_t offsets into the file for each block. To determine the length of a given block, read the next TOC entry and subtract the current offset from the next offset (this is why there is an extra TOC entry at the end). Each block is compressed using zlib. A standard zlib decompressor will decode them to a block of size block_size.

Unblob parses the TOC to determine end & start offset of the uzip file. It will find the compressed blocks, decompress them using zlib and parses them together to recover the decompressed file. Empty chunks are ignored, which is why the decompressed file with unlbob can be a little bit lighter than the original one.

[Sources]
https://github.com/freebsd/freebsd-src/blob/master/sys/geom/uzip/g_uzip.c

[qkaiser]

@vlaci what would be the easiest way to add pyzstd to unblob dependencies in Nix here ? It's not yet in upstream at https://github.com/NixOS/nixpkgs/blob/0fa90d642277de2c67e93204cc5870aba8af5878/pkgs/by-name/un/unblob/package.nix#L59 so we need a way to define it in this branch in the meantime.

[qkaiser]

@vlaci what would be the easiest way to add pyzstd to unblob dependencies in Nix here ? It's not yet in upstream at https://github.com/NixOS/nixpkgs/blob/0fa90d642277de2c67e93204cc5870aba8af5878/pkgs/by-name/un/unblob/package.nix#L59 so we need a way to define it in this branch in the meantime.

@rxpha3l I'm using this fix locally, but not sure if it's idiomatic Nix

diff --git a/overlay.nix b/overlay.nix
index 9c5051e..265cd79 100644
--- a/overlay.nix
+++ b/overlay.nix
@@ -29,6 +29,8 @@ final: prev:
         ];
       };
 
+      dependencies = (super.dependencies or []) ++ [ prev.python3.pkgs.pyzstd ];
+
       # remove this when packaging changes are upstreamed
       cargoDeps = final.rustPlatform.importCargoLock {
         lockFile = ./Cargo.lock;

[martonilles]

do we need to flush() the decompressor? probably not, but safer to do so

[e3krisztian]

The decompressor could hold on some uncompressed bytes, if given a buffer limit, but also in some other cases:

• lzma.LZMADecompressor: .eof, .needs_data, .unused_data, but only used if we gave a non-negative buffer size
• pyzstd.ZstdDecompressor has the same fields, and further says in the .decompress() docstring:

Decompress data, return a chunk of decompressed data if possible, or b'' otherwise.

It stops after a frame is decompressed.

• zlib.decompressorobj() also has .eof, unused_data, but .flush() and .unconsumed_tail as well

So at least pyzstd says it will never return more than a frame's worth of decompressed data at once.

Maybe the others also have some surprising logic, and we need to do some more work to decompress a stream properly (looking at the lzma.decompress function source it is also silently ignoring after eof input - we should probably check for that as well - it is potentially a problem to have extra bytes).