Expose zc level 0 (uncompressed) 🚀

[Galaxy4594]

Allow the creation of PNGs with uncompressed deflate streams via level 0 of libdeflate. If you want a glorified BMP with delta filters, this change will make your dreams come true 🙂.

[andrews05]

Thanks, this seems harmless enough 🙂
Curious why you would use this though?

[goodusername123]

Thanks, this seems harmless enough 🙂 Curious why you would use this though?

The possible use cases are niche but do exist, such as storing PNGs with compression methods other then deflate but in a way that doesn't break compatibility with pre-existing decoders. An example being to convert pre-existing PNGs to be uncompressed but with the delta filter strategies present in oxipng then compressing said PNGs inside of a container or filesystem that has compression methods with better ratios then deflate (such as brotli).

[andrews05]

Hm, just be aware that if the compression level is capped at 0, the filter evaluations (and some reductions) will all end up tied for which is best. E.g. if you run it with all filters -f 0-9 it will likely just end up choosing the first result every time, i.e. "None". You could pick just one, e.g. Brute, but that won't always be optimal for the external compressor.

[Galaxy4594]

Instead of trying every possible filter, I use a simple heuristic where I choose the same filter for deflate. I run oxipng -o max -P -vv input.png and select the filter that results in the smallest file size. Then, I run oxipng --zc 0 -f x --force input.png --out output.png && brotli -j -v output.png. With some simple scripting, I can generate PNGs compressed with Brotli/Zstd that are roughly 10-50% smaller. 😁

[andrews05]

Yeah, that could work. You could also drop the -P on the first run and then use --nx on the second, so it retains the best reductions from the first.

[edit] How does it compare to just using a newer format like JXL or (shudder) WebP?

[Galaxy4594]

How does it compare to just using a newer format like JXL or (shudder) WebP?

Not good, JXL and WebP compress about 10-20% more efficiently than Brotli/Zstd PNGs. For 16-bit content, JXL achieves around 50% better compression. For animated images, the improvement varies by 30-50%. Maybe compressing multiple images in a .tar would achieve better results? It probably won't make much of a difference.

While talking to a friend, we came up with a crazy idea to save bandwidth by any means necessary: serve uncompressed PNGs, which are then compressed via a CDN/web server and decompressed in the browser (assuming you can’t use WebP/JXL). 😅

[ace-dent]

@Galaxy4594 - I was experimenting with this a few years back. The theory is sound: (1) you have a container that has a better compression algorithm; and / or (2) you can exploit duplicated information across multiple images for higher compression. As you know for web, depending on server setup, this can happen at the transport layer using gzip and now brotli. For speed this may be disabled for png assets server side, but I can imagine embedded (base64) images in an html page may work out ...?

My interest is more in packaged pngs, often in zip archives, within a software distribution. Recently I analysed a few hundred pngs that are LZMA packed for the KolibriOS project. Contrary to theory (and as I've seen before too), ~99% of images do benefit from high level of png (/deflate) compression, and then being compressed in an archive (even deflating a pre-deflated stream!).

@andrews05 - I think expected behaviour here is good filter choices but just uncompressed in final output. Would it make sense to do the evaluations with some level of deflate compression? Or is it undesirable / unintuitive to override the User's choices in this way? Either way, it's pretty niche- and perhaps should be documented as such?

[Galaxy4594]

Recently I analysed a few hundred pngs that are LZMA packed for the KolibriOS project. Contrary to theory (and as I've seen before too), ~99% of images do benefit from high level of png (/deflate) compression, and then being compressed in an archive (even deflating a pre-deflated stream!).

I did my own tests I came to the opposite conclusion, uncompressed PNGs compressed better than regular deflate PNGs. I even tested on the data set you mentioned (KolibriOS) and these are the results I got. These were compressed at 7z level 9 with word size 273.

KolibriOS-uncompress-PNGs.7z ------- 44,165,338 bytes
KolibriOS-deflate-PNGs.7z  --------- 44,074,109 bytes
KolibriOS-uncompress-brute-PNGs.7z - 43,950,764 bytes

It may seem like recompressing deflate PNGs are better, but applying the right filter makes a difference. No manual setting of filters per image, I just applied the brute filter to every PNG, 😁.

[ace-dent]

@Galaxy4594 - thanks for sharing these interesting results. I should have been a bit more specific... KolibriOS uses its own packer (originally derived from 7z, but with some differences). I will certainly investigate again in the future 👍

[AlexTMjugador]

Thanks for the PR! ❤️

While I agree with the overall sentiment that the need for a compression level of 0 is pretty niche, the code looks good and the feature itself can be useful in those niche cases without being a maintenance burden, so I'm merging this!

[IlluminatiWave]

Thanks for adding this, although it would be better to expose a “none” flag (as we have the max flag) for a method without compression (by the way, is there a way to save the png in a single idat chunk with oxipng?)

Regarding the external compression, this works quite well, I usually get a ratio between 80 and 90% of the uncompressed size, and about 10 to 59% of the original file. Although this is very relative and depends on each image (compression level).

Use containers as external compression, I have experimented with this for a while (I made several scripts in fact, one of them I have as a repo, although currently it is code that I have decided to abandon for what I will say now), Apparently 7z works better when we compress files without compression than with compression, the compression is usually more aggressive if we use redundant content (for example b/w mangas, sets of images with variations), with little redundant content (for example photographs or images that have nothing to do with each other) lzma2 compression is usually not so extreme, however, usually using lzma2 /7z) per file compresses slightly more than using oxipng at max level (I'm talking about 7z at its maximum setting, using standard usually doesn't compress as well). Also, compressing with zlib level 0 (no compression / standard png) is slightly better than compressing the raw pixels (my repo), not to mention that we keep all the features of a standard png (my script does strange things).

Regarding using another compression algorithm on a png, it is possible (in fact I have another script that does just that, it joins all the idat chunks into one, decompresses the chunk deflate file which is the raw png data and recompresses it with other algorithms, it only affects the idat chunks so the content never really changes / I don't have the script repo).

I have to say that the compression is slightly identical when using zlib 0 + lzma2 max (7z), although of course, you won't be able to see anything in any image viewer because none of them can use lzma (or any other algorithm) on png images (but you can recover the image by doing the reverse process, unzip the chunk idat lzma and recompress it with zlib).

[andrews05]

Thanks for the observations @IlluminatiWave.

Oxipng will always write only a single IDAT chunk, regardless of how many are in the input and what optimisation or compression took place.

[AlexTMjugador]

To add to @andrews05's reply above, the PNG standard requires the use of Zlib (DEFLATE with additional headers) as the compression method for IDAT chunks, and only Zlib. This means you can't simply place the raw scanline data there or use other compression formats if you want to produce compliant PNG files.

However, sane DEFLATE compressors can automatically detect data that cannot be compressed and store it verbatim in the DEFLATE stream, adding only a small, constant overhead in such cases, so unless you happened to find a pathological case, or a case where other inter/intra file redundancies can be used to improve overall compression, there should be no need to tweak that. Of course, if you're using custom compression and decompression code, you can also preprocess IDAT contents losslessly to achieve better compression, but keep in mind that this preprocessed data does not conform, by definition, to a standard PNG stream.

[TPS]

the PNG standard requires the use of Zlib (DEFLATE with additional headers) as the compression method for IDAT chunks, and only Zlib.

@AlexTMjugador Per https://www.w3.org/TR/2003/REC-PNG-20031110/#10Compression, this isn't strictly true, though, in the wild, it very much is. I believe that specific language is why many of the early decoders checked for DEFLATE specifically, though many later 1s no longer bothered.

I remember having discussions ~30ya about, maybe now that a new GIF-killer was being developed, maybe there'd be the same for ZIP. It's pretty amazing that no ZIP-killer is as ubiquitous as PNG, so many software generations later.

[IlluminatiWave]

the PNG standard requires the use of Zlib (DEFLATE with additional headers) as the compression method for IDAT chunks, and only Zlib. This means you can't simply place the raw scanline data there or use other compression formats if you want to produce compliant PNG files.

It is actually possible, it is true that PNG requires the information to be compressed in some way and not in this raw form (even within a deflate lv 0 / the "none" flag). But if I remember correctly, within the ihdr chunk it is specified that there is a byte to indicate the compression algorithm (for 20 years this byte has been 0x00, that stands for compression method: deflate). Since there is no standard or suggestion, it could be used as byte 0x07 to indicate that lzma algorithm is used or 0x02 for brotli, etc. I don't know why there is no list to date suggesting which algorithm each byte would belong to (we have up to 256 possible valid algorithms for PNG)

And that surprises me even more since if I remember correctly, APNG already supported LZMA.

[AlexTMjugador]

@AlexTMjugador Per https://www.w3.org/TR/2003/REC-PNG-20031110/#10Compression, this isn't strictly true, though, in the wild, it very much is. I believe that specific language is why many of the early decoders checked for DEFLATE specifically, though many later 1s no longer bothered.

You're technically correct that the PNG datastream format specification allows for indicating a compression algorithm other than Zlib. However, since no specification defines the meaning of values other than 0, different software implementations don't need to agree on how to interpret those values. This lack of standardization severely limits interoperability, which undermines the purpose of a format with "Portable" in its name, and as a result, that's why I said that using Zlib is effectively "required" if you want your PNGs to be viewable by most decoders.

But indeed, you can find very creative uses to that field in niche cases where you're using PNG as a exchange format between well-known encoders and decoders... That's not how PNG was meant to be (ab)used in general, though 😉