Much faster standard PNG writing

[richgel999]

Using more modern zlib compression libs, it's possible to greatly speed up PNG writing while still writing fully standard PNG's that are reasonably compressed. The possible speed gains are large, I estimate around 2-15x vs. stb_image/lodepng/libpng. (Possibly even faster using libdeflate, a modern Deflate compressor, which I didn't have time to hook up.)

https://twitter.com/richgel999/status/1470696769470271493

Quick PNG writing bench, 2k 32bpp, ~7x faster vs. stbi_write_png:

miniz.h lv1g: .173 secs 9.99MB
lodepng+miniz f0: .256 9.99
miniz lvl3g: .498 8.91
miniz lvl5: .596 8.86
stbi f0 lvl 5: .895 11.41
stbi f0: .944 11.39
lodepng+miniz: .985 5.96
stbi: 1.247 8.15
lodepng: 2.311 5.89

[richgel999]

Additionally, it's also possible using multiple IDAT's to parallelize PNG encoding across multiple threads. Combining both techniques would result in enormous reductions in wall-block time for writing PNG.

The time it takes to read & write PNG is becoming a significant bottleneck in modern commercial environments that use 8K-16K or larger images/textures.

[DavidBuchanan314]

Given that it results in fully standard PNGs, using this technique doesn't require any changes to the spec, right?

[richgel999]

Given that it results in fully standard PNGs, using this technique doesn't require any changes to the spec, right?

Correct. No spec changes.

There's already a Rust implementation of the threaded writing idea:
https://github.com/brion/mtpng

[richgel999]

I modified miniz (a zlib alternative) to implement pixel-wise LZ compression: All LZ matches are aligned to 4 byte boundaries, and literals are always output in groups of 4. Compression is ~16% faster, or 8.6x faster than stbi_write_png:

miniz.h lvl1g+pixelwise LZ: .145 10.38

[richgel999]

Quick benchmark of various fast PNG writers vs. QOI:

[richgel999]

What we need is a new codec inside this red circle. Simpler and way faster than the existing filtering+Deflate method, and compresses better by 10-30%:

[randy408]

Which tool was used to generate the graphic?

I have cleaned up and published the spec I had for parallel encoding/decoding, it's fairly flexible: https://github.com/libspng/png-restart-marker

[palemieux]

What we need is a new codec inside this red circle. Simpler and way faster than the existing filtering+Deflate method, and compresses better by 10-30%:

I finally got around to setting up a framework for benchmarking lossless codecs, including JPEG 2000 Part 15 (HTJ2K) and JPEG XL. Initial results at:

https://www.lossless-benchmarks.com/

It is possible to achieve throughput comparable to QOI at lower file sizes.

[ProgramMax]

We discussed this in today's meeting.
We're open to the idea. But we need to see user interest--an implementation being used either widely or by a major player.

That might already exist.
I know of several implementations, including the ones linked above.
So this is a call to action for you all: Show us it being used [widely/by a major player].

[fintelia]

What specific action would be taken in response to widespread use of a faster compression strategy? The spec has never said that implementations need to use the specific zlib compression library written by Jean-loup Gailly and Mark Adler. So using a different/faster library has always been allowed.

I think at most it might warrant an informational note?

[ProgramMax]

Right, zlib isn't strictly required. Any standard deflate will do.
But this discussion is about adding things other than standard deflate, right?

[vrubleg]

This ticket is just about using a better and faster deflate library, keeping 100% compatibility. So there is nothing to standardize, it's mostly about updating libpng =)

[ProgramMax]

Oh, wait. I think I misunderstood & misspoke. Sorry about that.

I just checked -- "Does the spec require zlib specifically?? Surely it doesn't."
But it does require zlib formatted deflate streams.

So if we're talking about using a compatible library, go for it. (And if the spec isn't clear about "zlib format" vs. "zlib the library", we should indeed clarify.)