Different png RGBA values using Julia, C or Python

[t-bltg]

Per https://discourse.julialang.org/t/reading-png-rgb-channels-julia-vs-python/73599.

Reproducer at https://github.com/t-bltg/PngPixel.jl.

Xref: JuliaIO/ImageIO.jl#41 (comment), @johnnychen94.
For reference, the image used here is a scaled version of toucan.png from TestImages.jl: $ convert toucan.png -scale 5% img.png.

Tested with #47 but still fails on 1.6.5 or 1.7.1 (official binaries):

(@v1.6) pkg> st
      Status `~/.julia/environments/v1.6/Project.toml`
  [3da002f7] ColorTypes v0.11.0
  [6218d12a] ImageMagick v1.2.2
  [f57f5aa1] PNGFiles v0.3.12 `https://github.com/JuliaIO/PNGFiles.jl.git#paletted_images_fix`
  [295af30f] Revise v3.2.1

julia> versioninfo()
Julia Version 1.6.5
Commit 9058264a69 (2021-12-19 12:30 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, sandybridge)

julia> using PNGFiles, Downloads

julia> (x->(Int(x.r.i), Int(x.g.i), Int(x.b.i), Int(x.alpha.i)))(PNGFiles.load(Downloads.download("https://aws1.discourse-cdn.com/business5/uploads/julialang/original/3X/5/c/5c5cdaa477dcae516c7183c3e710ef65a0bb6ab0.png"))[3,3])
(233, 173, 4, 244)  # correct

julia> (x->(Int(x.r.i), Int(x.g.i), Int(x.b.i), Int(x.alpha.i)))(PNGFiles.load(Downloads.download("https://raw.githubusercontent.com/t-bltg/PngPixel.jl/master/img.png"))[3,3])
(245, 214, 39, 244)  # please use this url, somehow, the .png files are different in size ^^^^^^^^^^^^^^^^

@Drvi , this is getting weird, maybe I changed the file when uploading to discourse, sorry about the confusion:

$ identify -verbose img.png | grep Filesize
  Filesize: 481B
$ convert img.png txt:- | grep '2,2:'
2,2: (59881,44461,1028,62708)  #E9AD04F4  rgba(233,173,4,0.956863)
$ identify -verbose 5c5cdaa477dcae516c7183c3e710ef65a0bb6ab0.png | grep Filesize
  Filesize: 329B
$ convert 5c5cdaa477dcae516c7183c3e710ef65a0bb6ab0.png txt:- | grep '2,2:'
2,2: (59881,44461,1028,62708)  #E9AD04F4  srgba(233,173,4,0.956863)

OK, so this a Srgba vs rgba thing, let's use the SRGB -> linear transformation (https://en.wikipedia.org/wiki/SRGB#Transformation) (sorry i'm not very familiar with images theory, is that correct ?)

julia> using PNGFiles, Downloads
julia> lin(x) = x > .04045 ? ((x + .055) / 1.055)^2.4 : x / 12.92
julia> img = PNGFiles.load(Downloads.download("https://raw.githubusercontent.com/t-bltg/PngPixel.jl/master/img.png"))
julia> round.(Int, 255lin.([img[3,3].r, img[3,3].g, img[3,3].b]))
3-element Vector{Int64}:
 233
 171  # NOK
   5  # NOK

[Drvi]

@t-bltg Thanks for looking into this! Sadly, I'm not a real image theorist myself but the following is my understanding of the issue.

The two files are different in how they encode gamma information:

"https://aws1.discourse-cdn.com/business5/uploads/julialang/original/3X/5/c/5c5cdaa477dcae516c7183c3e710ef65a0bb6ab0.png"
does not contain neither a sRGB chunk, nor a gAMA chunk, in which case we follow libpngs advice and just assume the the gamma to be 0.45455 (1/2.2). This ends up being basically a no-op: the default guess for display_gamma is 2.2, the our deduced file_gamma is 1/2.2 in the end the "power transformation" has an exponent of 1 and we end up using the color values as they are encoded in the palette.

To get a little bit more explicit, this is how the gamma values are used (assuming 8-bit depth):

corrected = round(Int, 255 * ((raw / 255) ^ (1.0 #=viewing_gamma=# / (file_gamma * display_gamma))))

while

"https://raw.githubusercontent.com/t-bltg/PngPixel.jl/master/img.png" also doesn't have sRGB, but does contain a gAMA chunk with value 1.0.

which means that our formula ends up being

corrected = round(Int, 255 * ((raw / 255) ^ (1.0 / (file_gamma * display_gamma))))
corrected = round(Int, 255 * ((raw / 255) ^ (1.0 / (1.0 * 2.2))))
corrected = round(Int, 255 * ((raw / 255) ^ (0.45455)))

This is how we arrived at the (245, 214, 39, 244):

julia> round.(Int, 255 * ((UInt8[233, 173, 4] ./ 255) .^ (0.45455))) 
3-element Vector{Int64}:
 245
 214
  39

So what does this mean for our implementation?
Quoting from https://www.w3.org/TR/PNG-GammaAppendix.html:

If file_gamma is not 1.0, we know that gamma correction has been done on the sample values in the file, and we could call them "gamma corrected" samples.

Should we completely skip gamma correction when the file_gamma is 1.0? How about for file_gamma 0.99999? Are our results correct? Sadly, I'm not sure... maybe @IanButterworth, @johnnychen94 or @timholy, if they have a moment, could advise as they know much more than I do about images?

---

julia> round.(Int, 255lin.([img[3,3].r, img[3,3].g, img[3,3].b]))
3-element Vector{Int64}:
233
171 # NOK
5 # NOK

The transformation you are using, which is linear for small values of and x and then uses an exponent of 2.4 + some shift and scale, is in fact an sRGB compliant transformation, while gamma correction itself is just a simple power transformation. An exponent of 2.2 is generally used as a default for gamma correction and with this value the two transformations tend to be (visually) similar.

[t-bltg]

Thanks for the detailed explanations, I have a better understanding now.

Quoting from https://www.w3.org/TR/PNG-GammaAppendix.html:

Isn't the spec mentioned obsolete ? "PNG (Portable Network Graphics) Specification Version 1.0"

From what I understand, the current spec is 1.2, here are some selected bits:

sRGB

When the sRGB chunk is present, applications that recognize it and are capable of color management [ICC] must ignore the gAMA and cHRM chunks and use the sRGB chunk instead.

Gamma (not part of the PNG spec)

Decoder gamma handling

Decoders capable of full-fledged color management [ICC] will perform more sophisticated calculations than what is described here.

What does capable of mean here ? Is this the case for PNGFiles ?

The value of gamma can be taken directly from the gAMA chunk. Alternatively, an application may wish to allow the user to adjust the appearance of the displayed image by influencing the value of gamma. For example, the user could manually set a parameter called user_exponent that defaults to 1.0, and the application could set

Regardless of how an image was originally created, if an encoder or file format converter knows that the image has been displayed satisfactorily using a display system whose transfer function can be approximated by a power function with exponent display_exponent, then the image can be marked as having the gamma value:
gamma = 1 / display_exponent

Encoder gamma handling

It's better to write a gAMA chunk with an approximately right value than to omit the chunk and force PNG decoders to guess at an appropriate gamma.

$ identify -verbose 5c5cdaa477dcae516c7183c3e710ef65a0bb6ab0.png | grep 'gAMA\|Gamma\|sRGB'
  Colorspace: sRGB
  Gamma: 0.454545
    png:sRGB: intent=0 (Perceptual Intent)
$ identify -verbose img.png | grep 'gAMA\|Gamma\|sRGB'
  Gamma: 1
    png:gAMA: gamma=1 (See Gamma, above)

Not easy 😅 !

[Drvi]

Isn't the spec mentioned obsolete ? "PNG (Portable Network Graphics) Specification Version 1.0"

Good point (further illustrating my level of expertise here:))

The thing is, the two pngs we're discussing are not in sRGB:

This is the problematic png (notice the gAMA chunk):

This is the other one, which gives us correct value for pixel at [3,3] (also missing the gAMA chunk):

Compare it with an example output produced in tests, which clearly contains the sRGB chunk (highlighted):

What does capable of [ICC] mean here ? Is this the case for PNGFiles ?

I don't think we do so at the moment (possibly libpng has some default handling for that?), if you have a png file with an iCCP I can have a look at it, though.

Not easy 😅 !

Yes:) I'm still wondering what is the source of difference between PNGFiles and ImageMagick. Both recognise that the image has a gAMA and try to transform it accordingly, this gives me some confidence that we're actually doing things right, but they end up with slightly different values.

[t-bltg]

Ok, so let's forget about sRGB and focus on gAMA here.

Should we completely skip gamma correction when the file_gamma is 1.0? How about for file_gamma 0.99999?

Nearly all the plugins in imageio ignore the gAMA chunk when present in file, unless requested using a flag (imageio/imageio#366):

• pillow_legacy
• pillow
• freeimage

I've updated pngpixel.py and I must say that nearly all plugins (except PNG-FI) return gamma uncorrected pixel rgba(233,173,4,244) for the file containing the gAMA chunk.

[t-bltg]

Changing this line to if gamma > 0 and using gamma = -1. when using PNGFiles.load(...) gives the expected result using pngf.jl, so I don't have to use the sRGB -> CIE XYZ transformation:

$ julia pngf.jl 2 2 img.png -1.
rgba(233,173,4,244)

Now, we have to understand why png_get_PLTE returns slightly invalid values yielding (233, 171, 5, 244).
There is some rounding involved, thus the whole gamma transformation + sRGB -> CIE XYZ will never be bijective.
To me, the only valid solution is to avoid using gamma transformation, unless requested by the user.

[Drvi]

nearly all plugins (except PNG-FI) return gamma uncorrected pixel rgba(233,173,4,244) for the file containing the gAMA chunk.

Well, but is it desirable? If someone stored the pixels with a specific gAMA value, why should we ignore it? I'm still assuming here that gAMA corresponds to the file_gamma value in the equations above.

To turn off gamma correction in PNGFiles.jl, one can provide gamma=1.0 (meaning file_gamma * screen_gamma should cancel each other out and no correction should happen). Now looking at the code it probably should be changed to:

    elseif gamma != 1
        image_gamma[] = gamma / screen_gamma  # before: was image_gamma[] = gamma
        png_set_gamma(png_ptr, screen_gamma, image_gamma[])
    end

and make it explicit, that the gamma argument is the "end-to-end" gamma... But as a "turn-off" switch for gamma correction it works.

Is the source of difference between PNGFiles / ImageMagick and the python packages you mentioned the fact that we do gamma correction by default?

[t-bltg]

Well, but is it desirable? If someone stored the pixels with a specific gAMA value, why should we ignore it? I'm still assuming here that gAMA corresponds to the file_gamma value in the equations above.

Let's wait for the experts answer ;)

Now looking at the code it probably should be changed to:
elseif gamma != 1
image_gamma[] = gamma / screen_gamma # before: was image_gamma[] = gamma
png_set_gamma(png_ptr, screen_gamma, image_gamma[])
end

For this to work, you'd have to set screen_gamma = PNG_GAMMA_sRGB / PNG_FP_1, to get screen_gamma = 2.2 before, no ?

Is the source of difference between PNGFiles / ImageMagick and the python packages you mentioned the fact that we do gamma correction by default?

I'm digging into ReadOnePNGImage to see what really happens with gamma correction.

[Drvi]

For this to work, you'd have to set screen_gamma = PNG_GAMMA_sRGB / PNG_FP_1, to get screen_gamma = 2.2 before, no ?

Yes! I forgot screen_gamma does not actually contain the screen gamma value at this point.

I'm digging into ReadOnePNGImage to see what really happens with gamma correction.

I've attempted and gave up reading that 14k LOC file multiple times... Thanks for investigating this!

[johnnychen94]

Should we completely skip gamma correction when the file_gamma is 1.0? How about for file_gamma 0.99999? Are our results correct?

I don't know much about color space management, so my words here are opinioned from my own understanding of the specification.

We should skip gamma correction when it's 1.0 since the correction degenerate to an identity map (This could potentially give better decoding performance by telling more information to the libpng implementation but who knows 🤷‍♂️). If it's 0.99999 it has negligible differences so it doesn't matter in practice.

It's still my opinioned understanding, I assume our results are correct as it follows the specification suggestion. In https://www.w3.org/TR/2003/REC-PNG-20031110/#11gAMA "11.3.3 Colour space information", it says

If the adjustment is not performed, the error is usually small. Applications desiring high colour fidelity may wish to use an sRGB chunk or iCCP chunk.

An sRGB chunk or iCCP chunk, when present and recognized, overrides the gAMA chunk.

A PNG encoder that writes the iCCP chunk is encouraged to also write gAMA and cHRM chunks that approximate the ICC profile, to provide compatibility with applications that do not use the iCCP chunk.

PNG decoders that are used in an environment that is incapable of full-fledged colour management should use the gAMA and cHRM chunks if present.

I assume that we should stick to applying gamma correction by default, and also provide a way to override it (if we don't yet have it).

Personally, I would insist to say this inconsistency between Julia/C/Python is implementation differences since the specification doesn't say what is strictly correct. We can choose to make the gaps as close as possible. However, since there can be more unidentified cases like this and it can never be made so.

[johnnychen94]

I missed to quote another suggestion in the same chapter:

For gAMA these are the reference viewing conditions of the sRGB specification [IEC 61966-2-1], which are based on ISO 3664 [ISO-3664]. Adjustment for different viewing conditions is normally handled by a Colour Management System.

This seems to indicate that pure image IO backends (pillow, PNGFiles, etc) are discouraged to use the gAMA component?

[Drvi]

We should skip gamma correction when it's 1.0 since the correction degenerate to an identity map (This could potentially give better decoding performance by telling more information to the libpng implementation but who knows 🤷‍♂️). If it's 0.99999 it has negligible differences so it doesn't matter in practice.

Thanks for looking into this, @johnnychen94 . IIUC, there are two high-level component: the screen/display gamma and the file gamma. The gAMA chunk encodes the file gamma part. If the gAMA is 1.0, then the transformation is not an identity map; it collapses to the following (assuming screen gamma to be 2.2):

corrected = round(Int, 255 * ((raw / 255) ^ (1.0 / (file_gamma * display_gamma))))
corrected = round(Int, 255 * ((raw / 255) ^ (1.0 / (1.0 * 2.2))))
corrected = round(Int, 255 * ((raw / 255) ^ (0.45455)))

our current gamma argument is supposed to override the "total exponent" (file_gamma * display_gamma), gamma being = 1.0 meaning no gamma correction (the identity map case). I realized the implementation is not correct (#48 (comment)) apart from the "identity map" case, but that is not an issue here.

This seems to indicate that pure image IO backends (pillow, PNGFiles, etc) are discouraged to use the gAMA component?

I guess the question is: do we want to always pass pixels data in raw form (i.e. disregard gamma correction, but the pixels might be stored gamma corrected or not) or should we always do gamma correction (and then the pixels are always gamma corrected, or at least attempted to be). My gut feeling was that if data was stored in png (and not HDF5, or Arrow...), the the focus was on display-ability, but I think the spec is rich enough to accomodate even the data processing use case.

For different viewing conditions, the gamma correction could be adjusted for e.g. viewing in a dim room... not sure if that is what is meant in the quote?

[t-bltg]

The ImageMagick.jl pipeline looks like this:

ImageMagick.jl -> exportimagepixels! -> ExportImagePixels (ImageMagick/MagickCore/pixel.c) -> ExportCharPixel (ImageMagick/MagickCore/pixel.c)

Adding debugging to a custom build of ImageMagick in ExportCharPixel gives the following:
ExportCharPixel (2, 2) ScaleQuantumToChar(62984,55215,8638) = (245,215,34)

ScaleQuantumToChar(x) returns Uint8(55215 / 257 + 0.5) = 215 for the green channel which is the correct value, using a quantum depth value of 16 as is the default in ImageMagick. Does this mean that the channel value encoded in the image file is a 16bit value ?

Some answer elements from libpng-manual.txt

The channels are encoded in one of two ways:

a) As a small integer, value 0..255, contained in a single byte. For the
alpha channel the original value is simply value/255. For the color or
luminance channels the value is encoded according to the sRGB specification
and matches the 8-bit format expected by typical display devices.

The color/gray channels are not scaled (pre-multiplied) by the alpha
channel and are suitable for passing to color management software.

b) As a value in the range 0..65535, contained in a 2-byte integer, in
the native byte order of the platform on which the application is running.
All channels can be converted to the original value by dividing by 65535; all
channels are linear. Color channels use the RGB encoding (RGB end-points) of
the sRGB specification. This encoding is identified by the
PNG_FORMAT_FLAG_LINEAR flag below.

The samples are either contained directly in the image data, between 1 and 8
bytes per pixel according to the encoding, or are held in a color-map indexed
by bytes in the image data. In the case of a color-map the color-map entries
are individual samples, encoded as above, and the image data has one byte per
pixel to select the relevant sample from the color-map.
PNG_FORMAT_*

A format is built up using single bit flag values. All combinations are
valid. Formats can be built up from the flag values or you can use one of
the predefined values below. When testing formats always use the FORMAT_FLAG
macros to test for individual features - future versions of the library may
add new flags.

When reading or writing color-mapped images the format should be set to the
format of the entries in the color-map then png_image_{read,write}_colormap
called to read or write the color-map and set the format correctly for the
image data. Do not set the PNG_FORMAT_FLAG_COLORMAP bit directly!

The samples are either contained directly in the image data, between 1 and 8
bytes per pixel according to the encoding, or are held in a color-map indexed
by bytes in the image data. In the case of a color-map the color-map entries
are individual samples, encoded as above, and the image data has one byte per
pixel to select the relevant sample from the color-map.

From what I understand, if the file format if the PNG_FORMAT_FLAG_LINEAR was set during writing, then we should be able to get the 2-byte channel data.

EDIT1: getting close, from libpng, in png_handle_PLTE (pngrutil.c): (pal_ptr->red, pal_ptr->green, pal_ptr->blue) is assigned to

      i=13
        R=233
        G=173
        B=4
         reading 3 bytes

what we have just after calling png_get_PLTE are transformed values:
Int.(palette[3 * (14 - 1) + 1:3 * (14 - 1) + 3]) = [245, 214, 39].

EDIT2: this is were the palette is transformed pngrtran.c.

EDIT3: to build 214, this happens in png_gamma_8bit_correct: floor(255 * pow(0.678431, 0.454550) + .5) = 214.000000 -> 214, this means that my RGB to linear function should be linear(x) = floor(Int, 255((x / 255)^(1 / .45455))).

EDIT4:
ImageMagick uses uncorrected palette values from libpng, whereas PNGFiles uses gamma corrected ones (debug output from libpng):

ImageMagick:

   in PLTE retrieval function
         num_palette = 47
         get i=13
           R=233
           G=173
           B=4

PNGFiles:

   in PLTE retrieval function
         num_palette = 47
         get i=13
           R=245
           G=214
           B=39

In ImageMagick, if image_gamma > 0.75, the colorspace is set to RGBColorspace (see png.c) and the pixel gets encoded using the sRGB transformation pixel.c.

[t-bltg]

I've updated https://github.com/t-bltg/PngPixel.jl, the README.md resumes it all (I tried, to the best of my understanding, to make it as clear as possible).

I can get ImageMagick.jl and PNGFiles.jl to output the values that I consider correct (233,173,4,244) (uncorrected gamma):

• ImageMagick: uncorrect gamma using sRGB to CIE XYZ.
• PNGFiles: uncorrect gamma by taking ...^(file_gamma * screen_gamma) where file_gamma = 1. and screen_gamma = 2.2 as explained in Different png RGBA values using Julia, C or Python #48 (comment).

Since the two implementations give different results on gamma corrected values, should we not take gamma correction into account unless explicitly requested by the user as a flag in load(...) ?

This seems to indicate that pure image IO backends (pillow, PNGFiles, etc) are discouraged to use the gAMA component?

I think yes, it should be left to the CMS, unless explicitly requested.

Here are the two references quoted in the imageio issue resuming the situation:

• https://jonathannicol.com/blog/2006/12/01/fixing-png-gamma/
• https://hsivonen.fi/png-gamma/