[BUG] Different images are considered equal

[carlosame]

Description
I'm using this project as one of the image comparison methods being used in the EchoSVG tests, and noticed that one of the tests that should fail was systematically passing. For example, the following two images are identical (match) according to this project (version 4.4.0):

The last color is not the same (#c20272 vs #df0272), and the text located next to that color also varies (the images are transparent, so if you are using Github's dark theme you may need to download them and open elsewhere to read it).

To Reproduce
To be sure, I ran this separate piece of code to compare the two images (in case it was my testing infrastructure screwing things up):

BufferedImage expectedImage = ImageComparisonUtil.readImageFromResources("C:\\path\\to\\domSVGColor.png");
BufferedImage actualImage = ImageComparisonUtil.readImageFromResources("C:\\path\\to\\domSVGColor-new.png");
ImageComparison comparison = new ImageComparison(expectedImage, actualImage);
ImageComparisonResult result = comparison.compareImages();
assertTrue(ImageComparisonState.MATCH != result.getImageComparisonState());

and it failed (the different images do match). The above snippet is the obvious reproducer.

Expected behavior
The images are different and should not match.

Desktop

• OS: Windows 10
• Image-comparison version: 4.4.0.

[romankh3]

Hi @carlosame.
Thanks for opening a Bug. I will look at it and will respond you.

Also, feel free to do it by yourself if you want it.

Best regards, Roman.

[carlosame]

feel free to do it by yourself if you want it.

I rolled out my project's own image comparisons, given that it was the most natural thing to do: EchoSVG is a fork of Apache Batik, which already had image comparisons (and even assembled diff and side-to-side comparison images). Batik's image comparisons were done at the file stream level though, which is why I introduced your project in EchoSVG.

Now EchoSVG does its own image comparisons at the pixel level, and it appears to work well (thanks to it, I found several more images where image-comparison did not catch a font change). The new image comparison procedures are explained in the IMAGE_COMPARISONS.md top-level file at the EchoSVG repository.

Although I no longer use this project, I hope that you manage to fix this issue given that AFAIK this is the only general-purpose image comparison library written in Java (I haven't tested my stuff outside of my project's own files and therefore cannot endorse it as a generic solution).

[Cheese-Bar]

Hi, @romankh3 ,
I would like to work on this issues.

[romankh3]

Hi @Cheese-Bar, feel free to work on it. Wish you good luck 👍

[Cheese-Bar]

Hello @romankh3 ,
I have reproduced this bug, and I thought I have located the problem. It may caused by isDifferentPixels(expected.getRGB(x, y), actual.getRGB(x, y)). It failed to pick up on subtle changes in color and text in the two images. I also generate pure color picture to test it.

I found isDifferentPixels() function is related to variable pixelToleranceLevel = 0.1, so I try to adjust this configuration. When set pixelToleranceLevel = 0.07 can tell the difference between the two colors. But unless I set it to 0.0, It still can't detect changes in the number.

The picture below shows the result at pixelToleranceLevel = 0.07

The next picture shows the result at pixelToleranceLevel = 0.0

But unfortunately, such a change seems to affect the detection of other images, especially with .jpg picture.

In shouldProperlyCompareInJpeg() the result is a little different with result#21.png as you provided. Because it's fail to pass assertImagesEqual test. However, it looks give a good detection as human view, you can see below.

But in shouldProperlyCompareInJpeg(), It seems a big difference. It saw most of the image as different.

I carefully compared the two provided pictures of the Rubik's cube.

The problem should be that the two images are obtained in different ways, or there is a compression loss. 9.15KB for expected.jpg while 15.96KB for actual.jpg. In addition, due to the JPEG file format, the two images have different pixels in many details, especially where the color changes, Zoom in and you can see the difference with the human eye.

So maybe it's a useful proposition to specify the size of pixelToleranceLevel depending on the file format.
Or if the program is used to verify the reliability of documents, such as judicial evidence, set pixelToleranceLevel = 0.0 is useful.

How do you consider about it? I can provided my test case or do some further improvement.