SSIM result issue

[Jerry0103240]

Hi, I would like to confirm what metric is used in paper, cause I use SSIM get about 0.87, but use CW_SSIM get 0.93 (by Pyssim package) in GoPro gamma subset (1111 images for testing, 720 x 1280 resolutions), and the value in paper is about 0.9135
Thanks for replying!

[SeungjunNah]

SSIM functions from various packages have different implementations and options.
When I wrote this paper, I used MATLAB to measure SSIM of the RGB images which was wrong.
MATLAB has wrong SSIM implementation on RGB images as it extends its 1D implementation to 2D and 3D. RGB images are considered 3D array (H x W x C) and padded in C dimension before SSIM calculation to apply 11x11x11 kernel. (RRRRRGBBBBB)

These days, I don't use MATLAB and use skimage.metrics.structural_similarity.

from skimage.metrics import structural_similarity
ssim = structural_similarity(ref_im, res_im, multichannel=True, gaussian_weights=True, use_sample_covariance=False)

[Jerry0103240]

Thanks for the prompt reply and sorry for my late response, the answer is helpful!

BTW, I was testing with “scale_levels 1” which is an option to change in opts.lua file, but the results seems the same. (the downloaded weight file name are related to scale level 3, I’m not sure if this is the main reason). Is there anything I didn’t notice?

And what are the correct results if I change type Cudahalf to Cuda, (The inference time is longer, but results may not change a lot, It’s correct or not?)

Thanks for your patience!

[SeungjunNah]

When loading a saved model, the options related to the models are ignored. (including scale_levels)
The inference result of cudaHalf and cuda equal to each other. 16 bits are enough for general image restoration tasks at test time.

[Jerry0103240]

Thanks for your patience to this issue and clear explanation!

[hhrsn]

来自各种包的 SSIM 函数具有不同的实现和选项。当我写这篇论文时，我使用MATLAB来测量RGB图像的SSIM，这是错误的。 MATLAB 在将其 1D 实现扩展到 2D 和 3D 时，对 RGB 图像的 SSIM 实现有错误。RGB 图像被视为 3D 数组（高 x 宽 x 高），并在 SSIM 计算之前以 C 维填充以应用 11x11x11 内核。（RRRRRGBBBBB）

这些天，我不使用 MATLAB，而是使用 skimage.metrics.structural_similarity。

from skimage.metrics import structural_similarity
ssim = structural_similarity(ref_im, res_im, multichannel=True, gaussian_weights=True, use_sample_covariance=False)

Can you tell me which code to run to get the evaluation metrics after the test, looking forward to your reply?

[Jerry0103240]

Can you tell me which code to run to get the evaluation metrics after the test, looking forward to your reply?

@hhrsn

Hi, I open the issue about 3 years ago, some details might go wrong.

If I remember correctly, according to author's response, I tried MATLAB CW-SSIM or SSIM to get similar SSIM results mentioned in papaer.

you can try forward method first

[SeungjunNah]

@hhrsn You can refer to the code here https://github.com/SeungjunNah/DeepDeblur-PyTorch/blob/master/src/loss/metric.py#L30
It is a pytorch-ported version of skimage.metrics.structural_similarity. There will be minor numerical differences due to the difference between numpy and pytorch CudaTensor operations.