Benchmark against other IQA packages

[pythonmobile]

Do you know how the SSIM implementation in PIQA compares in speed with piq and pytorch_ssim? In CPU and GPU?

[francois-rozet]

Hello @pythonmobile,

I use a very simple script to benchmark my implementations against other IQA packages (piq, kornia, pytorch_msssim, ...). For instance, on my machine,

$ cd tests
$ python -O benchmark.py --metrics SSIM --device cuda
SSIM
----
        method     value      time
0    piqa.SSIM  0.724984  1.115653
1   vainf.SSIM  0.724984  1.201881
2     IQA.SSIM  0.725902  1.748422
3     piq.ssim  0.724968  1.822861
4  kornia.SSIM  0.731045  2.218630

where the value is the output of each method (for consistency) and time is the average runtime (in ms) over 420 one-image batches. The gap is larger if we activate Just-In-Time compilation (see #JIT):

$ PIQA_JIT=1 python -O benchmark.py --metrics SSIM --device cuda
SSIM
----
        method     value      time
0    piqa.SSIM  0.724984  0.864115
1   vainf.SSIM  0.724984  1.224833
2     IQA.SSIM  0.725902  1.752191
3     piq.ssim  0.724968  1.784100
4  kornia.SSIM  0.731045  2.211813

The performance boost comes from separating the 2-dimensional convolution kernel into two 1-dimensional kernels and applying them successively (see separable kernels), which makes GPU convolutions faster and allows to compute the SSIM for 3 or 4-dimensional images! Unfortunately, CPUs are not nearly as good for convolutions as GPUs, leading this optimization to actually increase the runtime on CPUs.

$ python -O benchmark.py --metrics SSIM --device cpu
SSIM
----
        method     value       time
0     IQA.SSIM  0.725902  29.809328
1     piq.ssim  0.724968  30.190178
2  kornia.SSIM  0.731045  37.022584
3    piqa.SSIM  0.724984  38.723981
4   vainf.SSIM  0.724984  40.865401

SSIM is the only metric for which this separation trick is implemented. For the other metrics, PIQA's implementation are faster on both GPU and CPU.

$ python -O benchmark.py --device cuda
TV
--
      method         value      time
0  kornia.tv  15553.015625  0.160524
1    piqa.TV  15553.015625  0.184596
2     piq.tv  15553.015625  0.199443

PSNR
----
        method      value      time
0    piqa.PSNR  27.567236  0.125523
1  kornia.PSNR -27.567261  0.157243
2     piq.psnr  27.567236  0.165018

SSIM
----
        method     value      time
0    piqa.SSIM  0.724984  0.852041
1   vainf.SSIM  0.724984  1.203600
2     IQA.SSIM  0.725902  1.748456
3     piq.ssim  0.724968  1.782655
4  kornia.SSIM  0.731045  2.212489

MS-SSIM
-------
          method     value      time
0   piqa.MS_SSIM  0.939777  1.917572
1    piq.ms_ssim  0.939775  3.137109
2    IQA.MS_SSIM  0.939794  4.009950
3  vainf.MS_SSIM  0.939777  4.196562

GMSD
----
      method     value      time
0  piqa.GMSD  0.094408  0.417131
1   piq.gmsd  0.094407  0.946302

MS-GMSD
-------
         method     value      time
0  piqa.MS_GMSD  0.098538  1.281921
1   piq.ms_gmsd  0.098537  3.477191

MDSI
----
      method     value      time
0  piqa.MDSI  0.331479  0.948595
1   piq.mdsi  0.331473  2.339903

HaarPSI
-------
         method     value      time
0  piqa.HaarPSI  0.793545  1.459722
1   piq.haarpsi  0.793538  3.060413

VSI
---
     method     value      time
0  piqa.VSI  0.975185  2.329966
1   piq.vsi  0.975762  6.605236

FSIM
----
      method     value       time
0  piqa.FSIM  0.939357   3.248637
1   piq.fsim  0.939357  22.771682