Learning Enriched Features for Real Image Restoration and Enhancement (ECCV 2020)

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, Ling Shao

---

News

• Keras Tutorial on MIRNet is available at https://keras.io/examples/vision/mirnet/

• Video on Tensorflow Youtube channel https://youtu.be/BMza5yrwZ9s

• Links to (unofficial) implementations are added here

---

Abstract: 为了从退化的图像中恢复高质量的图像内容，图像恢复在监控、计算摄影、医学成像和遥感等领域有着广泛的应用。近年来，卷积神经网络（CNN）在图像恢复方面取得了显著的进步。现有的基于CNN的方法通常在全分辨率或渐进低分辨率表示上运行。在前一种情况下，实现了空间上精确但上下文上不太可靠的结果，而在后一种情况下，生成了语义上可靠但空间上不太准确的输出。在本文中，我们提出了一种新的体系结构，其共同目标是在整个网络中保持空间精确的高分辨率表示，并从低分辨率表示中接收强上下文信息。我们方法的核心是一个多尺度剩余块，包含几个关键元素：（a）用于提取多尺度特征的并行多分辨率卷积流，（b）跨多分辨率流的信息交换，（c）用于捕获上下文信息的空间和通道注意机制，以及（d）基于注意的多尺度特征聚合。简而言之，我们的方法学习了一组丰富的特征，这些特征结合了来自多个尺度的上下文信息，同时保留了高分辨率的空间细节。在五个真实图像基准数据集上进行的大量实验表明，我们的方法MIRNet在图像去噪、超分辨率和图像增强等多种图像处理任务中取得了最先进的结果。

Network Architecture (click to expand)

Overall Framework of MIRNet

Selective Kernel Feature Fusion (SKFF)	Downsampling Module

Dual Attention Unit (DAU)	Upsampling Module

Installation

The model is built in PyTorch 1.1.0 and tested on Ubuntu 16.04 environment (Python3.7, CUDA9.0, cuDNN7.5).

For installing, follow these intructions

sudo apt-get install cmake build-essential libjpeg-dev libpng-dev
conda create -n pytorch1 python=3.7
conda activate pytorch1
conda install pytorch=1.1 torchvision=0.3 cudatoolkit=9.0 -c pytorch
pip install matplotlib scikit-image opencv-python yacs joblib natsort h5py tqdm

Training

1. Download the SIDD-Medium dataset from here
2. Generate image patches

python generate_patches_SIDD.py --ps 256 --num_patches 300 --num_cores 10

3. Download validation images of SIDD and place them in ../SIDD_patches/val

4. Install warmup scheduler

cd pytorch-gradual-warmup-lr; python setup.py install; cd ..

5. Train your model with default arguments by running

python train_denoising.py

Note: Our model is trained with 2 Nvidia Tesla-V100 GPUs. See #5 for changing the model parameters.

Evaluation

You can download, at once, the complete repository of MIRNet (including pre-trained models, datasets, results, etc) from this Google Drive link, or evaluate individual tasks with the following instructions:

Image Denoising

• Download the model and place it in ./pretrained_models/denoising/

Testing on SIDD dataset

• Download sRGB images of SIDD and place them in ./datasets/sidd/
• Run

python test_sidd_rgb.py --save_images

Testing on DND dataset

• Download sRGB images of DND and place them in ./datasets/dnd/
• Run

python test_dnd_rgb.py --save_images

Image Super-resolution

• Download the models and place them in ./pretrained_models/super_resolution/
• Download images of different scaling factor and place them in ./datasets/super_resolution/
• Run

python test_super_resolution.py --save_images --scale 3
python test_super_resolution.py --save_images --scale 4

Image Enhancement

Testing on LOL dataset

• Download the LOL model and place it in ./pretrained_models/enhancement/
• Download images of LOL dataset and place them in ./datasets/lol/
• Run

python test_enhancement.py --save_images --input_dir ./datasets/lol/ --result_dir ./results/enhancement/lol/ --weights ./pretrained_models/enhancement/model_lol.pth

Testing on Adobe-MIT FiveK dataset

• Download the FiveK model and place it in ./pretrained_models/enhancement/
• Download some sample images of fiveK dataset and place them in ./datasets/fivek_sample_images/
• Run

python test_enhancement.py --save_images --input_dir ./datasets/fivek_sample_images/ --result_dir ./results/enhancement/fivek/ --weights ./pretrained_models/enhancement/model_fivek.pth

Results

Experiments are performed on five real image datasets for different image processing tasks including, image denoising, super-resolution and image enhancement. Images produced by MIRNet can be downloaded from Google Drive link.

Image Denoising (click to expand)

Image Super-resolution (click to expand)

Image Enhancement (click to expand)

Other Implementations

• Tensorflow (Soumik Rakshit)
• Tensorflow-JS (Rishit Dagli)
• Tensorflow-TFLite (Sayak Paul)

Citation

If you use MIRNet, please consider citing:

@inproceedings{Zamir2020MIRNet,
    title={Learning Enriched Features for Real Image Restoration and Enhancement},
    author={Syed Waqas Zamir and Aditya Arora and Salman Khan and Munawar Hayat
            and Fahad Shahbaz Khan and Ming-Hsuan Yang and Ling Shao},
    booktitle={ECCV},
    year={2020}
}

Contact

Should you have any question, please contact waqas.zamir@inceptioniai.org

Our Related Works

• Restormer: Efficient Transformer for High-Resolution Image Restoration, CVPR 2022. Paper | Code
• Multi-Stage Progressive Image Restoration, CVPR 2021. Paper | Code
• CycleISP: Real Image Restoration via Improved Data Synthesis, CVPR 2020. Paper | Code