✨ RFDA-Pytorch ✨

Official Code for 'Recursive Fusion and Deformable Spatiotemporal Attention for Video Compression Artifact Reduction'

ACM Multimedia 2021 (ACMMM2021) Accepted Paper

Task: Video Quality Enhancement / Video Compression Artifact Reduction

The code will be gradually open source!

Open Source Scheduler

1 Release RF and DSTA core code within one month after camera ready [Done]

2 Release test code and models at five QP [Done]

3 Release train code (you know, in a mass ) [Done]

📧 Contact 📧

Feel free to contact me if you have any problems! zhaomy20@fudan.edu.cn

1. Pre-request

1.1. Environment

• Ubuntu 20.04/18.04
• CUDA 10.1
• PyTorch 1.6
• Packages: tqdm, lmdb, pyyaml, opencv-python, scikit-image

Suppose that you have installed CUDA 10.1, then:

$ git clone --depth=1 https://github.com/zhaominyiz/RFDA-PyTorch 
$ cd RFDA-PyTorch/
$ conda create -n video python=3.7 -y
$ conda activate video
$ python -m pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
$ python -m pip install tqdm lmdb pyyaml opencv-python scikit-image

1.2. DCNv2

Build DCNv2.

$ cd ops/dcn/
$ bash build.sh

(Optional) Check if DCNv2 works.

$ python simple_check.py

The DCNv2 source files here is different from the open-sourced version due to incompatibility. [issue]

🔥 2. Train 🔥

Build the dataset! check create_lmdb_mfqe_rf.py

Train RF only!

Train RFDA! check train_rf_ft.py!

⚡ 3. Test ⚡

3.1 Test MFQE 2.0 dataset

Please build the MFQE 2.0 dataset first (See Here), then run test_yuv_RF.py.

More instructions will coming soon!

$ python test_yuv_RF.py --opt_path config/****.yml

3.2 Test your own video clip

For yuv videos, you may refer to test_one_video_yuv_RF.py.

$ python test_one_video_yuv_RF.py --opt_path config/****.yml

For rgb videos, we will update new model and codes soon.

🌱 3.3 Pretrain models 🌱

RFDAQP22,27,32,37,42(trained on YUV space): BaiduDisk (RFDA) GoogleDisk

BRFQP37(trained on YUV space): ToBeDone!

🍻 Results 🍻

Comparison with State of the Art Methods

Speed and parameter size comparison

💖 Visualization Video Demo 💖

RFDA_Quick_DEMO.mp4

😉 Related Works 😉

· Boosting the performance of video compression artifact reduction with reference frame proposals and frequency domain information [Paper]

· Non-local convlstm for video compression artifact reduction [Paper] [Code]

😆 Citation 😆

If you find this project is useful for your research, please cite:

@inproceedings{zhao2021recursive,
  title={Recursive Fusion and Deformable Spatiotemporal Attention for Video Compression Artifact Reduction},
  author={Zhao, Minyi and Xu, Yi and Zhou, Shuigeng},
  booktitle={Proceedings of the 29th ACM International Conference on Multimedia},
  pages={5646--5654},
  year={2021}
}

👍 Special Thanks 👍

Our framework is based on STDF-Pytoch. Thank RyanXingQL for his work!