When trying to increase FPS (Frames Per Second) of an existing video, from 30fps to 60fps, for example, video interpolation will allow you adding these "missing" frames to a video.
These frames would need to be created by checking the previous + next frame, comparing them, and creating a new frame with the proper data.
AI is used to run this kind of calculation, which is very demanding from hardware perspective, specifically GPU power, and VRAM. In many cases, users would need to upgrade their GPU, or use their gaming PC video card for this purpose.
Using cloud resources will allow you to run the heavy lifting (all AI calculation) on another server. The output file would then be uploaded where you could download it when ready.
This approach will allow you running this process on virtually any PC or OS, including your laptop / Mac.
Running cloud workloads may vary in terms of pricing, and is based on [Type of hardware] x [time used].
-
Upload your
mp4file to a publicly available location
We will use this location on the next steps to download this file directly to the server we will use.
Notice this file should be accessible directly, i.e. link should end with...filename.mp4
Make sure link works by clicking on it, and see it works and start downloading the file. -
Prepare your cloud VM
1. Create an account (or use existing one) of Vultr, Linode. Both have options of GPU VMs with plenty of VRAM, charged by usage (hr).
2. VM size: Select the most appropriate plan for you (check the benchmark page for reference).
3. OS: Select Fedora 36, we will install docker + NVIDIA drivers later.
Once all ready - go and create the VM.
4. When VM is ready, login to it using SSH, if you're a windows user, you could use putty client. \ -
added fifth
This project is the implement of Real-Time Intermediate Flow Estimation for Video Frame Interpolation. Currently, our model can run 30+FPS for 2X 720p interpolation on a 2080Ti GPU. It supports arbitrary-timestep interpolation between a pair of images.
2022.7.4 - Our paper is accepted by ECCV2022 🎉. Thanks to all relevant authors, contributors and users!
From 2020 to 2022, we submitted RIFE for five submissions（rejected by CVPR21 ICCV21 AAAI22 CVPR22). Thanks to all anonymous reviewers, your suggestions have helped to significantly improve the paper!
Pinned Software: RIFE-App | FlowFrames | SVFI (中文)
16X interpolation results from two input images:
Flowframes | SVFI(中文) | Waifu2x-Extension-GUI | Autodesk Flame | SVP | MPV_lazy
RIFE-App(Paid) | Steam-VFI(Paid)
We are not responsible for and participating in the development of above software. According to the open source license, we respect the commercial behavior of other developers.
VapourSynth-RIFE | RIFE-ncnn-vulkan | VapourSynth-RIFE-ncnn-Vulkan
If you are a developer, welcome to follow Practical-RIFE, which aims to make RIFE more practical for users by adding various features and design new models with faster speed.
git clone git@github.com:megvii-research/ECCV2022-RIFE.git
cd ECCV2022-RIFE
pip3 install -r requirements.txt
-
Download the pretrained HD models from here. (百度网盘链接:https://pan.baidu.com/share/init?surl=u6Q7-i4Hu4Vx9_5BJibPPA 密码:hfk3，把压缩包解开后放在 train_log/*)
-
Unzip and move the pretrained parameters to train_log/*
-
This model is not reported by our paper, for our paper model please refer to evaluation.
Video Frame Interpolation
You can use our demo video or your own video.
python3 inference_video.py --exp=1 --video=video.mp4
(generate video_2X_xxfps.mp4)
python3 inference_video.py --exp=2 --video=video.mp4
(for 4X interpolation)
python3 inference_video.py --exp=1 --video=video.mp4 --scale=0.5
(If your video has very high resolution such as 4K, we recommend set --scale=0.5 (default 1.0). If you generate disordered pattern on your videos, try set --scale=2.0. This parameter control the process resolution for optical flow model.)
python3 inference_video.py --exp=2 --img=input/
(to read video from pngs, like input/0.png ... input/612.png, ensure that the png names are numbers)
python3 inference_video.py --exp=2 --video=video.mp4 --fps=60
(add slomo effect, the audio will be removed)
python3 inference_video.py --video=video.mp4 --montage --png
(if you want to montage the origin video and save the png format output)
Optical Flow Estimation
You may refer to #278.
Image Interpolation
python3 inference_img.py --img img0.png img1.png --exp=4
(2^4=16X interpolation results) After that, you can use pngs to generate mp4:
ffmpeg -r 10 -f image2 -i output/img%d.png -s 448x256 -c:v libx264 -pix_fmt yuv420p output/slomo.mp4 -q:v 0 -q:a 0
You can also use pngs to generate gif:
ffmpeg -r 10 -f image2 -i output/img%d.png -s 448x256 -vf "split[s0][s1];[s0]palettegen=stats_mode=single[p];[s1][p]paletteuse=new=1" output/slomo.gif
Place the pre-trained models in train_log/\*.pkl (as above)
Building the container:
docker build -t rife -f docker/Dockerfile .
Running the container:
docker run --rm -it -v $PWD:/host rife:latest inference_video --exp=1 --video=untitled.mp4 --output=untitled_rife.mp4
docker run --rm -it -v $PWD:/host rife:latest inference_img --img img0.png img1.png --exp=4
Using gpu acceleration (requires proper gpu drivers for docker):
docker run --rm -it --gpus all -v /dev/dri:/dev/dri -v $PWD:/host rife:latest inference_video --exp=1 --video=untitled.mp4 --output=untitled_rife.mp4
Download RIFE model or RIFE_m model reported by our paper.
UCF101: Download UCF101 dataset at ./UCF101/ucf101_interp_ours/
Vimeo90K: Download Vimeo90K dataset at ./vimeo_interp_test
MiddleBury: Download MiddleBury OTHER dataset at ./other-data and ./other-gt-interp
HD: Download HD dataset at ./HD_dataset. We also provide a google drive download link.
# RIFE
python3 benchmark/UCF101.py
# "PSNR: 35.282 SSIM: 0.9688"
python3 benchmark/Vimeo90K.py
# "PSNR: 35.615 SSIM: 0.9779"
python3 benchmark/MiddleBury_Other.py
# "IE: 1.956"
python3 benchmark/HD.py
# "PSNR: 32.14"
# RIFE_m
python3 benchmark/HD_multi_4X.py
# "PSNR: 22.96(544*1280), 31.87(720p), 34.25(1080p)"
Download Vimeo90K dataset.
We use 16 CPUs, 4 GPUs and 20G memory for training:
python3 -m torch.distributed.launch --nproc_per_node=4 train.py --world_size=4
2021.3.18 arXiv: Modify the main experimental data, especially the runtime related issues.
2021.8.12 arXiv: Remove pre-trained model dependency and propose privileged distillation scheme for frame interpolation. Remove census loss supervision.
2021.11.17 arXiv: Support arbitrary-time frame interpolation, aka RIFEm and add more experiments.
We sincerely recommend some related papers:
CVPR22 - Optimizing Video Prediction via Video Frame Interpolation
CVPR22 - Video Frame Interpolation with Transformer
CVPR22 - IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation
@inproceedings{huang2022rife,
title={Real-Time Intermediate Flow Estimation for Video Frame Interpolation},
author={Huang, Zhewei and Zhang, Tianyuan and Heng, Wen and Shi, Boxin and Zhou, Shuchang},
booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
year={2022}
}
Optical Flow: ARFlow pytorch-liteflownet RAFT pytorch-PWCNet
Video Interpolation: DVF TOflow SepConv DAIN CAIN MEMC-Net SoftSplat BMBC EDSC EQVI