Skip to content

Coherent Video Inpainting Using Optical Flow-Guided Efficient Diffusion

License

Notifications You must be signed in to change notification settings

NevSNev/FloED-main

Folders and files

NameName
Last commit message
Last commit date

Latest commit

66a8479 · Mar 19, 2025

History

57 Commits
Mar 19, 2025
Mar 12, 2025
Dec 9, 2024
Mar 19, 2025

Repository files navigation

FloED: Coherent Video Inpainting Using Optical Flow-Guided Efficient Diffusion

Bohai Gu   Hao Luo   Song Guo   Peiran Dong   Qihua Zhou

HKUST, DAMO Academy, Alibaba Group

arXiv Project Page

The text-guided video inpainting technique has significantly improved the performance of content generation applications. A recent family for these improvements uses diffusion models, which have become essential for achieving high-quality video inpainting results, yet they still face performance bottlenecks in temporal consistency and computational efficiency. This motivates us to propose a new video inpainting framework using optical Flow-guided Efficient Diffusion (FloED) for higher video coherence. Specifically, FloED employs a dual-branch architecture, where the time-agnostic flow branch restores corrupted flow first, and the multi-scale flow adapters provide motion guidance to the main inpainting branch. Besides, a training-free latent interpolation method is proposed to accelerate the multi-step denoising process using flow warping. With the flow attention cache mechanism, FLoED efficiently reduces the computational cost of incorporating optical flow. Extensive experiments on background restoration and object removal tasks show that FloED outperforms state-of-the-art diffusion-based methods in both quality and efficiency.

Method Overview

Video Demo

video demo

Please refer our project page for more details.

To-Do List

  • Release the inference code and weights

  • Release the training code

BibTeX

@article{gu2024advanced,
  title={Advanced Video Inpainting Using Optical Flow-Guided Efficient Diffusion},
  author={Gu, Bohai and Luo, Hao and Guo, Song and Dong, Peiran},
  journal={arXiv preprint arXiv:2412.00857},
  year={2024}
}

About

Coherent Video Inpainting Using Optical Flow-Guided Efficient Diffusion

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published