Skip to content

🐶 「Updating ...」Replacing Real-Time the High-Resolution Meeting Background Freely

License

Notifications You must be signed in to change notification settings

Charmve/TimeWarp

Repository files navigation

TimeWarp

TimeWarp - 在线视频背景替换

Replacing Real-Time the High-Resolution Meeting Background Freely.


Example comparison in a video call
Current video conferencing tools like Zoom can take an input feed (left) and replace the background, often introducing artifacts, as shown in the center result with close-ups of hair and glasses that still have the residual of the original background. Leveraging a frame of video without the subject (far left inset), our method produces real-time, high-resolution background matting without those common artifacts. The image on the right is our result with the corresponding close-ups, screenshot from our Zoom plugin implementation.

👉 See image results with interactive zoom-in

✨ Features

  • Real-Time High-Resolution Background Matting, 4K 30fps  HD 60fps;
  • Replacing our background in meeting screen;
  • Replace background with any size photo you want;
  • Cross mutil-platform, like zoom plugin, web-browser, miniprogram in PC, mobile and iPad, any platform with web service;
  • IFLYTEK voice translation between Chinese and English;
  • ...

Overview

Updates

  • [Jan 09 2021] PhotoMatte85 dataset is now published.
  • [Dec 21 2020] We updated our project to MIT License, which permits commercial use!

Download

Model / Weights

Video / Image Examples

Datasets

🔨 Build

  1. setup enviroment
# git clone https://github.com/Charmve/TimeWarp.git
# cd TimeWarp

# pip install -r requirements.txt
  1. download dataset and model Download

  2. run test code

  • Video
# cd TimeWarp
# bash test_video.sh
  • Image
# cd TimeWarp
# python3 test_image.py

Demo

🎧 Beta


demo Real-Time High-Resolution Background Matting[1]

Have a Try !

➡️ Web Demo by Gradio: Matting your own images from your browser.

Scripts

We provide several scripts in this repo for you to experiment with our model. More detailed instructions are included in the files.

  • inference_images.py: Perform matting on a directory of images.
  • inference_video.py: Perform matting on a video.
  • inference_webcam.py: An interactive matting demo using your webcam.

Notebooks

Additionally, you can try our notebooks in Google Colab for performing matting on images and videos.

Virtual Camera

We provide a demo application that pipes webcam video through our model and outputs to a virtual camera. The script only works on Linux system and can be used in Zoom meetings. For more information, checkout:

Web Demo

Developers in the community has helped us build a web demo. See Community Projects section below.

Usage / Documentation

You can run our model using PyTorch, TorchScript, TensorFlow, and ONNX. For detail about using our model, please check out the Usage / Documentation page.

中文解读 ->

Training

Training code will be released upon acceptance of the paper.

☕ Processing

gantt
        dateFormat  YYYY-MM-DD
        title TimeWarp Development Processing
        section image matting
        architecture design          :done,    des1, 2020-11-06,2020-12-20
        video matting                :active,  des2, 2021-01-07, 3d
        chrome extension plus        :         des3, after des2, 7d
        video meeting architecture   :         des4, after des3, 5d
Loading
  • More update processing details, please click HERE to see CHANGLOG.

🔔 Update Reminder

Subscribe -->

Business ✉️ email: yidazhang1@gmail.com

🍮 Community

🔗 Reference

[1] Real-Time High-Resolution Background Matting. Shanchuan Lin*, Andrey Ryabtsev*, Soumyadip Sengupta. [GitHub] | [arXiv] | [data]

[2] Background Matting: The World is Your Green Screen. Soumyadip Sengupta, Vivek Jayaram, Brian Curless, et.al. CVPR 2020. [GitHub] | [arXiv]

[3] MODNet: Is a Green Screen Really Necessary for Real-Time Portrait Matting? [GitHub] | [arXiv]

[4] High-Resolution Deep Image Matting. Haichao Yu, Ning Xu, Zilong Huang. [GitHub] | [arXiv]

[5] End-to-end Animal Image Matting. Jizhizi Li, Jing Zhang, Stephen J. Maybank. [GitHub] | [arXiv].

[6] BGMv2-webcam-plugin-linux. https://github.com/andreyryabtsev/BGMv2-webcam-plugin-linux

[7] Chrome-extensions-samples. https://github.com/GoogleChrome/chrome-extensions-samples

[8] Zoom Video Ingester. https://github.com/harvard-dce/zoom-recording-ingester

[9] Zoom Recording Downloader. https://github.com/ricardorodrigues-ca/zoom-recording-downloader

[10] PaperswithCode. Image Matting. https://paperswithcode.com/task/image-matting PaperswithCode image-matting

📜 License

This work is licensed under the Creative Commons Attribution NonCommercial ShareAlike 4.0 License.

💝 Acknowledge

Thanks for PeterL1n, et al.


TimeWarp
Feel free to ask any questions, open a PR if you feel something can be done differently!

🌟Star this repository🌟

Created by Charmve & maiwei.ai Community | Deployed on Gradio.app