Final project : erAIser - Remove an object in video using AI

Moving Object Replacement in Video(called 'AANet' in full project)

본 모델은 github.com/snapresearch의 생성모델 연구(Motion Representation for Articulated Animation)의 아키텍처를 활용합니다. 해당 아키텍처를 기반으로 새로운 task(동영상 내 지워진 물체를 대신하는 새로운 애니메이션 생성)를 실현하기위해 추가한 모듈과 함수는 아래와 같습니다.

Code Files for Object Replacement

aa_inference.py : 전체적인 inference 과정을 포함하는 class AAInference
preprocessing_aanet.py : inference에 필요한 함수들
preprocessing_davis.py : dataset의 정제를 위해 필요한 함수들.

Example

Original	AANet	Source

Original	AANet	Source

Dataset Used for Training

1. DAVIS

2. Taichi

3. TikTok

4. Cough

Architecture

---

Base Research for Object Replacement in video

Following part is copy and paste from original repository by snap-research

Object Replacement(AANET)을 구현하기 위해 사용된 Base model을 학습하는 코드를 위의 github에서 copy & paste 한 code입니다.

Training for base model in AANet

To train a model run:

CUDA_VISIBLE_DEVICES=0 python run.py --config config/dataset_name.yaml --device_ids 0

The code will create a folder in the log directory (each run will create a time-stamped new folder). Checkpoints will be saved to this folder. To check the loss values during training see log.txt. You can also check training data reconstructions in the train-vis subfolder. Then to train Animation via disentaglement (AVD) use:

CUDA_VISIBLE_DEVICES=0 python run.py --checkpoint log/{folder}/cpk.pth --config config/dataset_name.yaml --device_ids 0 --mode train_avd

Where {folder} is the name of the folder created in the previous step. (Note: use backslash '' before space.) This will use the same folder where checkpoint was previously stored. It will create a new checkpoint containing all the previous models and the trained avd_network. You can monitor performance in log file and visualizations in train-vis folder.

Training on your own dataset

1. Resize all the videos to the same size, e.g 256x256, the videos can be in '.gif', '.mp4' or folder with images. We recommend the latter, for each video make a separate folder with all the frames in '.png' format. This format is loss-less, and it has better i/o performance.

2. Create a folder data/dataset_name with 2 subfolders train and test, put training videos in the train and testing in the test.

3. Create a config file config/dataset_name.yaml. See description of the parameters in the config/vox256.yaml. Specify the dataset root in dataset_params specify by setting root_dir: data/dataset_name. Adjust other parameters as desired, such as the number of epochs for example. Specify id_sampling: False if you do not want to use id_sampling.

P.S

Object Replacement에 해당하는 파트에 대한 설명은 위와 같고, Object Replacement와 Object Remove에 대한 Implemetation 방법은 아래에 덧붙힌 Full Project(erAIser)의 Readme를 참고하세요.

---

Full Project(erAIser) and Implementation code

1. erAIser
2. Example
3. Demo screenshot
4. Usage
   • Environment setup
   • Run demo
5. References
6. Contributors

erAIser

‘erAIser’ is a service that provides a specific object erased video by using video object segmentation and video inpainting methods.

Most of video inpainting model need segmentation mask of objects. But it is hard to get in normal way. For your convenience, we used a deep learning model that allows users to easily obtain segmentation masks. We combined this video object segmentation model with the video inpainting model to increase usability. Additionally, we use AANet to change object to other object in object erased video.

Our team consists of nine members of ‘Tobigs’ who are interested in computer vision task.

All you have to do is draw the object bounding box that you want to erase in the first frame. Then 'erAIser' will make a video of the object being erased. Let’s make your own video of a specific object being erased with ‘erAIser’!

Example of erAIser(not using object replacement)

Demo screenshot

You can look around our web at the following link. For your information, the model is currently not working on the web due to the GPU environment.

web link.

Usage

Environment Setup

This code was tested in the following environments

• Ubuntu 18.04.5
• Python 3.7
• Pytorch 1.8.1
• CUDA 9.0
• GCC 5.5 (gnu c++14 compiler)

If you don't use gnu c++14 compiler, then you will encounter CUDA build error

1. Clone the repository & Setup

git clone https://github.com/sjinu96/object-replacement-in-video.git
cd object-replacement-in-video
conda create -n erAIser python=3.7 -y
conda activate erAIser
conda install cudatoolkit=9.0 -c pytorch -y
pip install -r requirements.txt
bash install.sh

2. Setup python path

export PYTHONPATH=$PWD:$PYTHONPATH
cd vos/
export PYTHONPATH=$PWD:$PYTHONPATH
cd ../vi/
export PYTHONPATH=$PWD:$PYTHONPATH
cd ../web/
export PYTHONPATH=$PWD:$PYTHONPATH
cd ../AANet/
export PYTHONPATH=$PWD:$PYTHONPATH
cd ../

Demo

1. Setup your environment
2. Download the Deep Video Inpainting model

cd vi/results/vinet_agg_rec

file_id="1_v0MBUjWFLD28oMfsG6YwG_UFKmgZ8_7"
file_name="save_agg_rec_512.pth"
curl -sc /tmp/cookie "https://drive.google.com/uc?export=download&id=${file_id}" > /dev/null
code="$(awk '/_warning_/ {print $NF}' /tmp/cookie)"
curl -Lb /tmp/cookie "https://drive.google.com/uc?export=download&confirm=${code}&id=${file_id}" -o ${file_name}

file_id="12TRCSwixAo9AqyZ0zXufNKhxuHJWG-RV"
file_name="save_agg_rec.pth"
curl -sc /tmp/cookie "https://drive.google.com/uc?export=download&id=${file_id}" > /dev/null
code="$(awk '/_warning_/ {print $NF}' /tmp/cookie)"
curl -Lb /tmp/cookie "https://drive.google.com/uc?export=download&confirm=${code}&id=${file_id}" -o ${file_name}

cd ../../../

3. Download the Siammask model

wget http://www.robots.ox.ac.uk/~qwang/SiamMask_DAVIS.pth

file_id="1IKZWpMeWXq-9osBqG7e7bTABumIZ32gB"
file_name="checkpoint_e19.pth"
curl -sc /tmp/cookie "https://drive.google.com/uc?export=download&id=${file_id}" > /dev/null
code="$(awk '/_warning_/ {print $NF}' /tmp/cookie)"
curl -Lb /tmp/cookie "https://drive.google.com/uc?export=download&confirm=${code}&id=${file_id}" -o ${file_name}

4. Download the AANet model

cd AANet/
mkdir checkpoints
cd checkpoints/

file_id="1DT6_SZHTkmuEWvCfs07F2mGLSkgxYplo"
file_name="4dataset384.pth"
curl -sc /tmp/cookie "https://drive.google.com/uc?export=download&id=${file_id}" > /dev/null
code="$(awk '/_warning_/ {print $NF}' /tmp/cookie)"
curl -Lb /tmp/cookie "https://drive.google.com/uc?export=download&confirm=${code}&id=${file_id}" -o ${file_name}

cd ../../

5. Make results directory for saving result video

mkdir results

results is defualt setting. You can change this.

6-1. Run inference.py for erasing

python3 inference.py --resume checkpoint_e19.pth --config config_inference.json

6-2. Run inference.py for change object to other (ex. person, animation character)

python3 inference.py --resume SiamMask_DAVIS.pth --config config_inference.json --using_aanet True

The result video will be saved in results.

---