Due to the privacy policies of the BioVid Database, we can only release the checkpoints, training code, and inference code. To minimize our effort, we are releasing the training and preprocessing code as is for reference purposes. We have only tested and verified the inference code, which includes a Gradio-based demo.
Install Inferno for the EMOCA decoder. Follow the instructions here and download the necessary models for facial reconstruction here. We have slightly modified the original code to generate useful latent variables for the face reconstruction app and to support the render_from_exp.py script.
Note: Installing pytorch3d might present compatibility issues due to mismatched versions of CUDA, PyTorch, and pytorch3d. If this occurs, install pytorch3d separately.
# Create and activate a new Conda environment
conda create python=3.10 -n paindiff
conda activate paindiff
# Install the required packages and pytorch3d
# Ensure the CUDA version matches your PyTorch installation and system configuration: https://pytorch.org/get-started/locally/
pip install -r requirements.txt
FORCE_CUDA=1 pip install git+https://github.com/facebookresearch/pytorch3d.git@stable
# Clone and set up the Inferno package
git clone https://github.com/damtien444/inferno inferno_package
cd inferno_package/
bash pull_submodules.sh
pip install -e .
# Download the pretrained EMOCA model for PainDiffusion
cd inferno_apps/FaceReconstruction
bash download_assets.shThanks to the Hugging Face Model Hub, you can run PainDiffusion with a single command.
gradio_demo.webm
-
Run the Demo
python online_run.py
-
Access at http://127.0.0.1:7860/ in your web browser.
For a better realistic avatar, we use PainDiffusion to drive Gaussian Avatars. Follow the step in this repository to do that.
drive_gaussianavatars.webm
We thank the previous author for their opensource code. This project is heavily based on the excellent implementations of diffusion models from:
@misc{dam2024paindiffusion,
title={PainDiffusion: Can robot express pain?},
author={Quang Tien Dam and Tri Tung Nguyen Nguyen and Dinh Tuan Tran and Joo-Ho Lee},
year={2024},
eprint={2409.11635},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2409.11635},
}