TriHuman: A Real-time and Controllable Tri-plane Representation for Detailed Human Geometry and Appearance Synthesis (TOG 2024)

Heming Zhu · Fangneng Zhan · Christian Theobalt · Marc Habermann‡

---

News

2024-11-29 The initial release, i.e., the Dataset is available, the Demo Code is on the way.

---

Dataset

Please find the Raw Data and Annotations, the Metadata, and the Checkpoints that is used in the paper from this link.

Raw data

The dataset contains long, dense multi-view video sequences performing everyday motions. The training sequences are ~20 minutes long (25fps), and the testing sequences are ~10 minutes long (25fps). Each sequnce is equipped with Foreground Segementation, and Per-frame Groundtruth 3D reconstruction generated by NeuS2. The raw data follows the folder structure below

|--- Subject0001
    |---training                            # training poses
    |   |--- images 
    |   |       |--- stream***.mp4          # the raw images
    |   |--- foregroundSegmentation 
    |   |       |--- stream***.mp4          # the foreground segementations             
    |   |--- recon_neus2 
    |           |--- *.ply                  # the Neus2 reconstruction for each frame
    |---testing
        |---...                             

Metadata

The raw data follows the folder structure below:

meta_data
|--- Subject0001
    |---skeletoolToGTPose                   # training poses
    |   |--- ... 
    |
    |---skeletoolToGTPoseTest               # Testing poses
    |   |--- ...
    | 
    |---cameras.calibration                 # 1K camera calibration
    |---ddc.character                       # character setting 
    |---ddc.obj                             # template mesh
    |---ddc_graph.obj                       # embedded graph
    |---ddc.skeleton                        # template skeleton
    |---ddc.png                             # template texture
    |---base_trans.pkl                      # translation for normalization
    |---dof_convert_dict.pkl                # dof with/without hands
    |---segmentation.txt                    # template segementation (hands / others) 
    |---segmentation_hand.txt               # template segementation (hands / others) 
    |---subdivision_dict_xyz_new.pkl        # subdivision information 

---

Installation

Clone the repo

git clone git@github.com:kv2000/TriHuman.git --recursive

Install the dependencies

The code is tested on Python 3.9, pytorch 2.10, and cuda 11.8.

Step 1. Setup DeepCharacters Pytorch

Firstly, install the underlying clothed human body model, 🎆DeepCharacters Pytorch🎆, which also consists the dependencies that needed for this repo.

Step 2. Other dependences

---

Todo list

• Demo Code
• Dataprocessing and Training

---

Contact

For questions, clarifications, feel free to get in touch with:
Heming Zhu: hezhu@mpi-inf.mpg.de
Marc Habermann: mhaberma@mpi-inf.mpg.de

---

License

Deep Characters Pyotrch is under CC-BY-NC license. The license applies to the pre-trained models and the metadata as well.

---

Acknowledgements

Christian Theobalt was supported by ERC Consolidator Grant 4DReply (No.770784). Adam Kortylewski was supported by the German Science Foundation (No.468670075). This project was also supported by the Saarbrucken Research Center for Visual Computing, Interaction, and AI.

Below are some resources that we benefit from (keep updating):

• 3D gaussian-splatting, bravo for the brilliant representation for real-time and high-quality rendering.
• DeepCharacters Pytorch for the human character model.
• Pytorch3D, Kaolin, and Kornia for the handy geometry library.
• NeuS for the project structure D:.