YouDream

YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals

Sandeep Mishra^✝︎, Oindrila Saha^✝︎, Alan C. Bovik
Accepted at NeurIPS 2024

---

[arXiv] | [Project Page]

---

---

Overview

YouDream introduces a novel approach for generating 3D animal models with anatomically accurate control, ensuring consistency and detail across generated outputs. This work combines text-to-3D generation with precise anatomical structures to cater to tasks requiring high-quality 3D assets in research and industry.

---

Git Log

To-Do List

• Release codebase
• Release a detailed documentation
• Release more Animal Prior configurations
• Release Pose Editor (manual pose editing -- suitable for pose-refinement and unseen animal generation)
• Release GPT-pose-editor (automatic pose editing -- suitable for known animals)

---

Code Documentation

Installation

1. Clone the repository:

   git clone https://github.com/YouDream3D/YouDream.git
   cd YouDream

2. Create and activate the conda environment:

   conda create -n youdream python=3.9 -y
   conda activate youdream

3. Install dependencies:

   conda install pytorch=1.13.0 torchvision pytorch-cuda=11.6 -c pytorch -c nvidia
   conda install -c iopath iopath
   conda install pytorch3d -c pytorch3d
   pip3 install -r requirements.txt

4. Download the Animal Pose ControlNet Checkpoint: Visit the Hugging Face model page to download the animal_pose_controlnet checkpoint. Place the downloaded checkpoint folder: "animal_pose_controlnet" in the root directory of YouDream.

Running the Code

1. Run the shell scripts to generate 3D models:

   ./run_all_tiger.sh 

---

Citation

If you find this work useful in your research or projects, please cite:

@inproceedings{NEURIPS2024_6fe5d7a2,
 author = {Mishra, Sandeep and Saha, Oindrila and Bovik, Alan},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {A. Globerson and L. Mackey and D. Belgrave and A. Fan and U. Paquet and J. Tomczak and C. Zhang},
 pages = {60669--60698},
 publisher = {Curran Associates, Inc.},
 title = {YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals},
 url = {https://proceedings.neurips.cc/paper_files/paper/2024/file/6fe5d7a2de090168917425fe89a6c1b8-Paper-Conference.pdf},
 volume = {37},
 year = {2024}
}

---

Acknowledgements

Our Code is based on DreamWaltz. We thank the DreamWaltz contributors for making the code available.

^✝︎ Equal contribution.