ControlNeXt: Powerful and Efficient Control for Image and Video Generation
- -# Corresponding author
- - -Overview
-TL;DR: This work proposes a light-weight controllable module for various base models (SD1.5, SDXL, SD3, SVD) and tasks (image / video generation with various conditions).
-⚠️ For the webpage transfer, we compress the images and videos to reduce file size. Please refer to the original files for full quality.
- -Hover on the picture to see the overlayed condition!
-.jpg)





Video Generation via Stable Video Diffusion
-User Input: First frame image & pose guidance sequence
-If you can't load the videos beacause of the network problem, you can also view them through BiliBili
-Image Generation with SDXL
-We trained a canny adapter on SDXL model.
-






.jpg)










.jpg)
.jpg)
Various Stylization & Editing
-


.jpg)

.jpg)



.jpg)









.jpg)

.jpg)

.jpg)


















Image Generation with 1.5
- - -Our method is also adaptable for community's LoRA weights
-









We trained a multiple adapters on SD1.5 model.
-









Image Super-Resolution with SD3
-We trained a Super-Resolution ControlNeXt on SD3 with degraded inputs.
-





Training convergence
-Our method achieves significantly faster convergence during training.
-It starts to learn the control abilities within hundreds of training steps.
-
Contact Us
-- Feel free to contact Bohao Peng at bhpeng22@cse.cuhk.edu.hk for any question,cooperation, and communication. -
- -- If you find this work useful, please consider citing: -
- -@article{peng2024controlnext, - title={ControlNeXt: Powerful and Efficient Control for Image and Video Generation}, - author={Peng, Bohao and Wang, Jian and Zhang, Yuechen and Li, Wenbo and Yang, Ming-Chang and Jia, Jiaya}, - journal={arXiv preprint}, - year={2024} -}-