Stable Diffusion NCNN

This is a public fork from EdVince/Stable-Diffusion-NCNN. All the credit goes to EdVince. As I cannot push LFS object to a public fork, I make this repo standalone.

Features appended:

1. remove OpenCV dependency
2. reduce memory usage from 7 GB to 5.5 GB
3. enable 4X super resolution with Real-ESRGAN
4. single header-only

Quick introduction

1. Checkout the readme from EdVince's repo for implementation details.
2. Download and extract the assets from the release of this repo, the layout should looks like:

YOUREXECUTABLE
assets
├── AutoencoderKL-fp16.bin
├── AutoencoderKL-fp16.param
├── FrozenCLIPEmbedder-fp16.bin
├── FrozenCLIPEmbedder-fp16.param
├── log_sigmas.bin
├── RealESRGAN_x4plus_anime_6B.fp32-sim-sim-opt.bin
├── RealESRGAN_x4plus_anime_6B.fp32-sim-sim-opt.param
├── UNetModel-fp16.bin
├── UNetModel-fp16.param
└── vocab.txt

Example usage:

Copy stable_diffusion.hpp and the downloaded assets to your working directory, and call C++ api:

void
stable_diffusion(   std::string positive_prompt,
                    std::string output_png_path,
                    int step,
                    int seed,
                    std::string negative_prompt );

Checkout the source file to find the default values to these arguments.

Example usage

Source code:

// test.cpp
#include "./stable_diffusion.hpp"
#include <string>

int main ()
{
    std::string positive_prompt = "floating hair, portrait, ((loli)), ((one girl)), cute face, hidden hands, asymmetrical bangs, beautiful detailed eyes, eye shadow, hair ornament, ribbons, bowties, buttons, pleated skirt, (((masterpiece))), ((best quality)), colorful";
    std::string output_png_path = "./output.png";
    stable_diffusion( std::string positive_prompt, std::string output_png_path );
    return 0;
}

Compile and link command

g++ -o test test.cpp -funsafe-math-optimizations -Ofast -flto=auto  -funroll-all-loops -pipe -march=native -std=c++20 -Wall -Wextra `pkg-config --cflags --libs ncnn`  -lstdc++ -pthread -Wl,--gc-sections -flto -fopt-info-vec-optimized

Run

./test

Dependency

This project only relies on ncnn. For Archlinux users, ncnn-git package from AUR is sufficient. For others, please install it manually.

To build a test application:

git clone https://github.com/fengwang/Stable-Diffusion-NCNN.git
cd Stable-Diffusion-NCNN
g++ -o test test.cpp -funsafe-math-optimizations -Ofast -flto=auto  -funroll-all-loops -pipe -march=native -std=c++20 -Wall -Wextra `pkg-config --cflags --libs ncnn` -lstdc++ -pthread -Wl,--gc-sections -flto -fopt-info-vec-optimized
./test

However, you need 5.5 GB memory to run.

Linking to NCNN

If ncnn is not installed in the standard paths (such as /usr/include/ncnn and /usr/lib/libncnn.so, please update the commandline to comply. For example:

g++ -o test test.cpp -funsafe-math-optimizations -Ofast -flto=auto  -funroll-all-loops -pipe -march=native -std=c++20 -Wall -Wextra -I/usr/local/include -L/usr/local/lib -lncnn -lstdc++ -pthread -Wl,--gc-sections -flto -fopt-info-vec-optimized

TODO

1. provide a GUI
2. provide a python binding (?)

9 Examples

Generating using the default positive_prompt and default negative_prompt with different seed in 20 steps

seed = 1

seed = 2

seed = 3

seed = 4

seed = 5

seed = 6

seed = 7

seed = 8

seed = 9