Prompt Sliders for Fine-Grained Control, Editing and Erasing of Concepts in Diffusion Models

We introduce the Prompt Slider method for precise manipulation, editing, and erasure of concepts in diffusion models. Project Page

Installing the dependencies

Before running the scripts, make sure to install the library's training dependencies:

You can install diffusers directly from pip or install from the latest version. To do this, execute one of the following steps in a new virtual environment:

Install with pip

pip install diffusers==0.27

Install from source

git clone https://github.com/huggingface/diffusers
cd diffusers
pip install .

Then cd in the promptsliders folder (you can also copy it to the examples folder in diffusers) and run:

pip install -r requirements.txt

And initialize an 🤗 Accelerate environment with:

accelerate config

Now we can launch the training using:

export MODEL_NAME="runwayml/stable-diffusion-v1-5"
export EMOTION="smiling"

accelerate launch textual_inversion.py \
    --pretrained_model_name_or_path=$MODEL_NAME \
    --learnable_property="object" \
    --placeholder_token="<$EMOTION-lora>" \
    --initializer_token="$EMOTION" \
    --mixed_precision="no" \
    --resolution=512 \
    --train_batch_size=1 \
    --gradient_accumulation_steps=1 \
    --max_train_steps=2000 \
    --learning_rate=5.0e-04 \
    --scale_lr \
    --lr_scheduler="constant" \
    --lr_warmup_steps=0 \
    --save_as_full_pipeline \
    --output_dir=outputs/$EMOTION-promptslider/ \
    --prompts_file="textsliders/data/prompts-$EMOTION.yaml"

Alternatively, one could run with default settings

bash prompt_slider_emotions.sh

A full training run takes ~1-2 hours on one A10 GPU.

Inference

If you have issues in running the code TypeError: unsupported operand type(s) for +: 'int' and 'NoneType' , install the earlier version of diffusers

pip install diffusers==0.20.2
pip install huggingface-hub==0.21

Once you have trained a model using above command, the inference can be done simply using the StableDiffusionPipeline or StableDiffusionXLPipeline wih the following script. Make sure to modify the concept name to your concept and the output is at output/age-slider_prompt/learned_embeds.safetensors.

python inference-promptsliders-sdxl.py age

To run inference with SD with default scale,

python inference_sd.py $path_to_the_saved_embedding $token_name

Acknowledgements

Thanks to diffusers and Concept Sliders!