Update readme

Author: chuanli11

doc/NeuRIPS.md

@@ -2,44 +2,78 @@
 
 ## Step One: Launch a Lambda Cloud instance
 
+The hands-on practice requires a Lambda Cloud account and a `1xA100` GPU instance, which costs `$1.10 USD/h`. 
+
+Remember to drop by our photo booth if you do not wish to sign up. Our staff will take photos and train a model for you. Leave us your email address, and we will send the results within 12 hours.
+
+If you wish to sign up, go to [https://cloud.lambdalabs.com/](https://cloud.lambdalabs.com/) and follow the signup step.
+
+Once signed in with your Lambda Cloud account, click the `Launch Instance` button.
+
+<img src="./images/lambda_cloud_dashboard.jpg" alt="drawing" style="width:480px;"/>
+
+Lambda Cloud will ask you for payment information when launching the first instance. Just follow the instructions and be aware that Lambda Cloud will place a __temporary__ `$10 USD pre-auth to verify your card, which will disappear within seven days. Once payment information is provided, you can launch an instance. For this workshop:
+* Choose 1xA100 instance (40GB SXM4 or 40GB PCIe are both fine).
+* Any region will work.
+* Don't need to attach a filesystem.
+* Follow the guide to add or generate an SSH key -- this step can not be skipped. However, this workshop won't use this key because all practice can be accomplished in the Cloud IDE (so no SSH session is needed).
+
+It takes about two mins to get the instance running (the green tick as shown in the picture below). 
+
+<img src="./images/lambda_cloud_dashboard_instance_ready.jpg" alt="drawing" style="width:480px;"/>
+
+You need to click the Cloud IDE `Launch` button (the purple button on the right end) to get access to the Jupyter Hub. If you see a message saying, "Your Jupyter Notebook is down," that means the Jupyter Hub isn't ready and please give it another minute or so. Eventually, it will look like this once it is ready:
+
+<img src="./images/lambda_cloud_jupyter_hub.jpg" alt="drawing" style="width:480px;"/>
 
-## Step Two: Download Notebooks
 
-In dash board, click `Cloud IDE` - `Launch`, this will bring you to the Jupyter hub pre-installed on the cloud instance.
 
-Create a new `Terminal` by clicking the `Terminal` button. In the terminal, run these two commands:
+
+## Step Two: Download Notebooks
+
+Create a terminal by clicking the `Terminal` icon, and run the following command in the terminal to download a few notebooks to your home directory.
 
 ```
 wget https://raw.githubusercontent.com/LambdaLabsML/dreambooth/neurips/setup.ipynb && \
 wget https://raw.githubusercontent.com/LambdaLabsML/dreambooth/neurips/train.ipynb && \
-wget https://raw.githubusercontent.com/LambdaLabsML/dreambooth/neurips/test.ipynb && \
+wget https://raw.githubusercontent.com/LambdaLabsML/dreambooth/neurips/test_param.ipynb && \
 wget https://raw.githubusercontent.com/LambdaLabsML/dreambooth/neurips/test_prompt.ipynb
 ```
 
-Click the refresh button in the `File Browser` (on the left side of the IDE), you should see `setup.ipynb`, `train.ipynb` and `test.ipynb`. 
+You need to click the refresh button in the `File Browser` (on the left side of the IDE) to see the notebooks. 
 
-## Step Three: Run Notebook
+<img src="./images/lambda_cloud_dashboard_download_ipynb.jpg" alt="drawing" style="width:480px;"/>
 
-### Run `setup.ipynb`
+Now you are ready to kick off the DreamBooth practice!
 
-This notebook will clone the DreamBooth repo, install a number of python packages needed for this pratice.
+## Step Three: Run Notebook
 
-It will create a folder at `/home/ubuntu/data` for you to upload the training photos.
+### Run `setup.ipynb`
 
-The last step in thie notebook will ask for a access token for downloading the Stable Diffusion model from Huggingface. You need to  
+This notebook will clone the DreamBooth repo and install several python packages. It will also create several folders in the home directory:
+* `/home/ubuntu/data`: This directory stores training photos.
+* `/home/ubuntu/model`: This is the directory where the trained model will be saved.
+* `/home/ubuntu/output`: This is the directory where the sampled images will be saved.
 
+The last step in this notebook will ask for an access token for downloading the Stable Diffusion model from Huggingface. You need to:  
 * Create a [huggingface](https://huggingface.co/) account if you don't have one.
-* Create your access token from "Settings - Access Tokens - New Token", and paste the token into the login field at the end of the notebook (see image below). 
+* Create your access token from "Settings - Access Tokens - New Token," and paste the token into the login field at the end of the notebook (see image below).
 <img src="./images/hf_token.jpg" alt="drawing" style="width:480px;"/>
 * Accept the [license of Stable Diffusion v1-4 Model Card](https://huggingface.co/CompVis/stable-diffusion-v1-4) if you agree. (Otherwise can't use the model)
 <img src="./images/hf_model_card.jpg" alt="drawing" style="width:480px;"/>
 
+### Upload Images
+
+We recommend preparing ~20 photos: ten close-ups of your face with various poses and facial expressions, five photos from your chest and up, and a few full-body shots.
+
 ### Run `train.ipynb`
-This notebook will train a DreamBooth model using the images you just uploaded.
+This notebook trains a DreamBooth model use the images inside of `/home/ubuntu/data`.
 
-Once trained, it will run some validations using the prompts in the `test_dreambooth.py` script.
+Once trained, it will also run a few inferences and display the prompts and sampled images at the end of the notebook.
 
 ### Run `test_prompt.ipynb` and `test.ipynb`
-You can use these notebook to play with the model you just trained. 
+You can use these notebooks to play with the model you just trained. 
+
+* `test_prompt.ipynb`: A notebook for prompt engineering. You will use fixed latent input to conduct controlled experiments for the impact of prompt engineering on the model output.
 
-## Q&A
+* `test_param.ipynb`: A notebook for trying different parameters for inference. Again, you will use fixed latent input to conduct controlled experiments for the impact of these parameters on the model output. 

doc/images/lambda_cloud_dashboard.jpg

@@ -77,13 +77,11 @@ def main():
     tests = {
             "1": ["photo, colorful cinematic portrait of " + token_class_str + ", armor, cyberpunk,background made of brain cells, back light, organic, art by greg rutkowski, ultrarealistic, leica 30mm", args.num_pred_steps, args.guide, "rutkowski"],
             "2": ["pencil sketch portrait of  " + token_class_str + " inpired by greg rutkowski, digital art by artgem", args.num_pred_steps, args.guide, "rutkowskiartgem"],
-            "3": ["photo, colorful cinematic portrait of  " + token_class_str + ", organic armor, cyberpunk, background brain cells mesh, art by greg rutkowski", args.num_pred_steps, args.guide, "rutkowskibraincells"],
-            "4": ["photo,colorful cinematic portrait of " + token_class_str + ", " + token_class_str + " with long hair, color lights, on stage, ultrarealistic", args.num_pred_steps, args.guide, "longhair"],
-            "5": ["photo, colorful cinematic portrait of  " + token_class_str + " with organic armor, cyberpunk background,  " + token_class_str + ", greg rutkowski", args.num_pred_steps, args.guide, "cyberpunkrutkowski"],
-            "6": ["photo portrait of  " + token_class_str + " astronaut, astronaut, helmet in alien world abstract oil painting, greg rutkowski, detailed face", args.num_pred_steps, args.guide, "astronautrutkowski"],
-            "7": ["photo portrait of  " + token_class_str + " as firefighter, helmet, ultrarealistic, leica 30mm", args.num_pred_steps, args.guide,  "firefighter"],
-            "8": ["photo portrait of  " + token_class_str + " as steampunk warrior, neon organic vines, digital painting", args.num_pred_steps, args.guide, "steampunk"],
-            "9": ["impressionist portrait painting of  " + token_class_str + " by Daniel F Gerhartz, (( " + token_class_str + " with painted in an impressionist style)), nature, trees", args.num_pred_steps, args.guide, "danielgerhartz"],
+            "3": ["photo,colorful cinematic portrait of " + token_class_str + ", " + token_class_str + " with long hair, color lights, on stage, ultrarealistic", args.num_pred_steps, args.guide, "longhair"],
+            "4": ["photo portrait of  " + token_class_str + " astronaut, astronaut, helmet in alien world abstract oil painting, greg rutkowski, detailed face", args.num_pred_steps, args.guide, "astronautrutkowski"],
+            "5": ["photo portrait of  " + token_class_str + " as firefighter, helmet, ultrarealistic, leica 30mm", args.num_pred_steps, args.guide,  "firefighter"],
+            "6": ["photo portrait of  " + token_class_str + " as steampunk warrior, neon organic vines, digital painting", args.num_pred_steps, args.guide, "steampunk"],
+            "7": ["impressionist portrait painting of  " + token_class_str + " by Daniel F Gerhartz, (( " + token_class_str + " with painted in an impressionist style)), nature, trees", args.num_pred_steps, args.guide, "danielgerhartz"],
     }
 
     if args.ddim:

doc/images/lambda_cloud_dashboard_download_ipynb.jpg

@@ -103,13 +103,31 @@
     ").half()"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6a325c88",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# No additional feature string\n",
+    "feature_str = \"\"\n",
+    "fill_placeholders = lambda x: x.replace(\"__token__\", token_name).replace(\"__class__\", class_str).replace(\"__feature__\", feature_str)\n",
+    "for prompt in prompts:\n",
+    "    prompt = fill_placeholders(prompt)\n",
+    "    image = pipe(prompt, num_inference_steps=num_pred_steps, guidance_scale=guide, latents = latents).images[0]\n",
+    "    print(prompt)\n",
+    "    save_and_display(image, predict_path)"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
    "id": "24372f4d-b650-41d1-8717-c0faf4a5c5fa",
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Use \"detailed face\" as the feature string\n",
     "feature_str = \", detailed face\"\n",
     "fill_placeholders = lambda x: x.replace(\"__token__\", token_name).replace(\"__class__\", class_str).replace(\"__feature__\", feature_str)\n",
     "for prompt in prompts:\n",
@@ -126,6 +144,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Use \"blue Punk Mohawk\" as the feature string\n",
     "feature_str = \", blue Punk Mohawk\"\n",
     "fill_placeholders = lambda x: x.replace(\"__token__\", token_name).replace(\"__class__\", class_str).replace(\"__feature__\", feature_str)\n",
     "for prompt in prompts:\n",
@@ -142,7 +161,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Negative prompt\n",
+    "# Negative prompt (the feature you want to get rid off)\n",
+    "# e.g. use \"glasses\" if you want to get rid of the glasses\n",
     "negative_prompt=\"glasses\"\n",
     "feature_str = \", blue Punk Mohawk\"\n",
     "fill_placeholders = lambda x: x.replace(\"__token__\", token_name).replace(\"__class__\", class_str).replace(\"__feature__\", feature_str)\n",
@@ -160,7 +180,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# repeat special token\n",
+    "# Repeat the special token and class string twice\n",
     "feature_str = \", blue Punk Mohawk\"\n",
     "fill_placeholders = lambda x: x.replace(\"__token__\", token_name + \" __class__, \" + token_name).replace(\"__class__\", class_str).replace(\"__feature__\", feature_str)\n",
     "for prompt in prompts:\n",

doc/images/lambda_cloud_dashboard_instance_ready.jpg

@@ -42,7 +42,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Run this command to train your DreamBooth Model\n",
+    "# Run this command to train your DreamBooth Model and generate a few inference results \n",
+    "\n",
+    "# Training (results will be saved to MODEL_DIR)\n",
     "!(python train_dreambooth.py \\\n",
     "  --config_file config.yaml \\\n",
     "  learning_rate=\"$LR\" \\\n",
@@ -54,68 +56,29 @@
     "  output_dir=\"$MODEL_DIR\" \\\n",
     "  max_train_steps=\"$MAX_NUM_STEPS\" \\\n",
     "  use_tf32=true \\\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "bf2124e9-c670-4eb0-8eab-c32c31d95038",
-   "metadata": {},
-   "source": [
-    "# Validate the model with prepared prompts"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "7d6cdc67-44aa-4612-8ff1-abb21e028b43",
-   "metadata": {
-    "scrolled": true,
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "# Results will be saved to PRED_DIR\n",
+    ")\n",
     "\n",
+    "# Inference (results will be saved to PRED_DIR)\n",
     "NUM_PRED=2 # number of predictions per prompt\n",
-    "\n",
     "!( python test_dreambooth.py \\\n",
     "    --model_path $MODEL_DIR \\\n",
     "    --pred_path $PRED_DIR \\\n",
     "    --num_preds $NUM_PRED \\\n",
     "    --ddim \\\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "ef07089c-910b-4d97-8c08-708fa9d5580f",
-   "metadata": {
-    "scrolled": true,
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
+    ")\n",
+    "\n",
     "import glob\n",
     "from IPython.display import Image, display\n",
     "for imageName in glob.glob(PRED_DIR +'/*.png'): #assuming JPG\n",
+    "    print(imageName)\n",
     "    display(Image(filename=imageName))\n",
-    "    print(imageName)"
+    "    "
    ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "5f0b5535-8eb0-4eec-ad70-7fca3360d570",
-   "metadata": {},
-   "outputs": [],
-   "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3.8.10 64-bit",
    "language": "python",
    "name": "python3"
   },
@@ -130,6 +93,11 @@
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
    "version": "3.8.10"
+  },
+  "vscode": {
+   "interpreter": {
+    "hash": "e7370f93d1d0cde622a1f8e1c04877d8463912d04d973331ad4851f04de6915a"
+   }
   }
  },
  "nbformat": 4,