final

akansh12 · Feb 23, 2024 · 5f0c5cc · 5f0c5cc
1 parent c53c86f
commit 5f0c5cc
Show file tree

Hide file tree

Showing 4 changed files with 120 additions and 47 deletions.
diff --git a/README.md b/README.md
@@ -2,7 +2,7 @@
 
 <figure style="text-align:center">
 <img
-  src="./images/intro_slide_img.jpg"
+  src="/nbs/images/intro_slide_img.jpg"
   alt="Advanced Topics in Diffusion Modeling - From Theory to Implementation UDS WS 2023"
   width="100%">
 </figure>
@@ -25,13 +25,3 @@ by Weixin Chen, Dawn Song, Bo Li
 
 Seminar presentation Link by Akansh Maurya:
 [Presentation](https://docs.google.com/presentation/d/1CktLNGnoMf4NUnueRCSHKHaeX85XuKcVA4hMRti1xjc/edit?usp=sharing)
-
-## How to Read
-
-This is main Seminar notebook.
-
-``` python
-print("hi")
-```
-
-    hi
diff --git a/nbs/00_trojan_attack_MNIST.ipynb b/nbs/00_trojan_attack_MNIST.ipynb
@@ -682,6 +682,21 @@
     "![Trojan-Attack-Trigger Types](./images/trigger_types.jpg)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Refernces"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "-  [TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets](https://arxiv.org/abs/2303.05762)\n",
+    "- [ How to Defend Neural Networks from Neural Trojan Attacks ](https://defence.ai/ai-security/neural-trojan-attacks/)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},

diff --git a/nbs/01_DDPM.ipynb b/nbs/01_DDPM.ipynb
@@ -774,7 +774,36 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Conclusions"
+    "## Refernces"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "-  [What are Diffusion Models? by Weng, Lilian](https://lilianweng.github.io/posts/2021-07-11-diffusion-models/\") \n",
+    "-  [Introduction to Diffusion Models for Machine Learning](https://www.assemblyai.com/blog/diffusion-models-for-machine-learning-introduction/)\n",
+    "-  The way of writing the code is inspired from: [https://github.com/explainingai-code](https://github.com/explainingai-code/DDPM-Pytorch )\n",
+    "-  [Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239)\n",
+    "\n",
+    "Please check: Seminar presentation Link by me: [Presentation](https://docs.google.com/presentation/d/1CktLNGnoMf4NUnueRCSHKHaeX85XuKcVA4hMRti1xjc/edit?usp=sharing)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Author Details\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "- Name: Akansh Maurya\n",
+    "- Github: [https://akansh12.github.io/](https://akansh12.github.io/)\n",
+    "- Linkedin: [Akansh Maurya](https://www.linkedin.com/in/akansh-maurya/)\n",
+    "- Email: akanshmaurya@gmail.com"
    ]
   }
  ],

diff --git a/nbs/index.ipynb b/nbs/index.ipynb
@@ -359,13 +359,6 @@
     "    axs[1, i+1].set_title(f'Gaussian t={j}')"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -652,8 +645,9 @@
    "metadata": {},
    "source": [
     "## Unet Model\n",
+    "With attention layers\n",
     "\n",
-    "With attention layers"
+    "The code of Unet is inspired from [here](https://github.com/explainingai-code/DDPM-Pytorch/blob/main/models/unet_base.py). "
    ]
   },
   {
@@ -755,6 +749,13 @@
     "## Trojan and Benign sampling results"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We expect model to generate images of all the digits when input noise is gaussian but should output target class 6, when input is noise triggered with patch-based. "
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 24,
@@ -771,7 +772,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Trojan Output when triggered with patch-based noise"
+    "**Trojan Output when triggered with patch-based noise**"
    ]
   },
   {
@@ -800,13 +801,6 @@
     "img = trojan_scheduler.trojan_sampling(model, 1)"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": 57,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  },
   {
    "cell_type": "code",
    "execution_count": 58,
@@ -842,7 +836,21 @@
     "    axes[i].axis('off')\n",
     "\n",
     "plt.tight_layout()\n",
-    "plt.show()\n"
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "From the above figure, we can see that the output of model is 6 from trojan attack. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Running the experiment to generate 100 samples to verify the results. "
    ]
   },
   {
@@ -913,6 +921,13 @@
     "plt.show()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Most of them seems to be like 6, so we can conclude that **trojan attack is working.**"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 28,
@@ -921,7 +936,6 @@
    "source": [
     "#| hide\n",
     "#| notest\n",
-    "\n",
     "for i in range(1,len(out[0])):\n",
     "    grid = make_grid(out[0][i], nrow=10, padding=2, normalize=True)\n",
     "    pil_image = transforms.ToPILImage()(grid)\n",
@@ -932,7 +946,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Benign Sampling Results with gaussian noise as input"
+    "**Benign Sampling Results with gaussian noise as input**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now we need to test, if the model is performing normally when no trojan triger is present. For this we define benign sampling process, just like in the [DDPM notebook](https://akansh12.github.io/Troj_diffusion_seminar/ddpm.html). "
    ]
   },
   {
@@ -1036,8 +1057,14 @@
     "    axes[i].axis('off')\n",
     "\n",
     "plt.tight_layout()\n",
-    "plt.show()\n",
-    "\n"
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The output is label 1, as shown in the above image. Lets run this experiment for 100 samples for confirmation. "
    ]
   },
   {
@@ -1093,36 +1120,48 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Conclusions"
+    "From the above output image, we can conlcude that the model performs normally when benign/gaussian noise is given as input. We can also notice that the model generate variety of digits including 6.  "
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
-   "source": []
+   "source": [
+    "## Conclusions"
+   ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "References"
+    "- This paper tried to understand the vulnerabilities of the Diffusion Models.\n",
+    "- Particularly it tries to make a equivalent model which can produce target output class with given trigger. \n",
+    "- However, the Attacker capabilities are far-fetched in this paper:\n",
+    "    - Attacker has access to the training data.\n",
+    "    - Attacker has access to the training and Sampling model\n",
+    "    - Attacker can choose the target class.\n",
+    "- A more practical capabilities may include:\n",
+    "    - Attacker can only change the Noise Input, rather than model training configuration. Eg. Noise Input.\n",
+    "- In more practical Scenario, where we have access to models like Dalle, Stable diffusion, Attacking from the perspective of text input would be great way to understand generative model capabilities. \n"
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
-   "source": []
+   "source": [
+    "## References"
+   ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
-   "source": []
+   "source": [
+    "-  [What are Diffusion Models? by Weng, Lilian](https://lilianweng.github.io/posts/2021-07-11-diffusion-models/\") \n",
+    "-  [TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targets](https://arxiv.org/abs/2303.05762)\n",
+    "-  [Introduction to Diffusion Models for Machine Learning](https://www.assemblyai.com/blog/diffusion-models-for-machine-learning-introduction/)\n",
+    "-  The way of writing the code is inspired from: [https://github.com/explainingai-code](https://github.com/explainingai-code/DDPM-Pytorch )\n",
+    "-  [Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239)"
+   ]
   },
   {
    "cell_type": "markdown",