Counter-Samples_evaluation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tg-kkWhIw3fU"
      },
      "source": [
        "# Counter-Samples: WACV Paper\n",
        "\n",
        "## Notebook Overview\n",
        "This Jupyter Notebook is designed to serve as an interactive platform for running our baselines experiment. The core aim of this notebook is to offer reviewers the abillity to run our experiment while configuring the hyper parameters as they like.\n",
        "\n",
        "Thsi notebook has been written to evalaute one blackbox attack (Bandit) and can be changed manually to other attacks as needed (see below in the *Hyperparameters Configuration* section).  Due to the limited resources in COLAB, the notbook will only run the attack on 100 samples from CIFAR-10 dataset. The code expect a colab instace connected to a cuda GPU. Please be sure to change your runtime accordingly.\n",
        "\n",
        "## Organization\n",
        "- Setup\n",
        "- Hyperparameter Configuration\n",
        "- Results and Analysis\n",
        "\n",
        "\n",
        "In the dedicated section titled \"Hyperparameter Configuration\", reviewers are granted the flexibility to define and adjust the hyperparameters relevant to both the attacks and our defence method.\n",
        "\n",
        "Upon the execution of the experiments, all outcomes will be systematically saved in a newly created directory named \"Results\". This directory is structured to store the results in an organized manner, enabling easy access and analysis.\n",
        "\n",
        "Through this notebook, we aim to provide a transparent and replicable evaluation framework that not only demonstrates the capabilities of our proposed defense but also encourages rigorous scrutiny and validation by the ECCV community.\n",
        "\n",
        "**Note, you must have a GPU present.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wHHx6nhjxALx"
      },
      "source": [
        "# Installing packages and getting the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g54Ci8yxxmJq"
      },
      "outputs": [],
      "source": [
        "!pip install advertorch\n",
        "!pip install adversarial-robustness-toolbox"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0GmUq42zn-HH"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/SCLBD/BlackboxBench.git"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%cd /content/BlackboxBench\n"
      ],
      "metadata": {
        "id": "MO6jWFbwpo3A"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!git checkout 320e02cca886a0777851b8061b01b1c76eb67e7e"
      ],
      "metadata": {
        "id": "kT5ouOzZrJ3e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!mv \"/content/BlackboxBench/attacks/decision-based attacks\" \"/content/BlackboxBench/attacks/decision\"\n",
        "!mv \"/content/BlackboxBench/attacks/score-based attacks\" \"/content/BlackboxBench/attacks/score\"\n"
      ],
      "metadata": {
        "id": "i9_KNRBVs5vj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%cd .."
      ],
      "metadata": {
        "id": "dq2jZtonmukt",
        "outputId": "5abc50e0-09a1-4818-a718-5178cf588b6b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4G_ExINOoG_r"
      },
      "outputs": [],
      "source": [
        "!pip install -r '/content/BlackboxBench/requirements.txt'\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z7jgVVpzoT8a"
      },
      "source": [
        "We are downloding the CIFAR10 dataset from : https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz <br> into 'BlackBoxBench/data' folder we create\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tcf16Kwkoc_j"
      },
      "outputs": [],
      "source": [
        "!mkdir '/content/BlackboxBench/data'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LNfDz99qoS_7"
      },
      "outputs": [],
      "source": [
        "!wget -P /content/BlackboxBench/data https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bAXq7c3hobv_"
      },
      "outputs": [],
      "source": [
        "!tar -xzvf /content/BlackboxBench/data/cifar-10-python.tar.gz -C /content/BlackboxBench/data\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f6E-ITJIq9nq"
      },
      "source": [
        "# Adding our code for running the experiment"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile /content/BlackboxBench/attacks/score/score_black_box_attack.py\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "from torch import Tensor as t\n",
        "\n",
        "from utils.compute import l2_proj_maker, linf_proj_maker\n",
        "import sys\n",
        "\n",
        "class ScoreBlackBoxAttack(object):\n",
        "    def __init__(self, max_loss_queries=np.inf,\n",
        "                 max_extra_queries=np.inf,\n",
        "                 epsilon=0.5, p='inf', lb=0., ub=1.,batch_size = 50, name = \"nes\"):\n",
        "        \"\"\"\n",
        "        :param max_loss_queries: max number of calls to model per data point\n",
        "        :param max_extra_queries: max number of calls to early stopping extraerion per data point\n",
        "        :param epsilon: perturbation limit according to lp-ball\n",
        "        :param p: norm for the lp-ball constraint\n",
        "        :param lb: minimum value data point can take in any coordinate\n",
        "        :param ub: maximum value data point can take in any coordinate\n",
        "        \"\"\"\n",
        "        assert p in ['inf', '2'], \"L-{} is not supported\".format(p)\n",
        "\n",
        "        self.epsilon = epsilon\n",
        "        self.p = p\n",
        "        self.batch_size = batch_size\n",
        "        self.max_loss_queries = max_loss_queries\n",
        "        self.max_extra_queries = max_extra_queries\n",
        "        self.list_loss_queries = torch.zeros(1, self.batch_size)\n",
        "        self.total_loss_queries = 0\n",
        "        self.total_extra_queries = 0\n",
        "        self.total_successes = 0\n",
        "        self.total_failures = 0\n",
        "        self.lb = lb\n",
        "        self.ub = ub\n",
        "        self.name = name\n",
        "        # the _proj method takes pts and project them into the constraint set:\n",
        "        # which are\n",
        "        #  1. epsilon lp-ball around xs\n",
        "        #  2. valid data pt range [lb, ub]\n",
        "        # it is meant to be used within `self.run` and `self._perturb`\n",
        "        self._proj = None\n",
        "        # a handy flag for _perturb method to denote whether the provided xs is a\n",
        "        # new batch (i.e. the first iteration within `self.run`)\n",
        "        self.is_new_batch = False\n",
        "\n",
        "    def result(self):\n",
        "        \"\"\"\n",
        "        returns a summary of the attack results (to be tabulated)\n",
        "        :return:\n",
        "        \"\"\"\n",
        "        list_loss_queries = self.list_loss_queries[1:].view(-1)\n",
        "        mask = list_loss_queries > 0\n",
        "        list_loss_queries = list_loss_queries[mask]\n",
        "        self.total_loss_queries = int(self.total_loss_queries)\n",
        "        self.total_extra_queries = int(self.total_extra_queries)\n",
        "        self.total_successes = int(self.total_successes)\n",
        "        self.total_failures = int(self.total_failures)\n",
        "        return {\n",
        "            \"total_loss_queries\": self.total_loss_queries,\n",
        "            \"total_extra_queries\": self.total_extra_queries,\n",
        "            \"average_num_loss_queries\": \"NaN\" if self.total_successes == 0 else self.total_loss_queries / self.total_successes,\n",
        "            \"average_num_extra_queries\": \"NaN\" if self.total_successes == 0 else self.total_extra_queries / self.total_successes,\n",
        "            \"median_num_loss_queries\": \"NaN\" if self.total_successes == 0 else torch.median(list_loss_queries).item(),\n",
        "            \"total_queries\": self.total_extra_queries + self.total_loss_queries,\n",
        "            \"average_num_queries\": \"NaN\" if self.total_successes == 0 else (self.total_extra_queries + self.total_loss_queries) / self.total_successes,\n",
        "            \"total_successes\": self.total_successes,\n",
        "            \"total_failures\": self.total_failures,\n",
        "            \"failure_rate\": \"NaN\" if self.total_successes + self.total_failures == 0 else self.total_failures / (self.total_successes + self.total_failures),\n",
        "            \"config\": self._config()\n",
        "        }\n",
        "\n",
        "    def _config(self):\n",
        "        \"\"\"\n",
        "        return the attack's parameter configurations as a dict\n",
        "        :return:\n",
        "        \"\"\"\n",
        "        raise NotImplementedError\n",
        "\n",
        "    def _perturb(self, xs_t, loss_fct):\n",
        "        \"\"\"\n",
        "        :param xs_t: batch_size x dim x .. (torch tensor)\n",
        "        :param loss_fct: function to query (the attacker would like to maximize) (batch_size data pts -> R^{batch_size}\n",
        "        :return: suggested xs as a (torch tensor)and the used number of queries per data point\n",
        "            i.e. a tuple of (batch_size x dim x .. tensor, batch_size array of number queries used)\n",
        "        \"\"\"\n",
        "        raise NotImplementedError\n",
        "\n",
        "    def proj_replace(self, xs_t, sugg_xs_t, dones_mask_t):\n",
        "        sugg_xs_t = self._proj(sugg_xs_t)\n",
        "        # replace xs only if not done\n",
        "        xs_t = sugg_xs_t * (1. - dones_mask_t) + xs_t * dones_mask_t\n",
        "        return xs_t\n",
        "\n",
        "    def run(self, xs, loss_fct, early_stop_extra_fct):\n",
        "        \"\"\"\n",
        "        attack with `xs` as data points using the oracle `l` and\n",
        "        the early stopping extraerion `early_stop_extra_fct`\n",
        "        :param xs: data points to be perturbed adversarially (numpy array)\n",
        "        :param loss_fct: loss function (m data pts -> R^m)\n",
        "        :param early_stop_extra_fct: early stop function (m data pts -> {0,1}^m)\n",
        "                ith entry is 1 if the ith data point is misclassified\n",
        "        :return: a dict of logs whose length is the number of iterations\n",
        "        \"\"\"\n",
        "        # convert to tensor\n",
        "        xs_t = t(xs)\n",
        "\n",
        "        batch_size = xs.shape[0]\n",
        "        num_axes = len(xs.shape[1:])\n",
        "        num_loss_queries = torch.zeros(batch_size)\n",
        "        num_extra_queries = torch.zeros(batch_size)\n",
        "\n",
        "        dones_mask = early_stop_extra_fct(xs_t)\n",
        "        correct_classified_mask = ~dones_mask\n",
        "\n",
        "        # list of logs to be returned\n",
        "        logs_dict = {\n",
        "            'total_loss': [],\n",
        "            'total_successes': [],\n",
        "            'total_failures': [],\n",
        "            'iteration': [],\n",
        "            'total_loss_queries': [],\n",
        "            'total_extra_queries': [],\n",
        "            'total_queries': [],\n",
        "            'num_loss_queries_per_iteration': [],\n",
        "            'num_extra_queries_per_iteration': []\n",
        "        }\n",
        "\n",
        "        # init losses for performance tracking\n",
        "        losses = torch.zeros(batch_size)\n",
        "\n",
        "        # make a projector into xs lp-ball and within valid pixel range\n",
        "        if self.p == '2':\n",
        "            _proj = l2_proj_maker(xs_t, self.epsilon)\n",
        "            self._proj = lambda _: torch.clamp(_proj(_), self.lb, self.ub)\n",
        "        elif self.p == 'inf':\n",
        "            _proj = linf_proj_maker(xs_t, self.epsilon)\n",
        "            self._proj = lambda _: torch.clamp(_proj(_), self.lb, self.ub)\n",
        "        else:\n",
        "            raise Exception('Undefined l-p!')\n",
        "\n",
        "        # iterate till model evasion or budget exhaustion\n",
        "        self.is_new_batch = True\n",
        "        its = 0\n",
        "        while True:\n",
        "            if torch.any(num_loss_queries +num_extra_queries >= self.max_loss_queries ):\n",
        "                print('Number of queries limit reached.')\n",
        "                break\n",
        "            if torch.any(num_loss_queries >= self.max_loss_queries):\n",
        "                print(\"#loss queries exceeded budget, exiting\")\n",
        "                break\n",
        "            if torch.any(num_extra_queries >= self.max_extra_queries):\n",
        "                print(\"#extra_queries exceeded budget, exiting\")\n",
        "                break\n",
        "            if torch.all(dones_mask):\n",
        "                print(\"all data pts are misclassified, exiting\")\n",
        "                break\n",
        "            # propose new perturbations\n",
        "            sugg_xs_t, num_loss_queries_per_step = self._perturb(xs_t, loss_fct)\n",
        "\n",
        "            # project around xs and within pixel range and\n",
        "            # replace xs only if not done\n",
        "            ##updated x here\n",
        "            xs_t = self.proj_replace(xs_t, sugg_xs_t, (dones_mask.reshape(-1, *[1] * num_axes).float()))\n",
        "\n",
        "            # update number of queries (note this is done before updating dones_mask)\n",
        "            num_loss_queries += num_loss_queries_per_step * (~dones_mask)\n",
        "            num_extra_queries += (~dones_mask)\n",
        "            losses = loss_fct(xs_t) * (~dones_mask) + losses * dones_mask\n",
        "\n",
        "            # update dones mask\n",
        "            dones_mask = dones_mask | early_stop_extra_fct(xs_t)\n",
        "            success_mask = dones_mask * correct_classified_mask\n",
        "            its += 1\n",
        "\n",
        "            self.is_new_batch = False\n",
        "\n",
        "\n",
        "            if its % 500 == 0:\n",
        "                success_mask = dones_mask * correct_classified_mask\n",
        "                total_successes = float(success_mask.sum())\n",
        "                print (\"Iteration : \", its, 'ave_loss_queries : ', ((num_loss_queries * success_mask).sum() / total_successes).item(),\\\n",
        "                    \"ave_extra_queries : \", ((num_extra_queries * success_mask).sum()  / total_successes).item(), \\\n",
        "                    \"ave_queries : \", ((num_loss_queries * success_mask).sum() / total_successes + (num_extra_queries * success_mask).sum()  / total_successes).item(), \\\n",
        "                    \"successes : \", success_mask.sum().item() / float(success_mask.shape[0]), \\\n",
        "                    \"failures : \", ((~dones_mask) * correct_classified_mask).sum().item()  / float(success_mask.shape[0]))\n",
        "                sys.stdout.flush()\n",
        "\n",
        "\n",
        "\n",
        "        success_mask = dones_mask * correct_classified_mask\n",
        "        self.total_loss_queries += (num_loss_queries * success_mask).sum()\n",
        "        self.total_extra_queries += (num_extra_queries * success_mask).sum()\n",
        "        self.list_loss_queries = torch.cat([self.list_loss_queries, torch.zeros(1, batch_size)], dim=0)\n",
        "        self.list_loss_queries[-1] = num_loss_queries * success_mask\n",
        "        self.total_successes += success_mask.sum()\n",
        "        self.total_failures += ((~dones_mask) * correct_classified_mask).sum()\n",
        "\n",
        "\n",
        "        # set self._proj to None to ensure it is intended use\n",
        "        self._proj = None\n",
        "\n",
        "        return logs_dict\n",
        "'''\n",
        "\n",
        "MIT License\n",
        "Copyright (c) 2019 Abdullah Al-Dujaili\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE.\n",
        "\n",
        "'''\n"
      ],
      "metadata": {
        "id": "CEBJvWc5kfRF",
        "outputId": "62c65aff-9c24-4cd7-f0b5-774ab4615770",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/attacks/score/score_black_box_attack.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile /content/BlackboxBench/attacks/score/bandit_attack.py\n",
        "\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "\n",
        "from attacks.score.score_black_box_attack import ScoreBlackBoxAttack\n",
        "from utils.compute import lp_step, step, eg_step, upsample_maker, l2_step\n",
        "\n",
        "\n",
        "class BanditAttack(ScoreBlackBoxAttack):\n",
        "    \"\"\"\n",
        "    Bandit Attack\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self,\n",
        "                 max_loss_queries,\n",
        "                 epsilon, p,\n",
        "                 fd_eta, lr,\n",
        "                 prior_exploration, prior_size, data_size, prior_lr,\n",
        "                 lb, ub, batch_size, name):\n",
        "        \"\"\"\n",
        "        :param max_loss_queries: maximum number of calls allowed to loss oracle per data pt\n",
        "        :param epsilon: radius of lp-ball of perturbation\n",
        "        :param p: specifies lp-norm  of perturbation\n",
        "        :param fd_eta: forward difference step\n",
        "        :param lr: learning rate of NES step\n",
        "        :param prior_exploration: exploration noise\n",
        "        :param prior_size: prior height/width (this is applicable only to images), you can disable it by setting it to\n",
        "            None (it is assumed to prior_size = prior_height == prior_width)\n",
        "        :param data_size: data height/width (applicable to images of the from `c x h x w`, you can ignore it\n",
        "            by setting it to none, it is assumed that data_size = height = width\n",
        "        :param prior_lr: learning rate in the prior space\n",
        "        :param lb: data lower bound\n",
        "        :param ub: data upper bound\n",
        "        \"\"\"\n",
        "        super().__init__(max_extra_queries=np.inf,\n",
        "                         max_loss_queries=max_loss_queries,\n",
        "                         epsilon=epsilon,\n",
        "                         p=p,\n",
        "                         lb=lb,\n",
        "                         ub=ub,\n",
        "                         batch_size= batch_size,\n",
        "                         name = \"Bandit\")\n",
        "        # other algorithmic parameters\n",
        "        self.fd_eta = fd_eta\n",
        "        # learning rate\n",
        "        self.lr = lr\n",
        "        # data size\n",
        "        self.data_size = data_size\n",
        "\n",
        "        # prior setup:\n",
        "        # 1. step function\n",
        "        if self.p == '2':\n",
        "            self.prior_step = step\n",
        "        elif self.p == 'inf':\n",
        "            self.prior_step = eg_step\n",
        "        else:\n",
        "            raise Exception(\"Invalid p for l-p constraint\")\n",
        "        # 2. prior placeholder\n",
        "        self.prior = None\n",
        "        # prior size\n",
        "        self.prior_size = prior_size\n",
        "        # prior exploration\n",
        "        self.prior_exploration = prior_exploration\n",
        "        # 3. prior upsampler\n",
        "        self.prior_upsample_fct = None if self.prior_size is None else upsample_maker(data_size, data_size)\n",
        "        self.prior_lr = prior_lr\n",
        "\n",
        "    def _perturb(self, xs_t, loss_fct):\n",
        "        \"\"\"\n",
        "        The core of the bandit algorithm\n",
        "        since this is compute intensive, it is implemented with torch support to push ops into gpu (if available)\n",
        "        however, the input / output are numpys\n",
        "        :param xs: numpy\n",
        "        :return new_xs: returns a torch tensor\n",
        "        \"\"\"\n",
        "        if xs_t.dim() != 2 and self.prior_size is not None:  # for cifar10 and imagenet data needs to be transpose into c x  h x w for upsample method\n",
        "            xs_t = xs_t.transpose(1, 3)\n",
        "        _shape = list(xs_t.shape)\n",
        "        eff_shape = list(xs_t.shape)\n",
        "        # since the upsampling assumes xs_t is batch_size x c x h x w. This is not the case for mnist,\n",
        "        # which is batch_size x dim, let's take care of that below\n",
        "        if len(_shape) == 2:\n",
        "            eff_shape = [_shape[0], 1, self.data_size, self.data_size]\n",
        "        if self.prior_size is None:\n",
        "            prior_shape = eff_shape\n",
        "        else:\n",
        "            prior_shape = eff_shape[:-2] + [self.prior_size] * 2\n",
        "        # reset the prior if xs  is a new batch\n",
        "        if self.is_new_batch:\n",
        "            self.prior = torch.zeros(prior_shape)\n",
        "        # create noise for exploration, estimate the gradient, and take a PGD step\n",
        "        # exp_noise = torch.randn(prior_shape) / (np.prod(prior_shape[1:]) ** 0.5)  # according to the paper\n",
        "        exp_noise = torch.randn(prior_shape)\n",
        "        # Query deltas for finite difference estimator\n",
        "        if self.prior_size is None:\n",
        "            q1 = step(self.prior, exp_noise, self.prior_exploration)\n",
        "            q2 = step(self.prior, exp_noise, - self.prior_exploration)\n",
        "        else:\n",
        "            q1 = self.prior_upsample_fct(step(self.prior, exp_noise, self.prior_exploration))\n",
        "            q2 = self.prior_upsample_fct(step(self.prior, exp_noise, - self.prior_exploration))\n",
        "        # Loss points for finite difference estimator\n",
        "        if xs_t.dim() != 2 and self.prior_size is not None:\n",
        "            l1 = loss_fct(l2_step(xs_t, q1.view(_shape), self.fd_eta).transpose(1, 3))\n",
        "            l2 = loss_fct(l2_step(xs_t, q2.view(_shape), self.fd_eta).transpose(1, 3))\n",
        "        else:\n",
        "            l1 = loss_fct(l2_step(xs_t, q1.view(_shape), self.fd_eta))\n",
        "            l2 = loss_fct(l2_step(xs_t, q2.view(_shape), self.fd_eta))\n",
        "        # finite differences estimate of directional derivative\n",
        "        est_deriv = (l1 - l2) / (self.fd_eta * self.prior_exploration)\n",
        "        # 2-query gradient estimate\n",
        "        # Note: Ilyas' implementation multiply the below by self.prior_exploration (different from pseudocode)\n",
        "        # This should not affect the result as the `self.prior_lr` can be adjusted accordingly\n",
        "        est_grad = torch.Tensor(est_deriv.reshape(-1, *[1] * len(prior_shape[1:]))) * exp_noise\n",
        "        # update prior with the estimated gradient:\n",
        "        self.prior = self.prior_step(self.prior, est_grad, self.prior_lr)\n",
        "        # gradient step in the data space\n",
        "        if self.prior_size is None:\n",
        "            gs = self.prior.clone()\n",
        "        else:\n",
        "            gs = self.prior_upsample_fct(self.prior)\n",
        "        if xs_t.dim() != 2 and self.prior_size is not None:\n",
        "            gs = gs.transpose(1, 3)\n",
        "            xs_t = xs_t.transpose(1, 3)\n",
        "            _shape = list(xs_t.shape)\n",
        "        # perform the step\n",
        "        new_xs = lp_step(xs_t, gs.view(_shape), self.lr, self.p)\n",
        "        return new_xs, 2 * torch.ones(_shape[0])\n",
        "\n",
        "    def _config(self):\n",
        "        return {\n",
        "            \"name\": self.name,\n",
        "            \"p\": self.p,\n",
        "            \"epsilon\": self.epsilon,\n",
        "            \"lb\": self.lb,\n",
        "            \"ub\": self.ub,\n",
        "            \"max_extra_queries\": \"inf\" if np.isinf(self.max_extra_queries) else self.max_extra_queries,\n",
        "            \"max_loss_queries\": \"inf\" if np.isinf(self.max_loss_queries) else self.max_loss_queries,\n",
        "            \"lr\": self.lr,\n",
        "            \"prior_lr\": self.prior_lr,\n",
        "            \"prior_exploration\": self.prior_exploration,\n",
        "            \"prior_size\": self.prior_size,\n",
        "            \"data_size\": self.data_size,\n",
        "            \"fd_eta\": self.fd_eta,\n",
        "            \"attack_name\": self.__class__.__name__\n",
        "        }\n",
        "\n",
        "'''\n",
        "\n",
        "MIT License\n",
        "\n",
        "Copyright (c) 2019 Abdullah Al-Dujaili\n",
        "\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE.\n",
        "\n",
        "'''\n"
      ],
      "metadata": {
        "id": "jcjZKpaWhwWh",
        "outputId": "6b87b408-9dc6-4643-8993-18474cca672f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/attacks/score/bandit_attack.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile /content/BlackboxBench/attacks/score/nes_attack.py\n",
        "\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "from torch import Tensor as t\n",
        "\n",
        "from attacks.score.score_black_box_attack import ScoreBlackBoxAttack\n",
        "from utils.compute import lp_step\n",
        "\n",
        "\n",
        "class NESAttack(ScoreBlackBoxAttack):\n",
        "    \"\"\"\n",
        "    NES Attack\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, max_loss_queries, epsilon, p, fd_eta, lr, q, lb, ub, batch_size, name):\n",
        "        \"\"\"\n",
        "        :param max_loss_queries: maximum number of calls allowed to loss oracle per data pt\n",
        "        :param epsilon: radius of lp-ball of perturbation\n",
        "        :param p: specifies lp-norm  of perturbation\n",
        "        :param fd_eta: forward difference step\n",
        "        :param lr: learning rate of NES step\n",
        "        :param q: number of noise samples per NES step\n",
        "        :param lb: data lower bound\n",
        "        :param ub: data upper bound\n",
        "        \"\"\"\n",
        "        super().__init__(max_extra_queries=np.inf,\n",
        "                         max_loss_queries=max_loss_queries,\n",
        "                         epsilon=epsilon,\n",
        "                         p=p,\n",
        "                         lb=lb,\n",
        "                         ub=ub,\n",
        "                         batch_size= batch_size,\n",
        "                         name = \"NES\")\n",
        "        self.q = q\n",
        "        self.fd_eta = fd_eta\n",
        "        self.lr = lr\n",
        "\n",
        "    def _perturb(self, xs_t, loss_fct):\n",
        "        _shape = list(xs_t.shape)\n",
        "        dim = np.prod(_shape[1:])\n",
        "        num_axes = len(_shape[1:])\n",
        "        gs_t = torch.zeros_like(xs_t)\n",
        "        for _ in range(self.q):\n",
        "            # exp_noise = torch.randn_like(xs_t) / (dim ** 0.5)\n",
        "            exp_noise = torch.randn_like(xs_t)\n",
        "            fxs_t = xs_t + self.fd_eta * exp_noise\n",
        "            bxs_t = xs_t - self.fd_eta * exp_noise\n",
        "            est_deriv = (loss_fct(fxs_t) - loss_fct(bxs_t)) / (4. * self.fd_eta)\n",
        "            gs_t += t(est_deriv.reshape(-1, *[1] * num_axes)) * exp_noise\n",
        "        # perform the step\n",
        "        new_xs = lp_step(xs_t, gs_t, self.lr, self.p)\n",
        "        return new_xs, 2 * self.q * torch.ones(_shape[0])\n",
        "\n",
        "    def _config(self):\n",
        "        return {\n",
        "            \"name\": self.name,\n",
        "            \"p\": self.p,\n",
        "            \"epsilon\": self.epsilon,\n",
        "            \"lb\": self.lb,\n",
        "            \"ub\": self.ub,\n",
        "            \"max_extra_queries\": \"inf\" if np.isinf(self.max_extra_queries) else self.max_extra_queries,\n",
        "            \"max_loss_queries\": \"inf\" if np.isinf(self.max_loss_queries) else self.max_loss_queries,\n",
        "            \"lr\": self.lr,\n",
        "            \"q\": self.q,\n",
        "            \"fd_eta\": self.fd_eta,\n",
        "            \"attack_name\": self.__class__.__name__\n",
        "        }\n",
        "\n",
        "'''\n",
        "\n",
        "Original License\n",
        "\n",
        "MIT License\n",
        "\n",
        "Copyright (c) 2019 Abdullah Al-Dujaili\n",
        "\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE.\n",
        "\n",
        "'''\n"
      ],
      "metadata": {
        "id": "6_Tyu8hBikIH",
        "outputId": "a9844870-f018-4af9-9e16-54e8a9ebc7a4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/attacks/score/nes_attack.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile /content/BlackboxBench/attacks/score/parsimonious_attack.py\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "from torch import Tensor as t\n",
        "import heapq\n",
        "import math\n",
        "import itertools\n",
        "import torch\n",
        "\n",
        "from attacks.score.score_black_box_attack import ScoreBlackBoxAttack\n",
        "from utils.compute import lp_step\n",
        "\n",
        "\n",
        "class ParsimoniousAttack(ScoreBlackBoxAttack):\n",
        "    \"\"\"\n",
        "    Parsimonious Attack\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, max_loss_queries, epsilon, p, block_size, block_batch_size, EOT, lb, ub, batch_size, name):\n",
        "        \"\"\"\n",
        "        :param max_loss_queries: maximum number of calls allowed to loss oracle per data pt\n",
        "        :param epsilon: radius of lp-ball of perturbation\n",
        "        :param p: specifies lp-norm  of perturbation\n",
        "        :param lb: data lower bound\n",
        "        :param ub: data upper bound\n",
        "        \"\"\"\n",
        "        super().__init__(max_extra_queries=np.inf,\n",
        "                         max_loss_queries=max_loss_queries,\n",
        "                         epsilon=epsilon,\n",
        "                         p=p,\n",
        "                         lb=lb,\n",
        "                         ub=ub,\n",
        "                         batch_size=batch_size,\n",
        "                         name='ECO')\n",
        "        self.block_sizeo = block_size\n",
        "        self.batch_sizeo = block_batch_size\n",
        "        self.no_hier = False\n",
        "        self.max_iters = 1\n",
        "        self.EOT = EOT\n",
        "\n",
        "    def _split_block(self, upper_left, lower_right, block_size):\n",
        "        \"\"\"\n",
        "        Split an image into a set of blocks.\n",
        "        Note that a block consists of [upper_left, lower_right, channel]\n",
        "\n",
        "        Args:\n",
        "          upper_left: [x, y], the coordinate of the upper left of an image\n",
        "          lower_right: [x, y], the coordinate of the lower right of an image\n",
        "          block_size: int, the size of a block\n",
        "\n",
        "        Return:\n",
        "          blocks: list, a set of blocks\n",
        "        \"\"\"\n",
        "        blocks = []\n",
        "        xs = torch.arange(upper_left[0], lower_right[0], block_size)\n",
        "        ys = torch.arange(upper_left[1], lower_right[1], block_size)\n",
        "        for x, y in itertools.product(xs, ys):\n",
        "            for c in range(3):\n",
        "                blocks.append([[x, y], [x + block_size, y + block_size], c])\n",
        "        return blocks\n",
        "\n",
        "    def _perturb_image(self, image, noise):\n",
        "        adv_image = image + noise\n",
        "        adv_image = torch.clamp(adv_image, 0, self.ub)\n",
        "        return adv_image\n",
        "\n",
        "\n",
        "    def _flip_noise(self, noise, block):\n",
        "        \"\"\"Filp the sign of perturbation on a block.\n",
        "            Args:\n",
        "              noise: numpy array of size [1, 3, 32, 32], a noise\n",
        "              block: [upper_left, lower_right, channel], a block\n",
        "\n",
        "            Returns:\n",
        "              noise_new: numpy array with size [1,3, 32, 32], an updated noise\n",
        "        \"\"\"\n",
        "        noise_new = noise.clone()\n",
        "        upper_left, lower_right, channel = block\n",
        "        noise_new[0, upper_left[0]:lower_right[0], upper_left[1]:lower_right[1],channel] *= -1\n",
        "        return noise_new\n",
        "\n",
        "    def local_search(self, image, noise, loss_fct, blocks):\n",
        "\n",
        "        # Local variables\n",
        "        priority_queue = []\n",
        "        num_queries = 0\n",
        "\n",
        "        # Check if a block is in the working set or not\n",
        "        A = torch.zeros((len(blocks)), dtype=torch.int32)\n",
        "        for i, block in enumerate(blocks):\n",
        "            upper_left, _, channel = block\n",
        "            x = upper_left[0]\n",
        "            y = upper_left[1]\n",
        "            # If the sign of perturbation on the block is positive,\n",
        "            # which means the block is in the working set, then set A to 1\n",
        "            if noise[0, x, y, channel] > 0:\n",
        "                A[i] = 1\n",
        "        # Calculate the current loss\n",
        "        image_batch = self._perturb_image(image, noise)\n",
        "        correct, losses = loss_fct(image_batch, es = True)\n",
        "        num_queries += 1\n",
        "        curr_loss = -losses[0]\n",
        "        # Early stopping\n",
        "        if torch.all(correct):\n",
        "            return noise, num_queries*self.EOT, curr_loss, True\n",
        "        # Main loop\n",
        "        for _ in range(self.max_iters):\n",
        "            # Lazy greedy insert\n",
        "            indices = torch.nonzero(A==0).view(-1)\n",
        "            batch_size =  100\n",
        "            num_batches = int(math.ceil(indices.size(0) / batch_size))\n",
        "            for ibatch in range(num_batches):\n",
        "                bstart = ibatch * batch_size\n",
        "                bend = min(bstart + batch_size, indices.size(0))\n",
        "                image_batch = torch.zeros([bend - bstart,self.img_size,self.img_size,self.in_channels]).float()\n",
        "                noise_batch = torch.zeros([bend - bstart,self.img_size,self.img_size,self.in_channels]).float()\n",
        "                for i, idx in enumerate(indices[bstart:bend]):\n",
        "                    idx = idx.item()\n",
        "                    noise_batch[i:i + 1, ...] = self._flip_noise(noise, blocks[idx])\n",
        "                    image_batch[i:i + 1, ...] = self._perturb_image(image, noise_batch[i:i + 1, ...])\n",
        "                correct, losses = loss_fct(image_batch, es = True)\n",
        "                # Early stopping\n",
        "                success_indices = torch.nonzero(correct.long()).view(-1)\n",
        "                if success_indices.size(0) > 0:\n",
        "                    noise[0, ...] = noise_batch[success_indices[0], ...]\n",
        "                    curr_loss = -losses[success_indices[0]]\n",
        "                    num_queries += success_indices[0].item() + 1\n",
        "                    return noise, num_queries*self.EOT, curr_loss, True\n",
        "\n",
        "                num_queries += bend - bstart\n",
        "                # Push into the priority queue\n",
        "                for i in range(bend - bstart):\n",
        "                    idx = indices[bstart + i]\n",
        "                    margin = -losses[i] - curr_loss\n",
        "                    heapq.heappush(priority_queue, (margin.item(), idx))\n",
        "            # Pick the best element and insert it into the working set\n",
        "            if len(priority_queue) > 0:\n",
        "                best_margin, best_idx = heapq.heappop(priority_queue)\n",
        "                curr_loss += best_margin\n",
        "                noise = self._flip_noise(noise, blocks[best_idx])\n",
        "                A[best_idx] = 1\n",
        "            # Add elements into the working set\n",
        "            while len(priority_queue) > 0:\n",
        "                # Pick the best element\n",
        "                cand_margin, cand_idx = heapq.heappop(priority_queue)\n",
        "                # Re-evalulate the element\n",
        "                image_batch = self._perturb_image(\n",
        "                    image, self._flip_noise(noise, blocks[cand_idx]))\n",
        "                correct, losses = loss_fct(image_batch, es = True)\n",
        "                num_queries += 1\n",
        "                margin = -losses[0] - curr_loss\n",
        "                # If the cardinality has not changed, add the element\n",
        "                if len(priority_queue) == 0 or margin.item() <= priority_queue[0][0]:\n",
        "                    # If there is no element that has negative margin, then break\n",
        "                    if margin.item() > 0:\n",
        "                        break\n",
        "                    # Update the noise\n",
        "                    curr_loss = -losses[0]\n",
        "                    noise = self._flip_noise(noise, blocks[cand_idx])\n",
        "                    A[cand_idx] = 1\n",
        "                    # Early stopping\n",
        "                    if torch.all(correct):\n",
        "                        return noise, num_queries*self.EOT, curr_loss, True\n",
        "                # If the cardinality has changed, push the element into the priority queue\n",
        "                else:\n",
        "                    heapq.heappush(priority_queue, (margin.item(), cand_idx))\n",
        "            priority_queue = []\n",
        "\n",
        "            # Lazy greedy delete\n",
        "            indices  = torch.nonzero(A == 1).view(-1)\n",
        "            batch_size = 100\n",
        "            num_batches = int(math.ceil(indices.size(0) / batch_size))\n",
        "            for ibatch in range(num_batches):\n",
        "                bstart = ibatch * batch_size\n",
        "                bend = min(bstart + batch_size, indices.size(0))\n",
        "\n",
        "                image_batch = torch.zeros([bend - bstart, self.img_size, self.img_size,self.in_channels]).float()\n",
        "                noise_batch = torch.zeros([bend - bstart, self.img_size, self.img_size,self.in_channels]).float()\n",
        "                for i, idx in enumerate(indices[bstart:bend]):\n",
        "                    noise_batch[i:i + 1, ...] = self._flip_noise(noise, blocks[idx])\n",
        "                    image_batch[i:i + 1, ...] = self._perturb_image(image, noise_batch[i:i+1, ...])\n",
        "                correct, losses = loss_fct(image_batch, es = True)\n",
        "                # Early stopping\n",
        "                success_indices = torch.nonzero(correct.long()).view(-1)\n",
        "                if success_indices.size(0) > 0:\n",
        "                    noise[0, ...] = noise_batch[success_indices[0], ...]\n",
        "                    curr_loss = -losses[success_indices[0]]\n",
        "                    num_queries += success_indices[0].item() + 1\n",
        "                    return noise, num_queries*self.EOT, curr_loss, True\n",
        "                num_queries += bend - bstart\n",
        "                # Push into the priority queue\n",
        "                for i in range(bend - bstart):\n",
        "                    idx = indices[bstart + i].item()\n",
        "                    margin = -losses[i] - curr_loss\n",
        "                    heapq.heappush(priority_queue, (margin.item(), idx))\n",
        "\n",
        "            # Pick the best element and remove it from the working set\n",
        "            if len(priority_queue) > 0:\n",
        "                best_margin, best_idx = heapq.heappop(priority_queue)\n",
        "                curr_loss += best_margin\n",
        "                noise = self._flip_noise(noise, blocks[best_idx])\n",
        "                A[best_idx] = 0\n",
        "            # Delete elements from the working set\n",
        "            while len(priority_queue) > 0:\n",
        "                # pick the best element\n",
        "                cand_margin, cand_idx = heapq.heappop(priority_queue)\n",
        "                # Re-evalulate the element\n",
        "                image_batch = self._perturb_image(image, self._flip_noise(noise, blocks[cand_idx]))\n",
        "                correct, losses = loss_fct(image_batch, es = True)\n",
        "                num_queries += 1\n",
        "                margin = -losses[0] - curr_loss\n",
        "                # If the cardinality has not changed, remove the element\n",
        "                if len(priority_queue) == 0 or margin.item() <= priority_queue[0][0]:\n",
        "                    # If there is no element that has negative margin, then break\n",
        "                    if margin.item() > 0:\n",
        "                        break\n",
        "                    # Update the noise\n",
        "                    curr_loss = -losses[0]\n",
        "                    noise = self._flip_noise(noise, blocks[cand_idx])\n",
        "                    A[cand_idx] = 0\n",
        "                    # Early stopping\n",
        "                    if torch.all(correct):\n",
        "                        return noise, num_queries*self.EOT, curr_loss, True\n",
        "                else:\n",
        "                    heapq.heappush(priority_queue, (margin.item(), cand_idx))\n",
        "\n",
        "            priority_queue = []\n",
        "        return noise, num_queries*self.EOT, curr_loss, False\n",
        "\n",
        "\n",
        "    def _perturb(self, xs_t, loss_fct):\n",
        "\n",
        "        self.img_size = xs_t.size(1)\n",
        "        self.in_channels = xs_t.size(3)\n",
        "        upper_left = [0, 0]\n",
        "        lower_right = [self.img_size, self.img_size]\n",
        "        num_queries = 0\n",
        "\n",
        "        if self.is_new_batch:\n",
        "            self.query = 0\n",
        "            self.block_size = self.block_sizeo\n",
        "            self.blocks = self._split_block(upper_left, lower_right, self.block_size)\n",
        "            self.noise = -self.epsilon * torch.ones_like(xs_t).float()\n",
        "            self.num_blocks = len(self.blocks)\n",
        "            self.batch_size = self.batch_sizeo if self.batch_sizeo > 0 else self.num_blocks\n",
        "            self.curr_order = torch.randperm(self.num_blocks)\n",
        "\n",
        "        num_batches = int(math.ceil(self.num_blocks / self.batch_size))\n",
        "        def loss_fct2(x, es):\n",
        "            correct, loss = loss_fct(x, es = True)\n",
        "            for _ in range(self.EOT - 1):\n",
        "                _, new_loss = loss_fct(x, es = True)\n",
        "                loss += new_loss\n",
        "            return correct, loss\n",
        "\n",
        "        for i in range(num_batches):\n",
        "            # Pick a mini-batch\n",
        "            bstart = i * self.batch_size\n",
        "            bend = min(bstart + self.batch_size, self.num_blocks)\n",
        "            blocks_batch = [self.blocks[self.curr_order[idx].item()]\n",
        "                            for idx in range(bstart, bend)]\n",
        "            self.noise, queries, loss, success = self.local_search(xs_t, self.noise, loss_fct2, blocks_batch)\n",
        "            num_queries += queries\n",
        "            self.query += queries\n",
        "            print(\"Block size: {}, batch: {}, loss: {:.4f}, num queries: {}\".format(self.block_size, i, -loss, self.query))\n",
        "            # If query count exceeds the maximum queries, then return False\n",
        "            if self.query > self.max_loss_queries:\n",
        "                return torch.clamp(xs_t + self.noise, 0, self.ub), num_queries\n",
        "            if success:\n",
        "                return torch.clamp(xs_t + self.noise, 0, self.ub), num_queries\n",
        "\n",
        "        # If block size is >= 2, then split the image into smaller blocks and reconstruct a batch\n",
        "        if not self.no_hier and self.block_size >= 2:\n",
        "            self.block_size //= 2\n",
        "            self.blocks = self._split_block(upper_left, lower_right, self.block_size)\n",
        "            self.num_blocks = len(self.blocks)\n",
        "            self.batch_size = self.batch_sizeo if self.batch_sizeo > 0 else self.num_blocks\n",
        "            self.curr_order = torch.randperm(self.num_blocks)\n",
        "        # Otherwise, shuffle the order of the batch\n",
        "        else:\n",
        "            self.curr_order = torch.randperm(self.num_blocks)\n",
        "\n",
        "        return torch.clamp(xs_t + self.noise, 0, self.ub), num_queries\n",
        "\n",
        "\n",
        "    def _config(self):\n",
        "        return {\n",
        "            \"name\": self.name,\n",
        "            \"p\": self.p,\n",
        "            \"epsilon\": self.epsilon,\n",
        "            \"lb\": self.lb,\n",
        "            \"ub\": self.ub,\n",
        "            \"max_extra_queries\": \"inf\" if np.isinf(self.max_extra_queries) else self.max_extra_queries,\n",
        "            \"max_loss_queries\": \"inf\" if np.isinf(self.max_loss_queries) else self.max_loss_queries,\n",
        "            \"attack_name\": self.__class__.__name__\n",
        "        }\n"
      ],
      "metadata": {
        "id": "WsHSyBgKi02S",
        "outputId": "13ba862e-5fe1-45af-8a92-c4d8c42ce37b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/attacks/score/parsimonious_attack.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile /content/BlackboxBench/attacks/score/sign_attack.py\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "\n",
        "from attacks.score.score_black_box_attack import ScoreBlackBoxAttack\n",
        "from utils.compute import lp_step, sign, norm\n",
        "\n",
        "\n",
        "class SignAttack(ScoreBlackBoxAttack):\n",
        "    \"\"\"\n",
        "    SignHunter\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, max_loss_queries, epsilon, p, fd_eta, lb, ub, batch_size, name):\n",
        "        \"\"\"\n",
        "        :param max_loss_queries: maximum number of calls allowed to loss oracle per data pt\n",
        "        :param epsilon: radius of lp-ball of perturbation\n",
        "        :param p: specifies lp-norm  of perturbation\n",
        "        :param fd_eta: forward difference step\n",
        "        :param lb: data lower bound\n",
        "        :param ub: data upper bound\n",
        "        \"\"\"\n",
        "        super().__init__(max_extra_queries=np.inf,\n",
        "                         max_loss_queries=max_loss_queries,\n",
        "                         epsilon=epsilon,\n",
        "                         p=p,\n",
        "                         lb=lb,\n",
        "                         ub=ub,\n",
        "                         batch_size= batch_size,\n",
        "                         name=\"Sign\")\n",
        "\n",
        "\n",
        "        self.fd_eta = fd_eta\n",
        "        self.best_est_deriv = None\n",
        "        self.xo_t = None\n",
        "        self.sgn_t = None\n",
        "        self.h = 0\n",
        "        self.i = 0\n",
        "\n",
        "    def _perturb(self, xs_t, loss_fct):\n",
        "        _shape = list(xs_t.shape)\n",
        "        dim = np.prod(_shape[1:])\n",
        "        # additional queries at the start\n",
        "        add_queries = 0\n",
        "        if self.is_new_batch:\n",
        "            self.xo_t = xs_t.clone()\n",
        "            self.h = 0\n",
        "            self.i = 0\n",
        "        if self.i == 0 and self.h == 0:\n",
        "            self.sgn_t = sign(torch.ones(_shape[0], dim))\n",
        "            fxs_t = lp_step(self.xo_t, self.sgn_t.view(_shape), self.epsilon, self.p)\n",
        "            bxs_t = self.xo_t\n",
        "            est_deriv = (loss_fct(fxs_t) - loss_fct(bxs_t)) / self.epsilon\n",
        "            self.best_est_deriv = est_deriv\n",
        "            add_queries = 3  # because of bxs_t and the 2 evaluations in the i=0, h=0, case.\n",
        "        chunk_len = np.ceil(dim / (2 ** self.h)).astype(int)\n",
        "        istart = self.i * chunk_len\n",
        "        iend = min(dim, (self.i + 1) * chunk_len)\n",
        "        self.sgn_t[:, istart:iend] *= - 1.\n",
        "        fxs_t = lp_step(self.xo_t, self.sgn_t.view(_shape), self.epsilon, self.p)\n",
        "        bxs_t = self.xo_t\n",
        "        est_deriv = (loss_fct(fxs_t) - loss_fct(bxs_t)) / self.epsilon\n",
        "\n",
        "        ### sign here\n",
        "        self.sgn_t[[i for i, val in enumerate(est_deriv < self.best_est_deriv) if val], istart: iend] *= -1.\n",
        "\n",
        "\n",
        "        self.best_est_deriv = (est_deriv >= self.best_est_deriv) * est_deriv + (\n",
        "                est_deriv < self.best_est_deriv) * self.best_est_deriv\n",
        "        # perform the step\n",
        "        new_xs = lp_step(self.xo_t, self.sgn_t.view(_shape), self.epsilon, self.p)\n",
        "        # update i and h for next iteration\n",
        "        self.i += 1\n",
        "        if self.i == 2 ** self.h or iend == dim:\n",
        "            self.h += 1\n",
        "            self.i = 0\n",
        "            # if h is exhausted, set xo_t to be xs_t\n",
        "            if self.h == np.ceil(np.log2(dim)).astype(int) + 1:\n",
        "                self.xo_t = xs_t.clone()\n",
        "                self.h = 0\n",
        "                print(\"new change\")\n",
        "\n",
        "#         if self.p == '2':\n",
        "#             import pdb\n",
        "#             pdb.set_trace()\n",
        "\n",
        "        return new_xs, torch.ones(_shape[0]) + add_queries\n",
        "\n",
        "    def get_gs(self):\n",
        "        \"\"\"\n",
        "        return the current estimated of the gradient sign\n",
        "        :return:\n",
        "        \"\"\"\n",
        "        return self.sgn_t\n",
        "\n",
        "    def _config(self):\n",
        "        return {\n",
        "            \"name\": self.name,\n",
        "            \"p\": self.p,\n",
        "            \"epsilon\": self.epsilon,\n",
        "            \"lb\": self.lb,\n",
        "            \"ub\": self.ub,\n",
        "            \"max_extra_queries\": \"inf\" if np.isinf(self.max_extra_queries) else self.max_extra_queries,\n",
        "            \"max_loss_queries\": \"inf\" if np.isinf(self.max_loss_queries) else self.max_loss_queries,\n",
        "            \"fd_eta\": self.fd_eta,\n",
        "            \"attack_name\": self.__class__.__name__\n",
        "        }\n",
        "\n",
        "'''\n",
        "\n",
        "Original License\n",
        "\n",
        "MIT License\n",
        "Copyright (c) 2019 Abdullah Al-Dujaili\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE.\n",
        "\n",
        "'''\n"
      ],
      "metadata": {
        "id": "dNOFzGpcjdUb",
        "outputId": "9fb17389-f88c-41d4-e091-2a8003c95602",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/attacks/score/sign_attack.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile /content/BlackboxBench/attacks/score/simple_attack.py\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "\n",
        "from attacks.score.score_black_box_attack import ScoreBlackBoxAttack\n",
        "from utils.compute import lp_step, sign\n",
        "\n",
        "\n",
        "class SimpleAttack(ScoreBlackBoxAttack):\n",
        "    \"\"\"\n",
        "    Simple Black-Box Attack\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, max_loss_queries, epsilon, p, lb, ub, delta, batch_size, name):\n",
        "        \"\"\"\n",
        "        :param max_loss_queries: maximum number of calls allowed to loss oracle per data pt\n",
        "        :param epsilon: radius of lp-ball of perturbation\n",
        "        :param p: specifies lp-norm  of perturbation\n",
        "        :param fd_eta: forward difference step\n",
        "        :param lb: data lower bound\n",
        "        :param ub: data upper bound\n",
        "        \"\"\"\n",
        "        super().__init__(max_extra_queries=np.inf,\n",
        "                         max_loss_queries=max_loss_queries,\n",
        "                         epsilon=epsilon,\n",
        "                         p=p,\n",
        "                         lb=lb,\n",
        "                         ub=ub,\n",
        "                         batch_size = batch_size,\n",
        "                         name = \"SimBA\")\n",
        "\n",
        "        #self.xo_t = None\n",
        "        self.delta = delta\n",
        "        self.perm = None\n",
        "        self.best_loss = None\n",
        "        self.i = 0\n",
        "\n",
        "    def _perturb(self, xs_t, loss_fct):\n",
        "        _shape = list(xs_t.shape)\n",
        "        dim = np.prod(_shape[1:])\n",
        "        b_sz = _shape[0]\n",
        "        add_queries = 0\n",
        "        if self.is_new_batch:\n",
        "            #self.xo_t = xs_t.clone()\n",
        "            self.i = 0\n",
        "            self.perm = torch.rand(b_sz, dim).argsort(dim=1)\n",
        "        if self.i == 0:\n",
        "            #self.sgn_t = sign(torch.ones(_shape[0], dim))\n",
        "            #fxs_t = lp_step(self.xo_t, self.sgn_t.view(_shape), self.epsilon, self.p)\n",
        "            #bxs_t = self.xo_t\n",
        "            loss = loss_fct(xs_t)\n",
        "            self.best_loss = loss\n",
        "            add_queries = 1\n",
        "        diff = torch.zeros(b_sz, dim)\n",
        "        # % if iterations are greater than dim\n",
        "        idx = self.perm[:, self.i % dim]\n",
        "        diff = diff.scatter(1, idx.unsqueeze(1), 1)\n",
        "        new_xs = xs_t.clone().contiguous().view(b_sz,-1)\n",
        "        # left attempt\n",
        "        left_xs = lp_step(xs_t, diff.view_as(xs_t), self.delta, self.p)\n",
        "        left_loss = loss_fct(left_xs)\n",
        "        replace_flag = torch.tensor((left_loss > self.best_loss).to(torch.float32)).unsqueeze(1)\n",
        "        #print(replace_flag.shape)\n",
        "        self.best_loss = replace_flag.squeeze(1) * left_loss + (1 - replace_flag.squeeze(1)) * self.best_loss\n",
        "        new_xs = replace_flag * left_xs.contiguous().view(b_sz,-1) + (1. - replace_flag) * new_xs\n",
        "        # right attempt\n",
        "        right_xs = lp_step(xs_t, diff.view_as(xs_t), - self.delta, self.p)\n",
        "        right_loss = loss_fct(right_xs)\n",
        "        # replace only those that have greater right loss and was not replaced\n",
        "        # in the left attempt\n",
        "        replace_flag = torch.tensor((right_loss > self.best_loss).to(torch.float32)).unsqueeze(1) * (1 - replace_flag)\n",
        "        #print(replace_flag.shape)\n",
        "        self.best_loss = replace_flag.squeeze(1) * right_loss + (1 - replace_flag.squeeze(1)) * self.best_loss\n",
        "        new_xs = replace_flag * right_xs.contiguous().view(b_sz,-1) + (1 - replace_flag) * new_xs\n",
        "        self.i += 1\n",
        "        # number of queries: add_queries (if first iteration to init best_loss) + left queries + (right queries if any)\n",
        "        num_queries = add_queries + torch.ones(b_sz) + torch.ones(b_sz) * replace_flag.squeeze(1)\n",
        "        return new_xs.view_as(xs_t), num_queries\n",
        "\n",
        "    def _config(self):\n",
        "        return {\n",
        "            \"name\": self.name,\n",
        "            \"p\": self.p,\n",
        "            \"epsilon\": self.epsilon,\n",
        "            \"delta\": self.delta,\n",
        "            \"lb\": self.lb,\n",
        "            \"ub\": self.ub,\n",
        "            \"max_extra_queries\": \"inf\" if np.isinf(self.max_extra_queries) else self.max_extra_queries,\n",
        "            \"max_loss_queries\": \"inf\" if np.isinf(self.max_loss_queries) else self.max_loss_queries,\n",
        "            \"attack_name\": self.__class__.__name__\n",
        "        }\n",
        "\n",
        "'''\n",
        "\n",
        "MIT License\n",
        "\n",
        "Copyright (c) 2019 Abdullah Al-Dujaili\n",
        "\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE.\n",
        "\n",
        "'''\n"
      ],
      "metadata": {
        "id": "YgoDdK_ZjjfA",
        "outputId": "67eb0c4f-0a0b-4d3e-c555-13f3b6246cec",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/attacks/score/simple_attack.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile /content/BlackboxBench/attacks/score/square_attack.py\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "from torch import Tensor as t\n",
        "import torch\n",
        "\n",
        "from attacks.score.score_black_box_attack import ScoreBlackBoxAttack\n",
        "from utils.compute import lp_step\n",
        "\n",
        "\n",
        "class SquareAttack(ScoreBlackBoxAttack):\n",
        "    \"\"\"\n",
        "    Square Attack\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, max_loss_queries, epsilon, p, p_init, lb, ub, batch_size, name):\n",
        "        \"\"\"\n",
        "        :param max_loss_queries: maximum number of calls allowed to loss oracle per data pt\n",
        "        :param epsilon: radius of lp-ball of perturbation\n",
        "        :param p: specifies lp-norm  of perturbation\n",
        "        :param lb: data lower bound\n",
        "        :param ub: data upper bound\n",
        "        \"\"\"\n",
        "        super().__init__(max_extra_queries=np.inf,\n",
        "                         max_loss_queries=max_loss_queries,\n",
        "                         epsilon=epsilon,\n",
        "                         p=p,\n",
        "                         lb=lb,\n",
        "                         ub=ub,\n",
        "                         batch_size= batch_size,\n",
        "                         name = \"Square\")\n",
        "\n",
        "        self.best_loss = None\n",
        "        self.i = 0\n",
        "        self.p_init = p_init\n",
        "\n",
        "    def p_selection(self, p_init, it, n_iters):\n",
        "        \"\"\" Piece-wise constant schedule for p (the fraction of pixels changed on every iteration). \"\"\"\n",
        "        it = int(it / n_iters * 10000)\n",
        "\n",
        "        if 10 < it <= 50:\n",
        "            p = p_init / 2\n",
        "        elif 50 < it <= 200:\n",
        "            p = p_init / 4\n",
        "        elif 200 < it <= 500:\n",
        "            p = p_init / 8\n",
        "        elif 500 < it <= 1000:\n",
        "            p = p_init / 16\n",
        "        elif 1000 < it <= 2000:\n",
        "            p = p_init / 32\n",
        "        elif 2000 < it <= 4000:\n",
        "            p = p_init / 64\n",
        "        elif 4000 < it <= 6000:\n",
        "            p = p_init / 128\n",
        "        elif 6000 < it <= 8000:\n",
        "            p = p_init / 256\n",
        "        elif 8000 < it <= 10000:\n",
        "            p = p_init / 512\n",
        "        else:\n",
        "            p = p_init\n",
        "\n",
        "        return p\n",
        "\n",
        "    def pseudo_gaussian_pert_rectangles(self, x, y):\n",
        "        delta = torch.zeros([x, y])\n",
        "        x_c, y_c = x // 2 + 1, y // 2 + 1\n",
        "\n",
        "        counter2 = [x_c - 1, y_c - 1]\n",
        "        for counter in range(0, max(x_c, y_c)):\n",
        "            delta[max(counter2[0], 0):min(counter2[0] + (2 * counter + 1), x),\n",
        "                max(0, counter2[1]):min(counter2[1] + (2 * counter + 1), y)] += 1.0 / (counter + 1) ** 2\n",
        "\n",
        "            counter2[0] -= 1\n",
        "            counter2[1] -= 1\n",
        "\n",
        "        delta /= torch.sqrt(torch.sum(delta ** 2, dim=1, keepdim=True))\n",
        "        return delta\n",
        "\n",
        "\n",
        "    def meta_pseudo_gaussian_pert(self, s):\n",
        "        delta = torch.zeros([s, s])\n",
        "        n_subsquares = 2\n",
        "        if n_subsquares == 2:\n",
        "            delta[:s // 2] = self.pseudo_gaussian_pert_rectangles(s // 2, s)\n",
        "            delta[s // 2:] = self.pseudo_gaussian_pert_rectangles(s - s // 2, s) * (-1)\n",
        "            delta /= torch.sqrt(torch.sum(delta ** 2, dim=1, keepdim=True))\n",
        "            if np.random.rand(1) > 0.5: delta = torch.transpose(delta, 0, 1)\n",
        "\n",
        "        elif n_subsquares == 4:\n",
        "            delta[:s // 2, :s // 2] = self.pseudo_gaussian_pert_rectangles(s // 2, s // 2) * np.random.choice([-1, 1])\n",
        "            delta[s // 2:, :s // 2] = self.pseudo_gaussian_pert_rectangles(s - s // 2, s // 2) * np.random.choice([-1, 1])\n",
        "            delta[:s // 2, s // 2:] = self.pseudo_gaussian_pert_rectangles(s // 2, s - s // 2) * np.random.choice([-1, 1])\n",
        "            delta[s // 2:, s // 2:] = self.pseudo_gaussian_pert_rectangles(s - s // 2, s - s // 2) * np.random.choice([-1, 1])\n",
        "            delta /= torch.sqrt(torch.sum(delta ** 2, dim=1, keepdim=True))\n",
        "\n",
        "        return delta\n",
        "\n",
        "    def _perturb(self, xs_t, loss_fct):\n",
        "\n",
        "        xs = xs_t.permute(0,3,1,2)\n",
        "        c, h, w = xs.shape[1:]\n",
        "        n_features = c*h*w\n",
        "        n_queries = torch.zeros(xs.shape[0])\n",
        "\n",
        "        if self.p == 'inf':\n",
        "            if self.is_new_batch:\n",
        "                self.x = xs.clone()\n",
        "                init_delta = t(np.random.choice([-self.epsilon, self.epsilon], size=[xs.shape[0], c, 1, w]))\n",
        "                xs = torch.clamp(xs + init_delta, self.lb, self.ub)\n",
        "                self.best_loss = loss_fct(xs.permute(0,2,3,1))\n",
        "                n_queries += torch.ones(xs.shape[0])\n",
        "                self.i = 0\n",
        "\n",
        "            deltas = xs - self.x\n",
        "            p = self.p_selection(self.p_init, self.i, 10000)\n",
        "            for i_img in range(xs.shape[0]):\n",
        "                s = int(round(np.sqrt(p * n_features / c)))\n",
        "                s = min(max(s, 1), h-1)  # at least c x 1 x 1 window is taken and at most c x h-1 x h-1\n",
        "                center_h = np.random.randint(0, h - s)\n",
        "                center_w = np.random.randint(0, w - s)\n",
        "\n",
        "                x_window = self.x[i_img, :, center_h:center_h+s, center_w:center_w+s]\n",
        "                x_best_window = xs[i_img, :, center_h:center_h+s, center_w:center_w+s]\n",
        "                # prevent trying out a delta if it doesn't change x_curr (e.g. an overlapping patch)\n",
        "                while torch.sum(torch.abs(torch.clamp(x_window + deltas[i_img, :, center_h:center_h+s, center_w:center_w+s], self.lb, self.ub) - x_best_window) < 10**-7) == c*s*s:\n",
        "                    deltas[i_img, :, center_h:center_h+s, center_w:center_w+s] = t(np.random.choice([-self.epsilon, self.epsilon], size=[c, 1, 1]))\n",
        "\n",
        "            x_new = torch.clamp(self.x + deltas, self.lb, self.ub).permute(0,2,3,1)\n",
        "\n",
        "        elif self.p == '2':\n",
        "            if self.is_new_batch:\n",
        "                self.x = xs.clone()\n",
        "                delta_init = torch.zeros(xs.shape)\n",
        "                s = h // 5\n",
        "                sp_init = (h - s * 5) // 2\n",
        "                center_h = sp_init + 0\n",
        "                for _ in range(h // s):\n",
        "                    center_w = sp_init + 0\n",
        "                    for _ in range(w // s):\n",
        "                        delta_init[:, :, center_h:center_h + s, center_w:center_w + s] += self.meta_pseudo_gaussian_pert(s).reshape(\n",
        "                            [1, 1, s, s]) * t(np.random.choice([-1, 1], size=[xs.shape[0], c, 1, 1]))\n",
        "                        center_w += s\n",
        "                    center_h += s\n",
        "                xs = torch.clamp(xs + delta_init / torch.sqrt(torch.sum(delta_init ** 2, dim=(1, 2, 3), keepdim=True)) * (self.epsilon), self.lb, self.ub)\n",
        "                self.best_loss = loss_fct(xs.permute(0,2,3,1))\n",
        "                n_queries += torch.ones(xs.shape[0])\n",
        "                self.i = 0\n",
        "\n",
        "            deltas = xs - self.x\n",
        "            p = self.p_selection(self.p_init, self.i, 10000)\n",
        "            s = max(int(round(np.sqrt(p * n_features / c))), 3)\n",
        "            if s % 2 == 0:\n",
        "                s += 1\n",
        "\n",
        "            s2 = s + 0\n",
        "            ### window_1\n",
        "            center_h = np.random.randint(0, h - s)\n",
        "            center_w = np.random.randint(0, w - s)\n",
        "            new_deltas_mask = torch.zeros(xs.shape)\n",
        "            new_deltas_mask[:, :, center_h:center_h + s, center_w:center_w + s] = 1.0\n",
        "\n",
        "            ### window_2\n",
        "            center_h_2 = np.random.randint(0, h - s2)\n",
        "            center_w_2 = np.random.randint(0, w - s2)\n",
        "            new_deltas_mask_2 = torch.zeros(xs.shape)\n",
        "            new_deltas_mask_2[:, :, center_h_2:center_h_2 + s2, center_w_2:center_w_2 + s2] = 1.0\n",
        "\n",
        "            ### compute total norm available\n",
        "            curr_norms_window = torch.sqrt(\n",
        "                torch.sum(((xs - self.x) * new_deltas_mask) ** 2, dim=(2, 3), keepdim=True))\n",
        "            curr_norms_image = torch.sqrt(torch.sum((xs - self.x) ** 2, dim=(1, 2, 3), keepdim=True))\n",
        "            mask_2 = torch.max(new_deltas_mask, new_deltas_mask_2)\n",
        "            norms_windows = torch.sqrt(torch.sum((deltas * mask_2) ** 2, dim=(2, 3), keepdim=True))\n",
        "\n",
        "            ### create the updates\n",
        "            new_deltas = torch.ones([self.x.shape[0], c, s, s])\n",
        "            new_deltas = new_deltas * self.meta_pseudo_gaussian_pert(s).reshape([1, 1, s, s])\n",
        "            new_deltas *= t(np.random.choice([-1, 1], size=[self.x.shape[0], c, 1, 1]))\n",
        "            old_deltas = deltas[:, :, center_h:center_h + s, center_w:center_w + s] / (1e-10 + curr_norms_window)\n",
        "            new_deltas += old_deltas\n",
        "            new_deltas = new_deltas / torch.sqrt(torch.sum(new_deltas ** 2, dim=(2, 3), keepdim=True)) * (\n",
        "                torch.max((self.epsilon) ** 2 - curr_norms_image ** 2, torch.zeros_like(curr_norms_image)) / c + norms_windows ** 2) ** 0.5\n",
        "            deltas[:, :, center_h_2:center_h_2 + s2, center_w_2:center_w_2 + s2] = 0.0  # set window_2 to 0\n",
        "            deltas[:, :, center_h:center_h + s, center_w:center_w + s] = new_deltas + 0  # update window_1\n",
        "\n",
        "            x_new = self.x + deltas / torch.sqrt(torch.sum(deltas ** 2, dim=(1, 2, 3), keepdim=True)) * (self.epsilon)\n",
        "            x_new = torch.clamp(x_new, self.lb, self.ub).permute(0,2,3,1)\n",
        "\n",
        "\n",
        "        new_loss = loss_fct(x_new)\n",
        "        n_queries += torch.ones(xs.shape[0])\n",
        "        idx_improved = new_loss > self.best_loss\n",
        "        self.best_loss = idx_improved * new_loss + ~idx_improved * self.best_loss\n",
        "        xs = xs.permute(0,2,3,1)\n",
        "        idx_improved = torch.reshape(idx_improved, [-1, *[1]*len(x_new.shape[:-1])])\n",
        "        x_new = idx_improved * x_new + ~idx_improved * xs\n",
        "        self.i += 1\n",
        "\n",
        "        return x_new, n_queries\n",
        "\n",
        "    def _config(self):\n",
        "        return {\n",
        "            \"p\": self.p,\n",
        "            \"epsilon\": self.epsilon,\n",
        "            \"lb\": self.lb,\n",
        "            \"ub\": self.ub,\n",
        "            \"max_extra_queries\": \"inf\" if np.isinf(self.max_extra_queries) else self.max_extra_queries,\n",
        "            \"max_loss_queries\": \"inf\" if np.isinf(self.max_loss_queries) else self.max_loss_queries,\n",
        "            \"attack_name\": self.__class__.__name__\n",
        "        }\n",
        "\n",
        "'''\n",
        "\n",
        "Copyright (c) 2019, Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, Matthias Hein\n",
        "All rights reserved.\n",
        "\n",
        "Redistribution and use in source and binary forms, with or without\n",
        "modification, are permitted provided that the following conditions are met:\n",
        "    * Redistributions of source code must retain the above copyright\n",
        "      notice, this list of conditions and the following disclaimer.\n",
        "    * Redistributions in binary form must reproduce the above copyright\n",
        "      notice, this list of conditions and the following disclaimer in the\n",
        "      documentation and/or other materials provided with the distribution.\n",
        "    * Neither the name of the copyright holder nor the\n",
        "      names of its contributors may be used to endorse or promote products\n",
        "      derived from this software without specific prior written permission.\n",
        "\n",
        "THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\n",
        "ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\n",
        "WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n",
        "DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY\n",
        "DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\n",
        "(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n",
        "LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND\n",
        "ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n",
        "(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\n",
        "SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n",
        "\n",
        "'''\n"
      ],
      "metadata": {
        "id": "GsxxdXvvjwqf",
        "outputId": "eaf4afd1-cb14-4956-b59e-687629ccaed9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/attacks/score/square_attack.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile /content/BlackboxBench/attacks/score/zo_sign_sgd_attack.py\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "from torch import Tensor as t\n",
        "import pdb\n",
        "\n",
        "\n",
        "from attacks.score.score_black_box_attack import ScoreBlackBoxAttack\n",
        "from utils.compute import lp_step\n",
        "\n",
        "\n",
        "class ZOSignSGDAttack(ScoreBlackBoxAttack):\n",
        "    \"\"\"\n",
        "    ZOSignSGD Attack\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, max_loss_queries, epsilon, p, fd_eta, lr, q, lb, ub, batch_size, name):\n",
        "        \"\"\"\n",
        "        :param max_loss_queries: maximum number of calls allowed to loss oracle per data pt\n",
        "        :param epsilon: radius of lp-ball of perturbation\n",
        "        :param p: specifies lp-norm  of perturbation\n",
        "        :param fd_eta: forward difference step\n",
        "        :param lr: learning rate of NES step\n",
        "        :param q: number of noise samples per NES step\n",
        "        :param lb: data lower bound\n",
        "        :param ub: data upper bound\n",
        "        \"\"\"\n",
        "        super().__init__(max_extra_queries=np.inf,\n",
        "                         max_loss_queries=max_loss_queries,\n",
        "                         epsilon=epsilon,\n",
        "                         p=p,\n",
        "                         lb=lb,\n",
        "                         ub=ub,\n",
        "                         batch_size=batch_size,\n",
        "                         name = \"zosignsgd\")\n",
        "        self.q = q\n",
        "        self.fd_eta = fd_eta\n",
        "        self.lr = lr\n",
        "\n",
        "    def _perturb(self, xs_t, loss_fct):\n",
        "        #pdb.set_trace()\n",
        "        _shape = list(xs_t.shape)\n",
        "        dim = np.prod(_shape[1:])\n",
        "        num_axes = len(_shape[1:])\n",
        "        gs_t = torch.zeros_like(xs_t)\n",
        "        for _ in range(self.q):\n",
        "            # exp_noise = torch.randn_like(xs_t) / (dim ** 0.5)\n",
        "            exp_noise = torch.randn_like(xs_t)\n",
        "            fxs_t = xs_t + self.fd_eta * exp_noise\n",
        "            bxs_t = xs_t\n",
        "            est_deriv = (loss_fct(fxs_t) - loss_fct(bxs_t)) / self.fd_eta\n",
        "            gs_t += est_deriv.reshape(-1, *[1] * num_axes) * exp_noise\n",
        "        # perform the sign step regardless of the lp-ball constraint\n",
        "        # this is the main difference in the method.\n",
        "        new_xs = lp_step(xs_t, gs_t, self.lr, 'inf')\n",
        "        # the number of queries required for forward difference is q (forward sample) + 1 at xs_t\n",
        "        return new_xs, (self.q + 1) * torch.ones(_shape[0])\n",
        "\n",
        "    def _config(self):\n",
        "        return {\n",
        "            'name': self.name,\n",
        "            \"p\": self.p,\n",
        "            \"epsilon\": self.epsilon,\n",
        "            \"lb\": self.lb,\n",
        "            \"ub\": self.ub,\n",
        "            \"max_extra_queries\": \"inf\" if np.isinf(self.max_extra_queries) else self.max_extra_queries,\n",
        "            \"max_loss_queries\": \"inf\" if np.isinf(self.max_loss_queries) else self.max_loss_queries,\n",
        "            \"lr\": self.lr,\n",
        "            \"q\": self.q,\n",
        "            \"fd_eta\": self.fd_eta,\n",
        "            \"attack_name\": self.__class__.__name__\n",
        "        }\n",
        "\n",
        "'''\n",
        "\n",
        "MIT License\n",
        "\n",
        "Copyright (c) 2019 Abdullah Al-Dujaili\n",
        "\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE.\n",
        "\n",
        "'''\n"
      ],
      "metadata": {
        "id": "sFSzMMBgkCrx",
        "outputId": "f420406e-f713-4dae-cfe6-db0cced37f66",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/attacks/score/zo_sign_sgd_attack.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile /content/BlackboxBench/attacks/decision/decision_black_box_attack.py\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "from torch import Tensor as t\n",
        "\n",
        "import sys\n",
        "\n",
        "class DecisionBlackBoxAttack(object):\n",
        "    def __init__(self, max_queries=np.inf, epsilon=0.5, p='inf', lb=0., ub=1., batch_size=1):\n",
        "        \"\"\"\n",
        "        :param max_queries: max number of calls to model per data point\n",
        "        :param epsilon: perturbation limit according to lp-ball\n",
        "        :param p: norm for the lp-ball constraint\n",
        "        :param lb: minimum value data point can take in any coordinate\n",
        "        :param ub: maximum value data point can take in any coordinate\n",
        "        \"\"\"\n",
        "        assert p in ['inf', '2'], \"L-{} is not supported\".format(p)\n",
        "\n",
        "        self.p = p\n",
        "        self.max_queries = max_queries\n",
        "        self.total_queries = 0\n",
        "        self.total_successes = 0\n",
        "        self.total_failures = 0\n",
        "        self.total_distance = 0\n",
        "        self.sigma = 0\n",
        "        self.EOT = 1\n",
        "        self.lb = lb\n",
        "        self.ub = ub\n",
        "        self.epsilon = epsilon / ub\n",
        "        self.batch_size = batch_size\n",
        "        self.list_loss_queries = torch.zeros(1, self.batch_size)\n",
        "\n",
        "    def result(self):\n",
        "        \"\"\"\n",
        "        returns a summary of the attack results (to be tabulated)\n",
        "        :return:\n",
        "        \"\"\"\n",
        "        list_loss_queries = self.list_loss_queries[1:].view(-1)\n",
        "        mask = list_loss_queries > 0\n",
        "        list_loss_queries = list_loss_queries[mask]\n",
        "        self.total_queries = int(self.total_queries)\n",
        "        self.total_successes = int(self.total_successes)\n",
        "        self.total_failures = int(self.total_failures)\n",
        "        return {\n",
        "            \"total_queries\": self.total_queries,\n",
        "            \"total_successes\": self.total_successes,\n",
        "            \"total_failures\": self.total_failures,\n",
        "            \"average_num_queries\": \"NaN\" if self.total_successes == 0 else self.total_queries / self.total_successes,\n",
        "            \"failure_rate\": \"NaN\" if self.total_successes + self.total_failures == 0 else self.total_failures / (self.total_successes + self.total_failures),\n",
        "            \"median_num_loss_queries\": \"NaN\" if self.total_successes == 0 else torch.median(list_loss_queries).item(),\n",
        "            \"config\": self._config()\n",
        "        }\n",
        "\n",
        "    def _config(self):\n",
        "        \"\"\"\n",
        "        return the attack's parameter configurations as a dict\n",
        "        :return:\n",
        "        \"\"\"\n",
        "        raise NotImplementedError\n",
        "\n",
        "    def distance(self, x_adv, x = None):\n",
        "        if x is None:\n",
        "            diff = x_adv.reshape(x_adv.size(0), -1)\n",
        "        else:\n",
        "            diff = (x_adv - x).reshape(x.size(0), -1)\n",
        "        if self.p == '2':\n",
        "            out = torch.sqrt(torch.sum(diff * diff)).item()\n",
        "        elif self.p == 'inf':\n",
        "            out = torch.sum(torch.max(torch.abs(diff), 1)[0]).item()\n",
        "        return out\n",
        "\n",
        "    def is_adversarial(self, x, y):\n",
        "        '''\n",
        "        check whether the adversarial constrain holds for x\n",
        "        '''\n",
        "        if self.targeted:\n",
        "            return self.predict_label(x) == y\n",
        "        else:\n",
        "            return self.predict_label(x) != y\n",
        "\n",
        "    def predict_label(self, xs):\n",
        "        if type(xs) is torch.Tensor:\n",
        "            x_eval = xs.permute(0,3,1,2)\n",
        "        else:\n",
        "            x_eval = torch.FloatTensor(xs.transpose(0,3,1,2))\n",
        "        x_eval = torch.clamp(x_eval, 0, 1)\n",
        "        # x_eval = x_eval + self.sigma * torch.randn_like(x_eval)\n",
        "        ### Defense ###\n",
        "        x_eval = self.defense_function(self.model,x_eval)\n",
        "        ### Defense ###\n",
        "        if self.ub == 255:\n",
        "            out = self.model(x_eval)\n",
        "        else:\n",
        "            out = self.model(x_eval)\n",
        "        l = out.argmax(dim=1)\n",
        "        return l.detach()\n",
        "\n",
        "    def _perturb(self, xs_t, ys):\n",
        "        raise NotImplementedError\n",
        "\n",
        "    def run(self, xs, ys_t, model, targeted, dset,activate_defense_function):\n",
        "        self.defense_function = activate_defense_function\n",
        "        self.model = model\n",
        "        self.targeted = targeted\n",
        "        self.train_dataset = dset\n",
        "\n",
        "        self.logs = {\n",
        "            'iteration': [0],\n",
        "            'query_count': [0]\n",
        "        }\n",
        "\n",
        "        xs = xs / self.ub\n",
        "        xs_t = t(xs)\n",
        "\n",
        "        # initialize\n",
        "        if self.targeted:\n",
        "            check = self.is_adversarial(xs_t, ys_t)\n",
        "            if torch.any(check):\n",
        "                print('Some original images already belong to the target class!')\n",
        "                return self.logs\n",
        "        else:\n",
        "            check = self.is_adversarial(xs_t, ys_t)\n",
        "            if torch.any(check):\n",
        "                print('Some original images do not belong to the original class!')\n",
        "                return self.logs\n",
        "\n",
        "        adv, q = self._perturb(xs_t, ys_t)\n",
        "        success = self.distance(adv,xs_t) < self.epsilon\n",
        "        self.total_queries += np.sum(q * success)\n",
        "        self.total_successes += np.sum(success)\n",
        "        self.total_failures += ys_t.shape[0] - success\n",
        "        self.list_loss_queries = torch.cat([self.list_loss_queries, torch.zeros(1, self.batch_size)], dim=0)\n",
        "        if type(q) is np.ndarray:\n",
        "            self.list_loss_queries[-1] = t(q * success)\n",
        "        else:\n",
        "            self.list_loss_queries[-1] = q * success\n",
        "        # self.total_distance += self.distance(adv,xs_t)\n",
        "\n",
        "        return self.logs\n",
        "'''\n",
        "\n",
        "MIT License\n",
        "Copyright (c) 2019 Abdullah Al-Dujaili\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE.\n",
        "\n",
        "'''\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c8xLkom0n2m-",
        "outputId": "41398c56-74aa-42bb-e6ce-4bb14c8f2a0b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/attacks/decision/decision_black_box_attack.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bESPqmzVMXhN",
        "outputId": "5d12f3ad-5d9e-4f3a-ed63-1c7bf2efb332"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /usr/local/lib/python3.10/dist-packages/advertorch/attacks/fast_adaptive_boundary.py\n"
          ]
        }
      ],
      "source": [
        "%%writefile /usr/local/lib/python3.10/dist-packages/advertorch/attacks/fast_adaptive_boundary.py\n",
        "\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "from __future__ import unicode_literals\n",
        "\n",
        "import torch\n",
        "import time\n",
        "\n",
        "try:\n",
        "    from torch import flip\n",
        "except ImportError:\n",
        "    from advertorch.utils import torch_flip as flip\n",
        "\n",
        "from advertorch.utils import replicate_input\n",
        "\n",
        "from .base import Attack\n",
        "from .base import LabelMixin\n",
        "\n",
        "DEFAULT_EPS_DICT_BY_NORM = {'Linf': .3, 'L2': 1., 'L1': 5.0}\n",
        "\n",
        "\n",
        "class FABAttack(Attack, LabelMixin):\n",
        "    \"\"\"\n",
        "    Fast Adaptive Boundary Attack (Linf, L2, L1)\n",
        "    https://arxiv.org/abs/1907.02044\n",
        "\n",
        "    :param predict:       forward pass function\n",
        "    :param norm:          Lp-norm to minimize ('Linf', 'L2', 'L1' supported)\n",
        "    :param n_restarts:    number of random restarts\n",
        "    :param n_iter:        number of iterations\n",
        "    :param eps:           epsilon for the random restarts\n",
        "    :param alpha_max:     alpha_max\n",
        "    :param eta:           overshooting\n",
        "    :param beta:          backward step\n",
        "    :param device:        device to use ('cuda' or 'cpu')\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(\n",
        "            self,\n",
        "            predict,\n",
        "            norm='Linf',\n",
        "            n_restarts=1,\n",
        "            n_iter=100,\n",
        "            eps=None,\n",
        "            alpha_max=0.1,\n",
        "            eta=1.05,\n",
        "            beta=0.9,\n",
        "            loss_fn=None,\n",
        "            verbose=False,\n",
        "    ):\n",
        "        \"\"\" FAB-attack implementation in pytorch \"\"\"\n",
        "\n",
        "        super(FABAttack, self).__init__(\n",
        "            predict, loss_fn=None, clip_min=0., clip_max=1.)\n",
        "\n",
        "        self.norm = norm\n",
        "        self.n_restarts = n_restarts\n",
        "        self.n_iter = n_iter\n",
        "        self.eps = eps if eps is not None else DEFAULT_EPS_DICT_BY_NORM[norm]\n",
        "        self.alpha_max = alpha_max\n",
        "        self.eta = eta\n",
        "        self.beta = beta\n",
        "        self.targeted = False\n",
        "        self.verbose = verbose\n",
        "\n",
        "    def check_shape(self, x):\n",
        "        return x if len(x.shape) > 0 else x.unsqueeze(0)\n",
        "\n",
        "    def get_diff_logits_grads_batch(self, imgs, la):\n",
        "        im = imgs.clone().requires_grad_()\n",
        "        with torch.enable_grad():\n",
        "            y = self.predict(im)\n",
        "\n",
        "        g2 = torch.zeros([y.shape[-1], *imgs.size()]).to(self.device)\n",
        "        grad_mask = torch.zeros_like(y)\n",
        "        for counter in range(y.shape[-1]):\n",
        "            zero_gradients(im)\n",
        "            grad_mask[:, counter] = 1.0\n",
        "            y.backward(grad_mask, retain_graph=True)\n",
        "            grad_mask[:, counter] = 0.0\n",
        "            g2[counter] = im.grad.data\n",
        "\n",
        "        g2 = torch.transpose(g2, 0, 1).detach()\n",
        "        y2 = self.predict(imgs).detach()\n",
        "        df = y2 - y2[torch.arange(imgs.shape[0]), la].unsqueeze(1)\n",
        "        dg = g2 - g2[torch.arange(imgs.shape[0]), la].unsqueeze(1)\n",
        "        df[torch.arange(imgs.shape[0]), la] = 1e10\n",
        "\n",
        "        return df, dg\n",
        "\n",
        "    def projection_linf(self, points_to_project, w_hyperplane, b_hyperplane):\n",
        "        t = points_to_project.clone()\n",
        "        w = w_hyperplane.clone()\n",
        "        b = b_hyperplane.clone()\n",
        "\n",
        "        ind2 = ((w * t).sum(1) - b < 0).nonzero().squeeze()\n",
        "        ind2 = self.check_shape(ind2)\n",
        "        w[ind2] *= -1\n",
        "        b[ind2] *= -1\n",
        "\n",
        "        c5 = (w < 0).float()\n",
        "        a = torch.ones(t.shape).to(self.device)\n",
        "        d = (a * c5 - t) * (w != 0).float()\n",
        "        a -= a * (1 - c5)\n",
        "\n",
        "        p = torch.ones(t.shape).to(self.device) * c5 - t * (2 * c5 - 1)\n",
        "        _, indp = torch.sort(p, dim=1)\n",
        "\n",
        "        b = b - (w * t).sum(1)\n",
        "        b0 = (w * d).sum(1)\n",
        "        b1 = b0.clone()\n",
        "\n",
        "        counter = 0\n",
        "        indp2 = flip(indp.unsqueeze(-1), dims=(1, 2)).squeeze()\n",
        "        u = torch.arange(0, w.shape[0])\n",
        "        ws = w[u.unsqueeze(1), indp2]\n",
        "        bs2 = - ws * d[u.unsqueeze(1), indp2]\n",
        "\n",
        "        s = torch.cumsum(ws.abs(), dim=1)\n",
        "        sb = torch.cumsum(bs2, dim=1) + b0.unsqueeze(1)\n",
        "\n",
        "        c = b - b1 > 0\n",
        "        b2 = sb[u, -1] - s[u, -1] * p[u, indp[u, 0]]\n",
        "        c_l = (b - b2 > 0).nonzero().squeeze()\n",
        "        c2 = ((b - b1 > 0) * (b - b2 <= 0)).nonzero().squeeze()\n",
        "        c_l = self.check_shape(c_l)\n",
        "        c2 = self.check_shape(c2)\n",
        "\n",
        "        lb = torch.zeros(c2.shape[0])\n",
        "        ub = torch.ones(c2.shape[0]) * (w.shape[1] - 1)\n",
        "        nitermax = torch.ceil(torch.log2(torch.tensor(w.shape[1]).float()))\n",
        "        counter2 = torch.zeros(lb.shape).long()\n",
        "\n",
        "        while counter < nitermax:\n",
        "            counter4 = torch.floor((lb + ub) / 2)\n",
        "            counter2 = counter4.long()\n",
        "            indcurr = indp[c2, -counter2 - 1]\n",
        "            b2 = sb[c2, counter2] - s[c2, counter2] * p[c2, indcurr]\n",
        "            c = b[c2] - b2 > 0\n",
        "            ind3 = c.nonzero().squeeze()\n",
        "            ind32 = (~c).nonzero().squeeze()\n",
        "            ind3 = self.check_shape(ind3)\n",
        "            ind32 = self.check_shape(ind32)\n",
        "            lb[ind3] = counter4[ind3]\n",
        "            ub[ind32] = counter4[ind32]\n",
        "            counter += 1\n",
        "\n",
        "        lb = lb.long()\n",
        "        counter2 = 0\n",
        "\n",
        "        if c_l.nelement() != 0:\n",
        "            lmbd_opt = (torch.max((b[c_l] - sb[c_l, -1]) / (-s[c_l, -1]),\n",
        "                                  torch.zeros(sb[c_l, -1].shape)\n",
        "                                  .to(self.device))).unsqueeze(-1)\n",
        "            d[c_l] = (2 * a[c_l] - 1) * lmbd_opt\n",
        "\n",
        "        lmbd_opt = (torch.max((b[c2] - sb[c2, lb]) / (-s[c2, lb]),\n",
        "                              torch.zeros(sb[c2, lb].shape)\n",
        "                              .to(self.device))).unsqueeze(-1)\n",
        "        d[c2] = torch.min(lmbd_opt, d[c2]) * c5[c2]\\\n",
        "            + torch.max(-lmbd_opt, d[c2]) * (1 - c5[c2])\n",
        "\n",
        "        return d * (w != 0).float()\n",
        "\n",
        "    def projection_l2(self, points_to_project, w_hyperplane, b_hyperplane):\n",
        "        t = points_to_project.clone()\n",
        "        w = w_hyperplane.clone()\n",
        "        b = b_hyperplane.clone()\n",
        "\n",
        "        c = (w * t).sum(1) - b\n",
        "        ind2 = (c < 0).nonzero().squeeze()\n",
        "        ind2 = self.check_shape(ind2)\n",
        "        w[ind2] *= -1\n",
        "        c[ind2] *= -1\n",
        "\n",
        "        u = torch.arange(0, w.shape[0]).unsqueeze(1)\n",
        "\n",
        "        r = torch.max(t / w, (t - 1) / w)\n",
        "        u2 = torch.ones(r.shape).to(self.device)\n",
        "        r = torch.min(r, 1e12 * u2)\n",
        "        r = torch.max(r, -1e12 * u2)\n",
        "        r[w.abs() < 1e-8] = 1e12\n",
        "        r[r == -1e12] = -r[r == -1e12]\n",
        "        rs, indr = torch.sort(r, dim=1)\n",
        "        rs2 = torch.cat((rs[:, 1:],\n",
        "                         torch.zeros(rs.shape[0], 1).to(self.device)), 1)\n",
        "        rs[rs == 1e12] = 0\n",
        "        rs2[rs2 == 1e12] = 0\n",
        "\n",
        "        w3 = w ** 2\n",
        "        w3s = w3[u, indr]\n",
        "        w5 = w3s.sum(dim=1, keepdim=True)\n",
        "        ws = w5 - torch.cumsum(w3s, dim=1)\n",
        "        d = -(r * w).clone()\n",
        "        d = d * (w.abs() > 1e-8).float()\n",
        "        s = torch.cat(((-w5.squeeze() * rs[:, 0]).unsqueeze(1),\n",
        "                       torch.cumsum((-rs2 + rs) * ws, dim=1) -\n",
        "                       w5 * rs[:, 0].unsqueeze(-1)), 1)\n",
        "\n",
        "        c4 = (s[:, 0] + c < 0)\n",
        "        c3 = ((d * w).sum(dim=1) + c > 0)\n",
        "        c6 = c4.nonzero().squeeze()\n",
        "        c2 = ((1 - c4.float()) * (1 - c3.float())).nonzero().squeeze()\n",
        "        c6 = self.check_shape(c6)\n",
        "        c2 = self.check_shape(c2)\n",
        "\n",
        "        counter = 0\n",
        "        lb = torch.zeros(c2.shape[0])\n",
        "        ub = torch.ones(c2.shape[0]) * (w.shape[1] - 1)\n",
        "        nitermax = torch.ceil(torch.log2(torch.tensor(w.shape[1]).float()))\n",
        "        counter2 = torch.zeros(lb.shape).long()\n",
        "\n",
        "        while counter < nitermax:\n",
        "            counter4 = torch.floor((lb + ub) / 2)\n",
        "            counter2 = counter4.long()\n",
        "            c3 = s[c2, counter2] + c[c2] > 0\n",
        "            ind3 = c3.nonzero().squeeze()\n",
        "            ind32 = (~c3).nonzero().squeeze()\n",
        "            ind3 = self.check_shape(ind3)\n",
        "            ind32 = self.check_shape(ind32)\n",
        "            lb[ind3] = counter4[ind3]\n",
        "            ub[ind32] = counter4[ind32]\n",
        "            counter += 1\n",
        "\n",
        "        lb = lb.long()\n",
        "        alpha = torch.zeros([1])\n",
        "\n",
        "        if c6.nelement() != 0:\n",
        "            alpha = c[c6] / w5[c6].squeeze(-1)\n",
        "            d[c6] = -alpha.unsqueeze(-1) * w[c6]\n",
        "\n",
        "        if c2.nelement() != 0:\n",
        "            alpha = (s[c2, lb] + c[c2]) / ws[c2, lb] + rs[c2, lb]\n",
        "            if torch.sum(ws[c2, lb] == 0) > 0:\n",
        "                ind = (ws[c2, lb] == 0).nonzero().squeeze().long()\n",
        "                ind = self.check_shape(ind)\n",
        "                alpha[ind] = 0\n",
        "            c5 = (alpha.unsqueeze(-1) > r[c2]).float()\n",
        "            d[c2] = d[c2] * c5 - alpha.unsqueeze(-1) * w[c2] * (1 - c5)\n",
        "\n",
        "        return d * (w.abs() > 1e-8).float()\n",
        "\n",
        "    def projection_l1(self, points_to_project, w_hyperplane, b_hyperplane):\n",
        "        t = points_to_project.clone()\n",
        "        w = w_hyperplane.clone()\n",
        "        b = b_hyperplane.clone()\n",
        "\n",
        "        c = (w * t).sum(1) - b\n",
        "        ind2 = (c < 0).nonzero().squeeze()\n",
        "        ind2 = self.check_shape(ind2)\n",
        "        w[ind2] *= -1\n",
        "        c[ind2] *= -1\n",
        "\n",
        "        r = torch.max(1 / w, -1 / w)\n",
        "        r = torch.min(r, 1e12 * torch.ones(r.shape).to(self.device))\n",
        "        rs, indr = torch.sort(r, dim=1)\n",
        "        _, indr_rev = torch.sort(indr)\n",
        "\n",
        "        u = torch.arange(0, w.shape[0]).unsqueeze(1)\n",
        "        u2 = torch.arange(0, w.shape[1]).repeat(w.shape[0], 1)\n",
        "        c6 = (w < 0).float()\n",
        "        d = (-t + c6) * (w != 0).float()\n",
        "        d2 = torch.min(-w * t, w * (1 - t))\n",
        "        ds = d2[u, indr]\n",
        "        ds2 = torch.cat((c.unsqueeze(-1), ds), 1)\n",
        "        s = torch.cumsum(ds2, dim=1)\n",
        "\n",
        "        c4 = s[:, -1] < 0\n",
        "        c2 = c4.nonzero().squeeze(-1)\n",
        "        c2 = self.check_shape(c2)\n",
        "\n",
        "        counter = 0\n",
        "        lb = torch.zeros(c2.shape[0])\n",
        "        ub = torch.ones(c2.shape[0]) * (s.shape[1])\n",
        "        nitermax = torch.ceil(torch.log2(torch.tensor(s.shape[1]).float()))\n",
        "        counter2 = torch.zeros(lb.shape).long()\n",
        "\n",
        "        while counter < nitermax:\n",
        "            counter4 = torch.floor((lb + ub) / 2)\n",
        "            counter2 = counter4.long()\n",
        "            c3 = s[c2, counter2] > 0\n",
        "            ind3 = c3.nonzero().squeeze()\n",
        "            ind32 = (~c3).nonzero().squeeze()\n",
        "            ind3 = self.check_shape(ind3)\n",
        "            ind32 = self.check_shape(ind32)\n",
        "            lb[ind3] = counter4[ind3]\n",
        "            ub[ind32] = counter4[ind32]\n",
        "            counter += 1\n",
        "\n",
        "        lb2 = lb.long()\n",
        "\n",
        "        if c2.nelement() != 0:\n",
        "            alpha = -s[c2, lb2] / w[c2, indr[c2, lb2]]\n",
        "            c5 = u2[c2].float() < lb.unsqueeze(-1).float()\n",
        "            u3 = c5[u[:c5.shape[0]], indr_rev[c2]]\n",
        "            d[c2] = d[c2] * u3.float().to(self.device)\n",
        "            d[c2, indr[c2, lb2]] = alpha\n",
        "\n",
        "        return d * (w.abs() > 1e-8).float()\n",
        "\n",
        "    def perturb(self, x, y=None):\n",
        "        \"\"\"\n",
        "        :param x:    clean images\n",
        "        :param y:    clean labels, if None we use the predicted labels\n",
        "        \"\"\"\n",
        "\n",
        "        self.device = x.device\n",
        "        self.orig_dim = list(x.shape[1:])\n",
        "        self.ndims = len(self.orig_dim)\n",
        "\n",
        "        x = x.detach().clone().float().to(self.device)\n",
        "        # assert next(self.predict.parameters()).device == x.device\n",
        "\n",
        "        y_pred = self._get_predicted_label(x)\n",
        "        if y is None:\n",
        "            y = y_pred.detach().clone().long().to(self.device)\n",
        "        else:\n",
        "            y = y.detach().clone().long().to(self.device)\n",
        "        pred = y_pred == y\n",
        "        corr_classified = pred.float().sum()\n",
        "        if self.verbose:\n",
        "            print('Clean accuracy: {:.2%}'.format(pred.float().mean()))\n",
        "        if pred.sum() == 0:\n",
        "            return x\n",
        "        pred = self.check_shape(pred.nonzero().squeeze())\n",
        "\n",
        "        startt = time.time()\n",
        "        # runs the attack only on correctly classified points\n",
        "        im2 = replicate_input(x[pred])\n",
        "        la2 = replicate_input(y[pred])\n",
        "        if len(im2.shape) == self.ndims:\n",
        "            im2 = im2.unsqueeze(0)\n",
        "        bs = im2.shape[0]\n",
        "        u1 = torch.arange(bs)\n",
        "        adv = im2.clone()\n",
        "        adv_c = x.clone()\n",
        "        res2 = 1e10 * torch.ones([bs]).to(self.device)\n",
        "        res_c = torch.zeros([x.shape[0]]).to(self.device)\n",
        "        x1 = im2.clone()\n",
        "        x0 = im2.clone().reshape([bs, -1])\n",
        "        counter_restarts = 0\n",
        "\n",
        "        while counter_restarts < self.n_restarts:\n",
        "            if counter_restarts > 0:\n",
        "                if self.norm == 'Linf':\n",
        "                    t = 2 * torch.rand(x1.shape).to(self.device) - 1\n",
        "                    x1 = im2 + (\n",
        "                        torch.min(\n",
        "                            res2,\n",
        "                            self.eps * torch.ones(res2.shape).to(self.device)\n",
        "                        ).reshape([-1, *([1] * self.ndims)])\n",
        "                    ) * t / (t.reshape([t.shape[0], -1]).abs()\n",
        "                             .max(dim=1, keepdim=True)[0]\n",
        "                             .reshape([-1, *([1] * self.ndims)])) * .5\n",
        "                elif self.norm == 'L2':\n",
        "                    t = torch.randn(x1.shape).to(self.device)\n",
        "                    x1 = im2 + (\n",
        "                        torch.min(\n",
        "                            res2,\n",
        "                            self.eps * torch.ones(res2.shape).to(self.device)\n",
        "                        ).reshape([-1, *([1] * self.ndims)])\n",
        "                    ) * t / ((t ** 2)\n",
        "                             .view(t.shape[0], -1)\n",
        "                             .sum(dim=-1)\n",
        "                             .sqrt()\n",
        "                             .view(t.shape[0], *([1] * self.ndims))) * .5\n",
        "                elif self.norm == 'L1':\n",
        "                    t = torch.randn(x1.shape).to(self.device)\n",
        "                    x1 = im2 + (torch.min(\n",
        "                        res2,\n",
        "                        self.eps * torch.ones(res2.shape).to(self.device)\n",
        "                    ).reshape([-1, *([1] * self.ndims)])\n",
        "                    ) * t / (t.abs().view(t.shape[0], -1)\n",
        "                             .sum(dim=-1)\n",
        "                             .view(t.shape[0], *([1] * self.ndims))) / 2\n",
        "\n",
        "                x1 = x1.clamp(0.0, 1.0)\n",
        "\n",
        "            counter_iter = 0\n",
        "            while counter_iter < self.n_iter:\n",
        "                with torch.no_grad():\n",
        "                    df, dg = self.get_diff_logits_grads_batch(x1, la2)\n",
        "                    if self.norm == 'Linf':\n",
        "                        dist1 = df.abs() / (1e-12 +\n",
        "                                            dg.abs()\n",
        "                                            .view(dg.shape[0], dg.shape[1], -1)\n",
        "                                            .sum(dim=-1))\n",
        "                    elif self.norm == 'L2':\n",
        "                        dist1 = df.abs() / (1e-12 + (dg ** 2)\n",
        "                                            .view(dg.shape[0], dg.shape[1], -1)\n",
        "                                            .sum(dim=-1).sqrt())\n",
        "                    elif self.norm == 'L1':\n",
        "                        dist1 = df.abs() / (1e-12 + dg.abs().reshape(\n",
        "                            [df.shape[0], df.shape[1], -1]).max(dim=2)[0])\n",
        "                    else:\n",
        "                        raise ValueError('norm not supported')\n",
        "                    ind = dist1.min(dim=1)[1]\n",
        "                    dg2 = dg[u1, ind]\n",
        "                    b = (- df[u1, ind] +\n",
        "                         (dg2 * x1).view(x1.shape[0], -1).sum(dim=-1))\n",
        "                    w = dg2.reshape([bs, -1])\n",
        "\n",
        "                    if self.norm == 'Linf':\n",
        "                        d3 = self.projection_linf(\n",
        "                            torch.cat((x1.reshape([bs, -1]), x0), 0),\n",
        "                            torch.cat((w, w), 0),\n",
        "                            torch.cat((b, b), 0))\n",
        "                    elif self.norm == 'L2':\n",
        "                        d3 = self.projection_l2(\n",
        "                            torch.cat((x1.reshape([bs, -1]), x0), 0),\n",
        "                            torch.cat((w, w), 0),\n",
        "                            torch.cat((b, b), 0))\n",
        "                    elif self.norm == 'L1':\n",
        "                        d3 = self.projection_l1(\n",
        "                            torch.cat((x1.reshape([bs, -1]), x0), 0),\n",
        "                            torch.cat((w, w), 0),\n",
        "                            torch.cat((b, b), 0))\n",
        "                    d1 = torch.reshape(d3[:bs], x1.shape)\n",
        "                    d2 = torch.reshape(d3[-bs:], x1.shape)\n",
        "                    if self.norm == 'Linf':\n",
        "                        a0 = d3.abs().max(dim=1, keepdim=True)[0]\\\n",
        "                            .view(-1, *([1] * self.ndims))\n",
        "                    elif self.norm == 'L2':\n",
        "                        a0 = (d3 ** 2).sum(dim=1, keepdim=True).sqrt()\\\n",
        "                            .view(-1, *([1] * self.ndims))\n",
        "                    elif self.norm == 'L1':\n",
        "                        a0 = d3.abs().sum(dim=1, keepdim=True)\\\n",
        "                            .view(-1, *([1] * self.ndims))\n",
        "                    a0 = torch.max(a0, 1e-8 * torch.ones(\n",
        "                        a0.shape).to(self.device))\n",
        "                    a1 = a0[:bs]\n",
        "                    a2 = a0[-bs:]\n",
        "                    alpha = torch.min(torch.max(a1 / (a1 + a2),\n",
        "                                                torch.zeros(a1.shape)\n",
        "                                                .to(self.device))[0],\n",
        "                                      self.alpha_max * torch.ones(a1.shape)\n",
        "                                      .to(self.device))\n",
        "                    x1 = ((x1 + self.eta * d1) * (1 - alpha) +\n",
        "                          (im2 + d2 * self.eta) * alpha).clamp(0.0, 1.0)\n",
        "\n",
        "                    is_adv = self._get_predicted_label(x1) != la2\n",
        "\n",
        "                    if is_adv.sum() > 0:\n",
        "                        ind_adv = is_adv.nonzero().squeeze()\n",
        "                        ind_adv = self.check_shape(ind_adv)\n",
        "                        if self.norm == 'Linf':\n",
        "                            t = (x1[ind_adv] - im2[ind_adv]).reshape(\n",
        "                                [ind_adv.shape[0], -1]).abs().max(dim=1)[0]\n",
        "                        elif self.norm == 'L2':\n",
        "                            t = ((x1[ind_adv] - im2[ind_adv]) ** 2)\\\n",
        "                                .view(ind_adv.shape[0], -1).sum(dim=-1).sqrt()\n",
        "                        elif self.norm == 'L1':\n",
        "                            t = (x1[ind_adv] - im2[ind_adv])\\\n",
        "                                .abs().view(ind_adv.shape[0], -1).sum(dim=-1)\n",
        "                        adv[ind_adv] = x1[ind_adv] * (t < res2[ind_adv]).\\\n",
        "                            float().reshape([-1, *([1] * self.ndims)]) \\\n",
        "                            + adv[ind_adv]\\\n",
        "                            * (t >= res2[ind_adv]).float().reshape(\n",
        "                            [-1, *([1] * self.ndims)])\n",
        "                        res2[ind_adv] = t * (t < res2[ind_adv]).float()\\\n",
        "                            + res2[ind_adv] * (t >= res2[ind_adv]).float()\n",
        "                        x1[ind_adv] = im2[ind_adv] + (\n",
        "                            x1[ind_adv] - im2[ind_adv]) * self.beta\n",
        "\n",
        "                    counter_iter += 1\n",
        "\n",
        "            counter_restarts += 1\n",
        "\n",
        "        ind_succ = res2 < 1e10\n",
        "        if self.verbose:\n",
        "            print('success rate: {:.0f}/{:.0f}'\n",
        "                  .format(ind_succ.float().sum(), corr_classified) +\n",
        "                  ' (on correctly classified points) in {:.1f} s'\n",
        "                  .format(time.time() - startt))\n",
        "\n",
        "        res_c[pred] = res2 * ind_succ.float() + 1e10 * (1 - ind_succ.float())\n",
        "        ind_succ = self.check_shape(ind_succ.nonzero().squeeze())\n",
        "        adv_c[pred[ind_succ]] = adv[ind_succ].clone()\n",
        "\n",
        "        return adv_c\n",
        "\n",
        "\n",
        "class LinfFABAttack(FABAttack):\n",
        "    \"\"\"\n",
        "    Linf - Fast Adaptive Boundary Attack\n",
        "    https://arxiv.org/abs/1907.02044\n",
        "\n",
        "    :param predict:       forward pass function\n",
        "    :param n_restarts:    number of random restarts\n",
        "    :param n_iter:        number of iterations\n",
        "    :param eps:           epsilon for the random restarts\n",
        "    :param alpha_max:     alpha_max\n",
        "    :param eta:           overshooting\n",
        "    :param beta:          backward step\n",
        "    :param device:        device to use ('cuda' or 'cpu')\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(\n",
        "            self,\n",
        "            predict,\n",
        "            n_restarts=1,\n",
        "            n_iter=100,\n",
        "            eps=None,\n",
        "            alpha_max=0.1,\n",
        "            eta=1.05,\n",
        "            beta=0.9,\n",
        "            loss_fn=None,\n",
        "            verbose=False,\n",
        "    ):\n",
        "        norm = 'Linf'\n",
        "        super(LinfFABAttack, self).__init__(\n",
        "            predict=predict, norm=norm, n_restarts=n_restarts,\n",
        "            n_iter=n_iter, eps=eps, alpha_max=alpha_max, eta=eta, beta=beta,\n",
        "            loss_fn=loss_fn, verbose=verbose)\n",
        "\n",
        "\n",
        "class L2FABAttack(FABAttack):\n",
        "    \"\"\"\n",
        "    L2 - Fast Adaptive Boundary Attack\n",
        "    https://arxiv.org/abs/1907.02044\n",
        "\n",
        "    :param predict:       forward pass function\n",
        "    :param n_restarts:    number of random restarts\n",
        "    :param n_iter:        number of iterations\n",
        "    :param eps:           epsilon for the random restarts\n",
        "    :param alpha_max:     alpha_max\n",
        "    :param eta:           overshooting\n",
        "    :param beta:          backward step\n",
        "    :param device:        device to use ('cuda' or 'cpu')\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(\n",
        "            self,\n",
        "            predict,\n",
        "            n_restarts=1,\n",
        "            n_iter=100,\n",
        "            eps=None,\n",
        "            alpha_max=0.1,\n",
        "            eta=1.05,\n",
        "            beta=0.9,\n",
        "            loss_fn=None,\n",
        "            verbose=False,\n",
        "    ):\n",
        "        norm = 'L2'\n",
        "        super(L2FABAttack, self).__init__(\n",
        "            predict=predict, norm=norm, n_restarts=n_restarts,\n",
        "            n_iter=n_iter, eps=eps, alpha_max=alpha_max, eta=eta, beta=beta,\n",
        "            loss_fn=loss_fn, verbose=verbose)\n",
        "\n",
        "\n",
        "class L1FABAttack(FABAttack):\n",
        "    \"\"\"\n",
        "    L1 - Fast Adaptive Boundary Attack\n",
        "    https://arxiv.org/abs/1907.02044\n",
        "\n",
        "    :param predict:       forward pass function\n",
        "    :param n_restarts:    number of random restarts\n",
        "    :param n_iter:        number of iterations\n",
        "    :param eps:           epsilon for the random restarts\n",
        "    :param alpha_max:     alpha_max\n",
        "    :param eta:           overshooting\n",
        "    :param beta:          backward step\n",
        "    :param device:        device to use ('cuda' or 'cpu')\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(\n",
        "            self,\n",
        "            predict,\n",
        "            n_restarts=1,\n",
        "            n_iter=100,\n",
        "            eps=None,\n",
        "            alpha_max=0.1,\n",
        "            eta=1.05,\n",
        "            beta=0.9,\n",
        "            loss_fn=None,\n",
        "            verbose=False,\n",
        "    ):\n",
        "        norm = 'L1'\n",
        "        super(L1FABAttack, self).__init__(\n",
        "            predict=predict, norm=norm, n_restarts=n_restarts,\n",
        "            n_iter=n_iter, eps=eps, alpha_max=alpha_max, eta=eta, beta=beta,\n",
        "            loss_fn=loss_fn, verbose=verbose)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Bp_ZbpRb0BIN",
        "outputId": "c5ed2f3f-f0da-4630-9e61-68d10121e932"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/datasets/dataset.py\n"
          ]
        }
      ],
      "source": [
        "%%writefile /content/BlackboxBench/datasets/dataset.py\n",
        "\n",
        "\"\"\"\n",
        "A wrapper for datasets\n",
        "    mnist, cifar10, imagenet\n",
        "\"\"\"\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "# from tensorflow.examples.tutorials.mnist import input_data\n",
        "\n",
        "import datasets.cifar10 as cifar10_input\n",
        "from datasets.imagenet import ImagenetValidData\n",
        "from utils.misc import data_path_join\n",
        "import torch\n",
        "import numpy as np\n",
        "from torchvision import transforms\n",
        "from torchvision.datasets import ImageFolder\n",
        "from torch.utils.data import DataLoader\n",
        "\n",
        "def load_imagenet(n_ex = 1000, size=224):\n",
        "    IMAGENET_SL = size\n",
        "    IMAGENET_PATH = data_path_join('imagenet/Sample_1000')\n",
        "    imagenet = ImageFolder(IMAGENET_PATH,\n",
        "                           transforms.Compose([\n",
        "                               transforms.Resize(IMAGENET_SL),\n",
        "                               transforms.CenterCrop(IMAGENET_SL),\n",
        "                               transforms.ToTensor()\n",
        "                           ]))\n",
        "    torch.manual_seed(0)\n",
        "\n",
        "    imagenet_loader = DataLoader(imagenet, batch_size=n_ex, shuffle=True, num_workers=1)\n",
        "    x_test, y_test = next(iter(imagenet_loader))\n",
        "    return np.array(x_test, dtype=np.float32), np.array(y_test)\n",
        "\n",
        "class Dataset(object):\n",
        "    def __init__(self, name, config):\n",
        "        \"\"\"\n",
        "        :param name: dataset name\n",
        "        :param config: dictionary whose keys are dependent on the dataset created\n",
        "         (see source code below)\n",
        "        \"\"\"\n",
        "        assert name in ['mnist', 'cifar10', 'cifar10aug', 'imagenet', 'imagenet_sub'], \"Invalid dataset\"\n",
        "        self.name = name\n",
        "        model_name = config['modeln']\n",
        "        # config = config['dset_config']\n",
        "\n",
        "        if self.name == 'cifar10':\n",
        "            data_path = data_path_join('/content/BlackboxBench/data')\n",
        "            self.data = cifar10_input.CIFAR10Data(data_path)\n",
        "        elif self.name == 'cifar10aug':\n",
        "            data_path = data_path_join('cifar10_data')\n",
        "            # raw_cifar = cifar10_input.CIFAR10Data(data_path)\n",
        "            # sess = config['sess']\n",
        "            # model = config['model']\n",
        "            # self.data = cifar10_input.AugmentedCIFAR10Data(raw_cifar, sess, model)\n",
        "        elif self.name == 'mnist':\n",
        "            self.data = input_data.read_data_sets(data_path_join('mnist_data'), one_hot=False)\n",
        "        elif self.name == 'imagenet':\n",
        "            data_path = data_path_join('imagenet_data')\n",
        "            self.data = ImagenetValidData(data_dir=data_path)\n",
        "        elif self.name == 'imagenet_sub':\n",
        "            if model_name == 'Inception':\n",
        "                self.x_test, self.y_test = load_imagenet(n_ex=1000, size=299)\n",
        "            else:\n",
        "                self.x_test, self.y_test = load_imagenet()\n",
        "            self.x_test = self.x_test.transpose(0, 2, 3, 1)\n",
        "\n",
        "    def get_next_train_batch(self, batch_size):\n",
        "        \"\"\"\n",
        "        Returns a tuple of (x_batch, y_batch)\n",
        "        \"\"\"\n",
        "        if self.name in ['cifar10', 'cifar10aug']:\n",
        "            return self.data.train_data.get_next_batch(batch_size, multiple_passes=True)\n",
        "        elif self.name == 'mnist':\n",
        "            return self.data.train.next_batch(batch_size)\n",
        "        elif self.name in ['imagenet', 'imagenet_sub']:\n",
        "            raise Exception(\n",
        "                'No training data for imagenet is needed (provided), the models are assumed to be pretrained!')\n",
        "\n",
        "    def get_eval_data(self, bstart, bend):\n",
        "        \"\"\"\n",
        "        :param bstart: batch start index\n",
        "        :param bend: batch end index\n",
        "        \"\"\"\n",
        "        if self.name in ['cifar10', 'cifar10aug']:\n",
        "            return self.data.eval_data.xs[bstart:bend, :], \\\n",
        "                   self.data.eval_data.ys[bstart:bend]\n",
        "        elif self.name == 'mnist':\n",
        "            return self.data.test.images[bstart:bend, :], \\\n",
        "                   self.data.test.labels[bstart:bend]\n",
        "        elif self.name == 'imagenet':\n",
        "            return self.data.get_eval_data(bstart, bend)\n",
        "        elif self.name == 'imagenet_sub':\n",
        "            return self.x_test[bstart:bend, :], \\\n",
        "                    self.y_test[bstart:bend]\n",
        "\n",
        "\n",
        "    @property\n",
        "    def min_value(self):\n",
        "        if self.name in ['cifar10', 'cifar10aug', 'mnist', 'imagenet', 'imagenet_sub']:\n",
        "            return 0.\n",
        "\n",
        "    @property\n",
        "    def max_value(self):\n",
        "        if self.name in ['cifar10', 'cifar10aug']:\n",
        "            return 255.\n",
        "        if self.name in ['mnist', 'imagenet', 'imagenet_sub']:\n",
        "            return 1.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3jqekQm8qI3r",
        "outputId": "3999d890-08e8-4340-9855-bf289acd4256"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/utils/model_loader.py\n"
          ]
        }
      ],
      "source": [
        "%%writefile /content/BlackboxBench/utils/model_loader.py\n",
        "\n",
        "from __future__ import division\n",
        "from __future__ import absolute_import\n",
        "\n",
        "import torch\n",
        "import os\n",
        "import json\n",
        "import torch.nn as nn\n",
        "from utils.misc import data_path_join\n",
        "# from utils_.utils import RecorderMeter\n",
        "# from utils_ import utils\n",
        "from models import resnet, wrn, vgg\n",
        "# , wrn_sap, wrn_adv_sap, model_zoo, vgg_rse, pni_model\n",
        "# from models.Resnet import resnet152_denoise, resnet101_denoise\n",
        "import numpy as np\n",
        "from torchvision import models, datasets\n",
        "\n",
        "def load_torch_models(model_name):\n",
        "\n",
        "    if torch.cuda.is_available():\n",
        "        device = torch.device('cuda')\n",
        "    else:\n",
        "        device = torch.device('cpu')\n",
        "\n",
        "    if model_name == \"resnet20\":\n",
        "        # TRAINED_MODEL_PATH = data_path_join('pretrained_models/pyramidnet_basic_110_84/00/')\n",
        "        # filename = 'model_best_state.pth'\n",
        "        # with open(os.path.join(TRAINED_MODEL_PATH, 'config.json')) as fr:\n",
        "        pretrained_model = torch.hub.load(\"chenyaofo/pytorch-cifar-models\", \"cifar10_resnet20\", pretrained=True)\n",
        "        # pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename))['state_dict'])\n",
        "    elif model_name == 'resnet50-AT':\n",
        "\n",
        "        from robustness import cifar_models\n",
        "        checkpoint = torch.load('cifar_linf_8.pt')\n",
        "        # Extract the model state dictionary from the OrderedDict (replace 'model' with the correct key)\n",
        "        model_state_dict = checkpoint['state_dict']\n",
        "        # Create an instance of the ResNet50 model\n",
        "        pretrained_model = cifar_models.resnet50()\n",
        "        mapped_state_dict = {k.replace('module.attacker.model.', ''): v for k, v in model_state_dict.items()}\n",
        "        # # Filter out keys that are not relevant to ResNet50\n",
        "        filtered_state_dict = {k: v for k, v in mapped_state_dict.items() if k in pretrained_model.state_dict()}\n",
        "        # Load the filtered state dictionary\n",
        "        pretrained_model.load_state_dict(filtered_state_dict)\n",
        "    elif model_name == 'vgg16':\n",
        "        # TRAINED_MODEL_PATH = data_path_join('pretrained_models/resnet_adv_4/cifar-10_linf/')\n",
        "        # filename = 'model_best_state.pth'\n",
        "        # with open(os.path.join(TRAINED_MODEL_PATH, 'config.json')) as fr:\n",
        "        pretrained_model = torch.hub.load(\"chenyaofo/pytorch-cifar-models\", \"cifar10_vgg16_bn\", pretrained=True)\n",
        "        # pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename)))\n",
        "    elif model_name == 'resnet':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/resnet_basic_110/00/')\n",
        "        filename = 'model_best_state.pth'\n",
        "        with open(os.path.join(TRAINED_MODEL_PATH, 'config.json')) as fr:\n",
        "            pretrained_model = resnet.Network(json.load(fr)['model_config'])\n",
        "            pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename))['state_dict'])\n",
        "    elif model_name == 'wrn':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/wrn_28_10/00/')\n",
        "        filename = 'model_best_state.pth'\n",
        "        with open(os.path.join(TRAINED_MODEL_PATH, 'config.json')) as fr:\n",
        "            pretrained_model = wrn.Network(json.load(fr)['model_config'])\n",
        "            pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename))['state_dict'])\n",
        "\n",
        "    elif model_name == 'dense':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/densenet_BC_100_12/00/')\n",
        "        filename = 'model_best_state.pth'\n",
        "        with open(os.path.join(TRAINED_MODEL_PATH, 'config.json')) as fr:\n",
        "            pretrained_model = densenet.Network(json.load(fr)['model_config'])\n",
        "            pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename))['state_dict'])\n",
        "\n",
        "\n",
        "    elif model_name == 'vgg_rse':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/rse_model/')\n",
        "        filename = 'cifar10_vgg_rse_005.pth'\n",
        "        # with open(os.path.join(TRAINED_MODEL_PATH, 'config.json')) as fr:\n",
        "            # pretrained_model = vgg.Network(json.load(fr)['model_config'])\n",
        "        noise_init = 0\n",
        "        noise_inner = 0\n",
        "        pretrained_model = nn.DataParallel(vgg_rse.VGG_RSE('VGG16', 10, noise_init, noise_inner, img_width=32), device_ids=range(1))\n",
        "        pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename)))\n",
        "\n",
        "    elif model_name == 'rse':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/rse_model/')\n",
        "        filename = 'cifar10_vgg_rse.pth'\n",
        "        # with open(os.path.join(TRAINED_MODEL_PATH, 'config.json')) as fr:\n",
        "            # pretrained_model = vgg.Network(json.load(fr)['model_config'])\n",
        "        noise_init = 0.2\n",
        "        noise_inner = 0.1\n",
        "        pretrained_model = nn.DataParallel(vgg_rse.VGG_RSE('VGG16', 10, noise_init, noise_inner, img_width=32), device_ids=range(1))\n",
        "        pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename)))\n",
        "\n",
        "\n",
        "\n",
        "    elif model_name == 'vgg':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/vgg_15_BN_64/00/')\n",
        "        filename = 'model_best_state.pth'\n",
        "        with open(os.path.join(TRAINED_MODEL_PATH, 'config.json')) as fr:\n",
        "            pretrained_model = vgg.Network(json.load(fr)['model_config'])\n",
        "            pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename),map_location=device)['state_dict'])\n",
        "\n",
        "\n",
        "    elif model_name == 'wrn_adv':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/wrn_adv/')\n",
        "        filename = 'adv_wrn16_linf.pth'\n",
        "        pretrained_model = wrn_adv.create_model(name='wide', num_classes = 10)\n",
        "        if hasattr(pretrained_model, 'module'):\n",
        "            pretrained_model = pretrained_model.module\n",
        "        pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename),map_location=device)['state_dict'])\n",
        "\n",
        "    elif model_name == 'wrn28':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/wrn_adv/')\n",
        "        filename = 'cifar_wrn_28.pth'\n",
        "        pretrained_model = wrn.WideNet()\n",
        "        pretrained_model = torch.nn.DataParallel(pretrained_model)\n",
        "        checkpoint = torch.load(os.path.join(TRAINED_MODEL_PATH, filename))\n",
        "        # if hasattr(pretrained_model, 'module'):\n",
        "        #     pretrained_model = pretrained_model.module\n",
        "        pretrained_model.load_state_dict(checkpoint['net'])\n",
        "\n",
        "    # elif model_name == 'wrn28_sap':\n",
        "    #     TRAINED_MODEL_PATH = data_path_join('pretrained_models/wrn_adv/')\n",
        "    #     filename = 'cifar_wrn_28.pth'\n",
        "    #     pretrained_model = wrn_sap.WideNet()\n",
        "    #     pretrained_model = torch.nn.DataParallel(pretrained_model)\n",
        "    #     checkpoint = torch.load(os.path.join(TRAINED_MODEL_PATH, filename))\n",
        "    #     # if hasattr(pretrained_model, 'module'):\n",
        "    #     #     pretrained_model = pretrained_model.module\n",
        "    #     pretrained_model.load_state_dict(checkpoint['net'])\n",
        "\n",
        "    elif model_name == 'wrn16_clean':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/wrn_adv/')\n",
        "        filename = 'wrn_clean.pth'\n",
        "        pretrained_model = wrn_adv.create_model(name='wide', num_classes = 10)\n",
        "        if hasattr(pretrained_model, 'module'):\n",
        "            pretrained_model = pretrained_model.module\n",
        "        pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename),map_location=device)['state_dict'])\n",
        "\n",
        "\n",
        "    elif model_name == 'wrn16_fine':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/wrn_adv/')\n",
        "        filename = 'wrn16_fine.pth'\n",
        "        pretrained_model = wrn_adv.create_model(name='wide16', num_classes = 10)\n",
        "        # if hasattr(pretrained_model, 'module'):\n",
        "        #     pretrained_model = pretrained_model.module\n",
        "        pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename),map_location=device))\n",
        "\n",
        "\n",
        "    elif model_name == 'wrn28_fine':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/wrn_adv/')\n",
        "        filename = 'wrn28_fine.pth'\n",
        "        pretrained_model = wrn.WideNet()\n",
        "        pretrained_model = torch.nn.DataParallel(pretrained_model)\n",
        "        # if hasattr(pretrained_model, 'module'):\n",
        "        #     pretrained_model = pretrained_model.module\n",
        "        pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename),map_location=device))\n",
        "\n",
        "\n",
        "\n",
        "    elif model_name == 'wrn16_clean_sap':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/wrn_adv/')\n",
        "        filename = 'wrn_clean.pth'\n",
        "        pretrained_model = wrn_adv_sap.create_model(name='wide', num_classes = 10)\n",
        "        if hasattr(pretrained_model, 'module'):\n",
        "            pretrained_model = pretrained_model.module\n",
        "        pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename),map_location=device)['state_dict'])\n",
        "\n",
        "\n",
        "    # elif model_name == 'wrn_gau':\n",
        "    #     TRAINED_MODEL_PATH = data_path_join('pretrained_models/wrn_adv/')\n",
        "    #     filename = 'wrn_005.pth'\n",
        "    #     # pretrained_model = nn.DataParallel(adv_model.create_model(name='wide_gau', num_classes = 10), device_ids=range(1))\n",
        "    #     # pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename)))\n",
        "    #     pretrained_model = wrn_adv.create_model(name='wide', num_classes = 10)\n",
        "    #     if hasattr(pretrained_model, 'module'):\n",
        "    #         pretrained_model = pretrained_model.module\n",
        "    #     pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename))['state_dict'])\n",
        "\n",
        "    elif model_name == 'wrn_01':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/wrn_adv/')\n",
        "        filename = '01checkpoint_200.pth'\n",
        "        # pretrained_model = nn.DataParallel(adv_model.create_model(name='wide_gau', num_classes = 10), device_ids=range(1))\n",
        "        # pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename)))\n",
        "        pretrained_model = wrn_adv.create_model(name='wide', num_classes = 10)\n",
        "        if hasattr(pretrained_model, 'module'):\n",
        "            pretrained_model = pretrained_model.module\n",
        "        pretrained_model.load_state_dict(torch.load(os.path.join(TRAINED_MODEL_PATH, filename))['state_dict'])\n",
        "\n",
        "    elif model_name == 'res18':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/resnet/')\n",
        "        filename = 'cifar_res18.pth'\n",
        "        pretrained_model = resnet.ResNet18()\n",
        "        pretrained_model = torch.nn.DataParallel(pretrained_model)\n",
        "        checkpoint = torch.load(os.path.join(TRAINED_MODEL_PATH, filename))\n",
        "        # if hasattr(pretrained_model, 'module'):\n",
        "        #     pretrained_model = pretrained_model.module\n",
        "        pretrained_model.load_state_dict(checkpoint['net'])\n",
        "\n",
        "\n",
        "    elif model_name == 'res50':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/resnet/')\n",
        "        filename = 'cifar_res50.pth'\n",
        "        pretrained_model = resnet.ResNet50()\n",
        "        pretrained_model = torch.nn.DataParallel(pretrained_model)\n",
        "        checkpoint = torch.load(os.path.join(TRAINED_MODEL_PATH, filename))\n",
        "        # if hasattr(pretrained_model, 'module'):\n",
        "        #     pretrained_model = pretrained_model.module\n",
        "        pretrained_model.load_state_dict(checkpoint['net'])\n",
        "\n",
        "\n",
        "    elif model_name == 'res101':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/resnet/')\n",
        "        filename = 'cifar_res101.pth'\n",
        "        pretrained_model = resnet.ResNet101()\n",
        "        pretrained_model = torch.nn.DataParallel(pretrained_model)\n",
        "        checkpoint = torch.load(os.path.join(TRAINED_MODEL_PATH, filename))\n",
        "        # if hasattr(pretrained_model, 'module'):\n",
        "        #     pretrained_model = pretrained_model.module\n",
        "        pretrained_model.load_state_dict(checkpoint['net'])\n",
        "\n",
        "    elif model_name == 'dense121':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/densenet/')\n",
        "        filename = 'cifar_dense121.pth'\n",
        "        pretrained_model = densenet.DenseNet121()\n",
        "        pretrained_model = torch.nn.DataParallel(pretrained_model)\n",
        "        checkpoint = torch.load(os.path.join(TRAINED_MODEL_PATH, filename))\n",
        "        # if hasattr(pretrained_model, 'module'):\n",
        "        #     pretrained_model = pretrained_model.module\n",
        "        pretrained_model.load_state_dict(checkpoint['net'])\n",
        "\n",
        "\n",
        "    elif model_name == 'best_adv_wrn34':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/wrn_adv/')\n",
        "        filename = 'cifar10_linf_wrn34-20.pt'\n",
        "        pretrained_model = model_zoo.WideResNet(\n",
        "            num_classes=10, depth=34, width=20,\n",
        "            activation_fn=model_zoo.Swish, mean=model_zoo.CIFAR10_MEAN,\n",
        "            std=model_zoo.CIFAR10_STD)\n",
        "        # dataset_fn = datasets.CIFAR10\n",
        "        params = torch.load(os.path.join(TRAINED_MODEL_PATH, filename))\n",
        "        pretrained_model.load_state_dict(params)\n",
        "\n",
        "    elif model_name == 'best_adv_wrn28':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/wrn_adv/')\n",
        "        filename = 'cifar10_linf_wrn28-10_with.pt'\n",
        "        pretrained_model = model_zoo.WideResNet(\n",
        "            num_classes=10, depth=28, width=10,\n",
        "            activation_fn=model_zoo.Swish, mean=model_zoo.CIFAR10_MEAN,\n",
        "            std=model_zoo.CIFAR10_STD)\n",
        "        # dataset_fn = datasets.CIFAR10\n",
        "        params = torch.load(os.path.join(TRAINED_MODEL_PATH, filename))\n",
        "        pretrained_model.load_state_dict(params)\n",
        "\n",
        "\n",
        "    elif model_name == 'pni':\n",
        "        TRAINED_MODEL_PATH = data_path_join('pretrained_models/pni_model/')\n",
        "        filename = 'checkpoint.pth.tar'\n",
        "        recorder = RecorderMeter(200)  # count number of epoches\n",
        "        pretrained_model = pni_model.noise_resnet20()\n",
        "        checkpoint = torch.load(os.path.join(TRAINED_MODEL_PATH, filename))\n",
        "        state_tmp = pretrained_model.state_dict()\n",
        "        if 'state_dict' in checkpoint.keys():\n",
        "            state_tmp.update(checkpoint['state_dict'])\n",
        "        else:\n",
        "            state_tmp.update(checkpoint)\n",
        "\n",
        "        pretrained_model.load_state_dict(state_tmp)\n",
        "\n",
        "    def normalize_fn(tensor, mean, std):\n",
        "        \"\"\"Differentiable version of torchvision.functional.normalize\"\"\"\n",
        "        # here we assume the color channel is in at dim=1\n",
        "        mean = mean[None, :, None, None]\n",
        "        std = std[None, :, None, None]\n",
        "        return tensor.sub(mean).div(std)\n",
        "\n",
        "    class NormalizeByChannelMeanStd(nn.Module):\n",
        "        def __init__(self, mean, std):\n",
        "            super(NormalizeByChannelMeanStd, self).__init__()\n",
        "            if not isinstance(mean, torch.Tensor):\n",
        "                mean = torch.tensor(mean)\n",
        "            if not isinstance(std, torch.Tensor):\n",
        "                std = torch.tensor(std)\n",
        "            self.register_buffer(\"mean\", mean)\n",
        "            self.register_buffer(\"std\", std)\n",
        "\n",
        "        def forward(self, tensor):\n",
        "            return normalize_fn(tensor, self.mean, self.std)\n",
        "\n",
        "        def extra_repr(self):\n",
        "            return 'mean={}, std={}'.format(self.mean, self.std)\n",
        "\n",
        "    if  (model_name == 'vgg_plain') or (model_name == 'vgg_rse') or (model_name == 'rse') or (model_name == 'pni') or (model_name == 'wrn_adv') or (model_name == 'wrn16_clean') or (model_name == 'wrn16_fine') or (model_name == 'wrn16_clean_sap') or (model_name == 'wrn_01') or (model_name == 'best_adv_wrn28') or (model_name == 'best_adv_wrn34') or (model_name == 'wrn_stop') or (model_name == 'sat_wrn16_l2') or (model_name == 'sat_wrn16_linf'):\n",
        "        net = pretrained_model\n",
        "\n",
        "    elif 'wrn28' in model_name:\n",
        "        mean = np.array([0.4914, 0.4822, 0.4465])\n",
        "        std = np.array([0.2023, 0.1994, 0.2010])\n",
        "\n",
        "        # std = np.array([0.2470, 0.2435, 0.2616])\n",
        "        normalize = NormalizeByChannelMeanStd(\n",
        "                mean=mean.tolist(), std=std.tolist())\n",
        "\n",
        "        net = nn.Sequential(\n",
        "            normalize,\n",
        "            pretrained_model\n",
        "        )\n",
        "\n",
        "    else:\n",
        "        mean = np.array([0.4914, 0.4822, 0.4465])\n",
        "        std = np.array([0.2470, 0.2435, 0.2616])\n",
        "\n",
        "        normalize = NormalizeByChannelMeanStd(\n",
        "                mean=mean.tolist(), std=std.tolist())\n",
        "\n",
        "        net = nn.Sequential(\n",
        "            normalize,\n",
        "            pretrained_model\n",
        "        )\n",
        "\n",
        "    if torch.cuda.is_available():\n",
        "        net = net.cuda()\n",
        "    net.eval()\n",
        "    return net\n",
        "\n",
        "\n",
        "class Permute(nn.Module):\n",
        "\n",
        "    def __init__(self, permutation = [2,1,0]):\n",
        "        super().__init__()\n",
        "        self.permutation = permutation\n",
        "\n",
        "    def forward(self, input):\n",
        "\n",
        "        return input[:, self.permutation]\n",
        "\n",
        "\n",
        "def load_torch_models_imagesub(model_name):\n",
        "    if model_name == \"VGG16\":\n",
        "        pretrained_model = models.vgg16_bn(pretrained=True)\n",
        "    elif model_name == 'Resnet18':\n",
        "        pretrained_model = models.resnet18(pretrained=True)\n",
        "    elif model_name == 'Resnet34':\n",
        "        pretrained_model = models.resnet34(pretrained=True)\n",
        "    elif model_name == 'Resnet50':\n",
        "        pretrained_model = models.resnet50(pretrained=True)\n",
        "    elif model_name == 'resnet50-AT':\n",
        "        checkpoint = torch.load('imagenet_linf_8.pt')\n",
        "        # Extract the model state dictionary from the OrderedDict (replace 'model' with the correct key)\n",
        "        model_state_dict = checkpoint['model']\n",
        "        # Create an instance of the ResNet50 model\n",
        "        pretrained_model = models.resnet50(pretrained=True)\n",
        "        mapped_state_dict = {k.replace('module.attacker.model.', ''): v for k, v in model_state_dict.items()}\n",
        "        # Filter out keys that are not relevant to ResNet50\n",
        "        filtered_state_dict = {k: v for k, v in mapped_state_dict.items() if k in pretrained_model.state_dict()}\n",
        "        # Load the filtered state dictionary\n",
        "        pretrained_model.load_state_dict(filtered_state_dict, strict=False)\n",
        "    elif model_name == 'Resnet101':\n",
        "        pretrained_model = models.resnet101(pretrained=True)\n",
        "    # elif model_name == 'Squeezenet':\n",
        "    #     pretrained_model = models.squeezenet1_1(pretrained=True)\n",
        "    elif model_name == 'Googlenet':\n",
        "        pretrained_model = models.googlenet(pretrained=True)\n",
        "    elif model_name == 'Inception':\n",
        "        pretrained_model = models.inception_v3(pretrained=True)\n",
        "    elif model_name == 'Widenet':\n",
        "        pretrained_model = models.wide_resnet50_2(pretrained=True)\n",
        "    # elif model_name == 'Adv_Denoise_Resnext101':\n",
        "    #     pretrained_model = resnet101_denoise()\n",
        "    #     loaded_state_dict = torch.load(os.path.join('./results/denoise/', model_name+\".pytorch\"))\n",
        "    #     pretrained_model.load_state_dict(loaded_state_dict)\n",
        "\n",
        "    # if 'defense' in state and state['defense']:\n",
        "    #     net = nn.Sequential(\n",
        "    #         Normalize(mean, std),\n",
        "    #         Permute([2,1,0]),\n",
        "    #         pretrained_model\n",
        "    #     )\n",
        "    # else:\n",
        "    # net = nn.Sequential(\n",
        "    #     Normalize(mean, std),\n",
        "    #     pretrained_model\n",
        "    # )\n",
        "\n",
        "    # from advertorch.utils import NormalizeByChannelMeanStd\n",
        "    def normalize_fn(tensor, mean, std):\n",
        "        \"\"\"Differentiable version of torchvision.functional.normalize\"\"\"\n",
        "        # here we assume the color channel is in at dim=1\n",
        "        mean = mean[None, :, None, None]\n",
        "        std = std[None, :, None, None]\n",
        "        return tensor.sub(mean).div(std)\n",
        "    class NormalizeByChannelMeanStd(nn.Module):\n",
        "        def __init__(self, mean, std):\n",
        "            super(NormalizeByChannelMeanStd, self).__init__()\n",
        "            if not isinstance(mean, torch.Tensor):\n",
        "                mean = torch.tensor(mean)\n",
        "            if not isinstance(std, torch.Tensor):\n",
        "                std = torch.tensor(std)\n",
        "            self.register_buffer(\"mean\", mean)\n",
        "            self.register_buffer(\"std\", std)\n",
        "\n",
        "        def forward(self, tensor):\n",
        "            return normalize_fn(tensor, self.mean, self.std)\n",
        "\n",
        "        def extra_repr(self):\n",
        "            return 'mean={}, std={}'.format(self.mean, self.std)\n",
        "\n",
        "    mean = np.array([0.485, 0.456, 0.406])\n",
        "    std = np.array([0.229, 0.224, 0.225])\n",
        "\n",
        "    normalize = NormalizeByChannelMeanStd(\n",
        "            mean=mean.tolist(), std=std.tolist())\n",
        "\n",
        "    if 'Denoise' in model_name:\n",
        "        net = nn.Sequential(\n",
        "            # Normalize(mean, std),\n",
        "            normalize,\n",
        "            Permute([2,1,0]),\n",
        "            pretrained_model\n",
        "        )\n",
        "\n",
        "    else:\n",
        "        net = nn.Sequential(\n",
        "            # Normalize(mean, std),\n",
        "            normalize,\n",
        "            pretrained_model\n",
        "        )\n",
        "\n",
        "\n",
        "    if torch.cuda.is_available():\n",
        "        net = net.cuda()\n",
        "    net.eval()\n",
        "    return net"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile /content/BlackboxBench/attacks/decision/sign_opt_attack.py\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "import time\n",
        "import scipy.spatial\n",
        "from scipy.linalg import qr\n",
        "import random\n",
        "from torch import Tensor as t\n",
        "\n",
        "from attacks.decision.decision_black_box_attack import DecisionBlackBoxAttack\n",
        "\n",
        "start_learning_rate = 1.0\n",
        "\n",
        "def quad_solver(Q, b):\n",
        "    \"\"\"\n",
        "    Solve min_a  0.5*aQa + b^T a s.t. a>=0\n",
        "    \"\"\"\n",
        "    K = Q.shape[0]\n",
        "    alpha = torch.zeros((K,))\n",
        "    g = b\n",
        "    Qdiag = torch.diag(Q)\n",
        "    for _ in range(20000):\n",
        "        delta = torch.maximum(alpha - g/Qdiag,0) - alpha\n",
        "        idx = torch.argmax(torch.abs(delta))\n",
        "        val = delta[idx]\n",
        "        if abs(val) < 1e-7:\n",
        "            break\n",
        "        g = g + val*Q[:,idx]\n",
        "        alpha[idx] += val\n",
        "    return alpha\n",
        "\n",
        "def sign(y):\n",
        "    \"\"\"\n",
        "    y -- numpy array of shape (m,)\n",
        "    Returns an element-wise indication of the sign of a number.\n",
        "    The sign function returns -1 if y < 0, 1 if x >= 0. nan is returned for nan inputs.\n",
        "    \"\"\"\n",
        "    y_sign = torch.sign(y)\n",
        "    y_sign[y_sign==0] = 1\n",
        "    return y_sign\n",
        "\n",
        "\n",
        "class SignOPTAttack(DecisionBlackBoxAttack):\n",
        "    \"\"\"\n",
        "    Sign_OPT\n",
        "    \"\"\"\n",
        "\n",
        "    def __init__(self, epsilon, p, alpha, beta, svm, momentum, max_queries, k, lb, ub, batch_size, sigma):\n",
        "        super().__init__(max_queries = max_queries,\n",
        "                         epsilon=epsilon,\n",
        "                         p=p,\n",
        "                         lb=lb,\n",
        "                         ub=ub,\n",
        "                         batch_size = batch_size)\n",
        "        self.alpha = alpha\n",
        "        self.beta = beta\n",
        "        self.svm = svm\n",
        "        self.momentum = momentum\n",
        "        self.k = k\n",
        "        self.sigma = sigma\n",
        "        self.query_count = 0\n",
        "\n",
        "\n",
        "    def _config(self):\n",
        "        return {\n",
        "            \"p\": self.p,\n",
        "            \"epsilon\": self.epsilon,\n",
        "            \"lb\": self.lb,\n",
        "            \"ub\": self.ub,\n",
        "            \"attack_name\": self.__class__.__name__\n",
        "        }\n",
        "\n",
        "    def attack_untargeted(self, x0, y0, alpha = 0.2, beta = 0.001):\n",
        "        \"\"\"\n",
        "        Attack the original image and return adversarial example\n",
        "        \"\"\"\n",
        "\n",
        "        y0 = y0[0]\n",
        "        self.query_count = 0\n",
        "\n",
        "        # Calculate a good starting point.\n",
        "        num_directions = 10\n",
        "        best_theta, g_theta = None, float('inf')\n",
        "        print(\"Searching for the initial direction on %d random directions: \" % (num_directions))\n",
        "        timestart = time.time()\n",
        "        for i in range(num_directions):\n",
        "            self.query_count += 1\n",
        "            theta = torch.randn_like(x0)\n",
        "            if self.predict_label(x0+theta)!=y0:\n",
        "                initial_lbd = torch.norm(theta)\n",
        "                theta /= initial_lbd\n",
        "                lbd, count = self.fine_grained_binary_search(x0, y0, theta, initial_lbd, g_theta)\n",
        "                self.query_count += count\n",
        "                if lbd < g_theta:\n",
        "                    best_theta, g_theta = theta, lbd\n",
        "                    # print(\"--------> Found distortion %.4f\" % g_theta)\n",
        "\n",
        "        timeend = time.time()\n",
        "        if g_theta == float('inf'):\n",
        "            print(\"Couldn't find valid initial, failed\")\n",
        "            return x0, self.query_count\n",
        "        # print(\"==========> Found best distortion %.4f in %.4f seconds \"\n",
        "        #       \"using %d queries\" % (g_theta, timeend-timestart, self.query_count))\n",
        "\n",
        "        # Begin Gradient Descent.\n",
        "        timestart = time.time()\n",
        "        xg, gg = best_theta, g_theta\n",
        "        vg = torch.zeros_like(xg)\n",
        "\n",
        "        for i in range(1500):\n",
        "            if self.svm == True:\n",
        "                sign_gradient, grad_queries = self.sign_grad_svm(x0, y0, xg, initial_lbd=gg, h=beta)\n",
        "            else:\n",
        "                sign_gradient, grad_queries = self.sign_grad_v1(x0, y0, xg, initial_lbd=gg, h=beta)\n",
        "            self.query_count += grad_queries\n",
        "            # Line search\n",
        "            min_theta = xg\n",
        "            min_g2 = gg\n",
        "            min_vg = vg\n",
        "            for _ in range(15):\n",
        "                if self.momentum > 0:\n",
        "                    new_vg = self.momentum*vg - alpha*sign_gradient\n",
        "                    new_theta = xg + new_vg\n",
        "                else:\n",
        "                    new_theta = xg - alpha * sign_gradient\n",
        "                new_theta /= torch.norm(new_theta)\n",
        "                new_g2, count = self.fine_grained_binary_search_local(x0, y0, new_theta, initial_lbd = min_g2, tol=beta/500)\n",
        "                self.query_count += count\n",
        "                alpha = alpha * 2\n",
        "                if new_g2 < min_g2:\n",
        "                    min_theta = new_theta\n",
        "                    min_g2 = new_g2\n",
        "                    if self.momentum > 0:\n",
        "                        min_vg = new_vg\n",
        "                else:\n",
        "                    break\n",
        "            if min_g2 >= gg:\n",
        "                for _ in range(15):\n",
        "                    alpha = alpha * 0.25\n",
        "                    if self.momentum > 0:\n",
        "                        new_vg = self.momentum*vg - alpha*sign_gradient\n",
        "                        new_theta = xg + new_vg\n",
        "                    else:\n",
        "                        new_theta = xg - alpha * sign_gradient\n",
        "                    new_theta /= torch.norm(new_theta)\n",
        "                    new_g2, count = self.fine_grained_binary_search_local(x0, y0, new_theta, initial_lbd = min_g2, tol=beta/500)\n",
        "                    self.query_count += count\n",
        "                    if new_g2 < gg:\n",
        "                        min_theta = new_theta\n",
        "                        min_g2 = new_g2\n",
        "                        if self.momentum > 0:\n",
        "                            min_vg = new_vg\n",
        "                        break\n",
        "            if alpha < 1e-4:\n",
        "                alpha = 1.0\n",
        "                print(\"Warning: not moving\")\n",
        "                beta = beta*0.1\n",
        "                if (beta < 1e-8):\n",
        "                    break\n",
        "\n",
        "            xg, gg = min_theta, min_g2\n",
        "            vg = min_vg\n",
        "\n",
        "\n",
        "            if self.query_count > self.max_queries:\n",
        "               break\n",
        "\n",
        "            dist = self.distance(gg*xg)\n",
        "            if dist < self.epsilon:\n",
        "                break\n",
        "\n",
        "            # print(\"Iteration %3d distortion %.4f num_queries %d\" % (i+1, dist, self.query_count))\n",
        "\n",
        "        dist = self.distance(gg*xg)\n",
        "        timeend = time.time()\n",
        "        # print(\"\\nAdversarial Example Found Successfully: distortion %.4f queries %d \\nTime: %.4f seconds\" % (dist, self.query_count, timeend-timestart))\n",
        "        return x0 + gg*xg, self.query_count\n",
        "\n",
        "    def sign_grad_v1(self, x0, y0, theta, initial_lbd, h=0.001, D=4, target=None):\n",
        "        \"\"\"\n",
        "        Evaluate the sign of gradient by formulat\n",
        "        sign(g) = 1/Q [ \\sum_{q=1}^Q sign( g(theta+h*u_i) - g(theta) )u_i$ ]\n",
        "        \"\"\"\n",
        "        K = self.k\n",
        "        sign_grad = torch.zeros_like(theta)\n",
        "        queries = 0\n",
        "        for _ in range(K):\n",
        "            u = torch.randn_like(theta)\n",
        "            u /= torch.norm(u)\n",
        "\n",
        "            sign = 1\n",
        "            new_theta = theta + h*u\n",
        "            new_theta /= torch.norm(new_theta)\n",
        "\n",
        "            # Targeted case.\n",
        "            if (target is not None and\n",
        "                self.predict_label(x0+initial_lbd*new_theta) == target):\n",
        "                sign = -1\n",
        "\n",
        "            # Untargeted case\n",
        "            if (target is None and\n",
        "                self.predict_label(x0+t(initial_lbd*new_theta)) != y0):\n",
        "                sign = -1\n",
        "            queries += 1\n",
        "            sign_grad += u*sign\n",
        "\n",
        "        sign_grad /= K\n",
        "\n",
        "        return sign_grad, queries\n",
        "\n",
        "    def sign_grad_svm(self, x0, y0, theta, initial_lbd, h=0.001, K=100, lr=5.0, target=None):\n",
        "        \"\"\"\n",
        "        Evaluate the sign of gradient by formulat\n",
        "        sign(g) = 1/Q [ \\sum_{q=1}^Q sign( g(theta+h*u_i) - g(theta) )u_i$ ]\n",
        "        \"\"\"\n",
        "        sign_grad = torch.zeros_like(theta)\n",
        "        queries = 0\n",
        "        dim = np.prod(theta.shape)\n",
        "        X = torch.zeros((dim, K))\n",
        "        for iii in range(K):\n",
        "            u = torch.randn_like(theta)\n",
        "            u /= torch.norm(u)\n",
        "\n",
        "            sign = 1\n",
        "            new_theta = theta + h*u\n",
        "            new_theta /= torch.norm(new_theta)\n",
        "\n",
        "            # Targeted case.\n",
        "            if (target is not None and\n",
        "                self.predict_label(x0+t(initial_lbd*new_theta)) == target):\n",
        "                sign = -1\n",
        "\n",
        "            # Untargeted case\n",
        "            if (target is None and\n",
        "                self.predict_label(x0+t(initial_lbd*new_theta)) != y0):\n",
        "                sign = -1\n",
        "\n",
        "            queries += 1\n",
        "            X[:,iii] = sign*u.reshape((dim,))\n",
        "\n",
        "        Q = X.transpose().dot(X)\n",
        "        q = -1*torch.ones((K,))\n",
        "        # G = torch.diag(-1*torch.ones((K,)))\n",
        "        # h = torch.zeros((K,))\n",
        "        ### Use quad_qp solver\n",
        "        #alpha = solve_qp(Q, q, G, h)\n",
        "        ### Use coordinate descent solver written by myself, avoid non-positive definite cases\n",
        "        alpha = quad_solver(Q, q)\n",
        "        sign_grad = (X.dot(alpha)).reshape(theta.shape)\n",
        "\n",
        "        return sign_grad, queries\n",
        "\n",
        "    def fine_grained_binary_search_local(self, x0, y0, theta, initial_lbd = 1.0, tol=1e-5):\n",
        "        nquery = 0\n",
        "        lbd = initial_lbd\n",
        "\n",
        "        if self.predict_label(x0+lbd*theta) == y0:\n",
        "            lbd_lo = lbd\n",
        "            lbd_hi = lbd*1.01\n",
        "            nquery += 1\n",
        "            while self.predict_label(x0+lbd_hi*theta) == y0:\n",
        "                lbd_hi = lbd_hi*1.01\n",
        "                nquery += 1\n",
        "                if lbd_hi > 20:\n",
        "                    return float('inf'), nquery\n",
        "        else:\n",
        "            lbd_hi = lbd\n",
        "            lbd_lo = lbd*0.99\n",
        "            nquery += 1\n",
        "            while self.predict_label(x0+lbd_lo*theta) != y0 :\n",
        "                lbd_lo = lbd_lo*0.99\n",
        "                nquery += 1\n",
        "                if nquery + self.query_count> self.max_queries:\n",
        "                    break\n",
        "\n",
        "        while (lbd_hi - lbd_lo) > tol:\n",
        "            lbd_mid = (lbd_lo + lbd_hi)/2.0\n",
        "            nquery += 1\n",
        "            if nquery + self.query_count> self.max_queries:\n",
        "                break\n",
        "            if self.predict_label(x0 + lbd_mid*theta) != y0:\n",
        "                lbd_hi = lbd_mid\n",
        "            else:\n",
        "                lbd_lo = lbd_mid\n",
        "        return lbd_hi, nquery\n",
        "\n",
        "    def fine_grained_binary_search(self, x0, y0, theta, initial_lbd, current_best):\n",
        "        nquery = 0\n",
        "        if initial_lbd > current_best:\n",
        "            if self.predict_label(x0+t(current_best*theta)) == y0:\n",
        "                nquery += 1\n",
        "                return float('inf'), nquery\n",
        "            lbd = current_best\n",
        "        else:\n",
        "            lbd = initial_lbd\n",
        "\n",
        "        lbd_hi = lbd\n",
        "        lbd_lo = 0.0\n",
        "\n",
        "        while (lbd_hi - lbd_lo) > 1e-5:\n",
        "            lbd_mid = (lbd_lo + lbd_hi)/2.0\n",
        "            nquery += 1\n",
        "            if nquery + self.query_count> self.max_queries:\n",
        "                break\n",
        "            if self.predict_label(x0 + t(lbd_mid*theta)) != y0:\n",
        "                lbd_hi = lbd_mid\n",
        "            else:\n",
        "                lbd_lo = lbd_mid\n",
        "        return lbd_hi, nquery\n",
        "\n",
        "    def attack_targeted(self, x0, target, alpha = 0.2, beta = 0.001):\n",
        "        \"\"\"\n",
        "        Attack the original image and return adversarial example\n",
        "        \"\"\"\n",
        "\n",
        "        target = target[0]\n",
        "\n",
        "        num_samples = 100\n",
        "        best_theta, g_theta = None, float('inf')\n",
        "        query_count = 0\n",
        "        ls_total = 0\n",
        "        sample_count = 0\n",
        "        print(\"Searching for the initial direction on %d samples: \" % (num_samples))\n",
        "        timestart = time.time()\n",
        "\n",
        "\n",
        "        # Iterate through training dataset. Find best initial point for gradient descent.\n",
        "        for i in range(500):\n",
        "            xi, _ = self.train_dataset.get_eval_data(i,i+1)\n",
        "            yi_pred = self.predict_label(xi)\n",
        "            query_count += 1\n",
        "            if yi_pred != target:\n",
        "                continue\n",
        "            theta = xi - x0\n",
        "            initial_lbd = torch.norm(theta)\n",
        "            theta /= initial_lbd\n",
        "            lbd, count = self.fine_grained_binary_search_targeted(x0, target, theta, initial_lbd, g_theta)\n",
        "            query_count += count\n",
        "            if lbd < g_theta:\n",
        "                best_theta, g_theta = theta, lbd\n",
        "                # print(\"--------> Found distortion %.4f\" % g_theta)\n",
        "\n",
        "            sample_count += 1\n",
        "            if sample_count >= num_samples:\n",
        "                break\n",
        "\n",
        "\n",
        "        timeend = time.time()\n",
        "        if g_theta == np.inf:\n",
        "            return x0, query_count\n",
        "        # print(\"==========> Found best distortion %.4f in %.4f seconds using %d queries\" %\n",
        "        #       (g_theta, timeend-timestart, query_count))\n",
        "\n",
        "        # Begin Gradient Descent.\n",
        "        timestart = time.time()\n",
        "        xg, gg = best_theta, g_theta\n",
        "\n",
        "        for i in range(1000):\n",
        "            if self.svm == True:\n",
        "                sign_gradient, grad_queries = self.sign_grad_svm(x0, 0, xg, initial_lbd=gg, h=beta, target=target)\n",
        "            else:\n",
        "                sign_gradient, grad_queries = self.sign_grad_v1(x0, 0, xg, initial_lbd=gg, h=beta, target=target)\n",
        "\n",
        "\n",
        "            # Line search\n",
        "            ls_count = 0\n",
        "            min_theta = xg\n",
        "            min_g2 = gg\n",
        "            for _ in range(15):\n",
        "                new_theta = xg - alpha * sign_gradient\n",
        "                new_theta /= torch.norm(new_theta)\n",
        "                new_g2, count = self.fine_grained_binary_search_local_targeted(x0, target, new_theta, initial_lbd = min_g2, tol=beta/500)\n",
        "                ls_count += count\n",
        "                alpha = alpha * 2\n",
        "                if new_g2 < min_g2:\n",
        "                    min_theta = new_theta\n",
        "                    min_g2 = new_g2\n",
        "                else:\n",
        "                    break\n",
        "\n",
        "            if min_g2 >= gg:\n",
        "                for _ in range(15):\n",
        "                    alpha = alpha * 0.25\n",
        "                    new_theta = xg - alpha * sign_gradient\n",
        "                    new_theta /= torch.norm(new_theta)\n",
        "                    new_g2, count = self.fine_grained_binary_search_local_targeted(x0, target, new_theta, initial_lbd = min_g2, tol=beta/500)\n",
        "                    ls_count += count\n",
        "                    if new_g2 < gg:\n",
        "                        min_theta = new_theta\n",
        "                        min_g2 = new_g2\n",
        "                        break\n",
        "\n",
        "            if alpha < 1e-4:\n",
        "                alpha = 1.0\n",
        "                print(\"Warning: not moving\")\n",
        "                beta = beta*0.1\n",
        "                if (beta < 1e-8):\n",
        "                    break\n",
        "\n",
        "            xg, gg = min_theta, min_g2\n",
        "\n",
        "            query_count += (grad_queries + ls_count)\n",
        "            ls_total += ls_count\n",
        "\n",
        "            if query_count > self.max_queries:\n",
        "                break\n",
        "\n",
        "            dist = self.distance(gg*xg)\n",
        "            if dist < self.epsilon:\n",
        "                break\n",
        "\n",
        "            if i%5==0:\n",
        "                print(\"Iteration %3d distortion %.4f num_queries %d\" % (i+1, dist, query_count))\n",
        "\n",
        "        adv_target = self.predict_label(x0 + t(gg*xg))\n",
        "        if (adv_target == target):\n",
        "            timeend = time.time()\n",
        "            print(\"\\nAdversarial Example Found Successfully: distortion %.4f target\"\n",
        "                  \" %d queries %d LS queries %d \\nTime: %.4f seconds\" % (dist, target, query_count, ls_total, timeend-timestart))\n",
        "\n",
        "            return x0 + t(gg*xg), query_count\n",
        "        else:\n",
        "            print(\"Failed to find targeted adversarial example.\")\n",
        "            return x0, query_count\n",
        "\n",
        "    def fine_grained_binary_search_local_targeted(self, x0, t, theta, initial_lbd=1.0, tol=1e-5):\n",
        "        nquery = 0\n",
        "        lbd = initial_lbd\n",
        "\n",
        "        if self.predict_label(x0 + t(lbd*theta)) != t:\n",
        "            lbd_lo = lbd\n",
        "            lbd_hi = lbd*1.01\n",
        "            nquery += 1\n",
        "            while self.predict_label(x0 + t(lbd_hi*theta)) != t:\n",
        "                lbd_hi = lbd_hi*1.01\n",
        "                nquery += 1\n",
        "                if lbd_hi > 100:\n",
        "                    return float('inf'), nquery\n",
        "        else:\n",
        "            lbd_hi = lbd\n",
        "            lbd_lo = lbd*0.99\n",
        "            nquery += 1\n",
        "            while self.predict_label(x0 + t(lbd_lo*theta)) == t:\n",
        "                lbd_lo = lbd_lo*0.99\n",
        "                nquery += 1\n",
        "\n",
        "        while (lbd_hi - lbd_lo) > tol:\n",
        "            lbd_mid = (lbd_lo + lbd_hi)/2.0\n",
        "            nquery += 1\n",
        "            if self.predict_label(x0 + t(lbd_mid*theta)) == t:\n",
        "                lbd_hi = lbd_mid\n",
        "            else:\n",
        "                lbd_lo = lbd_mid\n",
        "\n",
        "        return lbd_hi, nquery\n",
        "\n",
        "    def fine_grained_binary_search_targeted(self, x0, t, theta, initial_lbd, current_best):\n",
        "        nquery = 0\n",
        "        if initial_lbd > current_best:\n",
        "            if self.predict_label(x0 + t(current_best*theta)) != t:\n",
        "                nquery += 1\n",
        "                return float('inf'), nquery\n",
        "            lbd = current_best\n",
        "        else:\n",
        "            lbd = initial_lbd\n",
        "\n",
        "        lbd_hi = lbd\n",
        "        lbd_lo = 0.0\n",
        "\n",
        "        while (lbd_hi - lbd_lo) > 1e-5:\n",
        "            lbd_mid = (lbd_lo + lbd_hi)/2.0\n",
        "            nquery += 1\n",
        "            if self.predict_label(x0 + t(lbd_mid*theta)) != t:\n",
        "                lbd_lo = lbd_mid\n",
        "            else:\n",
        "                lbd_hi = lbd_mid\n",
        "        return lbd_hi, nquery\n",
        "\n",
        "\n",
        "    def _perturb(self, xs_t, ys):\n",
        "        if self.targeted:\n",
        "            adv, q = self.attack_targeted(xs_t, ys, self.alpha, self.beta)\n",
        "        else:\n",
        "            adv, q = self.attack_untargeted(xs_t, ys, self.alpha, self.beta)\n",
        "\n",
        "        return adv, q\n",
        "\n",
        "'''\n",
        "\n",
        "Original License\n",
        "\n",
        "MIT License\n",
        "\n",
        "Copyright (c) 2022 Minhao Cheng\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE.\n",
        "\n",
        "'''\n"
      ],
      "metadata": {
        "id": "znpx8UxnkuKX",
        "outputId": "043c3416-8fea-42f0-d999-6556c33bcce3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/attacks/decision/sign_opt_attack.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile /content/BlackboxBench/attacks/decision/hsja_attack.py\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "from torch import Tensor as t\n",
        "\n",
        "from attacks.decision.decision_black_box_attack import DecisionBlackBoxAttack\n",
        "\n",
        "\n",
        "class HSJAttack(DecisionBlackBoxAttack):\n",
        "    \"\"\"\n",
        "    HSJA\n",
        "    \"\"\"\n",
        "    def __init__(self, epsilon, p, max_queries, gamma, stepsize_search, max_num_evals, init_num_evals, EOT, sigma, lb, ub, batch_size):\n",
        "        super().__init__(max_queries = max_queries,\n",
        "                         epsilon=epsilon,\n",
        "                         p=p,\n",
        "                         lb=lb,\n",
        "                         ub=ub,\n",
        "                         batch_size = batch_size)\n",
        "        self.gamma = gamma\n",
        "        self.stepsize_search = stepsize_search\n",
        "        self.max_num_evals = max_num_evals\n",
        "        self.init_num_evals = init_num_evals\n",
        "        self.verbose = True\n",
        "        self.EOT = EOT\n",
        "        self.sigma = sigma\n",
        "\n",
        "    def _config(self):\n",
        "        return {\n",
        "            \"p\": self.p,\n",
        "            \"epsilon\": self.epsilon,\n",
        "            \"lb\": self.lb,\n",
        "            \"ub\": self.ub,\n",
        "            \"attack_name\": self.__class__.__name__\n",
        "        }\n",
        "\n",
        "    def hsja(self,input_xi,label_or_target):\n",
        "\n",
        "        d = int(np.prod(input_xi.shape))\n",
        "        # Set binary search threshold.\n",
        "        if self.p == '2':\n",
        "                theta = self.gamma / (np.sqrt(d) * d)\n",
        "        else:\n",
        "                theta = self.gamma / (d ** 2)\n",
        "\n",
        "        self.query = 0\n",
        "        # Initialize.\n",
        "        perturbed = self.initialize(input_xi, label_or_target)\n",
        "\n",
        "\n",
        "        # Project the initialization to the boundary.\n",
        "        perturbed, dist_post_update = self.binary_search_batch(input_xi, perturbed, label_or_target, theta)\n",
        "        dist = self.compute_distance(perturbed, input_xi)\n",
        "\n",
        "        for j in np.arange(10000):\n",
        "                # Choose delta.\n",
        "                if j==1:\n",
        "                    delta = 0.1 * (self.ub - self.lb)\n",
        "                else:\n",
        "                    if self.p == '2':\n",
        "                            delta = np.sqrt(d) * theta * dist_post_update\n",
        "                    elif self.p == 'inf':\n",
        "                            delta = d * theta * dist_post_update\n",
        "\n",
        "                # Choose number of evaluations.\n",
        "                num_evals = int(self.init_num_evals * np.sqrt(j+1))\n",
        "                num_evals = int(min([num_evals, self.max_num_evals]))\n",
        "\n",
        "                # approximate gradient.\n",
        "                gradf = self.approximate_gradient(perturbed, label_or_target, num_evals, delta)\n",
        "                if self.p == 'inf':\n",
        "                        update = np.sign(gradf)\n",
        "                else:\n",
        "                        update = gradf\n",
        "\n",
        "                # search step size.\n",
        "                if self.stepsize_search == 'geometric_progression':\n",
        "                        # find step size.\n",
        "                        epsilon = self.geometric_progression_for_stepsize(perturbed, label_or_target,\n",
        "                                update, dist, j+1)\n",
        "\n",
        "                        # Update the sample.\n",
        "                        perturbed = self.clip_image(perturbed + epsilon * update,\n",
        "                                self.lb, self.ub)\n",
        "\n",
        "                        # Binary search to return to the boundary.\n",
        "                        perturbed, dist_post_update = self.binary_search_batch(input_xi,\n",
        "                                perturbed[None], label_or_target, theta)\n",
        "\n",
        "                elif self.stepsize_search == 'grid_search':\n",
        "                        # Grid search for stepsize.\n",
        "                        epsilons = np.logspace(-4, 0, num=20, endpoint = True) * dist\n",
        "                        epsilons_shape = [20] + len(input_xi.shape) * [1]\n",
        "                        perturbeds = perturbed + epsilons.reshape(epsilons_shape) * update\n",
        "                        perturbeds = self.clip_image(perturbeds, self.lb, self.ub)\n",
        "                        idx_perturbed = self.decision_function(perturbeds, label_or_target)\n",
        "\n",
        "                        if np.sum(idx_perturbed) > 0:\n",
        "                                # Select the perturbation that yields the minimum distance # after binary search.\n",
        "                                perturbed, dist_post_update = self.binary_search_batch(input_xi,\n",
        "                                        perturbeds[idx_perturbed], label_or_target, theta)\n",
        "\n",
        "                # compute new distance.\n",
        "                dist = self.compute_distance(perturbed, input_xi)\n",
        "                if self.verbose:\n",
        "                        print('iteration: {:d}, {:s} distance {:.4f}'.format(j+1, self.p, dist))\n",
        "                if self.query > self.max_queries:\n",
        "                        break\n",
        "                if dist < self.epsilon:\n",
        "                        break\n",
        "\n",
        "        return t(perturbed).unsqueeze(0)\n",
        "\n",
        "    def decision_function(self, images, label):\n",
        "        \"\"\"\n",
        "        Decision function output 1 on the desired side of the boundary,\n",
        "        0 otherwise.\n",
        "        \"\"\"\n",
        "        images = torch.from_numpy(images).float()\n",
        "        if len(images.shape) == 3:\n",
        "                images = images.unsqueeze(0)\n",
        "        self.query += images.shape[0]\n",
        "        la = self.predict_label(images).cpu().numpy()\n",
        "        if self.targeted:\n",
        "                return (la==label)\n",
        "        else:\n",
        "                return (la!=label)\n",
        "\n",
        "    def clip_image(self, image, clip_min, clip_max):\n",
        "        # Clip an image, or an image batch, with upper and lower threshold.\n",
        "        return np.minimum(np.maximum(clip_min, image), clip_max)\n",
        "\n",
        "\n",
        "    def compute_distance(self, x_ori, x_pert):\n",
        "        # Compute the distance between two images.\n",
        "        if self.p == '2':\n",
        "                return np.linalg.norm(x_ori - x_pert)\n",
        "        elif self.p == 'inf':\n",
        "                return np.max(abs(x_ori - x_pert))\n",
        "\n",
        "\n",
        "    def approximate_gradient(self, sample, label_or_target, num_evals, delta):\n",
        "        # Generate random vectors.\n",
        "        noise_shape = [num_evals] + list(sample.shape)\n",
        "        if self.p == '2':\n",
        "                rv = np.random.randn(*noise_shape)\n",
        "        elif self.p == 'inf':\n",
        "                rv = np.random.uniform(low = -1, high = 1, size = noise_shape)\n",
        "\n",
        "        rv = rv / np.sqrt(np.sum(rv ** 2, axis = (1,2,3), keepdims = True))\n",
        "        perturbed = sample + delta * rv\n",
        "        perturbed = self.clip_image(perturbed, self.lb, self.ub)\n",
        "        rv = (perturbed - sample) / delta\n",
        "\n",
        "        # query the model.\n",
        "        decisions = self.decision_function(perturbed, label_or_target)\n",
        "        decision_shape = [len(decisions)] + [1] * len(sample.shape)\n",
        "        fval = 2 * decisions.astype(float).reshape(decision_shape) - 1.0\n",
        "\n",
        "        # Baseline subtraction (when fval differs)\n",
        "        if np.mean(fval) == 1.0: # label changes.\n",
        "                gradf = np.mean(rv, axis = 0)\n",
        "        elif np.mean(fval) == -1.0: # label not change.\n",
        "                gradf = - np.mean(rv, axis = 0)\n",
        "        else:\n",
        "                fval -= np.mean(fval)\n",
        "                gradf = np.mean(fval * rv, axis = 0)\n",
        "\n",
        "        # Get the gradient direction.\n",
        "        gradf = gradf / np.linalg.norm(gradf)\n",
        "\n",
        "        return gradf\n",
        "\n",
        "\n",
        "    def project(self, original_image, perturbed_images, alphas):\n",
        "        alphas_shape = [1] * len(original_image.shape)\n",
        "        alphas = alphas.reshape(alphas_shape)\n",
        "        if self.p == '2':\n",
        "                #print(alphas.shape,original_image.shape, perturbed_images.shape)\n",
        "                return (1-alphas) * original_image + alphas * perturbed_images\n",
        "        elif self.p == 'inf':\n",
        "                out_images = self.clip_image(\n",
        "                        perturbed_images,\n",
        "                        original_image - alphas,\n",
        "                        original_image + alphas\n",
        "                        )\n",
        "                return out_images\n",
        "\n",
        "\n",
        "    def binary_search_batch(self, original_image, perturbed_images, label_or_target, theta):\n",
        "        \"\"\" Binary search to approach the boundary. \"\"\"\n",
        "\n",
        "        # Compute distance between each of perturbed image and original image.\n",
        "        dists_post_update = np.array([\n",
        "                        self.compute_distance(\n",
        "                                original_image,\n",
        "                                perturbed_image\n",
        "                        )\n",
        "                        for perturbed_image in perturbed_images])\n",
        "        #print(dists_post_update)\n",
        "        # Choose upper thresholds in binary searchs based on constraint.\n",
        "        if self.p == 'inf':\n",
        "                highs = dists_post_update\n",
        "                # Stopping criteria.\n",
        "                thresholds = np.minimum(dists_post_update * theta, theta)\n",
        "        else:\n",
        "                highs = np.ones(len(perturbed_images))\n",
        "                thresholds = theta\n",
        "\n",
        "        lows = np.zeros(len(perturbed_images))\n",
        "\n",
        "\n",
        "\n",
        "        # Call recursive function.\n",
        "        while np.max((highs - lows) / thresholds) > 1:\n",
        "                # projection to mids.\n",
        "                mids = (highs + lows) / 2.0\n",
        "                mid_images = self.project(original_image, perturbed_images, mids)\n",
        "                # Update highs and lows based on model decisions.\n",
        "                decisions = self.decision_function(mid_images, label_or_target)\n",
        "                lows = np.where(decisions == 0, mids, lows)\n",
        "                highs = np.where(decisions == 1, mids, highs)\n",
        "                if self.query > self.max_queries:\n",
        "                        break\n",
        "\n",
        "        out_images = self.project(original_image, perturbed_images, highs)\n",
        "\n",
        "        # Compute distance of the output image to select the best choice.\n",
        "        # (only used when stepsize_search is grid_search.)\n",
        "        dists = np.array([\n",
        "                self.compute_distance(\n",
        "                        original_image,\n",
        "                        out_image\n",
        "                )\n",
        "                for out_image in out_images])\n",
        "        idx = np.argmin(dists)\n",
        "\n",
        "        dist = dists_post_update[idx]\n",
        "        out_image = out_images[idx]\n",
        "        return out_image, dist\n",
        "\n",
        "\n",
        "    def initialize(self, input_xi, label_or_target):\n",
        "        \"\"\"\n",
        "        Efficient Implementation of BlendedUniformNoiseAttack in Foolbox.\n",
        "        \"\"\"\n",
        "        success = 0\n",
        "        num_evals = 0\n",
        "        # Find a misclassified random noise.\n",
        "        while True:\n",
        "                random_noise = np.random.uniform(self.lb, self.ub, size = input_xi.shape)\n",
        "                success = self.decision_function(random_noise, label_or_target)[0]\n",
        "                if success:\n",
        "                        break\n",
        "                if self.query > self.max_queries:\n",
        "                        break\n",
        "                assert num_evals < 1e4,\"Initialization failed! \"\n",
        "                \"Use a misclassified image as `target_image`\"\n",
        "\n",
        "        # Binary search to minimize l2 distance to original image.\n",
        "        low = 0.0\n",
        "        high = 1.0\n",
        "        while high - low > 0.001:\n",
        "                mid = (high + low) / 2.0\n",
        "                blended = (1 - mid) * input_xi + mid * random_noise\n",
        "                success = self.decision_function(blended, label_or_target)\n",
        "                if success:\n",
        "                        high = mid\n",
        "                else:\n",
        "                        low = mid\n",
        "                if self.query > self.max_queries:\n",
        "                        break\n",
        "\n",
        "        initialization = (1 - high) * input_xi + high * random_noise\n",
        "\n",
        "        return initialization\n",
        "\n",
        "\n",
        "    def geometric_progression_for_stepsize(self, x, label_or_target, update, dist, j):\n",
        "        \"\"\"\n",
        "        Geometric progression to search for stepsize.\n",
        "        Keep decreasing stepsize by half until reaching\n",
        "        the desired side of the boundary,\n",
        "        \"\"\"\n",
        "        epsilon = dist / np.sqrt(j)\n",
        "\n",
        "        def phi(epsilon):\n",
        "                new = x + epsilon * update\n",
        "                success = self.decision_function(new, label_or_target)\n",
        "                return success\n",
        "\n",
        "        while not phi(epsilon):\n",
        "                epsilon /= 2.0\n",
        "                if self.query > self.max_queries:\n",
        "                        break\n",
        "\n",
        "        return epsilon\n",
        "\n",
        "\n",
        "    def _perturb(self, xs_t, ys_t):\n",
        "        xs = xs_t.cpu().numpy()\n",
        "        ys = ys_t.cpu().numpy()\n",
        "        adv = self.hsja(xs, ys)\n",
        "        return t(adv), self.query\n",
        "\n",
        "'''\n",
        "\n",
        "Original License\n",
        "\n",
        "MIT License\n",
        "\n",
        "Copyright (c) 2022 Minhao Cheng\n",
        "\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE.\n",
        "\n",
        "'''\n"
      ],
      "metadata": {
        "id": "q3jPStDik1RP",
        "outputId": "e6123e52-9c90-4f51-b394-e4f08968ea7e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/attacks/decision/hsja_attack.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile /content/BlackboxBench/attacks/decision/geoda_attack.py\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "import os\n",
        "import time\n",
        "from torch import Tensor as t\n",
        "from numpy import linalg\n",
        "import math\n",
        "\n",
        "from attacks.decision.decision_black_box_attack import DecisionBlackBoxAttack\n",
        "\n",
        "from math import cos, sqrt, pi\n",
        "\n",
        "\n",
        "def dct(x, y, v, u, n):\n",
        "    # Normalisation\n",
        "    def alpha(a):\n",
        "        if a == 0:\n",
        "            return sqrt(1.0 / n)\n",
        "        else:\n",
        "            return sqrt(2.0 / n)\n",
        "\n",
        "    return alpha(u) * alpha(v) * cos(((2 * x + 1) * (u * pi)) / (2 * n)) * cos(((2 * y + 1) * (v * pi)) / (2 * n))\n",
        "\n",
        "\n",
        "def generate_2d_dct_basis(sub_dim, n, path):\n",
        "    # Assume square image, so we don't have different xres and yres\n",
        "\n",
        "    # We can get different frequencies by setting u and v\n",
        "    # Here, we have a max u and v to loop over and display\n",
        "    # Feel free to adjust\n",
        "    maxU = sub_dim\n",
        "    maxV = sub_dim\n",
        "\n",
        "    dct_basis = []\n",
        "    for u in range(0, maxU):\n",
        "        for v in range(0, maxV):\n",
        "            basisImg = np.zeros((n, n))\n",
        "            for y in range(0, n):\n",
        "                for x in range(0, n):\n",
        "                    basisImg[y, x] = dct(x, y, v, u, max(n, maxV))\n",
        "            dct_basis.append(basisImg)\n",
        "    dct_basis = np.mat(np.reshape(dct_basis, (maxV*maxU, n*n))).transpose()\n",
        "    np.save(path, dct_basis)\n",
        "    return dct_basis\n",
        "\n",
        "class SubNoise(torch.nn.Module):\n",
        "    \"\"\"given subspace x and the number of noises, generate sub noises\"\"\"\n",
        "    # x is the subspace basis\n",
        "    def __init__(self, num_noises, x):\n",
        "        self.num_noises = num_noises\n",
        "        self.x = x\n",
        "        self.size = int(self.x.shape[0] ** 0.5)\n",
        "        super(SubNoise, self).__init__()\n",
        "\n",
        "    def forward(self):\n",
        "        r = torch.zeros([self.size ** 2, 3*self.num_noises], dtype=torch.float32)\n",
        "        noise = torch.randn([self.x.shape[1], 3*self.num_noises], dtype=torch.float32).cuda()\n",
        "        sub_noise = torch.transpose(torch.mm(self.x, noise), 0, 1)\n",
        "        r = sub_noise.view([self.num_noises, 3, self.size, self.size])\n",
        "        r_list = r.permute(0,2,3,1)\n",
        "        return r_list\n",
        "\n",
        "class GeoDAttack(DecisionBlackBoxAttack):\n",
        "    \"\"\"\n",
        "    GeoDA\n",
        "    \"\"\"\n",
        "    def __init__(self, epsilon, p, max_queries, sub_dim, tol, alpha, mu, search_space, grad_estimator_batch_size, lb, ub, batch_size, sigma):\n",
        "        super().__init__(max_queries = max_queries,\n",
        "                         epsilon=epsilon,\n",
        "                         p=p,\n",
        "                         lb=lb,\n",
        "                         ub=ub,\n",
        "                         batch_size = batch_size)\n",
        "        self.sub_dim = sub_dim\n",
        "        self.tol = tol\n",
        "        self.alpha = alpha\n",
        "        self.mu = mu\n",
        "        self.search_space = search_space\n",
        "        self.grad_estimator_batch_size = grad_estimator_batch_size\n",
        "        self.sigma = sigma\n",
        "\n",
        "    def _config(self):\n",
        "        return {\n",
        "            \"p\": self.p,\n",
        "            \"epsilon\": self.epsilon,\n",
        "            \"lb\": self.lb,\n",
        "            \"ub\": self.ub,\n",
        "            \"attack_name\": self.__class__.__name__\n",
        "        }\n",
        "\n",
        "    def opt_query_iteration(self, Nq, T, eta):\n",
        "        coefs=[eta**(-2*i/3) for i in range(0,T)]\n",
        "        coefs[0] = 1*coefs[0]\n",
        "        sum_coefs = sum(coefs)\n",
        "        opt_q=[round(Nq*coefs[i]/sum_coefs) for i in range(0,T)]\n",
        "        if opt_q[0]>80:\n",
        "            T = T + 1\n",
        "            opt_q, T = self.opt_query_iteration(Nq, T, eta)\n",
        "        elif opt_q[0]<50:\n",
        "            T = T - 1\n",
        "            opt_q, T = self.opt_query_iteration(Nq, T, eta)\n",
        "\n",
        "        return opt_q, T\n",
        "\n",
        "    def find_random_adversarial(self, x0, y0, epsilon=1000):\n",
        "\n",
        "        num_calls = 1\n",
        "\n",
        "        step = 0.02\n",
        "        perturbed = x0\n",
        "\n",
        "        while self.predict_label(perturbed) == y0[0]:\n",
        "            pert = torch.randn(x0.shape)\n",
        "\n",
        "            perturbed = x0 + num_calls*step* pert\n",
        "            perturbed = torch.clamp(perturbed, 0, 1)\n",
        "            num_calls += 1\n",
        "\n",
        "        return perturbed, num_calls\n",
        "\n",
        "    def bin_search(self, x_0, y_0, x_random, tol):\n",
        "        num_calls = 0\n",
        "        adv = x_random\n",
        "        cln = x_0\n",
        "        while True:\n",
        "\n",
        "            mid = (cln + adv) / 2.0\n",
        "            num_calls += 1\n",
        "\n",
        "            if self.predict_label(mid) != y_0[0]:\n",
        "                adv = mid\n",
        "            else:\n",
        "                cln = mid\n",
        "\n",
        "            if torch.norm(adv-cln)<tol:\n",
        "                break\n",
        "\n",
        "        return adv, num_calls\n",
        "\n",
        "    def go_to_boundary(self, x_0, y_0, grad):\n",
        "\n",
        "        epsilon = 1\n",
        "\n",
        "        num_calls = 1\n",
        "        perturbed = x_0\n",
        "\n",
        "        if self.p == '2':\n",
        "            grads = grad\n",
        "        elif self.p == 'inf':\n",
        "            grads = torch.sign(grad)/torch.norm(grad)\n",
        "\n",
        "        while self.predict_label(perturbed) == y_0[0]:\n",
        "            perturbed = x_0 + (num_calls*epsilon*grads[0])\n",
        "            perturbed = torch.clamp(perturbed, 0, 1)\n",
        "\n",
        "            num_calls += 1\n",
        "\n",
        "            if num_calls > 100:\n",
        "                print('falied ... ')\n",
        "                break\n",
        "        return perturbed, num_calls\n",
        "\n",
        "    def black_grad_batch(self, x_boundary, q_max, random_noises, y_0, sub_basis_torch):\n",
        "        grad_tmp = [] # estimated gradients in each estimate_batch\n",
        "        z = []        # sign of grad_tmp\n",
        "        outs = []\n",
        "        batch_size = self.grad_estimator_batch_size\n",
        "\n",
        "        num_batchs = math.ceil(q_max/batch_size)\n",
        "        last_batch = q_max - (num_batchs-1)*batch_size\n",
        "        EstNoise = SubNoise(batch_size, sub_basis_torch)\n",
        "        all_noises = []\n",
        "\n",
        "        for j in range(num_batchs):\n",
        "            if j == num_batchs-1:\n",
        "                EstNoise_last = SubNoise(last_batch, sub_basis_torch)\n",
        "                current_batch = EstNoise_last()\n",
        "                current_batch_np = current_batch.cpu().numpy()\n",
        "                noisy_boundary = [x_boundary[0,:,:,:].cpu().numpy()]*last_batch +self.alpha*current_batch.cpu().numpy()\n",
        "\n",
        "            else:\n",
        "                current_batch = EstNoise()\n",
        "                current_batch_np = current_batch.cpu().numpy()\n",
        "                noisy_boundary = [x_boundary[0,:,:,:].cpu().numpy()]*batch_size +self.alpha*current_batch.cpu().numpy()\n",
        "\n",
        "            all_noises.append(current_batch_np)\n",
        "\n",
        "            noisy_boundary_tensor = torch.tensor(noisy_boundary)\n",
        "\n",
        "            predict_labels = self.predict_label(noisy_boundary_tensor)\n",
        "\n",
        "            outs.append(predict_labels.cpu())\n",
        "\n",
        "        all_noise = np.concatenate(all_noises, axis=0)\n",
        "        outs = np.concatenate(outs, axis=0)\n",
        "\n",
        "\n",
        "        for i, predict_label in enumerate(outs):\n",
        "            if predict_label == y_0:\n",
        "                z.append(1)\n",
        "                grad_tmp.append(all_noise[i])\n",
        "            else:\n",
        "                z.append(-1)\n",
        "                grad_tmp.append(-all_noise[i])\n",
        "\n",
        "        grad = -(1/q_max)*sum(grad_tmp)\n",
        "\n",
        "        grad_f = torch.tensor(grad)[None, :,:,:]\n",
        "\n",
        "        return grad_f, sum(z)\n",
        "\n",
        "\n",
        "\n",
        "    def GeoDA(self, x_0, y_0, x_b, iteration, q_opt, sub_basis_torch):\n",
        "        q_num = 0\n",
        "        grad = 0\n",
        "\n",
        "        for i in range(iteration):\n",
        "\n",
        "            t1 = time.time()\n",
        "            random_vec_o = torch.randn(q_opt[i], x_0.shape[1], x_0.shape[2], x_0.shape[3])\n",
        "\n",
        "            grad_oi, _ = self.black_grad_batch(x_b, q_opt[i], random_vec_o, y_0[0], sub_basis_torch)\n",
        "            q_num = q_num + q_opt[i]\n",
        "\n",
        "            grad = grad_oi + grad\n",
        "            x_adv, qs = self.go_to_boundary(x_0, y_0, grad)\n",
        "            q_num = q_num + qs\n",
        "\n",
        "            x_adv, bin_query = self.bin_search(x_0, y_0, x_adv, self.tol)\n",
        "            q_num = q_num + bin_query\n",
        "\n",
        "            x_b = x_adv\n",
        "\n",
        "            t2 = time.time()\n",
        "\n",
        "            norm = self.distance(x_adv, x_0)\n",
        "            message = ' (took {:.5f} seconds)'.format(t2 - t1)\n",
        "            print('iteration -> ' + str(i) + str(message) + '     -- ' + self.p + ' norm is -> ' + str(norm))\n",
        "            if norm < self.epsilon:\n",
        "                break\n",
        "            if q_num > self.max_queries:\n",
        "                break\n",
        "\n",
        "\n",
        "        x_adv = torch.clamp(x_adv, 0, 1)\n",
        "\n",
        "        return x_adv, q_num, grad\n",
        "\n",
        "    def _perturb(self, xs_t, ys):\n",
        "\n",
        "        if self.search_space == 'sub':\n",
        "            print('Check if DCT basis available ...')\n",
        "\n",
        "            path = os.path.join(os.path.dirname(__file__), '2d_dct_basis_{}_{}.npy'.format(self.sub_dim,xs_t.size(2)))\n",
        "            if os.path.exists(path):\n",
        "                print('Yes, we already have it ...')\n",
        "                sub_basis = np.load(path).astype(np.float32)\n",
        "            else:\n",
        "                print('Generating dct basis ......')\n",
        "                sub_basis = generate_2d_dct_basis(self.sub_dim, xs_t.size(2), path).astype(np.float32)\n",
        "                print('Done!\\n')\n",
        "\n",
        "\n",
        "            sub_basis_torch = torch.from_numpy(sub_basis).cuda()\n",
        "\n",
        "\n",
        "        x_random, query_random_1 = self.find_random_adversarial(xs_t, ys, epsilon=100)\n",
        "\n",
        "        # Binary search\n",
        "\n",
        "        x_boundary, query_binsearch_2 = self.bin_search(xs_t, ys, x_random, self.tol)\n",
        "        x_b = x_boundary\n",
        "\n",
        "\n",
        "        query_rnd = query_binsearch_2 + query_random_1\n",
        "\n",
        "\n",
        "        iteration = round(self.max_queries/500)\n",
        "        q_opt_it = int(self.max_queries  - (iteration)*25)\n",
        "        q_opt_iter, iterate = self.opt_query_iteration(q_opt_it, iteration, self.mu)\n",
        "        q_opt_it = int(self.max_queries  - (iterate)*25)\n",
        "        q_opt_iter, iterate = self.opt_query_iteration(q_opt_it, iteration, self.mu)\n",
        "        print('Start: The GeoDA will be run for:' + ' Iterations = ' + str(iterate) + ', Query = ' + str(self.max_queries) + ', Norm = ' + str(self.p)+ ', Space = ' + str(self.search_space) )\n",
        "\n",
        "\n",
        "        t3 = time.time()\n",
        "        adv, query_o, _= self.GeoDA(xs_t, ys, x_b, iterate, q_opt_iter, sub_basis_torch)\n",
        "        t4 = time.time()\n",
        "        message = ' took {:.5f} seconds'.format(t4 - t3)\n",
        "        qmessage = ' with query = ' + str(query_o + query_rnd)\n",
        "\n",
        "\n",
        "        print('End: The GeoDA algorithm' + message + qmessage )\n",
        "\n",
        "\n",
        "        return adv, query_o + query_rnd\n"
      ],
      "metadata": {
        "id": "zniDyI2Zk703",
        "outputId": "a93d15de-e731-4839-d6b4-eb2c691f303f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting /content/BlackboxBench/attacks/decision/geoda_attack.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OfpQ3lwi_Cns",
        "outputId": "da5c9e21-8fa2-4dbb-cfa2-61293709328d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Writing /content/BlackboxBench/BaselinesExperiment.py\n"
          ]
        }
      ],
      "source": [
        "%%writefile /content/BlackboxBench/BaselinesExperiment.py\n",
        "from __future__ import absolute_import\n",
        "from __future__ import division\n",
        "from __future__ import print_function\n",
        "\n",
        "import json\n",
        "import math\n",
        "import os\n",
        "import time\n",
        "import sys\n",
        "sys.path.append(os.path.join(os.path.dirname(os.path.realpath(__file__)), \"..\"))\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "# import torch as ch\n",
        "import torch\n",
        "from advertorch.defenses import *\n",
        "from datasets.dataset import Dataset\n",
        "from utils.compute import tf_nsign, sign\n",
        "from utils.misc import config_path_join, src_path_join, create_dir, get_dataset_shape\n",
        "from utils.model_loader import load_torch_models, load_torch_models_imagesub\n",
        "from utils.compute import tf_nsign, sign, linf_proj_maker, l2_proj_maker\n",
        "from art.defences.preprocessor import Preprocessor, GaussianAugmentation, FeatureSqueezing\n",
        "import torch.nn as nn\n",
        "from attacks.score.bandit_attack import BanditAttack\n",
        "from attacks.score.zo_sign_sgd_attack import ZOSignSGDAttack\n",
        "from attacks.score.sign_attack import SignAttack\n",
        "from attacks.score.simple_attack import SimpleAttack\n",
        "from attacks.score.square_attack import SquareAttack\n",
        "from attacks.score.parsimonious_attack import ParsimoniousAttack\n",
        "from attacks.score.nes_attack import NESAttack\n",
        "from attacks.decision.sign_opt_attack import SignOPTAttack\n",
        "from attacks.decision.hsja_attack import HSJAttack\n",
        "from attacks.decision.geoda_attack import GeoDAttack\n",
        "import ast\n",
        "\n",
        "def Counter_Samples(iters,model, gaus, x_preprocessed, k):\n",
        "    import torch\n",
        "    # Apply noise first\n",
        "    use_cuda = True\n",
        "    device = torch.device(\"cuda\" if (use_cuda and torch.cuda.is_available()) else \"cpu\")\n",
        "    x_preprocessed = torch.from_numpy(gaus(x_preprocessed.detach().cpu().numpy())[0]).to(device)\n",
        "    x_preprocessed.requires_grad_(True)\n",
        "    x_preprocessed.retain_grad()\n",
        "    loss = nn.CrossEntropyLoss(reduction='none')\n",
        "    for iter in range(iters):\n",
        "        # predicting labels\n",
        "        model_output = model(x_preprocessed)\n",
        "        true_labels_indexes = torch.argmax(model_output, dim=1)\n",
        "        loss_comp = loss(model_output, true_labels_indexes)\n",
        "        loss_comp.backward(torch.ones_like(loss_comp))\n",
        "        # update the samples.\n",
        "        x_preprocessed = x_preprocessed - k * x_preprocessed.grad # No normalization.\n",
        "        x_preprocessed.retain_grad()\n",
        "    return x_preprocessed\n",
        "\n",
        "\n",
        "if __name__ == '__main__':\n",
        "\n",
        "\tprint(\"Running Baselines Experiment..\")\n",
        "\tversion_dict = {\n",
        "        'RND': False,\n",
        "        'CounterSamples_10': False,\n",
        "        'Counter_Samples_1': False,\n",
        "        'SND': False,\n",
        "        'FS': False,\n",
        "        'BS': False,\n",
        "        'AS': False,\n",
        "        'JPEG': False\n",
        "    }\n",
        "\tdevice = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "\tbaselines_json = os.sys.argv[2]\n",
        "\tdata_received = json.loads(baselines_json)\n",
        "\n",
        "\tbaselines = data_received[\"Baselines\"]\n",
        "\tprint(baselines)\n",
        "\tfor version in baselines:\n",
        "\t\tversion_dict[version] = True\n",
        "\n",
        "\t\t# Defense Selection\n",
        "\t\tFG = version_dict['Counter_Samples_1']\n",
        "\t\tPGD_FG = version_dict['CounterSamples_10']\n",
        "\t\tRND = version_dict['RND']\n",
        "\t\tgaussian_noise = version_dict['SND']\n",
        "\t\tfeature_squeezing = version_dict['FS']\n",
        "\t\tbit_squeezing = version_dict['BS']\n",
        "\t\tjpeg_compression = version_dict['JPEG']\n",
        "\t\taverage_smoothing = version_dict['AS']\n",
        "\n",
        "\n",
        "\t\tgaus = GaussianAugmentation(augmentation=False, sigma=0.01)\n",
        "\t\tbs = BitSqueezing(bit_depth=3)\n",
        "\t\tjc = JPEGFilter(quality=75)\n",
        "\t\tcs = AverageSmoothing2D(kernel_size=3, channels=3).to(device)\n",
        "\t\tfs = FeatureSqueezing(clip_values=(0,1),bit_depth = 3)\n",
        "\t\tconfig = os.sys.argv[1]\n",
        "\t\texp_id = config.split('/')[-1]\n",
        "\t\tres = {}\n",
        "\t\tcfs = [config]\n",
        "\t\tfor _cf in cfs:\n",
        "\t\t\tconfig_file = config_path_join(_cf)\n",
        "\n",
        "\t\t\twith open(config_file) as config_file:\n",
        "\t\t\t\tconfig = json.load(config_file)\n",
        "\n",
        "\t\t\tpgd_iters = config['defence_config']['k']\n",
        "\t\t\tstep_size_k = config['defence_config']['alpha']\n",
        "\n",
        "\t\t\t# for reproducibility\n",
        "\t\t\tnname = config['attack_name']\n",
        "\t\t\tprint(f'Now running the experiment on :\\n Attack: {nname} \\n Defence :{version}')\n",
        "\t\t\tdata_dir = f'Results/{nname}/{version}'\n",
        "\t\t\tcreate_dir(data_dir)\n",
        "\t\t\tprint(f'creating folder {data_dir}')\n",
        "\t\t\tseed = config['seed']\n",
        "\t\t\tnp.random.seed(seed)\n",
        "\t\t\ttorch.manual_seed(seed)\n",
        "\t\t\ttorch.cuda.manual_seed(seed)\n",
        "\n",
        "\t\t\tdset = Dataset(config['dset_name'], config)\n",
        "\n",
        "\t\t\tmodel_name = config['modeln']\n",
        "\t\t\tif config['dset_name'] == 'imagenet':\n",
        "\t\t\t\tmodel = load_torch_models_imagesub(model_name)\n",
        "\t\t\telse:\n",
        "\t\t\t\tmodel = load_torch_models(model_name)\n",
        "\n",
        "\t\t\tprint('The Black-Box model: {}'.format(config['modeln']))\n",
        "\n",
        "\n",
        "\t\t\tp_norm = config['attack_config']['p']\n",
        "\t\t\tprint(\"The attack norm constrain is: {} norm\".format(p_norm))\n",
        "\t\t\tepsilon = config['attack_config']['epsilon'] / 255.\n",
        "\n",
        "\t\t\t# set torch default device:\n",
        "\t\t\tif torch.cuda.is_available():\n",
        "\t\t\t\ttorch.set_default_tensor_type('torch.cuda.FloatTensor')\n",
        "\t\t\telse:\n",
        "\t\t\t\ttorch.set_default_tensor_type('torch.FloatTensor')\n",
        "\n",
        "\t\t\tdef cw_loss(logit, label, target=False):\n",
        "\t\t\t\tif target:\n",
        "\t\t\t\t\t# targeted cw loss: logit_t - max_{i\\neq t}logit_i\n",
        "\t\t\t\t\t_, argsort = logit.sort(dim=1, descending=True)\n",
        "\t\t\t\t\ttarget_is_max = argsort[:, 0].eq(label)\n",
        "\t\t\t\t\tsecond_max_index = target_is_max.long() * argsort[:, 1] + (~ target_is_max).long() * argsort[:, 0]\n",
        "\t\t\t\t\ttarget_logit = logit[torch.arange(logit.shape[0]), label]\n",
        "\t\t\t\t\tsecond_max_logit = logit[torch.arange(logit.shape[0]), second_max_index]\n",
        "\t\t\t\t\treturn target_logit - second_max_logit\n",
        "\t\t\t\telse:\n",
        "\t\t\t\t\t# untargeted cw loss: max_{i\\neq y}logit_i - logit_y\n",
        "\t\t\t\t\t_, argsort = logit.sort(dim=1, descending=True)\n",
        "\t\t\t\t\tgt_is_max = argsort[:, 0].eq(label)\n",
        "\t\t\t\t\tsecond_max_index = gt_is_max.long() * argsort[:, 1] + (~gt_is_max).long() * argsort[:, 0]\n",
        "\t\t\t\t\tgt_logit = logit[torch.arange(logit.shape[0]), label]\n",
        "\t\t\t\t\tsecond_max_logit = logit[torch.arange(logit.shape[0]), second_max_index]\n",
        "\t\t\t\t\treturn second_max_logit - gt_logit\n",
        "\n",
        "\t\t\tdef xent_loss(logit, label, target=False):\n",
        "\t\t\t\tif not target:\n",
        "\t\t\t\t\treturn torch.nn.CrossEntropyLoss(reduction='none')(logit, label)\n",
        "\t\t\t\telse:\n",
        "\t\t\t\t\treturn -torch.nn.CrossEntropyLoss(reduction='none')(logit, label)\n",
        "\n",
        "\t\t\t# criterion = torch.nn.CrossEntropyLoss(reduce=False)\n",
        "\t\t\t# criterion = xent_loss\n",
        "\t\t\tcriterion = cw_loss\n",
        "\n",
        "\n",
        "\t\t\tattacker = eval(config['attack_name'])(\n",
        "\t\t\t\t**config['attack_config'],\n",
        "\t\t\t\tlb=dset.min_value,\n",
        "\t\t\t\tub=dset.max_value\n",
        "\t\t\t)\n",
        "\n",
        "\t\t\tprint(attacker._config())\n",
        "\n",
        "\t\t\ttarget = config[\"target\"]\n",
        "\n",
        "\n",
        "\t\t\t# Iterate over the samples batch-by-batch\n",
        "\t\t\tnum_eval_examples = config['num_eval_examples']\n",
        "\t\t\teval_batch_size = config['attack_config']['batch_size']\n",
        "\t\t\tnum_batches = int(math.ceil(num_eval_examples / eval_batch_size))\n",
        "\n",
        "\n",
        "\t\t\tprint('Iterating over {} batches'.format(num_batches))\n",
        "\t\t\tstart_time = time.time()\n",
        "\t\t\tdef activate_defense(model, data, k=step_size_k, pgd_iters=pgd_iters):\n",
        "\t\t\t\t# k = step_size_k\n",
        "\t\t\t\tif RND:\n",
        "\t\t\t\t\tsigma = 0.02\n",
        "\t\t\t\t\tdata = data + sigma * torch.randn_like(data)\n",
        "\t\t\t\tif PGD_FG:\n",
        "\t\t\t\t\tdata = Counter_Samples(pgd_iters, model, gaus, data, k)\n",
        "\t\t\t\tif FG:\n",
        "\t\t\t\t\tdata = Counter_Samples(1, model, gaus, data, k)\n",
        "\t\t\t\tif gaussian_noise:\n",
        "\t\t\t\t\tdata = torch.from_numpy(gaus(data.detach().cpu().numpy())[0]).to(device)\n",
        "\t\t\t\tif feature_squeezing:\n",
        "\t\t\t\t\tdata = torch.tensor(fs(data.detach().cpu().numpy())[0]).to(device)\n",
        "\t\t\t\tif bit_squeezing:\n",
        "\t\t\t\t\t# bs = BitSqueezing(bit_depth=3)\n",
        "\t\t\t\t\tdata = bs(data)\n",
        "\t\t\t\tif jpeg_compression:\n",
        "\t\t\t\t\t# jc = JPEGFilter(quality=75)\n",
        "\t\t\t\t\tdata = jc(data).to(device)\n",
        "\t\t\t\tif average_smoothing:\n",
        "\t\t\t\t\t# cs = AverageSmoothing2D(kernel_size=3,channels=3)\n",
        "\t\t\t\t\tdata = cs(data).to(device)\n",
        "\n",
        "\t\t\t\treturn data\n",
        "\n",
        "\t\t\tdef get_clean_acc(model,data,y,k=step_size_k):\n",
        "\t\t\t\tdata = activate_defense(model,data,k=step_size_k,pgd_iters=pgd_iters)\n",
        "\t\t\t\tcorrect_classifcation = (torch.argmax(model(data), dim=1) == y).sum()\n",
        "\t\t\t\treturn correct_classifcation\n",
        "\t\t\tclean_acc = 0\n",
        "\n",
        "\t\t\tfor ibatch in range(num_batches):\n",
        "\t\t\t\tbstart = ibatch * eval_batch_size\n",
        "\t\t\t\tbend = min(bstart + eval_batch_size, num_eval_examples)\n",
        "\t\t\t\tprint('batch size: {}: ({}, {})'.format(bend - bstart, bstart, bend))\n",
        "\n",
        "\t\t\t\tx_batch, y_batch = dset.get_eval_data(bstart, bend)\n",
        "\t\t\t\ty_batch = torch.LongTensor(y_batch).to(device)\n",
        "\t\t\t\tx_ori = torch.FloatTensor(x_batch.copy().transpose(0,3,1,2) / 255.).to(device)\n",
        "\t\t\t\tclean_acc += get_clean_acc(model, x_ori, y_batch)\n",
        "\n",
        "\t\t\t\tif p_norm == 'inf':\n",
        "\t\t\t\t\tpass\n",
        "\t\t\t\telse:\n",
        "\t\t\t\t\tproj_2 = l2_proj_maker(x_ori, epsilon)\n",
        "\n",
        "\t\t\t\tdef get_label(target_type):\n",
        "\t\t\t\t\t_, logit = model(torch.FloatTensor(x_batch.transpose(0,3,1,2) / 255.))\n",
        "\t\t\t\t\tif target_type == 'random':\n",
        "\t\t\t\t\t\tlabel = torch.randint(low=0, high=logit.shape[1], size=label.shape).long().to(device)\n",
        "\t\t\t\t\telif target_type == 'least_likely':\n",
        "\t\t\t\t\t\tlabel = logit.argmin(dim=1)\n",
        "\t\t\t\t\telif target_type == 'most_likely':\n",
        "\t\t\t\t\t\tlabel = torch.argsort(logit, dim=1,descending=True)[:,1]\n",
        "\t\t\t\t\telif target_type == 'median':\n",
        "\t\t\t\t\t\tlabel = torch.argsort(logit, dim=1,descending=True)[:,4]\n",
        "\t\t\t\t\telif 'label' in target_type:\n",
        "\t\t\t\t\t\tlabel = torch.ones_like(y_batch) * int(target_type[5:])\n",
        "\t\t\t\t\treturn label.detach()\n",
        "\n",
        "\t\t\t\tif target:\n",
        "\t\t\t\t\ty_batch = get_label(config[\"target_type\"])\n",
        "\n",
        "\t\t\t\tif config['attack_name'] in [\"SignOPTAttack\",\"HSJAttack\",\"GeoDAttack\",\"OptAttack\",\"EvolutionaryAttack\",\n",
        "\t\t\t\t\t\t\t\t\t\t\t\"SignFlipAttack\",\"RaySAttack\",\"BoundaryAttack\"]:\n",
        "\t\t\t\t\tlogs_dict = attacker.run(x_batch, y_batch, model, target, dset,activate_defense)\n",
        "\n",
        "\t\t\t\telse:\n",
        "\t\t\t\t\tdef loss_fct(xs, es = False):\n",
        "\t\t\t\t\t\tif type(xs) is torch.Tensor:\n",
        "\t\t\t\t\t\t\tx_eval = (xs.permute(0,3,1,2)/ 255.).to(device)\n",
        "\t\t\t\t\t\telse:\n",
        "\t\t\t\t\t\t\tx_eval = (torch.FloatTensor(xs.transpose(0,3,1,2))/ 255.).to(device)\n",
        "\n",
        "\t\t\t\t\t\tif p_norm == 'inf':\n",
        "\t\t\t\t\t\t\tx_eval = torch.clamp(x_eval - x_ori, -epsilon, epsilon) + x_ori\n",
        "\t\t\t\t\t\telse:\n",
        "\t\t\t\t\t\t\t# proj_2 = l2_proj_maker(x_ori, epsilon)\n",
        "\t\t\t\t\t\t\tx_eval = proj_2(x_eval)\n",
        "\t\t\t\t\t\tx_eval = torch.clamp(x_eval, 0, 1)\n",
        "\t\t\t\t\t\tx_eval = activate_defense(model, x_eval,k= step_size_k)\n",
        "\t\t\t\t\t\ty_logit = model(x_eval.to(device))\n",
        "\t\t\t\t\t\tloss = criterion(y_logit, y_batch, target)\n",
        "\t\t\t\t\t\tif es:\n",
        "\t\t\t\t\t\t\ty_logit = y_logit.detach()\n",
        "\t\t\t\t\t\t\tcorrect = torch.argmax(y_logit, axis=1) == y_batch\n",
        "\t\t\t\t\t\t\tif target:\n",
        "\t\t\t\t\t\t\t\treturn correct, loss.detach()\n",
        "\t\t\t\t\t\t\telse:\n",
        "\t\t\t\t\t\t\t\treturn ~correct, loss.detach()\n",
        "\t\t\t\t\t\telse:\n",
        "\t\t\t\t\t\t\treturn loss.detach()\n",
        "\n",
        "\t\t\t\t\tdef early_stop_crit_fct(xs):\n",
        "\n",
        "\t\t\t\t\t\tif type(xs) is torch.Tensor:\n",
        "\t\t\t\t\t\t\tx_eval = xs.permute(0,3,1,2)/ 255.\n",
        "\t\t\t\t\t\telse:\n",
        "\t\t\t\t\t\t\tx_eval = torch.FloatTensor(xs.transpose(0,3,1,2))/ 255.\n",
        "\t\t\t\t\t\tx_eval = torch.clamp(x_eval, 0, 1)\n",
        "\t\t\t\t\t\tx_eval = activate_defense(model, x_eval, k= step_size_k)\n",
        "\t\t\t\t\t\ty_logit = model(x_eval.to(device))\n",
        "\t\t\t\t\t\ty_logit = y_logit.detach()\n",
        "\t\t\t\t\t\tcorrect = torch.argmax(y_logit, axis=1) == y_batch\n",
        "\n",
        "\t\t\t\t\t\tif target:\n",
        "\t\t\t\t\t\t\treturn correct\n",
        "\t\t\t\t\t\telse:\n",
        "\t\t\t\t\t\t\treturn ~correct\n",
        "\n",
        "\t\t\t\t\tlogs_dict = attacker.run(x_batch, loss_fct, early_stop_crit_fct)\n",
        "\n",
        "\t\t\t\tress = attacker.result()\n",
        "\t\t\t\tprint('The Black-Box model: {}'.format(config['modeln']))\n",
        "\t\t\tversion_dict[version] = False\n",
        "\n",
        "\t\t\tprint(\"Batches done after {} s\".format(time.time() - start_time))\n",
        "\t\t\tress['Clean Accuracy'] = (ress['total_successes'] + ress['total_failures']) / num_eval_examples\n",
        "\t\t\tif config['dset_name'] not in res:\n",
        "\t\t\t\tres[config['dset_name']] = [ress]\n",
        "\t\t\telse:\n",
        "\t\t\t\tres[config['dset_name']].append(ress)\n",
        "\n",
        "\t\t\t### Only For Small experiments.\n",
        "\t\t\tres_fname = data_dir + '/' +  'k_{}_res.json'.format(step_size_k)\n",
        "\t\t\tprint(\"Storing tabular data in {}\".format(res_fname))\n",
        "\t\t\twith open(res_fname, 'w') as f:\n",
        "\t\t\t\tjson.dump(res, f, indent=4, sort_keys=True)\n",
        "'''\n",
        "\n",
        "MIT License\n",
        "Copyright (c) 2019 Abdullah Al-Dujaili\n",
        "Permission is hereby granted, free of charge, to any person obtaining a copy\n",
        "of this software and associated documentation files (the \"Software\"), to deal\n",
        "in the Software without restriction, including without limitation the rights\n",
        "to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
        "copies of the Software, and to permit persons to whom the Software is\n",
        "furnished to do so, subject to the following conditions:\n",
        "The above copyright notice and this permission notice shall be included in all\n",
        "copies or substantial portions of the Software.\n",
        "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
        "AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
        "OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
        "SOFTWARE.\n",
        "\n",
        "'''\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F7Z_NAEhslp2"
      },
      "source": [
        "# Hyperparameters Configuration\n",
        "Change here the Hyper parameters as you like. <br>\n",
        "\n",
        "*Uncomment the correpsonding json configuration for the attack you want to run."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i88wiMKQrGEs"
      },
      "outputs": [],
      "source": [
        "## NES\n",
        "\n",
        "# attack_config_json = {\n",
        "#   \"_comment1\": \"===== DATASET CONFIGURATION =====\",\n",
        "#   'name_of_file' : 'nes',\n",
        "#   \"dset_name\": \"cifar10\",\n",
        "#   \"_comment2\": \"===== EVAL CONFIGURATION =====\",\n",
        "#   \"num_eval_examples\": 100,\n",
        "#   \"_comment3\": \"=====ADVERSARIAL EXAMPLES CONFIGURATION=====\",\n",
        "#   \"attack_name\": \"NESAttack\",\n",
        "#   \"attack_config\": {\n",
        "#     \"batch_size\" : 100,\n",
        "#     \"name\": \"NES\",\n",
        "#     \"epsilon\": 12.75,\n",
        "#     \"p\": \"inf\",\n",
        "#     \"fd_eta\": 2.55,\n",
        "#     \"lr\": 2.55,\n",
        "#     \"q\": 15,\n",
        "#     \"max_loss_queries\": 10 # The attacker's queries budget.\n",
        "#   },\n",
        "#     \"_comment4\": \"=====OUR METHOD CONFIGURATION=====\",\n",
        "#   \"defence_config\":{\n",
        "#       \"alpha\" : 0.03, # The step size\n",
        "#       \"k\" : 10 #Number of iterations.\n",
        "#   },\n",
        "#   \"modeln\": \"resnet20\",\n",
        "#   \"target\": False,\n",
        "#   \"target_type\": \"median\",\n",
        "#   \"seed\": 12345\n",
        "# }\n",
        "\n",
        "## Bandit\n",
        "\n",
        "attack_config_json ={\n",
        "  \"_comment1\": \"===== DATASET CONFIGURATION =====\",\n",
        "  'name_of_file' : 'bandit',\n",
        "  \"dset_name\": \"cifar10\",\n",
        "  \"_comment2\": \"===== EVAL CONFIGURATION =====\",\n",
        "  \"num_eval_examples\": 100,\n",
        "  \"_comment3\": \"=====ADVERSARIAL EXAMPLES CONFIGURATION=====\",\n",
        "  \"attack_name\": \"BanditAttack\",\n",
        "  \"attack_config\": {\n",
        "    \"batch_size\" :100,\n",
        "    \"name\": \"Bandit\",\n",
        "    \"epsilon\": 12.75,\n",
        "    \"p\": \"inf\",\n",
        "    \"lr\": 2.55,\n",
        "    \"fd_eta\": 2.55,\n",
        "    \"prior_lr\": 0.1,\n",
        "    \"prior_size\": 20,\n",
        "    \"data_size\": 32,\n",
        "    \"prior_exploration\": 0.1,\n",
        "    \"max_loss_queries\": 10000\n",
        "  },\n",
        "  \"_comment4\": \"=====OUR METHOD CONFIGURATION=====\",\n",
        "  \"defence_config\":{\n",
        "      \"alpha\" : 0.03, # The step size\n",
        "      \"k\" : 10 #Number of iterations.\n",
        "  },\n",
        "  \"modeln\": \"resnet20\",\n",
        "  \"target\": False,\n",
        "  \"target_type\": \"median\",\n",
        "  \"seed\": 123\n",
        "}\n",
        "\n",
        "## GeoD\n",
        "\n",
        "# attack_config_json ={\n",
        "#   \"_comment1\": \"===== DATASET CONFIGURATION =====\",\n",
        "#   'name_of_file' : 'geoda',\n",
        "#   \"dset_name\": \"cifar10\",\n",
        "#   \"_comment2\": \"===== EVAL CONFIGURATION =====\",\n",
        "#   \"num_eval_examples\": 100,\n",
        "#   \"_comment3\": \"=====ADVERSARIAL EXAMPLES CONFIGURATION=====\",\n",
        "#   \"attack_name\": \"GeoDAttack\",\n",
        "#   \"attack_config\": {\n",
        "#     \"batch_size\": 1,\n",
        "#     \"epsilon\": 12.75,\n",
        "#     \"p\": \"inf\",\n",
        "#     \"max_queries\": 10000,\n",
        "#     \"sub_dim\": 10,\n",
        "#     \"tol\": 0.0001,\n",
        "#     \"alpha\": 0.0002,\n",
        "#     \"mu\": 0.6,\n",
        "#     \"search_space\": \"sub\",\n",
        "#     \"grad_estimator_batch_size\": 40,\n",
        "#     \"sigma\": 0\n",
        "#   },\n",
        "#   \"_comment4\": \"=====OUR METHOD CONFIGURATION=====\",\n",
        "#   \"defence_config\":{\n",
        "#       \"alpha\" : 0.03, # The step size\n",
        "#       \"k\" : 10 #Number of iterations.\n",
        "#   },\n",
        "#   \"modeln\": \"resnet20\",\n",
        "#   \"target\": False,\n",
        "#   \"target_type\": \"median\",\n",
        "#   \"seed\":123\n",
        "# }\n",
        "\n",
        "## HopSkipJump (HSJ)\n",
        "\n",
        "# attack_config_json = {\n",
        "#   \"_comment1\": \"===== DATASET CONFIGURATION =====\",\n",
        "#   'name_of_file' : 'hsja',\n",
        "#   \"dset_name\": \"cifar10\",\n",
        "#   \"_comment2\": \"===== EVAL CONFIGURATION =====\",\n",
        "#   \"num_eval_examples\": 100,\n",
        "#   \"_comment3\": \"=====ADVERSARIAL EXAMPLES CONFIGURATION=====\",\n",
        "#   \"attack_name\": \"HSJAttack\",\n",
        "#   \"attack_config\": {\n",
        "#     \"batch_size\": 1,\n",
        "#     \"epsilon\": 12.75,\n",
        "#     \"p\": \"inf\",\n",
        "#     \"max_queries\": 10000,\n",
        "#     \"gamma\": 1.0,\n",
        "#     \"stepsize_search\": \"geometric_progression\",\n",
        "#     \"max_num_evals\": 1000,\n",
        "#     \"init_num_evals\": 1,\n",
        "#     \"EOT\": 0,\n",
        "#     \"sigma\": 0\n",
        "#   },\n",
        "#   \"_comment4\": \"=====OUR METHOD CONFIGURATION=====\",\n",
        "#   \"defence_config\":{\n",
        "#       \"alpha\" : 0.03, # The step size\n",
        "#       \"k\" : 10 #Number of iterations.\n",
        "#   },\n",
        "#   \"modeln\": \"resnet20\",\n",
        "#   \"target\": False,\n",
        "#   \"target_type\": \"median\",\n",
        "#   \"seed\":123\n",
        "# }\n",
        "\n",
        "## ECO\n",
        "\n",
        "# attack_config_json = {\n",
        "#   \"_comment1\": \"===== DATASET CONFIGURATION =====\",\n",
        "#   'name_of_file' : 'parm',\n",
        "#   \"dset_name\": \"cifar10\",\n",
        "#   \"_comment2\": \"===== EVAL CONFIGURATION =====\",\n",
        "#   \"num_eval_examples\": 100,\n",
        "#   \"_comment3\": \"=====ADVERSARIAL EXAMPLES CONFIGURATION=====\",\n",
        "#   \"attack_name\": \"ParsimoniousAttack\",\n",
        "#   \"attack_config\": {\n",
        "#     \"batch_size\" : 1,\n",
        "#     \"name\": \"ECO\",\n",
        "#     \"epsilon\": 12.75,\n",
        "#     \"p\": \"inf\",\n",
        "#     \"max_loss_queries\": 10000,\n",
        "#     \"EOT\": 1,\n",
        "#     \"block_size\": 4,\n",
        "#     \"block_batch_size\": 64\n",
        "#   },\n",
        "#   \"_comment4\": \"=====OUR METHOD CONFIGURATION=====\",\n",
        "#   \"defence_config\":{\n",
        "#       \"alpha\" : 0.03, # The step size\n",
        "#       \"k\" : 10 #Number of iterations.\n",
        "#   },\n",
        "#   \"sigma\": 0.0,\n",
        "#   \"modeln\": \"resnet20\",\n",
        "#   \"target\": False,\n",
        "#   \"target_type\": \"median\",\n",
        "#   \"seed\": 1023\n",
        "# }\n",
        "\n",
        "## SignHunter\n",
        "\n",
        "# attack_config_json = {\n",
        "#   \"_comment1\": \"===== DATASET CONFIGURATION =====\",\n",
        "#   'name_of_file' : 'sign',\n",
        "#   \"dset_name\": \"cifar10\",\n",
        "#   \"_comment2\": \"===== EVAL CONFIGURATION =====\",\n",
        "#   \"num_eval_examples\": 100,\n",
        "#   \"_comment3\": \"=====ADVERSARIAL EXAMPLES CONFIGURATION=====\",\n",
        "#   \"attack_name\": \"SignAttack\",\n",
        "#   \"attack_config\": {\n",
        "#     \"batch_size\" : 100,\n",
        "#     \"name\": \"Sign\",\n",
        "#     \"epsilon\": 12.75,\n",
        "#     \"p\": \"inf\",\n",
        "#     \"fd_eta\": 12.75,\n",
        "#     \"max_loss_queries\": 10000\n",
        "#   },\n",
        "#   \"_comment4\": \"=====OUR METHOD CONFIGURATION=====\",\n",
        "#   \"defence_config\":{\n",
        "#       \"alpha\" : 0.03, # The step size\n",
        "#       \"k\" : 10 #Number of iterations.\n",
        "#   },\n",
        "#   \"modeln\": \"resnet20\",\n",
        "#   \"target\": False,\n",
        "#   \"target_type\": \"median\",\n",
        "#   \"seed\": 123\n",
        "# }\n",
        "\n",
        "## SignOPT\n",
        "\n",
        "# attack_config_json = {\n",
        "#   \"_comment1\": \"===== DATASET CONFIGURATION =====\",\n",
        "#   'name_of_file' : 'signopt',\n",
        "#   \"dset_name\": \"cifar10\",\n",
        "#   \"_comment2\": \"===== EVAL CONFIGURATION =====\",\n",
        "#   \"num_eval_examples\": 100,\n",
        "#   \"_comment3\": \"=====ADVERSARIAL EXAMPLES CONFIGURATION=====\",\n",
        "#   \"attack_name\": \"SignOPTAttack\",\n",
        "#   \"attack_config\": {\n",
        "#     \"batch_size\": 1,\n",
        "#     \"epsilon\": 12.75,\n",
        "#     \"p\": \"inf\",\n",
        "#     \"alpha\": 0.2,\n",
        "#     \"beta\": 0.001,\n",
        "#     \"svm\": False,\n",
        "#     \"momentum\": 0,\n",
        "#     \"max_queries\": 10000,\n",
        "#     \"k\": 200,\n",
        "#     \"sigma\": 0\n",
        "#   },\n",
        "#   \"_comment4\": \"=====OUR METHOD CONFIGURATION=====\",\n",
        "#   \"defence_config\":{\n",
        "#       \"alpha\" : 0.03, # The step size\n",
        "#       \"k\" : 10 #Number of iterations.\n",
        "#   },\n",
        "#   \"modeln\": \"resnet20\",\n",
        "#   \"target\": False,\n",
        "#   \"target_type\": \"median\",\n",
        "#   \"seed\":123\n",
        "# }\n",
        "\n",
        "## SimBA\n",
        "\n",
        "# attack_config_json = {\n",
        "#   \"_comment1\": \"===== DATASET CONFIGURATION =====\",\n",
        "#   'name_of_file' : 'simple',\n",
        "#   \"dset_name\": \"cifar10\",\n",
        "#   \"_comment2\": \"===== EVAL CONFIGURATION =====\",\n",
        "#   \"num_eval_examples\": 100,\n",
        "#   \"_comment3\": \"=====ADVERSARIAL EXAMPLES CONFIGURATION=====\",\n",
        "#   \"attack_name\": \"SimpleAttack\",\n",
        "#   \"attack_config\": {\n",
        "#     \"name\": \"SimBA\",\n",
        "#     \"batch_size\": 100,\n",
        "#     \"epsilon\": 12.75,\n",
        "#     \"p\": \"inf\",\n",
        "#     \"delta\" : 2,\n",
        "#     \"max_loss_queries\": 10000\n",
        "#   },\n",
        "#   \"_comment4\": \"=====OUR METHOD CONFIGURATION=====\",\n",
        "#   \"defence_config\":{\n",
        "#       \"alpha\" : 0.03, # The step size\n",
        "#       \"k\" : 10 #Number of iterations.\n",
        "#   },\n",
        "#   \"modeln\": \"resnet20\",\n",
        "#   \"target\": False,\n",
        "#   \"target_type\": \"median\",\n",
        "#   \"seed\": 123\n",
        "# }\n",
        "\n",
        "\n",
        "## Square\n",
        "\n",
        "# attack_config_json = {\n",
        "#   \"_comment1\": \"===== DATASET CONFIGURATION =====\",\n",
        "#   'name_of_file' : 'square',\n",
        "#   \"dset_name\": \"cifar10\",\n",
        "#   \"_comment2\": \"===== EVAL CONFIGURATION =====\",\n",
        "#   \"num_eval_examples\": 100,\n",
        "#   \"_comment3\": \"=====ADVERSARIAL EXAMPLES CONFIGURATION=====\",\n",
        "#   \"attack_name\": \"SquareAttack\",\n",
        "#   \"attack_config\": {\n",
        "#     \"batch_size\" : 100,\n",
        "#     \"name\": \"Square\",\n",
        "#     \"epsilon\": 12.75,\n",
        "#     \"p\": \"inf\",\n",
        "#     \"p_init\": 0.05,\n",
        "#     \"max_loss_queries\": 10000\n",
        "#   },\n",
        "#   \"_comment4\": \"=====OUR METHOD CONFIGURATION=====\",\n",
        "#   \"defence_config\":{\n",
        "#       \"alpha\" : 0.03, # The step size\n",
        "#       \"k\" : 10 #Number of iterations.\n",
        "#   },\n",
        "#   \"modeln\": \"resnet20\",\n",
        "#   \"target\": False,\n",
        "#   \"target_type\": \"median\",\n",
        "#   \"seed\": 123\n",
        "# }\n",
        "\n",
        "##ZoSignSGD\n",
        "\n",
        "\n",
        "# attack_config_json = {\n",
        "#   \"_comment1\": \"===== DATASET CONFIGURATION =====\",\n",
        "#   'name_of_file' : 'zosignsgd',\n",
        "#   \"dset_name\": \"cifar10\",\n",
        "#   \"_comment2\": \"===== EVAL CONFIGURATION =====\",\n",
        "#   \"num_eval_examples\": 100,\n",
        "#   \"_comment3\": \"=====ADVERSARIAL EXAMPLES CONFIGURATION=====\",\n",
        "#   \"attack_name\": \"ZOSignSGDAttack\",\n",
        "#   \"attack_config\": {\n",
        "#     \"batch_size\" :100,\n",
        "#     \"name\": \"zosignsgd\",\n",
        "#     \"epsilon\": 12.75,\n",
        "#     \"p\": \"inf\",\n",
        "#     \"fd_eta\": 2.55,\n",
        "#     \"lr\": 1,\n",
        "#     \"q\": 30,\n",
        "#     \"max_loss_queries\": 1\n",
        "#   },\n",
        "#   \"_comment4\": \"=====OUR METHOD CONFIGURATION=====\",\n",
        "#   \"defence_config\":{\n",
        "#       \"alpha\" : 0.03, # The step size\n",
        "#       \"k\" : 10 #Number of iterations.\n",
        "#   },\n",
        "#   \"modeln\": \"resnet20\",\n",
        "#   \"target\": False,\n",
        "#   \"target_type\": \"median\",\n",
        "#   \"seed\": 123\n",
        "# }\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CmLAU9-U90Oa"
      },
      "outputs": [],
      "source": [
        "# Preparing the files for run.\n",
        "import json\n",
        "attack_json_path = f'cifar10_{attack_config_json[\"name_of_file\"]}_linf_config.json'\n",
        "# Saving the configuration file in the right place.\n",
        "temp_file_path = f'/content/BlackboxBench/config-jsons/{attack_json_path}'\n",
        "with open(temp_file_path, 'w') as json_file:\n",
        "    json.dump(attack_config_json, json_file, indent=2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ya1E_ru4PzWs"
      },
      "source": [
        "# Running the Experiment\n",
        "The results will be stored under *Results* folder.\n",
        "<br>\n",
        "<br>\n",
        "*For choosing the baselines defences methods you want to run the experiment on, please make sure that the correspoding value in 'baselines_dict' below is 'True'* <br>\n",
        "*Default - all methods*\n",
        "\n",
        "*Reminder*: <br>\n",
        "- RND : Random Noise\n",
        "- SND : Gaussian Noise\n",
        "- AS : Average Smoothing\n",
        "- BS : Bit Squeezing\n",
        "- FS : Feature Squeezing\n",
        "- JPEG : Jpeg Compression\n",
        "- Counter_Samples_10 : Our method with k = 10\n",
        "- Counter_Samples_1 : Our method with k = 1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "A4P8-rdNVjnR"
      },
      "outputs": [],
      "source": [
        "# Choose the baselines you want to run the experiment on (True means on) - Default is all methods.\n",
        "baselines_dict = {\n",
        "    'FS': True,\n",
        "    'RND' : True,\n",
        "    'CounterSamples_10' : True,\n",
        "    'Counter_Samples_1': True,\n",
        "    'AS' : True,\n",
        "    'BS' : True,\n",
        "    'SND' : True,\n",
        "    'JPEG' : True,\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mZ-WcQ6I754_"
      },
      "outputs": [],
      "source": [
        "# Preparing the file for run.\n",
        "import json\n",
        "baselines_chosed = [i for i in baselines_dict if baselines_dict[i] == True]\n",
        "baselines_json_dict = {\"Baselines\":baselines_chosed}\n",
        "baselines_json_string = json.dumps(baselines_json_dict)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ydYToVYgYrLM"
      },
      "outputs": [],
      "source": [
        "# Run command.\n",
        "command = f\"python /content/BlackboxBench/BaselinesExperiment.py /content/BlackboxBench/config-jsons/{attack_json_path} '{baselines_json_string}'\"\n",
        "!{command}"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "wHHx6nhjxALx",
        "f6E-ITJIq9nq"
      ],
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}