TOF.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yK-ONXiYqxMW"
      },
      "source": [
        "## **ToF Imaging**\n",
        "\n",
        "A **time-of-flight camera (ToF camera)** is a range imaging camera system for measuring distances between the camera and the subject for each point of the image based on time-of-flight (the round trip time of an artificial light signal, as provided by a laser or an LED). Laser-based time-of-flight cameras are part of a broader class of scannerless LIDAR, in which the entire scene is captured with each laser pulse, as opposed to point-by-point with a laser beam such as in scanning LIDAR systems."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!cat /proc/cpuinfo"
      ],
      "metadata": {
        "id": "8cAtVwV5G6jr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jwkgHOsnqxMX"
      },
      "source": [
        "The task involves **classifying the images into two classes**: background and robot."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yyNJMJxoqxMX"
      },
      "source": [
        "The dataset [Miniature mobile robot detection using an utra-low resolution time-of-flight sensor](https://ieee-dataport.org/documents/miniature-mobile-robot-detection-using-ultra-low-resolution-time-flight-sensor-dataset) is composed of ToF depth images acquired with the [ST VL53L5CX ToF sensor](https://www.st.com/en/imaging-and-photonics-solutions/vl53l5cx.html) and sampled by a miniature mobile robot navigating in a sand-like terrain. Each image has a resolution of 8x8 pixels. It comprises a total of 4.150 samples, with 2.062 samples belonging to the\n",
        "background class and 2.088 samples to the Robot class. The bit depth is 8 bits per pixel, where each pixel value represents the depth information."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FZvDBg5SqxMY"
      },
      "source": [
        "# **Download dataset**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Mh98JQrIqxMY"
      },
      "outputs": [],
      "source": [
        "import urllib.request\n",
        "\n",
        "url = \"https://www.dropbox.com/s/4txj0ob6ovy9jbr/time-of-flight.zip?dl=1\"\n",
        "u = urllib.request.urlopen(url)\n",
        "data = u.read()\n",
        "u.close()\n",
        "with open(\"./time-of-flight.zip\", \"wb\") as f :\n",
        "   f.write(data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kdfiN3mRqxMY"
      },
      "source": [
        "## Extract files:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vGGuY8EsqxMZ"
      },
      "outputs": [],
      "source": [
        "import zipfile\n",
        "\n",
        "with zipfile.ZipFile(\"time-of-flight.zip\",\"r\") as zip_ref:\n",
        "    zip_ref.extractall(\".\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C9o1ic6iqxMZ"
      },
      "source": [
        "## Remove the .DS_Store files:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2FooNHspqxMZ"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "for root, dirs, files in os.walk('./time-of-flight/'):\n",
        "    for file in files:\n",
        "        if file.endswith('.DS_Store'):\n",
        "            path = os.path.join(root, file)\n",
        "            print(\"Deleting: %s\" % (path))\n",
        "            if os.remove(path):\n",
        "                print(\"Unable to delete!\")\n",
        "            else:\n",
        "                print(\"Deleted...\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d2_s-MDlqxMa"
      },
      "source": [
        "## Load data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IoTMnJWDqxMa"
      },
      "outputs": [],
      "source": [
        "import sklearn.datasets\n",
        "\n",
        "dataset = sklearn.datasets.load_files('./time-of-flight', shuffle='True', encoding='utf-8')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8j8ppLxnqxMa"
      },
      "source": [
        "## Check the number of samples (should be 4150 for TOF dataset):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pqQ6aKsSqxMa"
      },
      "outputs": [],
      "source": [
        "samples_number = len(dataset.data)\n",
        "print(\"Number of samples: \", samples_number)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "knIUQ0G5qxMa"
      },
      "source": [
        "## Pre-process files content in order to extract just image values from the CSV structure:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B4werPLuqxMb"
      },
      "outputs": [],
      "source": [
        "for index, data in enumerate(dataset.data) :\n",
        "    data = data.split('\\n')\n",
        "    data = data[1].split(',')\n",
        "    dataset.data[index] = [int(i) for i in data]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jyn-YRKQqxMb"
      },
      "source": [
        "## Show some sample as a grayscale image:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UF3qKa8uqxMb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "953a27c6-ff2a-4fb9-9443-39559aba9dc6"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 800x800 with 20 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnUAAAKkCAYAAABmlcneAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABvG0lEQVR4nO3de3SU9b3v8c8kk0wCJBANigEMG5RAgCSGlsDZIMjJEdh7S2vbvVOKF7ygtd66VI7HLRZou4o9uhStVrqXR6mKC+yy22NbDm7uar3tom5EKkpKMBi5BCGBQGaSzO/8wcqQcJH5/ZLMM3nyfq2VtSDzfOf3m2c+8+SbJzPPL2CMMQIAAEC3luL1BAAAANBxNHUAAAA+QFMHAADgAzR1AAAAPkBTBwAA4AM0dQAAAD5AUwcAAOADNHUAAAA+QFMHAADgA75o6oYMGaKCggKVlJRo5MiR+sEPfqCGhgan+1q2bJm+/e1vd+4Eu8gf//hHTZkyxetpoBO1zXJBQYEefPDBs9YEAgEdOnTIapyqqiotXbrUcZZIZhwP4RccD+35oqmTpJUrV+rDDz/Uxx9/rLq6Oi1btsyzuTQ3N3s2Nrq/1iyvX79eixcv1nvvvdfpY/jpIIZTcTyEX3A8tOObpq5VJBLR0aNHlZOTo48++kgTJ05UaWmpCgsL9fOf/7zddvPmzdPo0aNVXFys6dOnn3JfNTU1+uY3v6lnnnlGkvTWW2+ppKREY8aM0fXXX6/i4mJt3LhRkjRlyhTdcccdmjBhgi6//HK1tLTE7n/06NG6/fbbFYlEJElz5szRkiVLYuPcc889WrhwoSRp4cKFqqio0BVXXKHCwkJNnTpVX331lSSpqalJP/rRj3TxxRdr3Lhx2rBhQxfsQSSLgQMHasSIEdq1a5d27Nih8vJyFRUVqaSkRK+88kq7bR9++GFdcsklGj58uJYvXx77/muvvabS0lIVFRVp8uTJ2rZtmyTphz/8obZv366SkhLNnDkzkQ8LCcTxEH7B8TBOxgfy8/PN8OHDTXFxsenbt6+ZOnWqaWpqMvX19aaxsdEYY8zRo0dNSUmJefvtt40xxixcuNDMnDkzdvu+ffuMMcY8++yz5lvf+pbZsmWLKSwsNK+99poxxphwOGwGDRpk1q9fb4wxZv369UaS2bBhgzHGmMmTJ5tp06aZSCRijDHm17/+tZk8ebJpbGw0TU1NZsaMGebBBx80xhhz7bXXmkcffTQ2/7vvvtssWLDAGGPMggULTH5+vqmtrTXGGFNRUWF+8YtfGGOMeeKJJ8zUqVNNOBw24XDYTJkyxUyePLkL9ii8kp+fbz744ANjjDF//etfzbBhw8y+ffvMuHHjzNKlS40xxnz66afmnHPOMVVVVcYYYySZ+fPnG2OMqaysNDk5OWbnzp1m79695pxzzjFbtmwxxhjzwgsvmJEjR5poNGo2bNhgiouLE/740PU4HsIvOB7a882ZutZTtLW1tRoyZIjuvfdeHTt2TDfeeKPGjBmj8ePHa9euXfrwww8lHX//xZ133qlQKCRJ6t+/f+y+Pv74Y82cOVMvvviiLr/8cknSJ598omAwqMsuu0ySdNlll2nYsGHt5nDVVVcpLS1NkrR27VrNmTNHoVBIwWBQc+fO1Zo1a+J6LNOnT9e5554rSZowYYIqKyslSevWrdM111yj9PR0paen6/rrr3fcW0hmFRUVGjlypAoLC3X77bcrIyND77//vm644QZJ0sUXX6yJEyfqjTfeiNXceOONkqShQ4fq0ksv1euvv653331XY8aM0ZgxYyRJs2fPVk1Njb744ovEPygkFMdD+AXHQztBryfQ2YLBoL773e9q3rx5qqurU25urj744AMFg0F95zvfUWNj41nvIy8vT+FwWOvXr1dxcfEZtwsEAu3+36dPn7i2DQaDamlpif2/sbGxXW1GRkbs36mpqWd8T8rJ48MfVq5cqZKSEq1du1ZXXHGFpk6deso2Z3vuyQYkjofo/jge2vHNmbq21q9fr4KCAh08eFCDBg1SMBjU9u3b2/1mOHPmTD322GMKh8OSpP3798duy8nJ0Zo1a/TKK6/opz/9qSSpoKBATU1N2rRpkyRp06ZN2rFjxxnnUF5erueee06RSETNzc16+umnY7/lXnTRRbE3ex44cECrVq2K63GVl5frhRdeUFNTkyKRiJ599lmLvYLupry8XLfccovmz5+v0tLS2PO9Y8cOvfnmm7r00ktj27beVlVVpTfeeEOTJk3S+PHj9dFHH2nr1q2SpBUrVmjgwIEaOHCgsrOzVVdXl/gHhYTjeAg/4HgYH9+cqauoqFBmZqaam5uVn5+vpUuXqra2VldffbV++9vfatiwYe06/HvvvVf333+/SktLlZaWpry8vHYHk6ysLK1evVpXXnml5s2bp4ceekgrVqzQrbfeqmg0qrFjx6qgoED9+vU77XxuuukmVVZWqrS0VNLxNw7/+Mc/jt32ve99TyNHjtTQoUM1fvz4uB7j3LlztXXrVhUWFionJ0eTJk3S5s2b3XYYuoUHHnhAF110kVatWqX7779fTzzxhAKBgJ5++mldeOGFse1aWlp0ySWXqKGhQY8//riGDBkiSVq+fLmuueYaNTc3KycnR7/73e8UCARUVFSkUaNGafTo0Ro6dKheffVVjx4hugLHQ/gRx8OzCxhjjNeT6C4OHz6srKwsSdJ//ud/aubMmaqsrFSvXr08nhkAJBbHQyD5+OZMXSK8/PLLevTRR2WMUTAY1PPPP88BDECPxPEQSD6cqQMAAPABX35QAgAAoKehqQMAAPABmjoAAAAfoKkDAADwgYR++jUajaqmpkZZWVk96grP3ZExRocPH1ZeXp5SUvzV+5PD7sHPGZTIYXdBDpEM4s1hQpu6mpoaDR48OJFDooOqq6s1aNAgr6fRqchh9+LHDErksLshh0gGZ8thQpu61gtV3nLLLbGFo+MRz/qEp/PVV19Z17iMdfDgQesaSTp06JB1jctjkqQjR45YbW+MUX19few585PWx/Tqq6+qd+/ecdc1NTU5jdd2Xct42T5fkvTuu+9a10jSggULrGvaXr3dhstSPH7MoHTicVVXVys7O9vj2XjPpbE4evSo01hnWj/26/g9h6+99prV8TAajTqN51JXW1trXfPWW29Z10hux8O8vDynsRoaGqxrzpZDp6buySef1EMPPaQ9e/aouLhYv/rVrzRu3Liz1rWe2g2FQlZNneul9NLT061rXH4AB4NuvXFqaqp1jevpf9fT6sl8Or6jOezdu3fSNnUuBz6b11RbLg1FInPhxwxKJx5XdnY2TZ3cnmdyeFxn5LB3797q06dP3GMmsqlzad576vHQukNYuXKl7rrrLi1YsEDvv/++iouLNW3aNO3bt895koAtcgivkUEkA3KItqybukceeURz587Vddddp8LCQi1dulS9evXSM8880xXzA06LHMJrZBDJgByiLaumLhKJaPPmzSovLz9xBykpKi8v19tvv93pkwNOhxzCa2QQyYAc4mRWbwarra1VS0uLzj///HbfP//88/XJJ5+csn04HFY4HI79v76+3nGawAnkEF6zzaBEDtH5yCFO1qUX3Vm8eLH69u0b++Jj0/ACOUQyIIdIBuTQ36yautzcXKWmpmrv3r3tvr93714NGDDglO3vu+8+1dXVxb6qq6s7NltA5BDes82gRA7R+cghTmbV1KWnp2vs2LFat25d7HvRaFTr1q3ThAkTTtk+FArFPq7Px/bRWcghvGabQYkcovORQ5zM+gJrd911l6699lp94xvf0Lhx47RkyRI1NDTouuuu64r5AadFDuE1MohkQA7RlnVTV1FRof379+snP/mJ9uzZo5KSEq1evfqUN2oCXYkcwmtkEMmAHKKtgHFdrsFBfX29+vbtqzvuuMPqas+un875/PPPrWtcVmxwWdJJkjZu3GhdY3PF77ZsVzUwxigcDquurs53p+dbc/jGG29Y7U/Xl4rLkkS/+MUvrGs++ugj6xrp+GURbLX99JwNm2XujDFqaWnxZQalEzmcPHmy1ao0rlevd1nZxMX69eud6s4991zrGpdlliS3/Po9h++9957V8dD1GODy89Jl+c6PP/7YukaS7rzzTqc6Fy6v5bPlsEs//QoAAIDEoKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfoKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfCHoyaDCoYDD+oZubm53GSU1Nta75wx/+YF2TmZlpXSPJah+0ikajTmPZMsYkZBwv9erVS7169Yp7e9ccpqTY/+7kMtaOHTusayQpPz/fusbltSVJoVAo7m2NMTp69KjTON1J7969lZaWFvf2LS0tTuO41tn6H//jfzjVpaenW9c0NjY6jRUIBOLeticcC6XjxymbY5XL8yW57c/du3db1/zrv/6rdY0k3XfffdY1rq+trsghZ+oAAAB8gKYOAADAB2jqAAAAfICmDgAAwAdo6gAAAHyApg4AAMAHaOoAAAB8gKYOAADAB2jqAAAAfICmDgAAwAdo6gAAAHyApg4AAMAH7FeU7wTRaNRqYXrXRez/9re/WdfYLKzdynV+iVwo2mbh4FZ+X8g6MzNTvXr16vJxXBYdnzBhgnXNCy+8YF0jSbt27XKqc5GTkxP3tn7PX6vzzjvPanF010XsI5GIdU1zc7N1TV1dnXWNJH355ZfWNf369XMaKxwOx72tMcb5GN+dpKWlWeXQlcsY5513nnVNSorbOSuX59rl52tX4UwdAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4QNCLQY0xVot1uy7s/fHHH1vXpKamWte4zs+lznXhYNuxesJi6mlpaUpLS4t7e9d977JA9NixY61rPv/8c+saSfrOd75jXfP73//eaaxgMP5DTk9YRF2SzjnnHIVCobi3b25udhqnsbHRuiYSiVjXuC6kPm3aNOsam9dvWzZzNMb0iCzaHg9dn2ebY0Cr/v37W9e4/CyX3I7ziewBzoYzdQAAAD5AUwcAAOADNHUAAAA+QFMHAADgAzR1AAAAPkBTBwAA4AM0dQAAAD5AUwcAAOADNHUAAAA+QFMHAADgAzR1AAAAPkBTBwAA4AM0dQAAAD4Q9GLQxsZGRaPRuLdPTU3twtl0nDHG6ynAQUpKilJS4v+9Jhh0e7lEIhHrmgsuuMC65ujRo9Y1khQKhaxrRo0a5TSWzf7uKfLz85WZmRn39q45rK2tta45fPiwdU16erp1jeT2OrHZb20dOXIk7m2NMWpubnYax89cX8sudS7HqBdffNG6RpLWrFljXfPoo486jRUIBJzqvg5HWAAAAB+gqQMAAPABq6Zu4cKFCgQC7b5GjBjRVXMDToscIhmQQ3iNDOJk1m/OGDVqlNauXXviDhzf3wF0BDlEMiCH8BoZRFvWz34wGNSAAQO6Yi5A3MghkgE5hNfIINqyfk/dZ599pry8PA0dOlSzZ8/W559/fsZtw+Gw6uvr230BnYEcIhmQQ3jNJoMSOfQ7q6aurKxMy5Yt0+rVq/XUU09p586dmjRp0hk/9r548WL17ds39jV48OBOmTR6NnKIZEAO4TXbDErk0O8CpgMXWTt06JDy8/P1yCOP6IYbbjjl9nA4rHA4HPt/fX29Bg8erJtvvtnqWkau1996+umnrWtcrolnc829juqK69qcTmss6urqlJ2dnZAxXbnmsLKyUllZWXGP4/pelYaGBuuar776yrqmsrLSukaSXnrpJeuaLVu2OI1lc620aDSq2trabpFByT2Hjz32mK+uU3fo0CHrGkmqrq62rtm8ebPTWPv37497W2OMwuFwt8jh2TIonTmH27ZtS8jx8Msvv7SuSUtLs6452xnLM0n269SdLYcdekdlv379NHz4cO3YseO0t4dCIaeLBgI2yCGSATmE186WQYkc+l2HrlN35MgRVVZWOl39Hugs5BDJgBzCa2QQVk3dPffco02bNqmqqkpvvfWWrrzySqWmpmrWrFldNT/gFOQQyYAcwmtkECez+vPr7t27NWvWLB04cED9+/fXxIkT9c4776h///5dNT/gFOQQyYAcwmtkECezaupWrFjRKYM2NzdbLerrunDwt771Lesal/catH3TqY0OfEalR+usHO7Zs8dqYW+XN+tKUktLi3WNy4eDGhsbrWtcub4mbTKf7K+Pzsphfn6+evfuHff2rjnMycmxrtm9e7d1jeuHCVwurWGz39py+dBIMuqsDLpwPQacf/751jV79+61rqmqqrKukY5/EMHW2LFjncay6TeMMYpEImfdjrVfAQAAfICmDgAAwAdo6gAAAHyApg4AAMAHaOoAAAB8gKYOAADAB2jqAAAAfICmDgAAwAdo6gAAAHyApg4AAMAHaOoAAAB8gKYOAADAB2jqAAAAfCDoxaDRaFTRaNRqexeZmZnWNePGjbOu2bp1q3WNJB08eNC6xnVfBAIBpzo/q6qqUq9eveLePjU11WmcSCTiVGfrwIEDTnUNDQ3WNc3NzU5j4VTGGBlj4t6+paXFaZxQKGRdk5eXZ13jkifp+OvRVkZGhtNYKSnxn8+weW66s0OHDlm9rvv06eM0jsuxIy0tzbrm7/7u76xrJOm//uu/rGvS09Odxurfv3/c20ajUdXU1Jx1O87UAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4QELXfm1dQ892LUzXdSabmpqsa1zGcl2PNZFrCrqO5cd1D1sf09GjR63qkn3t12PHjjnVubxOXNcfdVnz2Y8ZlLpHDl0y5ZpDl/m55tAmU63b+j2Htmv2uu4Pl+csHA5b19i+rlq55NC1R+mK42HAJDCpu3fv1uDBgxM1HDpBdXW1Bg0a5PU0OhU57F78mEGJHHY35BDJ4Gw5TGhTF41GVVNTo6ysLAUCgXa31dfXa/DgwaqurlZ2dnaippR0kmU/GGN0+PBh5eXlKSXFX3+lJ4dnlwz7wc8ZlMhhPJJhP/TUHCbDvk8WybAv4s1hQv/8mpKSctbfdLKzs3t8gKTk2A99+/b1dPyuQg7j5/V+8GsGJXJow+v90JNz6PW+TyZe74t4cui/XzsAAAB6IJo6AAAAH0iapi4UCmnBggUKhUKezWHIkCEqKChQSUmJCgoK9OCDD561JhAI6NChQ1bjVFVVaenSpae9LRn2Q0/Wkf3fNj8jR47UD37wA+tPlLVatmyZvv3tbzvVdgab/fDHP/5RU6ZM6fpJ9SDk8Dhy6J2O/iwih95IqqZu4cKFnjczK1eu1Icffqj169dr8eLFeu+99zp9jLM1dcmwH3qqju7/1vx8/PHHqqur07Jlyzp3ghZcP2YvkUOvkcPjyKF3OmPfk8PES5qmLtkMHDhQI0aM0K5du7Rjxw6Vl5erqKhIJSUleuWVV9pt+/DDD+uSSy7R8OHDtXz58tj3X3vtNZWWlqqoqEiTJ0/Wtm3bJEk//OEPtX37dpWUlGjmzJmJfFhIkEgkoqNHjyonJ0cfffSRJk6cqNLSUhUWFurnP/95u+3mzZun0aNHq7i4WNOnTz/lvmpqavTNb35TzzzzjCTprbfeUklJicaMGaPrr79excXF2rhxoyRpypQpuuOOOzRhwgRdfvnlamlpid3/6NGjdfvtt8euwzRnzhwtWbIkNs4999yjhQsXSpIWLlyoiooKXXHFFSosLNTUqVP11VdfSTp+Xbsf/ehHuvjiizVu3Dht2LChC/YgOgM5RDIgh4mT0E+/dieffPKJDhw4oClTpuif/umfdP311+vmm2/WZ599pvHjx+uSSy5Rfn6+pON/gv3ggw/0t7/9Td/4xjf093//9+rVq5d+8IMfaOPGjRozZoyWL1+u733ve/r444+1dOlS/fjHP9aHH37o7YNEp6uoqFBmZqaqqqo0duxY/cu//IuOHTumdevWKRQK6dixY/pv/+2/qby8XOPHj9fixYv16aefavPmzQqFQtq/f3+7+/voo4/0/e9/X48++qguv/xyRSIRVVRU6LnnntNll12mDRs26Nlnn21X8+mnn+r1119XWlqannrqKf3nf/6nNm/erNTUVM2cOVOPPvqo7r333rM+lnfffVebN2/Wueeeq+9///v6zW9+o/vuu0//9m//pu3bt+vjjz+WJE2bNq3zdiA6BTlEMiCHiceZupNUVFRo5MiRKiws1O23366MjAy9//77uuGGGyRJF198sSZOnKg33ngjVnPjjTdKkoYOHapLL71Ur7/+ut59912NGTNGY8aMkSTNnj1bNTU1+uKLLxL/oJAwrX9uqK2t1ZAhQ3Tvvffq2LFjuvHGGzVmzBiNHz9eu3btijX0f/zjH3XnnXfGTuv3798/dl8ff/yxZs6cqRdffFGXX365pOO/bASDQV122WWSpMsuu0zDhg1rN4errrpKaWlpkqS1a9dqzpw5CoVCCgaDmjt3rtasWRPXY5k+fbrOPfdcSdKECRNUWVkpSVq3bp2uueYapaenKz09Xddff73j3kJXIYdIBuQw8ThTd5KVK1eqpKREa9eu1RVXXKGpU6eess3JFwq1vR3+FwwG9d3vflfz5s1TXV2dcnNz9cEHHygYDOo73/mOGhsbz3ofeXl5CofDWr9+vYqLi8+43cl569OnT1zbBoPBdkv2NDY2tqvNyMiI/Ts1NfWM70kh78mLHCIZkMPESZozdU8++aSGDBmijIwMlZWVdckHFGyUl5frlltu0fz581VaWho7pbtjxw69+eabuvTSS2Pbtt5WVVWlN954Q5MmTdL48eP10UcfaevWrZKkFStWaODAgRo4cKCys7NVV1fXbryFCxcqEAi0+xoxYkSCHi2kzs/g+vXrVVBQoIMHD2rQoEEKBoPavn17u98MZ86cqcceeyy2tmHbPzfk5ORozZo1euWVV/TTn/5UklRQUKCmpiZt2rRJkrRp0ybt2LHjjHMoLy/Xc889p0gkoubmZj399NOx33Ivuuii2GM8cOCAVq1apY0bNyoQCGjRokV67LHHTpvD8vJyvfDCC2pqalIkEjnlzx3oGHJIDpNBT8/h8uXLtWjRonY5PN3P5GTLYVKcqVu5cqXuuusuLV26VGVlZVqyZImmTZum7du367zzzvNsXg888IAuuugirVq1Svfff7+eeOIJBQIBPf3007rwwgtj27W0tOiSSy5RQ0ODHn/8cQ0ZMkSStHz5cl1zzTVqbm5WTk6Ofve73ykQCKioqEijRo3S6NGjNXToUL366quSpFGjRmnt2rWx+w0Gk+Lp6RE6K4Ot7yFpbm5Wfn6+li5dqtraWl199dX67W9/q2HDhrU7+3vvvffq/vvvV2lpqdLS0pSXl6dVq1bFbs/KytLq1at15ZVXat68eXrooYe0YsUK3XrrrYpGoxo7dqwKCgrUr1+/087npptuUmVlpUpLSyUdf+Pwj3/849ht3/ve9zRy5EgNHTpU48ePV01NjUaNGqXp06errq5OP/vZzxQMBrVixYrYfc6dO1dbt25VYWGhcnJyNGnSJG3evNlib+NMyCE5TAbkcKgGDRqktLQ0bdmyRQ8//LDq6uq0ePHiU+4z6XJoksC4cePMrbfeGvt/S0uLycvLM4sXL/ZwVom1YMECU1xc7PU0eqzulMH6+vrYv9977z0zYMAA09DQ0Cn3TQ69RQ6PI4feIofdN4Oe//k1Eolo8+bNKi8vj30vJSVF5eXlevvttz2cWeJ99tlnysvL09ChQzV79mx9/vnnXk+pR+huGXz55ZdVXFysoqIi3XzzzXr++efVq1evTrt/cugNctgeOfQGOTyhO2bQ87/v1dbWqqWlReeff367759//vn65JNPPJpV4pWVlWnZsmUqKCjQl19+qUWLFmnSpEnaunWrsrKyvJ6er3W3DM6ZM0dz5szpkvsmh94hhyeQQ++Qw+O6awY9b+pw3IwZM2L/LioqUllZmfLz8/XSSy/FLqcCdDVyiGRADuG17ppBz//8mpubq9TUVO3du7fd9/fu3asBAwZ4NCvv9evXT8OHD//aT/Kgc5DBMyOHiUMOz4wcJg45PL3ukkHPm7r09HSNHTtW69ati30vGo1q3bp1mjBhgocz89aRI0dUWVmpCy64wOup+B4ZPDNymDjk8MzIYeKQw9PrNhn0+pMaxhizYsUKEwqFzLJly8y2bdvMTTfdZPr162f27Nnj9dQS5u677zYbN240O3fuNH/+859NeXm5yc3NNfv27fN6aj0CGTyOHHqLHB5HDr1FDrtvBpPiPXUVFRXav3+/fvKTn2jPnj0qKSnR6tWrT3mjpp/t3r1bs2bN0oEDB9S/f39NnDhR77zzTrtlUtB1yOBx5NBb5PA4cugtcth9MxgwxphEDRaNRlVTU6OsrCzPl9LA1zPG6PDhw8rLy1NKiud/pe9U5LB78HMGJXLYXZBDJIN4c5jQM3U1NTUaPHhwIodEB1VXV2vQoEFeT6NTkcPuxY8ZlMhhd0MOkQzOlsOENnWt13aprq5WdnZ23HVt13+zcaYFd7/O4cOHrWtc13p79913rWsqKyudxjp06JBTXTJfj8dV62P661//avX4/u3f/s1pvP/zf/6Pdc3Ro0etayKRiHWNJDU1NVnXuLy2pONnBWz5MYPSicf1H//xH+rdu3fcdenp6U7jDR8+3LqmpKTEuubIkSPWNZLiWtT9ZMeOHXMayyW/fs/hT3/603aL1p9NWlqa03gufxz89NNPrWt++ctfWtdIcmpwXXPocuw9Ww6dmronn3xSDz30kPbs2aPi4mL96le/0rhx485a13pqNzs726qpc3mxS24vXJfAhUIh6xpJSk1Nta5J9OnxZD4d39EcZmVlWeXQ5oDXlsufbFxqXJ8rl7pE5sKPGZROPK7evXurT58+cY/p2tTZZL1VorIrkcOO6IwcZmRkKDMzM+4xE9nUufyMdcm7lNgc2tS17rez1Vi/+loX+l2wYIHef/99FRcXa9q0adq3b5/tXQHOyCG8RgaRDMgh2rJu6h555BHNnTtX1113nQoLC7V06VL16tVLzzzzTFfMDzgtcgivkUEkA3KItqyaOtuFfsPhsOrr69t9AR1FDuE1l0XPySE6GznEyayauq9b6HfPnj2nbL948WL17ds39sUnbNAZyCG8ZptBiRyi85FDnKxLL7pz3333qa6uLvZVXV3dlcMBp0UOkQzIIZIBOfQ3q0+/2i70GwqFnD8ZCpwJOYTXXBY9J4fobOQQJ7M6U8dCv0gG5BBeI4NIBuQQJ7O+Tt1dd92la6+9Vt/4xjc0btw4LVmyRA0NDbruuuu6Yn7AaZFDeI0MIhmQQ7Rl3dSx0C+SATmE18ggkgE5RFsB43J5Z0f19fXq27ev6urqrK727LrElYuDBw9a12zevNlprB/96EdOdS6++uorq+2NMYpGo9bPVXfQmsMvv/zS6rH16tWrC2fV3nnnnWddEw6HncZyWXnFdZkwl6XM/JhB6UQOv/Wtb1ldnf9//+//7TTec889Z12zYMEC6xrX9VEbGhqsa1yXxrNZpcgYI2OM73O4evVqq+XqXJa4ktyWCnRZhuuf/umfrGuk4+9TtOWSXckuv/HmsEs//QoAAIDEoKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfoKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfCHo9gXhkZGQ41R06dMi6ZsCAAdY1t9xyi3WNJLW0tCSkRpKCQbun2hijSCTiNFZ30atXL/Xq1avLx9m9e7d1TUqK/e9bLjWudYFAoMvHMsbIGOM0TndSXFxsdYxzzeysWbOc6my55jA1NdW6xjWHNmMZY9Tc3Ow0jp+57pPf//731jVHjhyxrvmnf/on6xrJLVOuObSti+d4yJk6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfoKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfoKkDAADwAbtV3j1is9h1W71797auaWlpsa554oknrGsk6bbbbrOucV3g3HaR7Z6wkHqiDBo0yLrGJRuPPfaYdY3klnl0noMHDyoUCsW9vctxTZKCQfvD/dq1a61rotGodY2rRC2k3hPU1tbq6NGjcW//X//1X07jPPXUU051iZLIbNj8XDbGxPXa4kwdAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4gP0Kz91Ic3Ozdc2yZcusa26++WbrGkmqqKhwqnPhugg4vHHHHXdY1zz++ONOY7G4ubdSUlKsFvZOT093GicUClnXlJSUWNcYY6xrJMW1WDm6TiQSUTAYf0tQWVnZhbPxjksOk+kYypk6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfoKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfoKkDAADwAZo6AAAAHwh6PYGuFAgErGuuuOIK65q0tDTrGkn6l3/5F+uaTz/91GmsYNDuqTbGOI3TnUQiEUUikbi3T09P78LZtPf5558nbKxEsn1N9oQcLlq0SNnZ2XFv73Jck6RoNGpdk5uba12zcuVK6xpJuvLKK61rbF6/bdnkqidkUDr+OG0eq+3PlO4ikc93V+SQM3UAAAA+YNXULVy4UIFAoN3XiBEjumpuwGmRQyQDcgivkUGczPr86ahRo7R27doTd+DTU7BIbuQQyYAcwmtkEG1ZP/vBYFADBgzoirkAcSOHSAbkEF4jg2jL+j11n332mfLy8jR06FDNnj3bt2/oRnIjh0gG5BBeI4Noy+pMXVlZmZYtW6aCggJ9+eWXWrRokSZNmqStW7cqKyvrlO3D4bDC4XDs//X19R2fMXo8cohkQA7hNdsMSuTQ76yauhkzZsT+XVRUpLKyMuXn5+ull17SDTfccMr2ixcv1qJFizo+S6ANcohkQA7hNdsMSuTQ7zp0SZN+/fpp+PDh2rFjx2lvv++++1RXVxf7qq6u7shwwGmRQyQDcgivnS2DEjn0uw41dUeOHFFlZaUuuOCC094eCoWUnZ3d7gvobOQQyYAcwmtny6BEDv3Oqqm75557tGnTJlVVVemtt97SlVdeqdTUVM2aNaur5gecghwiGZBDeI0M4mRW76nbvXu3Zs2apQMHDqh///6aOHGi3nnnHfXv37+r5gecghwiGZBDeI0M4mRWTd2KFSu6ah5A3MghkgE5hNfIIE7WLS497fqRa5eFeV1qnnvuOesaSerdu7d1TWpqqtNYKSl2b5/sCYtYV1VVqU+fPnFvHwqFnMZpe/mAeP3tb39zGsuFywLxrletb2lpcarzs4aGBqvXda9evbpwNu01NjZa17geO9LT061rIpGI01gumfe7lpYWNTc3x719WVmZ0zhXX321dc3zzz/vNJaLaDSalGPF+7rq0AclAAAAkBxo6gAAAHyApg4AAMAHaOoAAAB8gKYOAADAB2jqAAAAfICmDgAAwAdo6gAAAHyApg4AAMAHaOoAAAB8gKYOAADAB2jqAAAAfICmDgAAwAeCXgz6pz/9Sb169Yp7+6ysLKdxotGodY0xxrqmb9++1jWudSkpbn14IBBwqvOzhx56SOnp6XFvv3//fqdxbrrpJuuaffv2JaRGcsthc3Oz01g2ry+X12J39PzzzysjIyPu7adPn+40TktLi3VNOBy2rpk0aZJ1jSR98cUX1jW5ublOY7k8Lr8zxli95vr06eM0zmWXXWZd88gjj1jX3HXXXdY1kttxx6XX6Ejd1+FMHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACADyR07dfWNdWOHj1qVee63mmi1n61fTytXNYfdFm/UbJ/XK3b+3H9zdbH1NTUZFXnut6pSz6OHTtmXVNfX29dI7k9x665cFn71Y8ZlE48rsbGRqu6I0eOOI2XqLVfXXPownXtTHJ4Quvjsj3muK4nHolErGtsXyNS9zgeujjbWAGTwNns3r1bgwcPTtRw6ATV1dUaNGiQ19PoVOSwe/FjBiVy2N2QQySDs+UwoU1dNBpVTU2NsrKyTuny6+vrNXjwYFVXVys7OztRU0o6ybIfjDE6fPiw8vLynM+UJityeHbJsB/8nEGJHMYjGfZDT81hMuz7ZJEM+yLeHCb0z68pKSln/U0nOzu7xwdISo790LdvX0/H7yrkMH5e7we/ZlAihza83g89OYde7/tk4vW+iCeH/vu1AwAAoAeiqQMAAPCBpGnqQqGQFixYoFAo5NkchgwZooKCApWUlKigoEAPPvjgWWsCgYAOHTpkNU5VVZWWLl162tuSYT/0ZF7v/2TIoOT9fujpvN7/5BAd3fdtMzRy5Ej94Ac/UENDg9N9LVu2TN/+9redajuDzb744x//qClTpnT9pM7EICY/P9988MEHxhhjdu/ebbKzs8277777tTWSzMGDB63G2bBhgykuLnabJHyNDCIZkEN0VNsMtbS0mH/4h38wTzzxhNN9Pfvss+Zb3/pWh+bT1NTUofp4/eEPfzCTJ09OyFinkzRn6pLNwIEDNWLECO3atUs7duxQeXm5ioqKVFJSoldeeaXdtg8//LAuueQSDR8+XMuXL499/7XXXlNpaamKioo0efJkbdu2TZL0wx/+UNu3b1dJSYlmzpyZyIeFboQMIhmQQ3RUJBLR0aNHlZOTo48++kgTJ05UaWmpCgsL9fOf/7zddvPmzdPo0aNVXFys6dOnn3JfNTU1+uY3v6lnnnlGkvTWW2+ppKREY8aM0fXXX6/i4mJt3LhRkjRlyhTdcccdmjBhgi6//HK1tLTE7n/06NG6/fbbY9fNmzNnjpYsWRIb55577tHChQslSQsXLlRFRYWuuOIKFRYWaurUqfrqq68kHb/e6Y9+9CNdfPHFGjdunDZs2NAFe9CCZ+1kEmr7m8Vf//pXM2zYMLNv3z4zbtw4s3TpUmOMMZ9++qk555xzTFVVlTHm+G+n8+fPN8YYU1lZaXJycszOnTvN3r17zTnnnGO2bNlijDHmhRdeMCNHjjTRaJTfTnFGZBDJgByio/Lz883w4cNNcXGx6du3r5k6dappamoy9fX1prGx0RhjzNGjR01JSYl5++23jTHGLFy40MycOTN2+759+4wxJ87UbdmyxRQWFprXXnvNGGNMOBw2gwYNMuvXrzfGGLN+/XojyWzYsMEYY8zkyZPNtGnTTCQSMcYY8+tf/9pMnjzZNDY2mqamJjNjxgzz4IMPGmOMufbaa82jjz4am//dd99tFixYYIwxZsGCBSY/P9/U1tYaY4ypqKgwv/jFL4wxxjzxxBNm6tSpJhwOm3A4bKZMmcKZumRSUVGhkSNHqrCwULfffrsyMjL0/vvv64YbbpAkXXzxxZo4caLeeOONWM2NN94oSRo6dKguvfRSvf7663r33Xc1ZswYjRkzRpI0e/Zs1dTU6Isvvkj8g0K3QgaRDMghOmrlypX68MMPVVtbqyFDhujee+/VsWPHdOONN2rMmDEaP368du3apQ8//FDS8fej3XnnnbH3rvXv3z92Xx9//LFmzpypF198UZdffrkk6ZNPPlEwGNRll10mSbrssss0bNiwdnO46qqrlJaWJklau3at5syZo1AopGAwqLlz52rNmjVxPZbp06fr3HPPlSRNmDBBlZWVkqR169bpmmuuUXp6utLT03X99dc77q3OkdDr1HUHK1euVElJidauXasrrrhCU6dOPWWbsy2P4rp8CiCRQSQHcojOEgwG9d3vflfz5s1TXV2dcnNz9cEHHygYDOo73/lOXMuA5eXlKRwOa/369SouLj7jdidnrk+fPnFtGwwG2y2l19jY2K42IyMj9u/U1NQzLhvpdeaT5kzdk08+qSFDhigjI0NlZWV67733PJ1PeXm5brnlFs2fP1+lpaV69tlnJUk7duzQm2++qUsvvTS2bettVVVVeuONNzRp0iSNHz9eH330kbZu3SpJWrFihQYOHKiBAwcqOztbdXV17cZbuHChAoFAu68RI0Yk6NFCIoMSOUwG5JAcJoPOzuH69etVUFCggwcPatCgQQoGg9q+fXu7M2UzZ87UY489FltzeP/+/bHbcnJytGbNGr3yyiv66U9/KkkqKChQU1OTNm3aJEnatGmTduzYccY5lJeX67nnnlMkElFzc7Oefvrp2Fm/iy66KPYYDxw4oOXLl2vRokUKBAJatGiRHnvssdNmsLy8XC+88IKampoUiURirwGvJMWZupUrV+quu+7S0qVLVVZWpiVLlmjatGnavn27zjvvPM/m9cADD+iiiy7SqlWrdP/99+uJJ55QIBDQ008/rQsvvDC2XUtLiy655BI1NDTo8ccf15AhQyRJy5cv1zXXXKPm5mbl5OTod7/7nQKBgIqKijRq1CiNHj1aQ4cO1auvvipJGjVqlNauXRu732AwKZ6eHoEMvhq7L3LoHXJIDpNBZ+WwoqJCmZmZam5uVn5+vpYuXara2lpdffXV+u1vf6thw4a1OwN877336v7771dpaanS0tKUl5enVatWxW7PysrS6tWrdeWVV2revHl66KGHtGLFCt16662KRqMaO3asCgoK1K9fv9PO56abblJlZaVKS0slHf8gxY9//OPYbd/73vc0cuRIDR06VIMGDVJaWpq2bNmihx9+WHV1dVq8ePEp9zl37lxt3bpVhYWFysnJ0aRJk7R58+a491Gn8+zdfG2MGzfO3HrrrbH/t7S0mLy8PLN48WIPZ5VYCxYs4A3DHiKDx5FDb5HD48iht7pTDuvr62P/fu+998yAAQNMQ0NDh++3u2bQ8z+/RiIRbd68WeXl5bHvpaSkqLy8XG+//baHM0u8zz77THl5eRo6dKhmz56tzz//3Osp9QhksD1y6A1y2B459EZ3y+HLL7+s4uJiFRUV6eabb9bzzz+vXr16dcp9d8cMet7U1dbWqqWlReeff367759//vnas2ePR7NKvLKyMi1btkyrV6/WU089pZ07d2rSpEk6fPiw11PzPTJ4Ajn0Djk8gRx6p7vlcM6cOfqv//ovbdmyRe+//367ZrQjumsGeZNCkpgxY0bs30VFRSorK1N+fr5eeuml2CUEgK5GDpEMyCG81l0z6PmZutzcXKWmpmrv3r3tvr93714NGDDAo1l5r1+/fho+fPjXfpIHnYMMnhk5TBxyeGbkMHHI4el1lwx63tSlp6dr7NixWrduXex70WhU69at04QJEzycmbeOHDmiyspKXXDBBV5PxffI4JmRw8Qhh2dGDhOHHJ5et8mg15/UMMaYFStWmFAoZJYtW2a2bdtmbrrpJtOvXz+zZ88er6eWMHfffbfZuHGj2blzp/nzn/9sysvLTW5ubmyZFHQtMngcOfQWOTyOHHqLHHbfDCb0PXXRaFQ1NTXKyspqd9XlGTNm6Gc/+5nmz5+vvXv3qqioSC+//LIyMzNVX1+fyCl6ZufOnaqoqNBXX32l3NxcTZgwQWvWrFEoFPJkHxhjdPjwYeXl5SklxfMTup3qdDkkg8clUw79nEGJHH4dcpg45PD0kimDUvw5DBhjTKImtXv3bg0ePDhRw6ETVFdXa9CgQV5Po1ORw+7FjxmUyGF3Qw6RDM6Ww4SeqcvKypIk/eIXv2i3jtrZuP521HaJkXj9z//5P53GcjFw4EDrmtblU2w1NTU51bU+Z37S+pgeeOABqxxmZmZ21ZROcaZ1Bb9OdXW101hvvfWWdc327dudxjpy5Ih1jR8zKJ14XA899JBVtnr37u00Xuui5jZczki0XRXCRlVVlXXNgQMHnMayyaExRo2Njb7P4Ztvvvm166Se7ODBg07jHTp0yLrmjTfesK5xzcbjjz9uXTNs2DCnsWpra61rzpZDp6buySef1EMPPaQ9e/aouLhYv/rVrzRu3Liz1rWe2s3IyLA6iLk2dTY/sFtlZ2c7jeXCZeHfRC8W7PXixF+nM3Lop6YuFAo5jZWammpdk8hc+DGD0onHlZmZaZUt1wurujR1Lr8MuowjueXQ9WdDdzj22uiMHPbp08eqcXU9UeBS53JsS09Pt66R3HqARP5Z/mw5tJ5J65pwCxYs0Pvvv6/i4mJNmzZN+/btc54kYIscwmtkEMmAHKIt66bukUce0dy5c3XdddepsLBQS5cuVa9evfTMM890xfyA0yKH8BoZRDIgh2jLqqnrbmvCwZ/IIbxGBpEMyCFOZvWeuq9bE+6TTz45ZftwONzujf095aPQ6FrkEF6zzaBEDtH5yCFO1qXv7lu8eLH69u0b++Jj0/ACOUQyIIdIBuTQ36yaOts14e677z7V1dXFvlwvuQC0RQ7hNZf1MckhOhs5xMmsmjrbNeFCoZCys7PbfQEdRQ7hNZf1MckhOhs5xMmsr1N311136dprr9U3vvENjRs3TkuWLFFDQ4Ouu+66rpgfcFrkEF4jg0gG5BBtWTd1FRUV2r9/v37yk59oz549Kikp0erVq095oybQlcghvEYGkQzIIdpyWlHitttu02233eY8aFlZmdVyJC5LC0nShRde6FSXKC5XvI5Go05j2dYZY9TS0uI0VqJ0NId5eXlWV+dP5IoSLsvBVVZWOo3lslSNaw5trsqfwGWpnXU0g5JUWlpqdTwcNWqU0zguV73/f//v/zmN5cJlBSDXVR5sjm09JYfBYFDBYPwtgevP5WPHjlnXuCwtdvL7DJORzf6O92dy4ta2AAAAQJehqQMAAPABmjoAAAAfoKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfoKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB8IejHoiBEjlJ2dHff2n3zyidM4gwYNcqpLlNraWuuaPn36dMFMeqaioiKr/Xn06FGncYJB+5fZvn37rGu+/PJL6xpJSkmx/90uEAg4jWVbZ4xxGqc76d27t1UOXZ4vVy5jNTc3d8FMTi81NdWpLhqNxr1tT8igJGVnZ1v9XLbZtq2JEyda1/zjP/6jdc3vf/976xpXrhlxPY5+Hc7UAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACAD9ivNN4JUlJSrBaKdl0g2mWR3a5YYLczuS7mneyPywsXXXSR1aLU1dXVTuM0NTVZ1wSD9i/Nr776yrpGktLT061rXOYn2eewJyymnpmZqczMzC4f58CBA9Y1kUjEumbVqlXWNZKUm5trXRMOh53GamlpiXvbnpBB6fhr0+b1WVpa2oWz6bhEPm+uPUpX4EwdAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4gNuq3B3U0tJitaCyy4LoktuCvnfeead1zdatW61rJGn9+vXWNYlaSL0n2Llzp7KysuLe3nXRZpcc7tu3z7omGo1a10hSWlqadU1qaqrTWDb7goXUvef6PLsIh8PWNa4Zsd3fPSGL2dnZys7O9noap3Xs2DHrmptuuslprB/+8IfWNYn42RDvtpypAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfoKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfoKkDAADwAZo6AAAAH6CpAwAA8IGgF4MaY2SMiXv7yspKp3FKS0uta4JB+12Snp5uXeMqJcWtD7d9XMYYNTU1OY3VXTQ3N1s9Rtf9EY1GrWsyMjKsa+rq6qxrJOnAgQPWNbW1tU5jpaWlxb2tMUYtLS1O43QnkUhEkUiky8c599xzrWtSU1O7YCanl5mZaV0TDoedxrL5+WOzLc6uurraumbEiBFdMBPvdUUOOVMHAADgAzR1AAAAPmDV1C1cuFCBQKDdl19PiyJ5kUMkA3IIr5FBnMz6DWSjRo3S2rVrT9yBw3vQgI4ih0gG5BBeI4Noy/rZDwaDGjBgQFfMBYgbOUQyIIfwGhlEW9bvqfvss8+Ul5enoUOHavbs2fr888/PuG04HFZ9fX27L6AzkEMkA3IIr9lkUCKHfmfV1JWVlWnZsmVavXq1nnrqKe3cuVOTJk3S4cOHT7v94sWL1bdv39jX4MGDO2XS6NnIIZIBOYTXbDMokUO/C5gOXITn0KFDys/P1yOPPKIbbrjhlNvD4XC76wjV19dr8ODB2r9/v7Kzs+Me5/e//73T/P75n//ZumbevHnWNdu2bbOukaTVq1db1+Tm5jqNdezYMavtjTE6duyY6urqrJ4rL7jm8C9/+Yv69OkT9ziJvE7d2X7bPp1FixZZ10jSnj17rGtcrjUluV2nrjtkUHLP4ccff6ysrKy4x0nkD+HXXnvNumbatGlOY5133nnWNa7XqTty5Ejc27ZeV7U75PBsGZTOnMNEPT6uU3eCTX6NMYpGo2d9njr0jsp+/fpp+PDh2rFjx2lvD4VCCoVCHRkCOCtyiGRADuG1s2VQIod+16Hr1B05ckSVlZW64IILOms+gDVyiGRADuE1Mgirpu6ee+7Rpk2bVFVVpbfeektXXnmlUlNTNWvWrK6aH3AKcohkQA7hNTKIk1n9+XX37t2aNWuWDhw4oP79+2vixIl655131L9//66aH3AKcohkQA7hNTKIk1k1dStWrOiUQVtaWqwW6v77v/97p3FcPgPi8qZxlzfCu0pJYWW3zsphY2Oj1YU6Xd+U7cLmAwWtXK9V9fHHH1vXZGRkOI3lJ52Vw3A4rPT09E65r87m8t6r73//+05jXXzxxdY1n332mdNYfrmMR2dlUJJ+85vfKDMzM+7t//t//+9O44wdO9a6pgOf5+xx6BAAAAB8gKYOAADAB2jqAAAAfICmDgAAwAdo6gAAAHyApg4AAMAHaOoAAAB8gKYOAADAB2jqAAAAfICmDgAAwAdo6gAAAHyApg4AAMAHaOoAAAB8IOjFoMYYGWPi3r5fv35O43z00UfWNX369LGuqa2tta5xFY1GEzaW3x06dEhNTU1dPo7LcxYM2r80J0+ebF0jSWvXrrWusXn9tpWammo1RktLi9M43UljY6PT850IU6ZMSUiNJJWUlFjXJOL121McOnRIjY2NcW//zDPPOI1z9OhR65rMzEzrGtdjVCJ/xtrMMd5tOVMHAADgAzR1AAAAPkBTBwAA4AM0dQAAAD5AUwcAAOADNHUAAAA+QFMHAADgAzR1AAAAPkBTBwAA4AM0dQAAAD5AUwcAAOADCV1wsHXtssOHD1vVua7/eOTIEeuaSCRiXdPc3GxdI0n19fXWNa7r0tmug9e6vev6ecms9TG5rEHowuU5c9nvNus2dnSsROTCzxmUTjyuhoYGqzqX40Z34HKcT2Q2/J7DcDhsVWe7fSuX/CbyGJWsmYr3eBgwCXwEu3fv1uDBgxM1HDpBdXW1Bg0a5PU0OhU57F78mEGJHHY35BDJ4Gw5TGhTF41GVVNTo6ysLAUCgXa31dfXa/DgwaqurlZ2dnaippR0kmU/GGN0+PBh5eXlKSXFX3+lJ4dnlwz7wc8ZlMhhPJJhP/TUHCbDvk8WybAv4s1hQv/8mpKSctbfdLKzs3t8gKTk2A99+/b1dPyuQg7j5/V+8GsGJXJow+v90JNz6PW+TyZe74t4cui/XzsAAAB6IJo6AAAAH0iapi4UCmnBggUKhULWtUOGDFFBQYFKSko0cuRI/eAHP7D+RFmrZcuW6dvf/rZTbWew2Q9//OMfNWXKlK6fVA9CDo8jh97qSA47Q9ssFxQU6MEHHzxrTSAQ0KFDh6zGqaqq0tKlS894u9f7oSdLhn1PDh0YH8jPzzcffPCBMcaYlpYW8w//8A/miSeecLqvZ5991nzrW9/q0Hyampo6VB+vP/zhD2by5MkJGQtnRw7hF22zvHv3bpOdnW3efffdr62RZA4ePGg1zoYNG0xxcbHbJOF75NBe0pyp6yyRSERHjx5VTk6OPvroI02cOFGlpaUqLCzUz3/+83bbzZs3T6NHj1ZxcbGmT59+yn3V1NTom9/8pp555hlJ0ltvvaWSkhKNGTNG119/vYqLi7Vx40ZJ0pQpU3THHXdowoQJuvzyy9XS0hK7/9GjR+v222+PXQNvzpw5WrJkSWyce+65RwsXLpQkLVy4UBUVFbriiitUWFioqVOn6quvvpIkNTU16Uc/+pEuvvhijRs3Ths2bOiCPYjOQA7hFwMHDtSIESO0a9cu7dixQ+Xl5SoqKlJJSYleeeWVdts+/PDDuuSSSzR8+HAtX7489v3XXntNpaWlKioq0uTJk7Vt2zZJ0g9/+ENt375dJSUlmjlzZiIfFroZchgnr7vKzpCfn2+GDx9uiouLTd++fc3UqVNNU1OTqa+vN42NjcYYY44ePWpKSkrM22+/bYwxZuHChWbmzJmx2/ft22eMOXGGZMuWLaawsNC89tprxhhjwuGwGTRokFm/fr0xxpj169cbSWbDhg3GGGMmT55spk2bZiKRiDHGmF//+tdm8uTJprGx0TQ1NZkZM2aYBx980BhjzLXXXmseffTR2Pzvvvtus2DBAmOMMQsWLDD5+fmmtrbWGGNMRUWF+cUvfmGMMeaJJ54wU6dONeFw2ITDYTNlyhTOkCQRcgi/aHuG5K9//asZNmyY2bdvnxk3bpxZunSpMcaYTz/91JxzzjmmqqrKGHP8DMn8+fONMcZUVlaanJwcs3PnTrN3715zzjnnmC1bthhjjHnhhRfMyJEjTTQa9dUZEnQ+cmjPN2fqVq5cqQ8//FC1tbUaMmSI7r33Xh07dkw33nijxowZo/Hjx2vXrl368MMPJR1/H9Cdd94Z+xt5//79Y/f18ccfa+bMmXrxxRd1+eWXS5I++eQTBYNBXXbZZZKkyy67TMOGDWs3h6uuukppaWmSpLVr12rOnDkKhUIKBoOaO3eu1qxZE9djmT59us4991xJ0oQJE1RZWSlJWrduna655hqlp6crPT1d119/vePeQlchh/CLiooKjRw5UoWFhbr99tuVkZGh999/XzfccIMk6eKLL9bEiRP1xhtvxGpuvPFGSdLQoUN16aWX6vXXX9e7776rMWPGaMyYMZKk2bNnq6amRl988UXiHxS6HXJoJ6HXqUuEYDCo7373u5o3b57q6uqUm5urDz74QMFgUN/5znfiWkopLy9P4XBY69evV3Fx8Rm3O/mCoX369Ilr22Aw2G5JnMbGxna1GRkZsX+npqaecRmyk8dH8iCH6O5WrlypkpISrV27VldccYWmTp16yjZne+7JBjqKHNpJmjN1Tz75pIYMGaKMjAyVlZXpvffec76v9evXq6CgQAcPHtSgQYMUDAa1ffv2dmcoZs6cqcceeyy2ft3+/ftjt+Xk5GjNmjV65ZVX9NOf/lSSVFBQoKamJm3atEmStGnTJu3YseOMcygvL9dzzz2nSCSi5uZmPf3007GzLRdddFHs8R04cECrVq3Sxo0bFQgEtGjRIj322GMKBAIaMWLEKff5wgsvqKmpSZFIRM8++6zzPsKpOjODEjmEm87OYUeVl5frlltu0fz581VaWhp7vnfs2KE333xTl156aWzb1tuqqqr0xhtvaNKkSRo/frw++ugjbd26VZK0YsUKDRw4UAMHDlR2drbq6upOGXPhwoUKBALtvk7OIbpWT89hd81gUpypW7lype666y4tXbpUZWVlWrJkiaZNm6bt27frvPPOi+s+KioqlJmZqebmZuXn52vp0qWqra3V1Vdfrd/+9rcaNmxYuw7/3nvv1f3336/S0lKlpaUpLy9Pq1atit2elZWl1atX68orr9S8efP00EMPacWKFbr11lsVjUY1duxYFRQUqF+/fqedz0033aTKykqVlpZKOv4G9h//+Mex2773ve9p5MiRGjp0qMaPH6+amhqNGjVK06dPV11dnX72s58pGAxqxYoVsfucO3eutm7dqsLCQuXk5GjSpEnavHmz5d7G6XRGBiVyiI7prBx2tgceeEAXXXSRVq1apfvvv19PPPGEAoGAnn76aV144YWx7VpaWnTJJZeooaFBjz/+uIYMGSJJWr58ua655ho1NzcrJydHv/vd7xQIBFRUVKRRo0Zp9OjRGjp0qF599dXYfY0aNUpr166N/T8YTIofVz0COTyew26ZQa/f1GeMMePGjTO33npr7P8tLS0mLy/PLF682MNZnaq+vj727/fee88MGDDANDQ0dMp9L1iwwDdv1OyOuksGjSGHftadctiVyKG3yGH3zaDnf36NRCLavHmzysvLY99LSUlReXm53n77bQ9ndqqXX35ZxcXFKioq0s0336znn39evXr16rT7/+yzz5SXl6ehQ4dq9uzZ+vzzzzvtvnFm3SmDEjn0q+6Ww65GDr1BDk/ojhn0/FxibW2tWlpadP7557f7/vnnn69PPvnEo1md3pw5czRnzpwuue+ysjItW7ZMBQUF+vLLL7Vo0SJNmjRJW7duVVZWVpeMieO6UwYlcuhX3S2HXYkceoccHtddM+h5U4fjZsyYEft3UVGRysrKlJ+fr5deein20W2gq5FDJANyCK911wx6/ufX3Nxcpaamau/eve2+v3fvXg0YMMCjWXmvX79+Gj58+Nd+shGdgwyeGTlMHHJ4ZuQwccjh6XWXDHre1KWnp2vs2LFat25d7HvRaFTr1q3ThAkTPJyZt44cOaLKykpdcMEFXk/F98jgmZHDxCGHZ0YOE4ccnl63yaDXn9QwxpgVK1aYUChkli1bZrZt22Zuuukm069fP7Nnzx6vp5Ywd999t9m4caPZuXOn+fOf/2zKy8tNbm5ubNkodC0yeBw59BY5PI4ceoscdt8MJsV76ioqKrR//3795Cc/0Z49e1RSUqLVq1ef8kZNP9u9e7dmzZqlAwcOqH///po4caLeeeeddstGoeuQwePIobfI4XHk0FvksPtmMGCMMYkaLBqNqqamRllZWT1q2Y7uyBijw4cPKy8vTykpnv+VvlORw+7BzxmUyGF3QQ6RDOLNYULP1NXU1Gjw4MGJHBIdVF1drUGDBnk9jU5FDrsXP2ZQIofdDTlEMjhbDhPa1LVe2+Xf//3f1bt377jrzj33XKfxDh8+bF3z7rvvWte0XULJxsGDB61r6uvrncY6evSo1fbGGIXD4aS+Ho+r1sf0/vvvWz2+1vVZbdXU1FjXfPDBB9Y1rteQ+stf/mJdU11d7TTWoUOHrGv8mEHpxOP65S9/qYyMjLjrotGo03gux462S9bF6z/+4z+sayQ5NRbHjh1zGqupqcm6xu85/Nd//VerHLpe8Ly5udm6xuVnZdt1tG38+7//u3WN68/llpYW65qz5dCpqXvyySf10EMPac+ePSouLtavfvUrjRs37qx1rad2e/fubdXU9enTx2WacvnLsk2oW6WmplrXSHI6le96+t/1tHoyn47vaA6zsrKsDtTp6elO87TJeiuXHLrOzyW/icyFHzMonXhcGRkZyszMjHtM16YuEolY17isdZmdnW1dI7k9z+TwuM7Koc1xxyazbbk01KFQyLrG9XjY3XNo3SG0LvS7YMECvf/++youLta0adO0b98+50kCtsghvEYGkQzIIdqybuoeeeQRzZ07V9ddd50KCwu1dOlS9erVS88880xXzA84LXIIr5FBJANyiLasmjrbhX7D4bDq6+vbfQEdRQ7hNZdFz8khOhs5xMmsmrqvW+h3z549p2y/ePFi9e3bN/bFJ2zQGcghvGabQYkcovORQ5ysSy+6c99996muri725fqJOaAjyCGSATlEMiCH/mb10SbbhX5DoZDTp1aAr0MO4TWXRc/JITobOcTJrM7UsdAvkgE5hNfIIJIBOcTJrC9CdNddd+naa6/VN77xDY0bN05LlixRQ0ODrrvuuq6YH3Ba5BBeI4NIBuQQbVk3dSz0i2RADuE1MohkQA7RltOKErfddptuu+0250EvvPBCqyv55+XlOY3z1ltvWdc0NDRY1/Tv39+6xpXr1eRtlyMxxqixsdFprETpaA4bGxuVlpYW9/Yu2ZCk119/3brmyy+/tK5xWYJLcluSzPUTczZL9xljnJbRSaSOZlCSxo8fb7VqzoEDB5zGKSwstK5xzZQLl5VNXFa8kOyOoz0lh5deeqlVDl1/FtXV1VnXVFZWWte4LBMqub2+XFfXsM1hPKtkdemnXwEAAJAYNHUAAAA+QFMHAADgAzR1AAAAPkBTBwAA4AM0dQAAAD5AUwcAAOADNHUAAAA+QFMHAADgAzR1AAAAPkBTBwAA4AM0dQAAAD5AUwcAAOADQS8GTU1NVWpqqhdDn1V9fb11zapVq5zGmjFjhnXNsWPHnMZqbGy02j4ajTqN053k5OQoOzs77u3r6uqcxrnlllusa9LS0qxrrrrqKusaV8YYp7pgMP5DjjFGLS0tTuN0JwUFBVY5/NOf/uQ0zrnnnmtdk5GR4TSWC5tstEpJcTsvEQgEnOr8LBQKKRQKxb29688Il7ovvvjCusb12HHllVda17gcryWpubk57m3jPR5ypg4AAMAHaOoAAAB8gKYOAADAB2jqAAAAfICmDgAAwAdo6gAAAHyApg4AAMAHaOoAAAB8gKYOAADAB2jqAAAAfICmDgAAwAdo6gAAAHzAfgXlzhg0GHReANdGbm6udc3ixYu7YCanl5WVZV3jusC268LXfpaVlWX1HLgsOC5Jffv2daqzdejQoYSMI9ktRN0WC6mfyhgjY0zc2w8ZMqTrJnOSqVOnJmysvXv3Wtf069ev8yfSQ6Wnpys9PT3u7aPRqNM4Lj/DbObV6txzz7Wukdxy6DI/SQqHw3Fva4xRS0vLWbfjJz0AAIAP0NQBAAD4AE0dAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPiA2wrlHdTc3KympqYuHycSiXT5GB3hMr+//OUvTmPl5uZabe+6WLOfFRYWej2Fr3XfffclbKx4FpY+HZtc2Sxy351Fo1Gr/XLgwAHncWw988wz1jUPPPCAdY0kbdiwwbomNTXVaaxAIOBU52dpaWlWC9O7/owIhULWNUePHrWu+etf/2pd4yotLc2pzia/xpi4+ibO1AEAAPgATR0AAIAP0NQBAAD4AE0dAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPhA0ItBm5qa1NTU1OXjpKamdvkYHVFfX29d83d/93dOY0UiEavtjTFO46BztLS0WNf88z//s9NYgUDAuiYajTqNZVPXUzJojLF6rOeee67zOLbOOecc65ovvvjCuibRbH42GGMS8vPKa6mpqVb7JRQKOY2TkmJ/LmnkyJHWNW+88YZ1TUfqXGRlZcW9bbyvX87UAQAA+IBVU7dw4UIFAoF2XyNGjOiquQGnRQ6RDMghvEYGcTLrP7+OGjVKa9euPXEHQU/+gosejhwiGZBDeI0Moi3rZz8YDGrAgAFdMRcgbuQQyYAcwmtkEG1Zv6fus88+U15enoYOHarZs2fr888/74p5AV+LHCIZkEN4jQyiLaszdWVlZVq2bJkKCgr05ZdfatGiRZo0aZK2bt162k9xhMNhhcPh2P9dPu0JnIwcIhmQQ3jNNoMSOfQ7q6ZuxowZsX8XFRWprKxM+fn5eumll3TDDTecsv3ixYu1aNGijs8SaIMcIhmQQ3jNNoMSOfS7Dl3SpF+/fho+fLh27Nhx2tvvu+8+1dXVxb6qq6s7MhxwWuQQyYAcwmtny6BEDv2uQ03dkSNHVFlZqQsuuOC0t4dCIWVnZ7f7AjobOUQyIIfw2tkyKJFDv7Nq6u655x5t2rRJVVVVeuutt3TllVcqNTVVs2bN6qr5Aacgh0gG5BBeI4M4mdV76nbv3q1Zs2bpwIED6t+/vyZOnKh33nlH/fv376r5Aacgh0gG5BBeI4M4mVVTt2LFiq6aBxA3cohkQA7hNTKIk3ly6eljx45ZLRzsav/+/dY1LgsHu3L5KHljY6PTWLZ1PWUxdRvHjh1zqnNZDDwvL8+6xvU5CwQCTnXoHC0tLWppaYl7+wMHDjiN4/I879mzx2msRHFZHB6nl5qaavVzOS0tzWmczMxM65rLL7/cuua8886zrkm0rjj28ooAAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfoKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfoKkDAADwgaAXg9bX1ysajXb5OL169eryMSRpwoQJTnWHDh2yrjly5IjTWE1NTU51frZkyRJlZGTEvX1jY6PTOL/85S+ta4wxTmO5cBnLdX42dYncB16KRqNWx8PS0lKncV588UXrGpdjqOuxfdSoUdY1PSUjiRCJRBSJROLevk+fPk7j9O/f37omJcX+/JNrNoYMGWJdU1VV5TRWIBBwqvs6nKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfoKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPCBhK792roWW0NDg1VdfX2903gu66S6jNXc3GxdI7mtkZjotQ79uLZi62OyXcvVde3XRK6tmiiJXPs12feFq9bHdfjwYau6lpYWp/GOHTtmXWOzFmgr1+Ohy+NyXWeWHJ7Q+rhsf14Gg27tg8tx1GXtV9s+o5VLplx7lK7IYcAkMKm7d+/W4MGDEzUcOkF1dbUGDRrk9TQ6FTnsXvyYQYkcdjfkEMngbDlMaFMXjUZVU1OjrKwsBQKBdrfV19dr8ODBqq6uVnZ2dqKmlHSSZT8YY3T48GHl5eU5/ZaUzMjh2SXDfvBzBiVyGI9k2A89NYfJsO+TRTLsi3hzmNA/v6akpJz1N53s7OweHyApOfZD3759PR2/q5DD+Hm9H/yaQYkc2vB6P/TkHHq975OJ1/sinhz679cOAACAHoimDgAAwAeSpqkLhUJasGCBQqGQZ3MYMmSICgoKVFJSooKCAj344INnrQkEAjp06JDVOFVVVVq6dOlpb0uG/dCTdWT/t83PyJEj9YMf/MD5E1jLli3Tt7/9bafazmCzH/74xz9qypQpXT+pHsTr40AyHAsl7/dDT9bRfc/x0CMGMfn5+eaDDz4wxhize/duk52dbd59992vrZFkDh48aDXOhg0bTHFxsdskkbTa5qelpcX8wz/8g3niiSec7uvZZ5813/rWtzo0n6ampg7Vx+sPf/iDmTx5ckLGQmJwLERHcTz0RtKcqUs2AwcO1IgRI7Rr1y7t2LFD5eXlKioqUklJiV555ZV22z788MO65JJLNHz4cC1fvjz2/ddee02lpaUqKirS5MmTtW3bNknSD3/4Q23fvl0lJSWaOXNmIh8WEiQSiejo0aPKycnRRx99pIkTJ6q0tFSFhYX6+c9/3m67efPmafTo0SouLtb06dNPua+amhp985vf1DPPPCNJeuutt1RSUqIxY8bo+uuvV3FxsTZu3ChJmjJliu644w5NmDBBl19+uVpaWmL3P3r0aN1+++2xa4/NmTNHS5YsiY1zzz33aOHChZKkhQsXqqKiQldccYUKCws1depUffXVV5KkpqYm/ehHP9LFF1+scePGacOGDV2wB5EsOBaiozgeJpBn7WQSavubxV//+lczbNgws2/fPjNu3DizdOlSY4wxn376qTnnnHNMVVWVMeb4b6fz5883xhhTWVlpcnJyzM6dO83evXvNOeecY7Zs2WKMMeaFF14wI0eONNFolN9OfSo/P98MHz7cFBcXm759+5qpU6eapqYmU19fbxobG40xxhw9etSUlJSYt99+2xhjzMKFC83MmTNjt+/bt88Yc+I30y1btpjCwkLz2muvGWOMCYfDZtCgQWb9+vXGGGPWr19vJJkNGzYYY4yZPHmymTZtmolEIsYYY37961+byZMnm8bGRtPU1GRmzJhhHnzwQWOMMddee6159NFHY/O/++67zYIFC4wxxixYsMDk5+eb2tpaY4wxFRUV5he/+IUxxpgnnnjCTJ061YTDYRMOh82UKVM4U+czHAvRURwPvcGZupNUVFRo5MiRKiws1O23366MjAy9//77uuGGGyRJF198sSZOnKg33ngjVnPjjTdKkoYOHapLL71Ur7/+ut59912NGTNGY8aMkSTNnj1bNTU1+uKLLxL/oJAwK1eu1Icffqja2loNGTJE9957r44dO6Ybb7xRY8aM0fjx47Vr1y59+OGHko6//+LOO++MvVejf//+sfv6+OOPNXPmTL344ou6/PLLJUmffPKJgsGgLrvsMknSZZddpmHDhrWbw1VXXaW0tDRJ0tq1azVnzhyFQiEFg0HNnTtXa9asieuxTJ8+Xeeee64kacKECaqsrJQkrVu3Ttdcc43S09OVnp6u66+/3nFvIZlxLERHcTxMvIRep647WLlypUpKSrR27VpdccUVmjp16inbnHyhUNvb4X/BYFDf/e53NW/ePNXV1Sk3N1cffPCBgsGgvvOd78S1VE5eXp7C4bDWr1+v4uLiM253ct769OkT17bBYLDd0kyNjY3tajMyMmL/Tk1NPePyT+TdnzgWorNwPEycpDlT9+STT2rIkCHKyMhQWVmZ3nvvPU/nU15erltuuUXz589XaWmpnn32WUnSjh079Oabb+rSSy+Nbdt6W1VVld544w1NmjRJ48eP10cffaStW7dKklasWKGBAwdq4MCBys7OVl1dXbvxFi5cqEAg0O5rxIgRCXq0kDo/g+vXr1dBQYEOHjyoQYMGKRgMavv27e1+M5w5c6Yee+wxhcNhSdL+/ftjt+Xk5GjNmjV65ZVX9NOf/lSSVFBQoKamJm3atEmStGnTJu3YseOMcygvL9dzzz2nSCSi5uZmPf3007Hfci+66KLYYzxw4IBWrVqljRs3KhAIaNGiRXrsscdOm8Py8nK98MILampqUiQSieUfnaOnHwsljofJoKcfD5cvX65Fixa1Ox6eLoPJdjxMijN1K1eu1F133aWlS5eqrKxMS5Ys0bRp07R9+3add955ns3rgQce0EUXXaRVq1bp/vvv1xNPPKFAIKCnn35aF154YWy7lpYWXXLJJWpoaNDjjz+uIUOGSJKWL1+ua665Rs3NzcrJydHvfvc7BQIBFRUVadSoURo9erSGDh2qV199VZI0atQorV27Nna/rgsmw15nZbCiokKZmZlqbm5Wfn6+li5dqtraWl199dX67W9/q2HDhrU743Hvvffq/vvvV2lpqdLS0pSXl6dVq1bFbs/KytLq1at15ZVXat68eXrooYe0YsUK3XrrrYpGoxo7dqwKCgrUr1+/087npptuUmVlpUpLSyUdf+Pwj3/849ht3/ve9zRy5EgNHTpU48ePV01NjUaNGqXp06errq5OP/vZzxQMBrVixYrYfc6dO1dbt25VYWGhcnJyNGnSJG3evNlib+NMOBa+Grsvjofe4Xg4VIMGDVJaWpq2bNmihx9+WHV1dVq8ePEp95l0x0PP3s3Xxrhx48ytt94a+39LS4vJy8szixcv9nBWibVgwQLeMOyh7pTB+vr62L/fe+89M2DAANPQ0NAp900OvdWdctiVyKG3ulMOu+p42F0z6PmfXyORiDZv3qzy8vLY91JSUlReXq63337bw5kl3meffaa8vDwNHTpUs2fP1ueff+71lHqE7pbBl19+WcXFxSoqKtLNN9+s559/Xr169eq0+yeH3uhuOexq5NAb3S2HXXk87I4Z9Px8dm1trVpaWnT++ee3+/7555+vTz75xKNZJV5ZWZmWLVumgoICffnll1q0aJEmTZqkrVu3Kisry+vp+Vp3y+CcOXM0Z86cLrlvcuid7pbDrkQOvdPdcthVx8PumkHPmzocN2PGjNi/i4qKVFZWpvz8fL300kuxSwgAXY0cIhmQQ3itu2bQ8z+/5ubmKjU1VXv37m33/b1792rAgAEezcp7/fr10/Dhw7/2kzzoHGTwzMhh4pDDMyOHiUMOT6+7ZNDzpi49PV1jx47VunXrYt+LRqNat26dJkyY4OHMvHXkyBFVVlbqggsu8HoqvkcGz4wcJg45PDNymDjk8PS6TQa9/qSGMcasWLHChEIhs2zZMrNt2zZz0003mX79+pk9e/Z4PbWEufvuu83GjRvNzp07zZ///GdTXl5ucnNzY8ukoGuRwePIobfI4XHk0FvksPtmMKHvqYtGo6qpqVFWVla7qy7PmDFDP/vZzzR//nzt3btXRUVFevnll5WZman6+vpETtEzO3fuVEVFhb766ivl5uZqwoQJWrNmjUKhkCf7wBijw4cPKy8vTykpnp/Q7VSnyyEZPC6ZcujnDErk8OuQw8Qhh6eXTBmU4s9hwBhjEjWp3bt3a/DgwYkaDp2gurpagwYN8noanYocdi9+zKBEDrsbcohkcLYcJvRMXevHgCdNmmR1dfBErqUWiUSsa/70pz85jTVw4EDrmmPHjjmN1XZNOxvJ/NFtV62P6aGHHlJmZmbcda45dPm96dChQ9Y1b775pnWNJO3atSshNZJ09OhR6xo/ZlA68bjefPPNr12f8mSuZ4sOHjxoXbN69Wrrmv/1v/6XdY0kp2XADh8+7DTWkSNHrGv8nsNdu3YpOzs77rpoNOo0nsvP2HjWhj3Z//2//9e6RpLTMl9/+9vfnMZyye/ZcujU1D355JN66KGHtGfPHhUXF+tXv/qVxo0bd9a61h+KwWAwaZs6l6DavBDacnlciV4s2OvFib9OR3OYmZmZtE2dy0EsLS3NukY6vji1rUTmwo8ZlE48rj59+lg1DK5NXVNTk3VN20XM4+V6PHR5XOTwuM7IYXZ2dtI2denp6dY1Nsf2trr78dD6VdS6JtyCBQv0/vvvq7i4WNOmTdO+ffucJwnYIofwGhlEMiCHaMu6qXvkkUc0d+5cXXfddSosLNTSpUvVq1cvPfPMM10xP+C0yCG8RgaRDMgh2rJq6rrbmnDwJ3IIr5FBJANyiJNZvafOdk24cDiscDgc+39P+Sg0uhY5hNdc1sckh+hs5BAn69KL7ixevFh9+/aNffGxaXiBHCIZkEMkA3Lob1ZNne2acPfdd5/q6upiX9XV1R2bLSByCO+5rI9JDtHZyCFOZtXU2a4JFwqFYh+Ttv24NHAm5BBec1kfkxyis5FDnMz6OnV33XWXrr32Wn3jG9/QuHHjtGTJEjU0NOi6667rivkBp0UO4TUyiGRADtGWdVNXUVGh/fv36yc/+Yn27NmjkpISrV69+pQ3agJdiRzCa2QQyYAcoi2nFSVuu+023Xbbbc6DnnfeeVZXiHa9WnPbT/jE68svv3QaK1FcryZvuw8TuCSws47mMBQKWV0x3+VK6JLb6hAubFZpaat3797WNeecc47TWDb70Bij5uZmp3ESpaMZlI7vf5vnwPVK/g0NDdY1H3zwgXXN/PnzrWuk42uQ2krEaj494VgoHX9tuh7jbLj+DLN12WWXOdVt2bLFusbltSXZvZaNMXGNk5i9CwAAgC5FUwcAAOADNHUAAAA+QFMHAADgAzR1AAAAPkBTBwAA4AM0dQAAAD5AUwcAAOADNHUAAAA+QFMHAADgAzR1AAAAPkBTBwAA4AM0dQAAAD4Q9GLQiy66SBkZGXFvHw6HncbZvn27dc2mTZusawYNGmRdI8lqH7RqampyGislxa5/N8YoGo06jdVdNDc3W+1P133f0NBgXVNfX29d06tXL+saSUpNTbWuCYVCTmNlZmbGva0xRocPH3YapzsJBoMKBuM/FDc3NzuN89lnn1nXzJ8/37rmH//xH61rJOk3v/mNdY1LdiUpEAhYbW+McRqnO8nIyLD6meSaw0gkYl1j8/polZ6ebl0jSQUFBdY127ZtcxrL5jgfjUbj+lnCmToAAAAfoKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPABmjoAAAAfoKkDAADwAZo6AAAAH6CpAwAA8AGaOgAAAB+gqQMAAPAB+1VyO8H06dPVp0+fuLevqqpyGucvf/mLdc33v/99p7FcuCxSbLsQdUfr/CwcDislJf7fa8LhsNM4TU1N1jUui17v3r3bukaSdu3aZV3jui9s9ndPWERdknr16qXevXvHvf2xY8ecxtm/f791jcsxKhqNWte4SktLc6qzzWFLS4vTON1JNBq1eu5cn2eXY1tzc7N1jc1z3FZhYaF1zdtvv+001ueffx73tvHub87UAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACAD9iv1twJzjvvPGVlZcW9/ZEjR5zGcVnE/s4777SuyczMtK6RpDVr1ljXuC5S7LIv/K6lpcVqoe6mpiancVwWo7ZZ6LnVJ598Yl0juS2wbYxxGsumznUMv/vqq6+c6lyOHf3797euSeSxhuNa52lsbFR6enqXj+NyPHTJbjQata6RpJycHOuaAQMGOI1l0zvE+7OKM3UAAAA+QFMHAADgAzR1AAAAPkBTBwAA4AM0dQAAAD5AUwcAAOADNHUAAAA+QFMHAADgAzR1AAAAPkBTBwAA4AM0dQAAAD5AUwcAAOADNHUAAAA+EPRi0JaWFrW0tMS9fWZmptM4AwYMsK656667rGvmzp1rXSNJzz77rHXNOeec4zRWJBJxqvOzcDisQCAQ9/Y2me1o3Ysvvmhd45qNaDRqXWOMcRor2cZIBtFo1Oo56Nevn9M4waD94T47O9u65vbbb7eukaT58+c71blIS0tL2FjdRXp6utLT0+Pe/siRI07juOSwubnZusbluCZJvXv3dqpzkZIS/3m1eI+HnKkDAADwAZo6AAAAH7Bq6hYuXKhAINDua8SIEV01N+C0yCGSATmE18ggTmb9x+1Ro0Zp7dq1J+7A4e/jQEeRQyQDcgivkUG0Zf3sB4NBpw8gAJ2JHCIZkEN4jQyiLev31H322WfKy8vT0KFDNXv2bH3++edn3DYcDqu+vr7dF9AZyCGSATmE12wyKJFDv7Nq6srKyrRs2TKtXr1aTz31lHbu3KlJkybp8OHDp91+8eLF6tu3b+xr8ODBnTJp9GzkEMmAHMJrthmUyKHfWTV1M2bM0D//8z+rqKhI06ZN06pVq3To0CG99NJLp93+vvvuU11dXeyrurq6UyaNno0cIhmQQ3jNNoMSOfS7Dr2jsl+/fho+fLh27Nhx2ttDoZBCoVBHhgDOihwiGZBDeO1sGZTIod916Dp1R44cUWVlpS644ILOmg9gjRwiGZBDeI0Mwqqpu+eee7Rp0yZVVVXprbfe0pVXXqnU1FTNmjWrq+YHnIIcIhmQQ3iNDOJkVn9+3b17t2bNmqUDBw6of//+mjhxot555x3179+/q+YHnIIcIhmQQ3iNDOJkVk3dihUrOmXQlpYWq0XOe/Xq5TTO1VdfbV3zb//2b9Y1X/em1K+zatUq65rMzEynsRoaGpzqklFn5bC5udlqoWibzLZls2hzq5tuusm6xvWio01NTdY18S4u3ZE61zESpbNy2KtXL6tjXEZGhtM4ubm51jU33nijdc2hQ4esa1wNGjQoYWMlo87KoCRFIhFFIpG4t3c93kSjUesal2Ova9/gwvXnq83+jncfsPYrAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACAD9DUAQAA+ABNHQAAgA/Q1AEAAPgATR0AAIAP0NQBAAD4AE0dAACADwS9GLSpqUlNTU1xb5+Tk+M0TmNjo3XNtddea13zpz/9ybpGksLhsHVNZmam01g4VXNzs5qbm+PevqWlxWkcY4x1TW5urnXNVVddZV0jSVVVVdY1r7/+utNYNvvQZb91R9u2bVOfPn3i3r53795O4xQXF1vX9O3b17pmxYoV1jWS9M1vftOpzkUwGP+PPmOM82u/O0lJSVFKStef54lGo9Y1qampXTCT04tEItY1hw4dchrLpkeJd79xpg4AAMAHaOoAAAB8gKYOAADAB2jqAAAAfICmDgAAwAdo6gAAAHyApg4AAMAHaOoAAAB8gKYOAADAB2jqAAAAfICmDgAAwAcSuvZr61qOR44ccaqzZTuOJDU0NFjXuKwV51rnsm6eZL8PW7f34/qbrY/Jdu1d1/UfXepcsuGylrAkq3WYWyUih37OoHTicdkec1z3vcvx8OjRo9Y1LnmS3F4n5LDjWh/X4cOHrepc973NetutXDPlwuV14jo/m33Yuu3ZcpjQpq41NJdeemkih0UHHD582GlR72TWmsNf/vKXHs8E8fBjBqUTOSwvL/d4JoiH33N40UUXeTwTxONsOQyYBP76EY1GVVNTo6ysLAUCgXa31dfXa/DgwaqurlZ2dnaippR0kmU/GGN0+PBh5eXlKSXFX3+lJ4dnlwz7wc8ZlMhhPJJhP/TUHCbDvk8WybAv4s1hQs/UpaSkaNCgQV+7TXZ2do8PkJQc+8GPv5VK5NCG1/vBrxmUyKENr/dDT86h1/s+mXi9L+LJof9+7QAAAOiBaOoAAAB8IGmaulAopAULFigUCnk9FU+xH7zF/j+O/eAt9v9x7AfvsO9P6E77IqEflAAAAEDXSJozdQAAAHBHUwcAAOADNHUAAAA+QFMHAADgA0nT1D355JMaMmSIMjIyVFZWpvfee8/rKSXUwoULFQgE2n2NGDHC62n1KD09gxI5TAbkkBwmg56ew+6awaRo6lauXKm77rpLCxYs0Pvvv6/i4mJNmzZN+/bt83pqCTVq1Ch9+eWXsa8333zT6yn1GGTwBHLoHXJ4Ajn0Djk8rjtmMCmaukceeURz587Vddddp8LCQi1dulS9evXSM8884/XUEioYDGrAgAGxr9zcXK+n1GOQwRPIoXfI4Qnk0Dvk8LjumEHPm7pIJKLNmzervLw89r2UlBSVl5fr7bff9nBmiffZZ58pLy9PQ4cO1ezZs/X55597PaUegQy2Rw69QQ7bI4feIIcndMcMet7U1dbWqqWlReeff367759//vnas2ePR7NKvLKyMi1btkyrV6/WU089pZ07d2rSpEk6fPiw11PzPTJ4Ajn0Djk8gRx6hxwe110zGPR6AjhuxowZsX8XFRWprKxM+fn5eumll3TDDTd4ODP0JOQQyYAcwmvdNYOen6nLzc1Vamqq9u7d2+77e/fu1YABAzyalff69eun4cOHa8eOHV5PxffI4JmRw8Qhh2dGDhOHHJ5ed8mg501denq6xo4dq3Xr1sW+F41GtW7dOk2YMMHDmXnryJEjqqys1AUXXOD1VHyPDJ4ZOUwccnhm5DBxyOHpdZsMmiSwYsUKEwqFzLJly8y2bdvMTTfdZPr162f27Nnj9dQS5u677zYbN240O3fuNH/+859NeXm5yc3NNfv27fN6aj0CGTyOHHqLHB5HDr1FDrtvBpPiPXUVFRXav3+/fvKTn2jPnj0qKSnR6tWrT3mjpp/t3r1bs2bN0oEDB9S/f39NnDhR77zzjvr37+/11HoEMngcOfQWOTyOHHqLHHbfDAaMMcbrSQAAAKBjPH9PHQAAADqOpg4AAMAHaOoAAAB8gKYOAADAB2jqAAAAfICmDgAAwAdo6gAAAHyApg4AAMAHaOoAAAB8gKYOAADAB2jqAAAAfICmDgAAwAf+Pzzj0R5IHJ5zAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import random\n",
        "\n",
        "w = 10\n",
        "h = 10\n",
        "fig = plt.figure(figsize=(8, 8))\n",
        "columns = 4\n",
        "rows = 5\n",
        "\n",
        "for i in range(1, columns*rows +1):\n",
        "    index = random.randint(0, samples_number)\n",
        "    img = np.array(dataset.data[index]).reshape(8,8)\n",
        "    img = img.astype(float)\n",
        "    img = img / np.max(img)\n",
        "    fig.add_subplot(rows, columns, i)\n",
        "    plt.imshow(img, cmap='gray')\n",
        "    if(dataset.target[index] == 0):\n",
        "        plt.title(\"Background\", fontsize=8)\n",
        "    else:\n",
        "        plt.title(\"Robot\", fontsize=8)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zku4ylweqxMb"
      },
      "source": [
        "## Prepare the dataset for training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BRa_Y7wBqxMb"
      },
      "outputs": [],
      "source": [
        "X = np.array(dataset.data)/255\n",
        "y = np.array(dataset.target)\n",
        "print(\"Shape X: \", X.shape)\n",
        "print(\"Shape y: \", y.shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sW8xyWBbqxMb"
      },
      "outputs": [],
      "source": [
        "print(\"Samples number 1: \", np.count_nonzero(y == 1))\n",
        "print(\"Samples number 0: \", np.count_nonzero(y == 0))"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Get the **height**, **width** and **number of channels** of the images."
      ],
      "metadata": {
        "id": "25pky3Vh7tIX"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jCLHFd-M3IzC"
      },
      "outputs": [],
      "source": [
        "for index, data in enumerate(dataset.data) :\n",
        "    dataset.data[index] = np.array(dataset.data[index]).reshape((8,8))\n",
        "\n",
        "height,width=dataset.data[0].shape\n",
        "number_of_channels=1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "08rjB07ZqxMc"
      },
      "source": [
        "##Splitting dataset into training set, validation set and test set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qUG_p07arS3S"
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "X = np.array(dataset.data, dtype='float32')\n",
        "y = np.array(dataset.target)\n",
        "\n",
        "X_train_full, X_test, y_train_full, y_test = train_test_split(X, y, test_size=0.15, random_state=1)\n",
        "\n",
        "X_train, X_valid, y_train, y_valid = train_test_split(X_train_full, y_train_full, test_size=0.15, random_state=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dRgt_rzpRQ7u"
      },
      "outputs": [],
      "source": [
        "linear_classifier_accuracy=[]\n",
        "MLP_classifier_accuracy=[]\n",
        "CNN_classifier_accuracy=[]\n",
        "DCNN_classifier_accuracy=[]\n",
        "NN_model_sizes=[]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w2ou5uc7qxMe"
      },
      "source": [
        "# **NN models**\n",
        "In this section we implement all general functions that can be used by all NN models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c9qUtpPD_W-e"
      },
      "outputs": [],
      "source": [
        "pip install -q -U keras-tuner"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Epochs and batch size configuration"
      ],
      "metadata": {
        "id": "02oo8O8tu14A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "epochs=150\n",
        "batch_size=16"
      ],
      "metadata": {
        "id": "mLDH-s6Mu6lx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Import needed libraries"
      ],
      "metadata": {
        "id": "s9lpEAQq8Nb1"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9HcTz4l1FV8a"
      },
      "outputs": [],
      "source": [
        "!pip install scikit-optimize\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "from sklearn.metrics import accuracy_score\n",
        "from sklearn.linear_model import SGDClassifier\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "import time\n",
        "import tracemalloc\n",
        "import sklearn\n",
        "from keras.backend import binary_crossentropy\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "import keras_tuner as kt\n",
        "from keras_tuner.tuners import BayesianOptimization\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from sklearn.metrics import confusion_matrix\n",
        "import joblib\n",
        "import os\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Generalized functions"
      ],
      "metadata": {
        "id": "_z44ZJQ_vOuy"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6i1nNvyeXnNq"
      },
      "outputs": [],
      "source": [
        "def NN_hyperparameter_fit(hyperparameter,Type_model):\n",
        "  if (Type_model==\"CNN\"):\n",
        "    model=build_model(hyperparameter)\n",
        "  if (Type_model==\"DCNN\"):\n",
        "    model=build_model_DCNN(hyperparameter)\n",
        "  model.summary()\n",
        "  history = model.fit(X_train, y_train, epochs=epochs,batch_size=batch_size, validation_data=(X_valid, y_valid))\n",
        "  plot_model_performance(history,epochs)\n",
        "  return model,history\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pC_E7_A_qcs4"
      },
      "outputs": [],
      "source": [
        "def evaluate_NN_models(model_list):\n",
        "  loss, accuracy , y_pred ,precision, recall ,f1_score ,support ,confusion_matrix_list = [], [], [], [], [], [], [], []\n",
        "  for model in model_list:\n",
        "    loss_value,accuracy_value=model.evaluate(X_test, y_test)\n",
        "    loss.append(loss_value)\n",
        "    accuracy.append(accuracy_value)\n",
        "    y_pred_value=np.round(model.predict(X_test))\n",
        "    y_pred.append(y_pred_value)\n",
        "    precision_value, recall_value, f1_score_value, support_value = sklearn.metrics.precision_recall_fscore_support(y_test , y_pred_value, average= 'binary' )\n",
        "    precision.append(precision_value)\n",
        "    recall.append(recall_value)\n",
        "    f1_score.append(f1_score_value)\n",
        "    support.append(support_value)\n",
        "    confusion_matrix_value=confusion_matrix(y_test, y_pred_value)\n",
        "    confusion_matrix_list.append(confusion_matrix_value)\n",
        "  return [accuracy ,precision, recall ,f1_score],confusion_matrix_list,y_pred\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a9geD5BAKJUg"
      },
      "outputs": [],
      "source": [
        "def plot_model_performace_accuracy(history):\n",
        "  fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(20, 5))\n",
        "  ax1.plot(history[0].history['accuracy'], 'g', label='Training Accuracy')\n",
        "  ax1.plot(history[0].history['val_accuracy'], color='orange', label='Validation Accuracy')\n",
        "  ax1.set_title('training and validation')\n",
        "  ax1.set_xlabel('epoch')\n",
        "  ax1.set_ylabel('accuracy')\n",
        "  ax1.set_ylim(0.7, 1)\n",
        "  ax1.grid(True)\n",
        "  ax1.legend()\n",
        "  ax2.plot(history[1].history['accuracy'], 'g', label='Training Accuracy')\n",
        "  ax2.plot(history[1].history['val_accuracy'], color='orange', label='Validation Accuracy')\n",
        "  ax2.set_title('training and validation')\n",
        "  ax2.set_xlabel('epoch')\n",
        "  ax2.set_ylabel('accuracy')\n",
        "  ax2.set_ylim(0.7, 1)\n",
        "  ax2.grid(True)\n",
        "  ax2.legend()\n",
        "  ax3.plot(history[2].history['accuracy'], 'g', label='Training Accuracy')\n",
        "  ax3.plot(history[2].history['val_accuracy'], color='orange', label='Validation Accuracy')\n",
        "  ax3.set_title('training and validation')\n",
        "  ax3.set_xlabel('epoch')\n",
        "  ax3.set_ylabel('accuracy')\n",
        "  ax3.set_ylim(0.7, 1)\n",
        "  plt.grid(True)\n",
        "  plt.legend()\n",
        "  plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XTVrOjZnYxzq"
      },
      "outputs": [],
      "source": [
        "def plot_model_performance(history,number_of_epochs):\n",
        "  plt.figure()\n",
        "  plt.plot(history.history['loss'], 'g', label='Training Loss')\n",
        "  plt.plot(history.history['val_loss'], color='orange', label='Validation Loss')\n",
        "  plt.title('Training and Validation Loss')\n",
        "  plt.ylabel('Loss')\n",
        "  plt.xlabel('epoch')\n",
        "  plt.legend()\n",
        "  plt.grid(True)\n",
        "  plt.tight_layout()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7-9-qnH9Q_nR"
      },
      "outputs": [],
      "source": [
        "def plot_confusion_matrix(ax, conf_matrix, title, cmap):\n",
        "    sns.heatmap(conf_matrix, annot=True, fmt='d', cmap=cmap, cbar=False, ax=ax)\n",
        "    ax.set_title(title)\n",
        "\n",
        "def draw_confusion_matrix(matrix,name):\n",
        "  colormap=[\"Blues\",\"Greens\",\"Oranges\"]\n",
        "  fig, axes = plt.subplots(1, 3, figsize=(15, 5))\n",
        "  for i in range(len(matrix)):\n",
        "    plot_confusion_matrix(axes[i], matrix[i], \"Confusion Matrix  \"+name+ str(i+1), colormap[i])\n",
        "  plt.tight_layout()\n",
        "  plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BF_BtmSWi9Ky"
      },
      "outputs": [],
      "source": [
        "from tabulate import tabulate\n",
        "def print_table(data,headers):\n",
        "  table_data = list(zip(*data))\n",
        "  table = tabulate(table_data, headers=headers, tablefmt='grid')\n",
        "  print(table)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "82pFChydzcPy"
      },
      "outputs": [],
      "source": [
        "def calculate_ML_size(model,file_name):\n",
        "  model_filename = \"/content/\"+file_name+\".joblib\"\n",
        "  joblib.dump(model, model_filename)\n",
        "  file_size = os.path.getsize(model_filename)\n",
        "  return file_size"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9ryKW2XHGO8B"
      },
      "outputs": [],
      "source": [
        "def ML_hyperparameters_study(classifier, param_grid, X_train_ML, X_test_ML, y_train_full_ML, y_test_full_ML,random=False):\n",
        "    classifier.random_state = 42\n",
        "    grid_search = GridSearchCV(classifier, param_grid, cv=5, verbose=1)\n",
        "    grid_search.fit(X_train_ML, y_train_full_ML)\n",
        "    results = grid_search.cv_results_\n",
        "    mean_training_score_all = []\n",
        "    params_all = []\n",
        "    testing_accuracy = []\n",
        "    models_size = []\n",
        "    execution_time_list = []\n",
        "    total_allocated_memory=[]\n",
        "    average_allocated_memory=[]\n",
        "    precision_list, recall_list, f1_score_list=[],[],[]\n",
        "\n",
        "    def fit_and_measure_memory(model, X_train_ML, y_train_full_ML):\n",
        "      tracemalloc.start()\n",
        "      start_time = time.time()\n",
        "      model.fit(X_train_ML, y_train_full_ML)\n",
        "      snapshot = tracemalloc.take_snapshot()\n",
        "      tracemalloc.stop()\n",
        "      end_time = time.time()\n",
        "      execution_time_list.append(end_time - start_time)\n",
        "      return snapshot\n",
        "\n",
        "    for mean_score, params in zip(results['mean_test_score'], results['params']):\n",
        "        mean_training_score_all.append(mean_score)\n",
        "        params_all.append(params)\n",
        "        hyper_model = classifier\n",
        "        if random:\n",
        "            hyper_model.random_state = 42\n",
        "        hyper_model.set_params(**params)\n",
        "        snap = fit_and_measure_memory(hyper_model, X_train_ML, y_train_full_ML)\n",
        "        # Calculate the total allocated memory from the snapshot\n",
        "        total_memory=sum(stat.size for stat in snap.statistics('lineno'))\n",
        "        total_allocated_memory.append(format_memory(total_memory))\n",
        "        # Calculate the average allocated memory\n",
        "        num_snapshots = len(snap.statistics('lineno'))\n",
        "        average_allocated_memory.append(format_memory(total_memory / num_snapshots))\n",
        "        y_pred = hyper_model.predict(X_test_ML)\n",
        "        testing_accuracy.append(accuracy_score(y_test_full_ML, y_pred))\n",
        "        precision_value, recall_value, f1_score_value, support_value = sklearn.metrics.precision_recall_fscore_support(y_test_full_ML , y_pred, average= 'weighted' )\n",
        "        precision_list.append(precision_value)\n",
        "        recall_list.append(recall_value)\n",
        "        f1_score_list.append(f1_score_value)\n",
        "        models_size.append(format_memory(calculate_ML_size(hyper_model, \"model(\" + str(mean_score) + \")\")))\n",
        "\n",
        "    best_params = grid_search.best_params_\n",
        "    return [params_all,testing_accuracy,precision_list,recall_list,f1_score_list, models_size, execution_time_list,total_allocated_memory,average_allocated_memory], best_params ,mean_training_score_all"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dLjH4-0ftyDD"
      },
      "outputs": [],
      "source": [
        "def convert_params_to_list(params_all):\n",
        "  all_keys = set()\n",
        "  for d in params_all:\n",
        "      all_keys.update(d.keys())\n",
        "\n",
        "  data_values = [[d.get(key, None) for d in params_all] for key in all_keys]\n",
        "  return data_values,all_keys"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GwHHi40t55Z2"
      },
      "outputs": [],
      "source": [
        "def format_memory(memory_bytes):\n",
        "    if memory_bytes < 1024:\n",
        "        return f\"{memory_bytes} B\"\n",
        "    elif memory_bytes < 1024 * 1024:\n",
        "        return f\"{memory_bytes / 1024:.2f} KB\"\n",
        "    elif memory_bytes < 1024 * 1024 * 1024:\n",
        "        return f\"{memory_bytes / (1024 * 1024):.2f} MB\"\n",
        "    else:\n",
        "        return f\"{memory_bytes / (1024 * 1024 * 1024):.2f} GB\""
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def calculate_model_sizes(model):\n",
        "  input_size = model.input_shape[1]\n",
        "  output_size = model.output_shape[1]\n",
        "  num_params = sum(tf.keras.backend.count_params(p) for p in model.trainable_variables)\n",
        "  activation_units = 0\n",
        "  for layer in model.layers:\n",
        "      if isinstance(layer, tf.keras.layers.Dense):\n",
        "          activation_units += layer.units\n",
        "      elif isinstance(layer, tf.keras.layers.Conv2D):\n",
        "          activation_units += layer.filters * (layer.input_shape[1] // layer.strides[0]) * (layer.input_shape[2] // layer.strides[1])\n",
        "  return {\n",
        "      \"input_size\": input_size,\n",
        "      \"output_size\": output_size,\n",
        "      \"num_params\": num_params,\n",
        "      \"activation_units\": activation_units\n",
        "  }"
      ],
      "metadata": {
        "id": "ZWfN9Z08oKKU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Shallow ML Classifiers (Baseline)**"
      ],
      "metadata": {
        "id": "60T0RwetvooX"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3tmEM5ucRcMW"
      },
      "outputs": [],
      "source": [
        "X_train_full_ML, X_test_ML, y_train_full_ML, y_test_full_ML = train_test_split(X, y, test_size=0.15, random_state=1)\n",
        "X_train_ML= np.array([np.array(x_train).reshape(-1) for x_train in X_train_full_ML])\n",
        "X_test_ML=np.array([np.array(x_test).reshape(-1) for x_test in X_test_ML])\n",
        "\n",
        "print('Shape:', X_train_ML.shape)\n",
        "print('Type:', X_train_ML.dtype)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iGsbnuckqxMc"
      },
      "source": [
        "## **Linear SVM Classifier**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "adHRRJNOqxMc"
      },
      "outputs": [],
      "source": [
        "param_grid = {\n",
        "    'alpha': [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0]\n",
        "}\n",
        "\n",
        "models_info_linear,best_params_linear,mean_training_score_all_linear = ML_hyperparameters_study(SGDClassifier(),param_grid,X_train_ML, X_test_ML, y_train_full_ML, y_test_full_ML,random=True)\n",
        "headers_linear=list(param_grid.keys())\n",
        "\n",
        "#### visulaize all hyperparameters with acuuracy and model size in one table\n",
        "\n",
        "headers_linear=['models','hyperparameters',' accuracy ','precision','recall','f1_score','model size','Runtime','total allocated memory','average allocated memory']\n",
        "models_name=[]\n",
        "for i in range(1, len(param_grid['alpha'])+1):\n",
        "  models_name.append(\"Linear model \" + str(i))\n",
        "data_linear=[models_name]\n",
        "data_linear.extend(models_info_linear)\n",
        "print_table(data_linear,headers_linear)\n",
        "linear_classifier_accuracy=models_info_linear[1]\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m0abDXWnaomn"
      },
      "source": [
        "## **Random Forest Classifier**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hHidNlrdavsA"
      },
      "outputs": [],
      "source": [
        "\n",
        "param_grid = {\n",
        "    'n_estimators': [50],\n",
        "    'criterion': ['gini'],\n",
        "    'max_depth': [None, 5, 10, 20, 30]\n",
        "}\n",
        "\n",
        "models_info_RF,best_params_RF,mean_training_score_all_RF  = ML_hyperparameters_study(RandomForestClassifier(),param_grid,X_train_ML, X_test_ML, y_train_full_ML, y_test_full_ML,random=True)\n",
        "\n",
        "headers_RF=['models','hyperparameters',' accuracy ','precision','recall','f1_score','model size','Runtime','total allocated memory','average allocated memory']\n",
        "models_name=[]\n",
        "for i in range(1, 6):\n",
        "  models_name.append(\"RF model \" + str(i))\n",
        "data_RF=[models_name]\n",
        "data_RF.extend(models_info_RF)\n",
        "print_table(data_RF,headers_RF)\n",
        "print(\"best hyperparameter : \", best_params_RF)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w_sElX_29SoE"
      },
      "source": [
        "## **KNN Classifier**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SdYEQGQh9SoJ"
      },
      "outputs": [],
      "source": [
        "param_grid = [{'weights': [\"uniform\", \"distance\"], 'n_neighbors': [3, 4, 5]}]\n",
        "\n",
        "models_info_KNN,best_params_KNN,mean_training_score_all_KNN = ML_hyperparameters_study(KNeighborsClassifier(),param_grid,X_train_ML, X_test_ML, y_train_full_ML, y_test_full_ML,random=False)\n",
        "\n",
        "\n",
        "#### visulaize all hyperparameters with acuuracy and model size in one table\n",
        "\n",
        "headers_KNN=['models','hyperparameters',' accuracy ','precision','recall','f1_score','model size','Runtime','total allocated memory','average allocated memory']\n",
        "models_name=[]\n",
        "for i in range(1, 7):\n",
        "  models_name.append(\"KNN model \" + str(i))\n",
        "data_KNN=[models_name]\n",
        "data_KNN.extend(models_info_KNN)\n",
        "print_table(data_KNN,headers_KNN)\n",
        "print(\"best hyperparameter : \", best_params_KNN)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ljcvgMjlivyr"
      },
      "source": [
        "# **Fully connected NN (MLP)**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ix-RR64vizZ7"
      },
      "outputs": [],
      "source": [
        "def build_model_fc(number_of_hidden_layers,number_of_units,learning_rate=0.0001):\n",
        "    model = tf.keras.Sequential()\n",
        "    model.add(tf.keras.layers.InputLayer(input_shape=(width, height)))\n",
        "    model.add(tf.keras.layers.Flatten())\n",
        "    for i in range(number_of_hidden_layers):\n",
        "      model.add(tf.keras.layers.Dense( number_of_units[i], activation='relu',kernel_initializer='glorot_uniform'))\n",
        "      model.add(tf.keras.layers.Dropout(rate = 0.3))\n",
        "    model.add(tf.keras.layers.Dense(1,activation='sigmoid'))\n",
        "    optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)\n",
        "     # Compile the model with the hyperparameters\n",
        "    model.compile(optimizer=optimizer,\n",
        "                  loss='binary_crossentropy',\n",
        "                  metrics=['accuracy'],)\n",
        "    model.summary()\n",
        "    history = model.fit(X_train, y_train, epochs=epochs,batch_size=batch_size, validation_data=(X_valid, y_valid))\n",
        "    return model,history"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Creating grid of hyperparameters values"
      ],
      "metadata": {
        "id": "Zh8j8vb_Ojxv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import random\n",
        "\n",
        "min_num_hidden_layers = 1  # Minimum number of hidden layers\n",
        "max_num_hidden_layers = 5  # Maximum number of hidden layers\n",
        "min_length = 32  # Minimum length of each hidden layer\n",
        "max_length = 256  # Maximum length of each hidden layer\n",
        "num_hidden_layers=[]\n",
        "hidden_layer_lengths = []\n",
        "num_layers = random.randint(min_num_hidden_layers, max_num_hidden_layers)\n",
        "number_of_models=10\n",
        "for _ in range(number_of_models):\n",
        "    num_layers = random.randint(min_num_hidden_layers, max_num_hidden_layers)\n",
        "    num_hidden_layers.append(num_layers)\n",
        "    # Generate a list of random numbers between min_length and max_length and sort them in descending order\n",
        "    layer_lengths = sorted([random.randint(min_length, max_length) for _ in range(num_layers)], reverse=True)\n",
        "    hidden_layer_lengths.append(layer_lengths)\n",
        "\n",
        "\n",
        "for i in range(len(num_hidden_layers)):\n",
        "  print(f\"hidden layer :{num_hidden_layers[i]} , units: {hidden_layer_lengths[i]}\")"
      ],
      "metadata": {
        "id": "qYi5zHzpOp-v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QAyYZPImu9hv"
      },
      "source": [
        "## **build** and **train** 10 different MLP models each with different hyperparameters values"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IpLOsBcOqopo"
      },
      "outputs": [],
      "source": [
        "models_list_MLP=[]\n",
        "results_MLP=[]\n",
        "for i in range(len(num_hidden_layers)):\n",
        "  model,results=build_model_fc(num_hidden_layers[i],hidden_layer_lengths[i])\n",
        "  models_list_MLP.append(model)\n",
        "  results_MLP.append(results)\n",
        "  plot_model_performance(results,epochs)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Choose the best three MLP models"
      ],
      "metadata": {
        "id": "AJ29ZnS4O_eH"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zlxMOxyXq-Gs"
      },
      "outputs": [],
      "source": [
        "result_all_models_FC,confusion_matrix_FC,predicitions_FC=evaluate_NN_models(models_list_MLP)\n",
        "def find_top_three_indexes(lst):\n",
        "    arr = np.array(lst)\n",
        "    top_indexes = arr.argsort()[-3:][::-1]\n",
        "    return top_indexes\n",
        "\n",
        "# Example usage:\n",
        "top_three_indexes = find_top_three_indexes(result_all_models_FC[0])\n",
        "print(\"Indexes of the top three values:\", top_three_indexes)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Train 3 models with the entire training set:"
      ],
      "metadata": {
        "id": "GccTxLr9PKkR"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uROK3FZ_q-ou"
      },
      "outputs": [],
      "source": [
        "MLP_best_models=[models_list_MLP[top_three_indexes[0]],models_list_MLP[top_three_indexes[1]],models_list_MLP[top_three_indexes[2]]]\n",
        "model1=MLP_best_models[0]\n",
        "model2=MLP_best_models[1]\n",
        "model3=MLP_best_models[2]\n",
        "results_total_train1 = model1.fit(X_train_full, y_train_full, epochs=epochs,batch_size=batch_size)\n",
        "results_total_train2 = model2.fit(X_train_full, y_train_full, epochs=epochs,batch_size=batch_size)\n",
        "results_total_train3 = model3.fit(X_train_full, y_train_full, epochs=epochs,batch_size=batch_size)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "models_list_FC = []\n",
        "models_list_FC.append(model1)\n",
        "models_list_FC.append(model2)\n",
        "models_list_FC.append(model3)"
      ],
      "metadata": {
        "id": "kgW2e2BJPSdK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QnKNnhz8rElV"
      },
      "outputs": [],
      "source": [
        "plot_model_performace_accuracy([results_MLP[top_three_indexes[0]],results_MLP[top_three_indexes[0]],results_MLP[top_three_indexes[0]]])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QyXelBx3vL5u"
      },
      "source": [
        "## Evaluate the best three MLP models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1zkgO5oTss-Q"
      },
      "outputs": [],
      "source": [
        "result_all_models_FC,confusion_matrix_FC,predicitions_FC=evaluate_NN_models(MLP_best_models)\n",
        "\n",
        "## create a table with all models results\n",
        "headers=['models','hyperparameter','accuracy','precision','recall','f1_score']\n",
        "\n",
        "hyperparameters_values=[]\n",
        "models=[]\n",
        "for i in range(len(top_three_indexes)):\n",
        "  hyperparameters_values.append(f\"{num_hidden_layers[top_three_indexes[i]]},{hidden_layer_lengths[top_three_indexes[i]]}\")\n",
        "  models.append(f\"MLP Model{i}\")\n",
        "data_normal_FC=[models]\n",
        "data_normal_FC.append(hyperparameters_values)\n",
        "data_normal_FC.extend(result_all_models_FC)\n",
        "print_table(data_normal_FC,headers)\n",
        "MLP_classifier_accuracy=result_all_models_FC[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "81AM5PEcuou1"
      },
      "outputs": [],
      "source": [
        "draw_confusion_matrix(confusion_matrix_FC,\"MLP model\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MHN6N68et-_P"
      },
      "source": [
        "# **CNN**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "knA-lZadHHEv"
      },
      "outputs": [],
      "source": [
        "def build_model(hp,kernel_size=3):\n",
        "\n",
        "    # Tune the learning rate for the optimizer\n",
        "    hp_learning_rate = hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4,1e-5])\n",
        "    # Tune the optimizer\n",
        "    hp_optimizer = hp.Choice('optimizer', values=['adam'])\n",
        "    # Tune the number of filters in the Conv2D layer\n",
        "    hp_filters = hp.Int('filters', min_value=16, max_value=64, step=8)\n",
        "    #tune the dropout values\n",
        "    dropout_rate = hp.Choice('dropout', values=[0.3,0.5])\n",
        "    #tuning weights initialization\n",
        "    kernel_initializer = hp.Choice('kernel_initializer', values=['glorot_uniform', 'he_normal', 'lecun_normal'])\n",
        "\n",
        "    model = tf.keras.Sequential()\n",
        "    model.add(tf.keras.layers.InputLayer(input_shape=(width, height)))\n",
        "    model.add(tf.keras.layers.Reshape(target_shape=(width, height, number_of_channels)))\n",
        "    model.add(tf.keras.layers.Conv2D(filters = hp_filters , kernel_size=(kernel_size, kernel_size), activation='relu', kernel_initializer=kernel_initializer))\n",
        "    model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
        "    model.add(tf.keras.layers.Dropout(rate = dropout_rate))\n",
        "    model.add(tf.keras.layers.Flatten())\n",
        "    model.add(tf.keras.layers.Dense(32, activation='relu'))\n",
        "    model.add(tf.keras.layers.Dense(1, kernel_initializer=kernel_initializer))\n",
        "    model.add(tf.keras.layers.Activation('sigmoid'))\n",
        "\n",
        "     # Compile the model with the hyperparameters\n",
        "    model.compile(optimizer=hp_optimizer,\n",
        "                  loss='binary_crossentropy',\n",
        "                  metrics=['accuracy'],)\n",
        "    return model\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oyKybPUZG-RV"
      },
      "source": [
        "## Observe the most performing models and identify their hyperparameters:"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's use a Bayesian approach to conduct the search.\n",
        "tuner = BayesianOptimization(\n",
        "    build_model,\n",
        "    objective='val_accuracy',\n",
        "    max_trials=20,\n",
        "    directory='tuner_single_dense',\n",
        "    project_name='training_tuner_single_dense2'\n",
        ")\n",
        "\n",
        "#Let's start training models with different hyperparameters.\n",
        "tuner.search(X_train, y_train, epochs=75,batch_size=batch_size, validation_data=(X_valid, y_valid))"
      ],
      "metadata": {
        "id": "0-zqjOFH-Alq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xxJO0iYoHi9U"
      },
      "outputs": [],
      "source": [
        "num_trials = 3\n",
        "best_hps2 = tuner.get_best_hyperparameters(num_trials=num_trials)\n",
        "for idx, hyperparameters in enumerate(best_hps2):\n",
        "    print(f\"Set {idx + 1}: {hyperparameters.values}\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4aznstOiVV5M"
      },
      "outputs": [],
      "source": [
        "best_hyperparameter_CNN=[best_hps2[3],best_hps2[1],best_hps2[2]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QL1DrH7wjPfC"
      },
      "source": [
        "## Train 3 CNN models and plot performance"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sdI_yVdlHsYu"
      },
      "outputs": [],
      "source": [
        "model4,results4=NN_hyperparameter_fit(best_hyperparameter_CNN[0],\"CNN\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zXnpHDyNJFLo"
      },
      "outputs": [],
      "source": [
        "model5,results5=NN_hyperparameter_fit(best_hyperparameter_CNN[1],\"CNN\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "29nxon6fJnUT"
      },
      "outputs": [],
      "source": [
        "model6,results6=NN_hyperparameter_fit(best_hyperparameter_CNN[2],\"CNN\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V5VMP0aGr3QE"
      },
      "outputs": [],
      "source": [
        "plot_model_performace_accuracy([results4,results5,results6])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oaqmNKfujPfF"
      },
      "source": [
        "## Let's train the three models with the entire training set:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nOyltvFRLzZy"
      },
      "outputs": [],
      "source": [
        "results_total_train4 = model4.fit(X_train_full, y_train_full, epochs=epochs,batch_size=batch_size)\n",
        "results_total_train5 = model5.fit(X_train_full, y_train_full, epochs=epochs,batch_size=batch_size)\n",
        "results_total_train6 = model6.fit(X_train_full, y_train_full, epochs=epochs,batch_size=batch_size)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WX0fxFp75eHl"
      },
      "outputs": [],
      "source": [
        "models_list_CNN = []\n",
        "models_list_CNN.append(model4)\n",
        "models_list_CNN.append(model5)\n",
        "models_list_CNN.append(model6)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uF562MkZjPfG"
      },
      "source": [
        "## Evaluate the CNN models on test set:\n",
        "Evaluate All models and print out a table with all parameters (model , hyperparameter, accyracy, precision, recall, f1_score)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aTipflbguwdp"
      },
      "outputs": [],
      "source": [
        "result_all_models_CNN,confusion_matrix_CNN,predicitions=evaluate_NN_models(models_list_CNN)\n",
        "## create a table with all models results\n",
        "headers=['models','hyperparameter','accuracy','precision','recall','f1_score']\n",
        "data_normal_CNN=[['CNN model 1',' CNN model 2', 'CNN model 3']]\n",
        "hyperparameters_values=[]\n",
        "for idx, hyperparameters in enumerate(best_hyperparameter_CNN):\n",
        "  hyperparameters_values.append(hyperparameters.values)\n",
        "data_normal_CNN.append(hyperparameters_values)\n",
        "data_normal_CNN.extend(result_all_models_CNN)\n",
        "print_table(data_normal_CNN,headers)\n",
        "CNN_classifier_accuracy=result_all_models_CNN[0]\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kkb3ECFBadPY"
      },
      "source": [
        "Draw confusion matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2laHmAfCafQm"
      },
      "outputs": [],
      "source": [
        "draw_confusion_matrix(confusion_matrix_CNN,\"CNN model\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J4dgcL2Xd8Bx"
      },
      "source": [
        "# **Deeper CNN (DCNN)**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Kl1zAgVepvHl"
      },
      "outputs": [],
      "source": [
        "### number_of_convolutional : the number of convolutional layer\n",
        "### kernal_size : a list of kernal sizes for each convolutional layer repespectively\n",
        "def build_model_DCNN(hp,number_of_convolutional=2,kernal_size=[3,3]):\n",
        "    # Tune the learning rate for the optimizer\n",
        "    hp_learning_rate = hp.Choice('learning_rate', values=[1e-2,1e-3,1e-4])\n",
        "    # Tune the optimizer\n",
        "    hp_optimizer = hp.Choice('optimizer', values=['adam'])\n",
        "    # Tune the number of filters in the Conv2D layer\n",
        "    hp_filters = hp.Int('filters', min_value=16, max_value=64, step=8)\n",
        "    #tune the dropout values\n",
        "    dropout_rate = hp.Choice('dropout', values=[0.3,0.5])\n",
        "    #tuning weights initialization\n",
        "    kernel_initializer = hp.Choice('kernel_initializer', values=['glorot_uniform', 'he_normal', 'lecun_normal'])\n",
        "\n",
        "    model = tf.keras.Sequential()\n",
        "    model.add(tf.keras.layers.InputLayer(input_shape=(width, height)))\n",
        "    model.add(tf.keras.layers.Reshape(target_shape=(width, height, number_of_channels)))\n",
        "\n",
        "    for i in range(number_of_convolutional - 1):\n",
        "      model.add(tf.keras.layers.Conv2D(filters = hp_filters , kernel_size=(kernal_size[i], kernal_size[i]), activation='relu',padding='same', kernel_initializer=kernel_initializer))\n",
        "      model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
        "      model.add(tf.keras.layers.Dropout(rate = dropout_rate))\n",
        "\n",
        "    model.add(tf.keras.layers.Conv2D(filters = hp_filters , kernel_size=(kernal_size[len(kernal_size)-1], kernal_size[len(kernal_size)-1]), activation='relu',padding='same', kernel_initializer=kernel_initializer))\n",
        "\n",
        "\n",
        "    model.add(tf.keras.layers.Flatten())\n",
        "    model.add(tf.keras.layers.Dense(32, activation='relu'))\n",
        "    model.add(tf.keras.layers.Dense(1, kernel_initializer=kernel_initializer))\n",
        "    model.add(tf.keras.layers.Activation('sigmoid'))\n",
        "\n",
        "    # Compile the model with the hyperparameters\n",
        "    model.compile(optimizer=hp_optimizer,\n",
        "                  loss='binary_crossentropy',\n",
        "                  metrics=['accuracy'])\n",
        "    return model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Observe the most performing models and identify their hyperparameters:"
      ],
      "metadata": {
        "id": "55wJBmnY6gMb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's use a Bayesian approach to conduct the search.\n",
        "tuner = BayesianOptimization(\n",
        "    build_model_DCNN,\n",
        "    objective='val_accuracy',\n",
        "    max_trials=5,\n",
        "    #executions_per_trial=2,\n",
        "    directory='tuner_single_dense',\n",
        "    project_name='training_tuner_single_dense17'\n",
        ")\n",
        "\n",
        "#Let's start training models with different hyperparameters.\n",
        "tuner.search(X_train, y_train, epochs=75,batch_size=batch_size, validation_data=(X_valid, y_valid))"
      ],
      "metadata": {
        "id": "_Dbj7qR2-d7P"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "trssd2pSyOUT"
      },
      "outputs": [],
      "source": [
        "num_trials = 6\n",
        "best_hps_DCNN = tuner.get_best_hyperparameters(num_trials=num_trials)\n",
        "for idx, hyperparameters in enumerate(best_hps_DCNN):\n",
        "    print(f\"Set {idx + 1}: {hyperparameters.values}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gHg-f69aIcNX"
      },
      "outputs": [],
      "source": [
        "best_hyperparameter_DCNN=[best_hps_DCNN[0],best_hps_DCNN[1],best_hps_DCNN[2]]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JfPtfsRjyyTf"
      },
      "source": [
        "## Train 3 models with the best hyperparameters and study the performance of the model with validation dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rITqbBjgycvC"
      },
      "outputs": [],
      "source": [
        "model7,results7=NN_hyperparameter_fit(best_hyperparameter_DCNN[0],\"DCNN\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IpLrskL5yp1d"
      },
      "outputs": [],
      "source": [
        "model8,results8=NN_hyperparameter_fit(best_hyperparameter_DCNN[1],\"DCNN\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D2dFvtJMyp93"
      },
      "outputs": [],
      "source": [
        "model9,results9=NN_hyperparameter_fit(best_hyperparameter_DCNN[2],\"DCNN\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CDGmglKZzCOq"
      },
      "outputs": [],
      "source": [
        "plot_model_performace_accuracy([results7,results8,results9])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tT2l5Hpcy9aD"
      },
      "source": [
        "## Train the model on all the entire dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CmXW-fQUzA0j"
      },
      "outputs": [],
      "source": [
        "results_total_train7 = model7.fit(X_train_full, y_train_full, epochs=epochs,batch_size=batch_size)\n",
        "results_total_train8 = model8.fit(X_train_full, y_train_full, epochs=epochs,batch_size=batch_size)\n",
        "results_total_train9 = model9.fit(X_train_full, y_train_full, epochs=epochs,batch_size=batch_size)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dlZ0S2HrzhPR"
      },
      "outputs": [],
      "source": [
        "models_list_DCNN = []\n",
        "models_list_DCNN.append(model7)\n",
        "models_list_DCNN.append(model8)\n",
        "models_list_DCNN.append(model9)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6bXDmagDzvnu"
      },
      "source": [
        "## Evaluate All models and print out a table with all parameters (model , hyperparameter, accyracy, precision, recall, f1_score)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bEieZUhFzwT0"
      },
      "outputs": [],
      "source": [
        "result_all_models_DCNN,confusion_matrix_DCNN,predicitions=evaluate_NN_models(models_list_DCNN)\n",
        "## create a table with all models results\n",
        "headers=['models','hyperparameter','accuracy','precision','recall','f1_score']\n",
        "data_normal_DCNN=[['DCNN model 1',' DCNN model 2', 'DCNN model 3']]\n",
        "hyperparameters_values=[]\n",
        "for idx, hyperparameters in enumerate(best_hyperparameter_DCNN):\n",
        "  hyperparameters_values.append(hyperparameters.values)\n",
        "data_normal_DCNN.append(hyperparameters_values)\n",
        "data_normal_DCNN.extend(result_all_models_DCNN)\n",
        "print_table(data_normal_DCNN,headers)\n",
        "DCNN_classifier_accuracy=result_all_models_DCNN[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "33BXAYYZ0P0S"
      },
      "source": [
        "Draw confusion matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IMuVC7UJ0RuW"
      },
      "outputs": [],
      "source": [
        "draw_confusion_matrix(confusion_matrix_DCNN,\"DCNN model\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UUU_a3XeqxMq"
      },
      "source": [
        "\n",
        "# **TensorFlow Lite model**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fAK7VfJsbGfQ"
      },
      "source": [
        "## Functions to evaluate and print analysis for  TFlite / quantized models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D7SYQ3xYqxMr"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "def evaluate(model_file, X, y, categoricalAccuarcy):\n",
        "    if(categoricalAccuarcy):\n",
        "      accuracy = tf.keras.metrics.SparseCategoricalAccuracy()\n",
        "    else:\n",
        "      accuracy = tf.keras.metrics.BinaryAccuracy()\n",
        "\n",
        "    interpreter = tf.lite.Interpreter(model_path = model_file)\n",
        "    interpreter.allocate_tensors()\n",
        "    input_details = interpreter.get_input_details()[0]\n",
        "    output_details = interpreter.get_output_details()[0]\n",
        "\n",
        "    y_preds = []  # To store the predicted labels\n",
        "    y_real = []\n",
        "\n",
        "    for x, y_true in zip(X,y):\n",
        "        if input_details['dtype'] == np.uint8:\n",
        "            input_scale, input_zero_point = input_details[\"quantization\"]\n",
        "            x = x / input_scale + input_zero_point\n",
        "        x = np.expand_dims(x, axis=0).astype(input_details[\"dtype\"])\n",
        "        interpreter.set_tensor(input_details[\"index\"], x)\n",
        "        interpreter.invoke()\n",
        "        y_pred = interpreter.get_tensor(output_details[\"index\"])[0]\n",
        "        accuracy.update_state(y_true, y_pred)\n",
        "         # Collect the predicted labels\n",
        "        if(categoricalAccuarcy):\n",
        "            y_preds.append(np.argmax(y_pred))\n",
        "        else:\n",
        "            y_preds.append(np.round(y_pred[0]))\n",
        "\n",
        "        y_real.append(y_true)\n",
        "    return accuracy.result(), y_preds, y_real"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def calculate_accuracy_quantization_full(y_prediction, y_real):\n",
        "    if len(y_prediction) != len(y_real):\n",
        "        raise ValueError(\"y_prediction and y_real must have the same length.\")\n",
        "\n",
        "    num_correct = sum(1 for pred, real in zip(y_prediction, y_real) if pred == real)\n",
        "    total_predictions = len(y_prediction)\n",
        "\n",
        "    accuracy = num_correct / total_predictions\n",
        "\n",
        "    return accuracy"
      ],
      "metadata": {
        "id": "X_CJYIyaQTyB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5AwkNvbTqxMr"
      },
      "outputs": [],
      "source": [
        "\n",
        "def evaluate_tflite_quantized_models(models_list,base_filename_model,models_files,full=False):\n",
        "  tflite_model_accuracy,tflite_model_predicitons,y_true,precisions_tf_lite_model,recalls_tf_lite_model,f1_scores_tf_lite_model,confusion_tflite_matrix_list = [], [], [] ,[],[], [], []\n",
        "  for i in range(len(models_list)):\n",
        "    path = base_filename_model + '/' + models_files[i].name\n",
        "    print(\"path\",path)\n",
        "    accuracy, y_pred, y_real = evaluate(path, X_test, y_test, False)\n",
        "    accuracy=accuracy.numpy()\n",
        "    tflite_model_predicitons.append(y_pred)\n",
        "    y_true.append(y_real)\n",
        "    # print(f\"TFLite model {i+1} accuracy = {tflite_model_accuracy[i]:.4f}\")\n",
        "    if full==True:\n",
        "      y_pred = [np.round(value / 255) for value in y_pred]\n",
        "      accuracy = calculate_accuracy_quantization_full(y_pred, y_real)\n",
        "    tflite_model_accuracy.append(accuracy)\n",
        "    precision_value, recall_value, f1_score_value, support_value = sklearn.metrics.precision_recall_fscore_support(y_real , y_pred, average= 'binary' )\n",
        "    precisions_tf_lite_model.append(precision_value)\n",
        "    recalls_tf_lite_model.append(recall_value)\n",
        "    f1_scores_tf_lite_model.append(f1_score_value)\n",
        "    confusion_matrix_value=confusion_matrix(y_test, y_pred)\n",
        "    confusion_tflite_matrix_list.append(confusion_matrix_value)\n",
        "  return [tflite_model_accuracy,precisions_tf_lite_model,recalls_tf_lite_model,f1_scores_tf_lite_model],confusion_tflite_matrix_list,tflite_model_predicitons\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1a-Z7foywbsu"
      },
      "source": [
        "## Converting all models to TFlite model using tensorflow lite converter"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TVC6TsGTt5ku"
      },
      "outputs": [],
      "source": [
        "models_list=[model1,model2,model3,model4,model5,model6,model7,model8,model9]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S7cNq4bhqxMq"
      },
      "outputs": [],
      "source": [
        "tflite_models = []\n",
        "for index in range(len(models_list)):\n",
        "  converter = tf.lite.TFLiteConverter.from_keras_model(models_list[index])\n",
        "  tflite_models.append(converter.convert())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VeAumtFjqxMq"
      },
      "source": [
        "## It's now a TensorFlow Lite model, but it's still using 32-bit float values for all parameter data. We can store the models in a file in order to estimate its size:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hN1FkWBkqxMq"
      },
      "outputs": [],
      "source": [
        "import pathlib\n",
        "import os\n",
        "tflite_models_dir = pathlib.Path(\"./\")\n",
        "base_filename_model_tflite = 'tflite_model'\n",
        "path = tflite_models_dir/base_filename_model_tflite\n",
        "\n",
        "if not os.path.exists(path):\n",
        "  os.makedirs(path)\n",
        "\n",
        "tflite_model_files = []\n",
        "tflite_model_size = []\n",
        "\n",
        "for i in range(1, len(models_list) +1):\n",
        "  filename = f\"{base_filename_model_tflite}_{i}.tflite\"\n",
        "  tflite_model_files.append(path/filename)\n",
        "  tflite_model_files[i-1].write_bytes(tflite_models[i-1])\n",
        "  tflite_model_size.append(os.path.getsize(tflite_model_files[i -1]) / float(2**10))\n",
        "  print(\"TFlite model in KB:\", tflite_model_size[i-1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JF5adl6njPfJ"
      },
      "source": [
        "## Evaluate and Calculate Accuracy, Precision, Recall, F1 Tf_lite_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Mo3mhz-OZGfD"
      },
      "outputs": [],
      "source": [
        "tflite_all_results, confusion_tflite_all, tflite_predictions = evaluate_tflite_quantized_models(models_list,base_filename_model_tflite,tflite_model_files)\n",
        "## create a table with all tf lite models results\n",
        "headers=['models','hyperparameter','accuracy','precision','recall','f1_score','model size KB']\n",
        "data_tflite=[['tflite MLP model 1','tflite MLP model 2', 'tflite MLP model 3',\n",
        "              'tflite CNN model 1','tflite CNN model 2', 'tflite CNN model 3',\n",
        "              'tflite DCNN model 1','tflite DCNN model 2', 'tflite DCNN model 3']]\n",
        "hyperparameters_values=[]\n",
        "\n",
        "for i in range(len(top_three_indexes)):\n",
        "  hyperparameters_values.append(f\"{num_hidden_layers[top_three_indexes[i]]},{hidden_layer_lengths[top_three_indexes[i]]}\")\n",
        "\n",
        "\n",
        "for idx, hyperparameters in enumerate(best_hyperparameter_CNN):\n",
        "  hyperparameters_values.append(hyperparameters.values)\n",
        "\n",
        "\n",
        "for idx, hyperparameters in enumerate(best_hyperparameter_DCNN):\n",
        "  hyperparameters_values.append(hyperparameters.values)\n",
        "\n",
        "data_tflite.append(hyperparameters_values)\n",
        "data_tflite.extend(tflite_all_results)\n",
        "data_tflite.append(tflite_model_size)\n",
        "print_table(data_tflite,headers)\n",
        "NN_model_sizes=tflite_model_size"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8JRZqEpUqxMr"
      },
      "source": [
        "# **Post-training quantization**\n",
        "\n",
        "We can enable the default optimizations flag to quantize all fixed parameters (weights and biases) to 8-bit integers. Notice that scale and zero point for weights and bias can be calculated before the inference, becouse their ranges are already available. But how we can calculate scale and zero point for activations? We can use the **dynamic quantization**, in which scale and zero point for activations are calculated on-the-fly (online during inference). This means that the activations are always stored in float 32 and they are converted to integers while processing and back to floating point after the processing is done.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xLvlcFezxB7z"
      },
      "source": [
        "## **Dynamic Quantization**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KvE7tJamqxMr"
      },
      "outputs": [],
      "source": [
        "tflite_dynamic_quantized_models = []\n",
        "\n",
        "for i in range (0 , len(models_list)):\n",
        "  converter = tf.lite.TFLiteConverter.from_keras_model(models_list[i])\n",
        "  converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
        "  tflite_dynamic_quantized_models.append(converter.convert())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZfbztuEiqxMs"
      },
      "outputs": [],
      "source": [
        "tflite_dynamic_quantized_model_files = []\n",
        "tflite_dynamic_quantized_model_size = []\n",
        "\n",
        "base_filename_dynamic = 'tflite_dynamic_quantized_model'\n",
        "path = tflite_models_dir/base_filename_dynamic\n",
        "\n",
        "if not os.path.exists(path):\n",
        "  os.makedirs(path)\n",
        "\n",
        "for i in range (1, len(models_list) + 1):\n",
        "  file_name =  f\"{base_filename_dynamic}_{i}.tflite\"\n",
        "  tflite_dynamic_quantized_model_files.append(path/file_name)\n",
        "  tflite_dynamic_quantized_model_files[i-1].write_bytes(tflite_dynamic_quantized_models[i-1])\n",
        "  tflite_dynamic_quantized_model_size.append(os.path.getsize(tflite_dynamic_quantized_model_files[i - 1]) / float(2**10))\n",
        "  print(\"TFlite dynamic quantized model in KB:\", tflite_dynamic_quantized_model_size[i-1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uGcAO554criq"
      },
      "source": [
        "### Evaluate and Calculate Accuracy, Precision, Recall, F1 Tf_lite_model_dynamic_models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h3JMRv6MXD46"
      },
      "outputs": [],
      "source": [
        "dynamic_quantized_all_results, confusion_dynamic_quantized_all, dynamic_quantized_predictions = evaluate_tflite_quantized_models(models_list,base_filename_dynamic,tflite_dynamic_quantized_model_files)\n",
        "## create a table with all tf lite models results\n",
        "headers=['models','hyperparameter','accuracy','precision','recall','f1_score','model size KB']\n",
        "data_dynamic_quantized=[['Dynamic quantized MLP model 1','Dynamic quantized MLP model 2', 'Dynamic quantized MLP model 3',\n",
        "                         'Dynamic quantized CNN model 1','Dynamic quantized CNN model 2', 'Dynamic quantized CNN model 3',\n",
        "                         'Dynamic quantized DCNN model 1','Dynamic quantized DCNN model 2', 'Dynamic quantized DCNN model 3']]\n",
        "data_dynamic_quantized.append(hyperparameters_values)\n",
        "data_dynamic_quantized.extend(dynamic_quantized_all_results)\n",
        "data_dynamic_quantized.append(tflite_dynamic_quantized_model_size)\n",
        "print_table(data_dynamic_quantized,headers)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VnsT3qFDjPfP"
      },
      "source": [
        "## **Static Quantization**\n",
        "The model is now smaller with quantized weights with some decrease in the accuracy, but other variable data are still in float format. To quantize variable data (input/output and intermediates between layers), we can use **static quantization** to pre-computes scales and zero points also for all variable data in order to eliminate this overhead. However, we need some representative data in order to collect the distribution statistics for all the variable data and compute an estime of scales and zero points. The short-coming is that if the data is not representative, the scales and zero points computed might not reflect the true scenario during inference, and the inference accuracy will be harmed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dnE_UlZkqxMt"
      },
      "outputs": [],
      "source": [
        "def representative_data_gen():\n",
        "  for input_value in tf.data.Dataset.from_tensor_slices(X_train).batch(1).take(100):\n",
        "    yield [input_value]\n",
        "\n",
        "tflite_static_quantized_models = []\n",
        "\n",
        "for i in range (0 , len(models_list)):\n",
        "  converter = tf.lite.TFLiteConverter.from_keras_model(models_list[i])\n",
        "  converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
        "  converter.representative_dataset = representative_data_gen\n",
        "  tflite_static_quantized_models.append(converter.convert())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DXfL2hewqxMt"
      },
      "outputs": [],
      "source": [
        "tflite_static_quantized_model_files = []\n",
        "tflite_static_quantized_model_size = []\n",
        "\n",
        "base_filename_static = 'tflite_static_quantized_model'\n",
        "path = tflite_models_dir/base_filename_static\n",
        "\n",
        "if not os.path.exists(path):\n",
        "  os.makedirs(path)\n",
        "\n",
        "for i in range (1, len(models_list) + 1):\n",
        "  file_name =  f\"{base_filename_static}_{i}.tflite\"\n",
        "  tflite_static_quantized_model_files.append(path/file_name)\n",
        "  tflite_static_quantized_model_files[i - 1].write_bytes(tflite_static_quantized_models[i - 1])\n",
        "  tflite_static_quantized_model_size.append(os.path.getsize(tflite_static_quantized_model_files[i - 1]) / float(2**10))\n",
        "  print(\"TFlite static quantized model in KB:\", tflite_static_quantized_model_size[i-1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mx5wtz8xjPfQ"
      },
      "source": [
        "### Evaluate and Calculate Accuracy, Precision, Recall, F1 Tf_lite_model_static_models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ef3jL6hKqxMt"
      },
      "outputs": [],
      "source": [
        "static_quantized_all_results, confusion_static_quantized_all, static_quantized_predictions = evaluate_tflite_quantized_models(models_list,base_filename_static,tflite_static_quantized_model_files)\n",
        "## create a table with all tf lite models results\n",
        "headers=['models','hyperparameter','accuracy','precision','recall','f1_score','model size KB']\n",
        "data_static_quantized=[['Static quantized MLP model 1','Static quantized MLP model 2', 'Static quantized MLP model 3',\n",
        "                        'Static quantized CNN model 1','Static quantized CNN model 2', 'Static quantized CNN model 3',\n",
        "                        'Static quantized DCNN model 1','Static quantized DCNN model 2', 'Static quantized DCNN model 3']]\n",
        "data_static_quantized.append(hyperparameters_values)\n",
        "data_static_quantized.extend(static_quantized_all_results)\n",
        "data_static_quantized.append(tflite_static_quantized_model_size)\n",
        "print_table(data_static_quantized,headers)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V3_niyOBqxMt"
      },
      "source": [
        "## **Full Static Quantization**\n",
        "Now all weights and variable data are quantized. However, to maintain compatibility with applications that traditionally use float model, the TensorFlow Lite Converter leaves the model input and output tensors in float. This is good for compatibility, but it won't be compatible with devices that perform only integer-based operations. To ensure end-to-end integer-only model, we need to specify some parameters to the converter."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MRfn9vIMqxMu"
      },
      "outputs": [],
      "source": [
        "tflite_full_static_quantized_models = []\n",
        "\n",
        "for i in range(0, len(models_list)):\n",
        "  converter = tf.lite.TFLiteConverter.from_keras_model(models_list[i])\n",
        "  converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
        "  converter.representative_dataset = representative_data_gen\n",
        "  converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]\n",
        "  converter.inference_input_type = tf.uint8\n",
        "  converter.inference_output_type = tf.uint8\n",
        "  tflite_full_static_quantized_models.append(converter.convert())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xFtAoEXSqxMu"
      },
      "source": [
        "### The internal quantization remains the same as above, but we can see the input and output tensors are now integer format:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_iCvHPMMqxMu"
      },
      "outputs": [],
      "source": [
        "interpreter = tf.lite.Interpreter(model_content = tflite_full_static_quantized_models[0])\n",
        "input_type = interpreter.get_input_details()[0]['dtype']\n",
        "print('input: ', input_type)\n",
        "output_type = interpreter.get_output_details()[0]['dtype']\n",
        "print('output: ', output_type)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NLJ3UUXPqxMu"
      },
      "outputs": [],
      "source": [
        "tflite_full_static_quantized_model_files = []\n",
        "tflite_full_static_quantized_model_size = []\n",
        "\n",
        "base_filename_full_static = 'tflite_full_static_quantized_model'\n",
        "path = tflite_models_dir/base_filename_full_static\n",
        "\n",
        "if not os.path.exists(path):\n",
        "  os.makedirs(path)\n",
        "\n",
        "for i in range(1, len(models_list) + 1) :\n",
        "  file_name =  f\"{base_filename_full_static}_{i}.tflite\"\n",
        "  tflite_full_static_quantized_model_files.append(path/file_name)\n",
        "  tflite_full_static_quantized_model_files[i - 1].write_bytes(tflite_full_static_quantized_models[i - 1])\n",
        "  tflite_full_static_quantized_model_size.append(os.path.getsize(tflite_full_static_quantized_model_files[i - 1]) / float(2**10))\n",
        "  print(\"TFlite full static quantized model in KB:\", tflite_full_static_quantized_model_size[i-1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uyYekcEYjPfT"
      },
      "source": [
        "### Evaluate and Calculate Accuracy, Precision, Recall, F1 Tf_lite_model_full_static_models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P2KDJ1qtqxMu"
      },
      "outputs": [],
      "source": [
        "full_static_quantized_all_results, confusion_full_static_quantized_all, full_static_quantized_predictions = evaluate_tflite_quantized_models(models_list,base_filename_full_static,tflite_full_static_quantized_model_files,full=True)\n",
        "## create a table with all tf lite models results\n",
        "headers=['models','hyperparameter','accuracy','precision','recall','f1_score','model size KB']\n",
        "data_full_static_quantized=[['Full static quantized MLP model 1','full static quantized MLP model 2', 'full static quantized MLP model 3',\n",
        "                             'Full static quantized CNN model 1','full static quantized CNN model 2', 'full static quantized CNN model 3',\n",
        "                             'Full static quantized DCNN model 1','full static quantized DCNN model 2', 'full static quantized DCNN model 3']]\n",
        "data_full_static_quantized.append(hyperparameters_values)\n",
        "data_full_static_quantized.extend(full_static_quantized_all_results)\n",
        "data_full_static_quantized.append(tflite_full_static_quantized_model_size)\n",
        "print_table(data_full_static_quantized,headers)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HmCqpGjTf-DT"
      },
      "source": [
        "# **Best models**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Jla4e1q_2qv6"
      },
      "outputs": [],
      "source": [
        "linear_index=linear_classifier_accuracy.index(max(linear_classifier_accuracy))\n",
        "MLP_index=MLP_classifier_accuracy.index(max(MLP_classifier_accuracy))\n",
        "CNN_index=CNN_classifier_accuracy.index(max(CNN_classifier_accuracy))\n",
        "DCNN_index=DCNN_classifier_accuracy.index(max(DCNN_classifier_accuracy))\n",
        "print(linear_index,MLP_index,CNN_index,DCNN_index)\n",
        "\n",
        "tflite_modelfiles_best=[tflite_model_files[MLP_index],tflite_model_files[CNN_index + 3],tflite_model_files[DCNN_index + 6]]\n",
        "dynamic_modelfiles_best=[tflite_dynamic_quantized_model_files[MLP_index],tflite_dynamic_quantized_model_files[CNN_index + 3],tflite_dynamic_quantized_model_files[DCNN_index + 6]]\n",
        "static_modelfiles_best=[tflite_static_quantized_model_files[MLP_index],tflite_static_quantized_model_files[CNN_index + 3],tflite_static_quantized_model_files[DCNN_index + 6]]\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aU0BlqEQjPfU"
      },
      "source": [
        "# **Quantization Aware Training**\n",
        "Quantization introduces information loss and therefore the inference accuracy from the quantized integer models are inevitably lower than that from the floating point models. Such information loss is due to that the floating points after quantization and de-quantization is not exactly recoverable. The idea of quantization aware training is to ask the neural network to take the effect of such information loss into account during training. We can use the TensorFlow Model Optimization toolkit and passing in input the Keras model. What that API is doing is extending that network with the ability to mimic the quantized behavior that would be happening during the inference time, during the training time.# Aware Quantization"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_80xQHD3rr8m"
      },
      "outputs": [],
      "source": [
        "!pip install tensorflow-model-optimization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cPAnK6Q0qKqx"
      },
      "source": [
        "## Functions for building aware MLP, CNN and DCNN models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YEDJvoWOqxMv"
      },
      "outputs": [],
      "source": [
        "def build_model_aware_CNN(quantizator,kernel_size=3):\n",
        "\n",
        "    model = tf.keras.Sequential()\n",
        "    model.add(tf.keras.layers.InputLayer(input_shape=(width, height)))\n",
        "    model.add(tf.keras.layers.Reshape(target_shape=(width, height, number_of_channels)))\n",
        "    model.add(tf.keras.layers.Conv2D(24 , kernel_size=(kernel_size, kernel_size), activation='relu', kernel_initializer=\"glorot_uniform\"))\n",
        "    model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
        "    model.add(tf.keras.layers.Dropout(rate = 0.3))\n",
        "    model.add(tf.keras.layers.Flatten())\n",
        "    model.add(tf.keras.layers.Dense(32, activation='relu'))\n",
        "    model.add(tf.keras.layers.Dense(1, kernel_initializer=\"glorot_uniform\"))\n",
        "    model.add(tf.keras.layers.Activation('sigmoid'))\n",
        "    optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)\n",
        "    quantization_aware_model = quantizator(model)\n",
        "    # Compile the model with the hyperparameters\n",
        "    quantization_aware_model.compile(optimizer=optimizer,\n",
        "                  loss='binary_crossentropy',\n",
        "                  metrics=['accuracy'],)\n",
        "    quantization_aware_model.summary()\n",
        "\n",
        "    return quantization_aware_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S7kxZ1mI9i9u"
      },
      "outputs": [],
      "source": [
        "def build_model_aware_fc(quantizator,number_of_hidden_layers=1,number_of_units=128):\n",
        "\n",
        "    model = tf.keras.Sequential()\n",
        "    model.add(tf.keras.layers.InputLayer(input_shape=(width, height)))\n",
        "    model.add(tf.keras.layers.Flatten())\n",
        "    for i in range(1,number_of_hidden_layers+1):\n",
        "      model.add(tf.keras.layers.Dense( number_of_units/i, activation='relu'))\n",
        "      model.add(tf.keras.layers.Dropout(rate = 0.3))\n",
        "    model.add(tf.keras.layers.Dense(1,activation='sigmoid'))\n",
        "\n",
        "    quantization_aware_model = quantizator(model)\n",
        "     # Compile the model with the hyperparameters\n",
        "    quantization_aware_model.compile(optimizer=\"adam\",\n",
        "                  loss='binary_crossentropy',\n",
        "                  metrics=['accuracy'],)\n",
        "    quantization_aware_model.summary()\n",
        "    return quantization_aware_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5VJoDOK6qdOG"
      },
      "outputs": [],
      "source": [
        "def build_model_aware_DCNN(quantizator,number_of_convolutional=2,kernal_size=[5,3]):\n",
        "\n",
        "\n",
        "    model = tf.keras.Sequential()\n",
        "    model.add(tf.keras.layers.InputLayer(input_shape=(width, height)))\n",
        "    model.add(tf.keras.layers.Reshape(target_shape=(width, height, number_of_channels)))\n",
        "\n",
        "    for i in range(1,number_of_convolutional):\n",
        "      model.add(tf.keras.layers.Conv2D(64 , kernel_size=(kernal_size[i], kernal_size[i]), activation='relu', kernel_initializer=\"glorot_uniform\"))\n",
        "      model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
        "      model.add(tf.keras.layers.Dropout(rate = 0.3))\n",
        "\n",
        "    model.add(tf.keras.layers.Conv2D(64 , kernel_size=(kernal_size[len(kernal_size)-1], kernal_size[len(kernal_size)-1]), activation='relu', kernel_initializer=\"glorot_uniform\"))\n",
        "\n",
        "\n",
        "    model.add(tf.keras.layers.Flatten())\n",
        "    model.add(tf.keras.layers.Dense(32, activation='relu'))\n",
        "    model.add(tf.keras.layers.Dense(1, kernel_initializer=\"glorot_uniform\"))\n",
        "    model.add(tf.keras.layers.Activation('sigmoid'))\n",
        "    optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)\n",
        "    quantization_aware_model = quantizator(model)\n",
        "    quantization_aware_model.compile(optimizer=optimizer,\n",
        "                  loss='binary_crossentropy',\n",
        "                  metrics=['accuracy'],)\n",
        "    quantization_aware_model.summary()\n",
        "    return quantization_aware_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S_XJABxLzEaG"
      },
      "outputs": [],
      "source": [
        "## since our study on three different comination of the hyperparameters we will create three aware training quantizations models in order to compare the results\n",
        "\n",
        "import tensorflow_model_optimization as tfmot\n",
        "quantizator = tfmot.quantization.keras.quantize_model\n",
        "quantization_aware_model1=build_model_aware_fc(quantizator)\n",
        "quantization_aware_model2=build_model_aware_CNN(quantizator)\n",
        "quantization_aware_model3=build_model_aware_DCNN(quantizator)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M6ccqFczqxMv"
      },
      "source": [
        "## When we train the networks, it is implicitly learning to be resilient to the quantization error."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IL-MNu4OqxMv"
      },
      "outputs": [],
      "source": [
        "aware_results1 = quantization_aware_model1.fit(X_train, y_train, epochs=epochs,batch_size=batch_size, validation_data=(X_valid, y_valid))\n",
        "aware_results2 = quantization_aware_model2.fit(X_train, y_train, epochs=epochs,batch_size=batch_size, validation_data=(X_valid, y_valid))\n",
        "aware_results3 = quantization_aware_model3.fit(X_train, y_train, epochs=epochs,batch_size=batch_size, validation_data=(X_valid, y_valid))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nu6zTcY_qxMw"
      },
      "outputs": [],
      "source": [
        "plot_model_performace_accuracy([aware_results1,aware_results2,aware_results3])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yRs0p9y1A32P"
      },
      "outputs": [],
      "source": [
        "aware_model_list=[quantization_aware_model1,quantization_aware_model2,quantization_aware_model3]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zcylnWd8qxMw"
      },
      "outputs": [],
      "source": [
        "tflite_aware_quantized_models = []\n",
        "for i in range(0, len(aware_model_list)):\n",
        "  converter = tf.lite.TFLiteConverter.from_keras_model(aware_model_list[i])\n",
        "  converter.optimizations = [tf.lite.Optimize.DEFAULT]\n",
        "  converter.representative_dataset = representative_data_gen\n",
        "  converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]\n",
        "  converter.inference_input_type = tf.uint8\n",
        "  converter.inference_output_type = tf.uint8\n",
        "  tflite_aware_quantized_models.append(converter.convert())\n",
        "\n",
        "interpreter = tf.lite.Interpreter(model_content = tflite_aware_quantized_models[0])\n",
        "input_type = interpreter.get_input_details()[0]['dtype']\n",
        "print('input: ', input_type)\n",
        "output_type = interpreter.get_output_details()[0]['dtype']\n",
        "print('output: ', output_type)\n",
        "\n",
        "tflite_aware_quantized_model_files = []\n",
        "tflite_aware_quantized_model_size = []\n",
        "\n",
        "base_filename_aware = 'tflite_aware_quantized_model'\n",
        "path = tflite_models_dir/base_filename_aware\n",
        "\n",
        "if not os.path.exists(path):\n",
        "  os.makedirs(path)\n",
        "\n",
        "for i in range(1, len(aware_model_list) + 1) :\n",
        "  file_name =  f\"{base_filename_aware}_{i}.tflite\"\n",
        "  tflite_aware_quantized_model_files.append(path/file_name)\n",
        "  tflite_aware_quantized_model_files[i - 1].write_bytes(tflite_aware_quantized_models[i - 1])\n",
        "  tflite_aware_quantized_model_size.append(os.path.getsize(tflite_aware_quantized_model_files[i - 1]) / float(2**10))\n",
        "  print(\"TFlite aware quantized model in KB:\", tflite_aware_quantized_model_size[i-1])\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ouJIZDJRjPfW"
      },
      "source": [
        "## Calculate Accuracy, Precision, Recall, F1 Tf_lite_aware_quantized_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ygt18mgzqxMw"
      },
      "outputs": [],
      "source": [
        "aware_quantized_all_results, confusion_aware_quantized_all, aware_quantized_predictions = evaluate_tflite_quantized_models(aware_model_list,base_filename_aware,tflite_aware_quantized_model_files,full=True)\n",
        "## create a table with all tf lite models results\n",
        "headers=['models''accuracy','precision','recall','f1_score','model size KB']\n",
        "data_aware_quantized=[['Aware quantized MLP model ','Aware quantized model CNN ', 'Aware quantized DCNN model ']]\n",
        "data_aware_quantized.extend(aware_quantized_all_results)\n",
        "data_aware_quantized.append(tflite_aware_quantized_model_size)\n",
        "print_table(data_aware_quantized,headers)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **GFLOPS**"
      ],
      "metadata": {
        "id": "-rbJYvCafLIU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from tensorflow.python.profiler.option_builder import ProfileOptionBuilder\n",
        "from tensorflow.python.profiler.model_analyzer import profile\n",
        "models_list.extend(aware_model_list)\n",
        "def get_flops_tfv2_1(model):\n",
        "    forward_pass = tf.function(model.call,\n",
        "                        input_signature=[tf.TensorSpec(shape=(1,) + model.input_shape[1:])])\n",
        "    graph_info = profile(forward_pass.get_concrete_function().graph,\n",
        "                            options=ProfileOptionBuilder.float_operation())\n",
        "    flops = graph_info.total_float_ops // 2\n",
        "    return flops\n",
        "\n",
        "GFlOPS=[]\n",
        "for model in models_list:\n",
        "  gflops=get_flops_tfv2_1(model)\n",
        "  GFlOPS.append(gflops)\n",
        "  print(f'#Operations = {gflops}')"
      ],
      "metadata": {
        "id": "bxt1KoelfP8e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3ffK-qn1EZE3"
      },
      "source": [
        "# **Benchmark Models with STM tools**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MZrtTZTi01aq"
      },
      "outputs": [],
      "source": [
        "# download from github requirement.txt\n",
        "import requests\n",
        "\n",
        "github_file_url = 'https://github.com/STMicroelectronics/stm32ai-modelzoo/blob/main/requirements.txt'\n",
        "local_file_path = 'requirements.txt'\n",
        "\n",
        "response = requests.get(github_file_url)\n",
        "\n",
        "if response.status_code == 200:\n",
        "    json_data = response.json()  # Converti la risposta in JSON\n",
        "    file_content = json_data['payload']['blob']['rawLines']  # Estrai il contenuto del file\n",
        "\n",
        "    with open(local_file_path, 'w') as file:\n",
        "        for line in file_content:\n",
        "            file.write(line + '\\n')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tQr7_l_YFXwE"
      },
      "outputs": [],
      "source": [
        "#install package defined in requirement.txt\n",
        "!pip install -r ./requirements.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VoSVfdhuEgGu"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "os.environ[\"STM32AI_USERNAME\"] = \"ali.dabbous@edu.unige.it\"\n",
        "os.environ[\"STM32AI_PASSWORD\"] = \"Moustafa2365!\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oB2Lz7IcKOfu"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "!{sys.executable} -m pip install pycurl seaborn numpy matplotlib\n",
        "!{sys.executable} -m pip install ipywidgets\n",
        "!{sys.executable} -m pip install gitdir\n",
        "!{sys.executable} -m pip install shutils"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-kYxsji4L-No"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import shutil\n",
        "# Get STM32Cube.AI Developer Cloud\n",
        "!gitdir https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/common/stm32ai_dc\n",
        "\n",
        "# Reorganize local folders\n",
        "if os.path.exists('./stm32ai_dc'):\n",
        "    shutil.rmtree('./stm32ai_dc')\n",
        "shutil.move('./common/stm32ai_dc', './stm32ai_dc')\n",
        "shutil.rmtree('./common')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B10qsQH1MJC2"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "import ipywidgets as widgets\n",
        "\n",
        "sys.path.append(os.path.abspath('stm32ai'))\n",
        "os.environ['STATS_TYPE'] = 'jupyter_devcloud'\n",
        "\n",
        "os.makedirs('models', exist_ok=True)\n",
        "os.makedirs('outputs', exist_ok=True)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zQpJKoM-paeN"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "import os\n",
        "\n",
        "# Append sys.path in order to add import folder for STM32AI\n",
        "dir_name = os.path.dirname('./')\n",
        "sys.path.insert(0, os.path.abspath(os.path.join(dir_name, '..')))\n",
        "sys.path.append(os.path.abspath('../../../common'))\n",
        "from stm32ai_dc import Stm32Ai, CloudBackend, CliParameters\n",
        "from stm32ai_dc.errors import ParameterError, BenchmarkServerError\n",
        "\n",
        "# Get username/password from environment\n",
        "username = os.environ.get('STM32AI_USERNAME', None)\n",
        "password = os.environ.get('STM32AI_PASSWORD', None)\n",
        "\n",
        "results = []\n",
        "\n",
        "# Create STM32AI Class with Cloud Backend, given a username/password and a possible version\n",
        "# Version set to \"None\" will use the latest version available in Developer Cloud\n",
        "ai = Stm32Ai(CloudBackend(username, password, version=None))\n",
        "\n",
        "# List boards available for a benchmark in STM32Cube.AI Developer Cloud\n",
        "boards = ai.get_benchmark_boards()\n",
        "\n",
        "\n",
        "# Boards length should be greater than zero\n",
        "# A length equals to zero mean a current maintenance or a failure\n",
        "if len(boards) == 0:\n",
        "    print(\"No board detected remotely, can't start benchmark\")\n",
        "    sys.exit(0)\n",
        "\n",
        "boards\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1-X3Yp3epgXt"
      },
      "source": [
        "# **Benchmark models on STM boards**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AQ0WexL1aU0c"
      },
      "outputs": [],
      "source": [
        "#uploading models to ST\n",
        "def upload_model(number_of_models,base_file_name_model,model_file_names):\n",
        "\n",
        "  model_names=[]\n",
        "  for i in range(number_of_models):\n",
        "    path_model = base_file_name_model + '/' + model_file_names[i].name\n",
        "    ai.upload_model(path_model)\n",
        "    model_names.append(model_file_names[i].name)\n",
        "  return model_names"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nd0yH0VxbFlr"
      },
      "outputs": [],
      "source": [
        "#deleting model from ST\n",
        "def delete_model(number_of_models,model_names):\n",
        "  for i in range(number_of_models):\n",
        "    model_name=model_names[i]\n",
        "    ai.delete_model(model_name)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2SXwGGe2bhtH"
      },
      "outputs": [],
      "source": [
        "### Analyze and benchmark models\n",
        "def analyze_benchmark_ST(board_name,model_name):\n",
        "  activation_size,weights_size,macc,rom_size,ram_size,execution_time=[], [], [], [], [], []\n",
        "  for i in range(len(model_name)):\n",
        "    analyzing=ai.analyze(CliParameters(model=model_name[i]))\n",
        "    print(analyzing)\n",
        "    activation_size.append(analyzing.activations_size)\n",
        "    weights_size.append(analyzing.weights)\n",
        "    macc.append(analyzing.macc)\n",
        "    rom_size.append(analyzing.rom_size)\n",
        "    ram_size.append(analyzing.ram_size)\n",
        "    benchamrking=ai.benchmark(CliParameters(model = model_name[i]), board_name)\n",
        "    print(benchamrking)\n",
        "    execution_time.append(benchamrking.graph['exec_time']['duration_ms'])\n",
        "  return [activation_size,weights_size,macc,rom_size,ram_size,execution_time]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4VQHmRuJ4j5a"
      },
      "outputs": [],
      "source": [
        "##### benchmark and analyze models\n",
        "def process_models_st(models_number,base_file_name,models_name,model_type):\n",
        "  tflite_model_names=upload_model(models_number,base_file_name,models_name)\n",
        "  data_tflite_board1,data_tflite_board2,data_tflite_board3,data_tflite_board4=[],[],[],[]\n",
        "  print(\"models names\",tflite_model_names)\n",
        "\n",
        "  ######### benchmark and analyze dynamic quantized model with 'B_U585I_IOT02A' 4\n",
        "  B_U585I_IOT02A_tflite_models_results=analyze_benchmark_ST('B-U585I-IOT02A',tflite_model_names)\n",
        "  headers=[\"models\",\"activation_size\",\"weights_size\",\"macc\",\"rom_size\",\"ram_size\",\"execution_time\"]\n",
        "  data_tflite_board1=[[model_type+\" MLP model \",model_type+\" CNN model \",model_type+\" DCNN model \"]]\n",
        "  data_tflite_board1.extend(B_U585I_IOT02A_tflite_models_results)\n",
        "\n",
        "  ######### benchmark and analyze dynamic quantized model with 'NUCLEO-F401RE' 0\n",
        "  NUCLEO_F401RE_tflite_models_results=analyze_benchmark_ST('NUCLEO-F401RE',tflite_model_names)\n",
        "  data_tflite_board2=[[model_type+\" MLP model \",model_type+\" CNN model \",model_type+\" DCNN model \"]]\n",
        "  data_tflite_board2.extend(NUCLEO_F401RE_tflite_models_results)\n",
        "\n",
        "  ######### benchmark and analyze dynamic quantized model with  'STM32L4R9I-DISCO' 1\n",
        "  STM32L4R9I_DISCO_tflite_models_results=analyze_benchmark_ST('STM32L4R9I-DISCO',tflite_model_names)\n",
        "  data_tflite_board3=[[model_type+\" MLP model \",model_type+\" CNN model \",model_type+\" DCNN model \"]]\n",
        "  data_tflite_board3.extend(STM32L4R9I_DISCO_tflite_models_results)\n",
        "\n",
        "  ######### benchmark and analyze dynamic quantized model with  'STM32H7B3I-DK' 2\n",
        "  STM32H7B3I_DK_tflite_models_results=analyze_benchmark_ST('STM32H7B3I-DK',tflite_model_names)\n",
        "  data_tflite_board4=[[model_type+\" MLP model \",model_type+\" CNN model \",model_type+\" DCNN model \"]]\n",
        "  data_tflite_board4.extend(STM32H7B3I_DK_tflite_models_results)\n",
        "\n",
        "  delete_model(models_number,tflite_model_names)\n",
        "  print(\"B-U585I-IOT02A\")\n",
        "  print_table(data_tflite_board1,headers)\n",
        "  print(\"NUCLEO-F401RE\")\n",
        "  print_table(data_tflite_board2,headers)\n",
        "  print(\"STM32L4R9I-DISCO\")\n",
        "  print_table(data_tflite_board3,headers)\n",
        "  print(\"STM32H7B3I-D\")\n",
        "  print_table(data_tflite_board4,headers)\n",
        "  return B_U585I_IOT02A_tflite_models_results, NUCLEO_F401RE_tflite_models_results,STM32L4R9I_DISCO_tflite_models_results,STM32H7B3I_DK_tflite_models_results,data_tflite_board1,data_tflite_board2,data_tflite_board3,data_tflite_board4"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Analyze and Benchmark best models on STM boards"
      ],
      "metadata": {
        "id": "rppx4P8NQ9P_"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-9JSxSC0djKe"
      },
      "outputs": [],
      "source": [
        "############# tflite model hardware analysis and benchmarking\n",
        "B_U585I_IOT02A_tflite_models_result, NUCLEO_F401RE_tflite_models_result,STM32L4R9I_DISCO_tflite_models_result,STM32H7B3I_DK_tflite_models_result,data_tflite_model_board1,data_tflite_model_board2,data_tflite_model_board3,data_tflite_model_board4=process_models_st(len(tflite_modelfiles_best),base_filename_model_tflite,tflite_modelfiles_best,\"tflite\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KHmwqkAa6Y2N"
      },
      "outputs": [],
      "source": [
        "############# dynamic quantized tflite model hardware analysis and benchmarking\n",
        "B_U585I_IOT02A_dynamic_quantized_tflite_models_result, NUCLEO_F401RE_dynamic_quantized_tflite_models_result,STM32L4R9I_DISCO_dynamic_quantized_tflite_models_result,STM32H7B3I_DK_dynamic_quantized_tflite_models_result,data_dynamic_model_board1,data_dynamic_model_board2,data_dynamic_model_board3,data_dynamic_model_board4=process_models_st(len(dynamic_modelfiles_best),base_filename_dynamic,dynamic_modelfiles_best,\"dynamic quantized tflite\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Sf0Nzd2X6z6_"
      },
      "outputs": [],
      "source": [
        "############# static quantized tflite model hardware analysis and benchmarking\n",
        "B_U585I_IOT02A_static_quantized_tflite_models_result, NUCLEO_F401RE_static_quantized_tflite_models_result,STM32L4R9I_DISCO_static_quantized_tflite_models_result,STM32H7B3I_DK_static_quantized_tflite_models_result,data_static_model_board1,data_static_model_board2,data_static_model_board3,data_static_model_board4=process_models_st(len(static_modelfiles_best),base_filename_static,static_modelfiles_best,\"static quantized tflite\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LBXkRi-SHSr9"
      },
      "outputs": [],
      "source": [
        "############# aware quantized tflite model hardware analysis and benchmarking\n",
        "B_U585I_IOT02A_aware_quantized_tflite_models_result, NUCLEO_F401RE_aware_quantized_tflite_models_result,STM32L4R9I_DISCO_aware_quantized_tflite_models_result,STM32H7B3I_DK_aware_quantized_tflite_models_result,data_aware_model_board1,data_aware_board2,data_aware_model_board3,data_aware_model_board4=process_models_st(len(tflite_aware_quantized_model_files),base_filename_aware,tflite_aware_quantized_model_files,\"aware quantized tflite\")\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Generate C Code from your model**"
      ],
      "metadata": {
        "id": "kNKIOTHzp3dO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "##### Generate C code for your model\n",
        "def generate_C_Code(models_number,base_file_name,models_name,model_type):\n",
        "  tflite_model_names=upload_model(models_number,base_file_name,models_name)\n",
        "  print(\"models names\",tflite_model_names)\n",
        "  for i in range(len(tflite_model_names)):\n",
        "    model_output=f\"./model_output {i}\"\n",
        "    res= ai.generate(CliParameters(model=tflite_model_names[i], output=model_output))\n",
        "    print(\"Result from local file:\", res, flush=True)\n",
        "  delete_model(models_number,tflite_model_names)"
      ],
      "metadata": {
        "id": "CWLhuQPfqHoZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "generate_C_Code(len(static_modelfiles_best),base_filename_static,static_modelfiles_best,\"static quantized tflite\")"
      ],
      "metadata": {
        "id": "GOthc6wNvPGI"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [
        "yK-ONXiYqxMW",
        "FZvDBg5SqxMY",
        "kdfiN3mRqxMY",
        "C9o1ic6iqxMZ",
        "d2_s-MDlqxMa",
        "8j8ppLxnqxMa",
        "knIUQ0G5qxMa",
        "jyn-YRKQqxMb",
        "Zku4ylweqxMb",
        "25pky3Vh7tIX",
        "08rjB07ZqxMc",
        "w2ou5uc7qxMe",
        "02oo8O8tu14A",
        "s9lpEAQq8Nb1",
        "_z44ZJQ_vOuy",
        "60T0RwetvooX",
        "iGsbnuckqxMc",
        "m0abDXWnaomn",
        "w_sElX_29SoE",
        "ljcvgMjlivyr",
        "Zh8j8vb_Ojxv",
        "QAyYZPImu9hv",
        "AJ29ZnS4O_eH",
        "GccTxLr9PKkR",
        "QyXelBx3vL5u",
        "MHN6N68et-_P",
        "oyKybPUZG-RV",
        "QL1DrH7wjPfC",
        "oaqmNKfujPfF",
        "uF562MkZjPfG",
        "J4dgcL2Xd8Bx",
        "55wJBmnY6gMb",
        "JfPtfsRjyyTf",
        "tT2l5Hpcy9aD",
        "6bXDmagDzvnu",
        "UUU_a3XeqxMq",
        "fAK7VfJsbGfQ",
        "1a-Z7foywbsu",
        "VeAumtFjqxMq",
        "JF5adl6njPfJ",
        "8JRZqEpUqxMr",
        "xLvlcFezxB7z",
        "uGcAO554criq",
        "VnsT3qFDjPfP",
        "mx5wtz8xjPfQ",
        "V3_niyOBqxMt",
        "xFtAoEXSqxMu",
        "uyYekcEYjPfT",
        "HmCqpGjTf-DT",
        "aU0BlqEQjPfU",
        "cPAnK6Q0qKqx",
        "M6ccqFczqxMv",
        "ouJIZDJRjPfW",
        "-rbJYvCafLIU",
        "3ffK-qn1EZE3",
        "1-X3Yp3epgXt",
        "rppx4P8NQ9P_",
        "kNKIOTHzp3dO"
      ],
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.3"
    },
    "nav_menu": {
      "height": "279px",
      "width": "309px"
    },
    "toc": {
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": false,
      "toc_cell": false,
      "toc_position": {},
      "toc_section_display": "block",
      "toc_window_display": false
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}