RandomFerns.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Random Ferns </h3>\n",
    "Random ferns is an ensemble probabilistic method for classification. \n",
    "\n",
    "In optimal Bayesian classifier:\n",
    "\n",
    "$ H(F) = \\underset{k}{argmax} \\frac{P(F|c_k)P(c_k)}{P(F)}= \\underset{k}{argmax} P(F|c_k)P(c_k)$\n",
    "\n",
    "where $[F]_{d \\times 1}$ is a $d$ dimensional input vector and $P(F|c_k)$ is the probability of $F$ given class label $k$. The problem is calculating $P(F|c_k)$ for high dimensional data (d) is not possible, because:\n",
    "\n",
    "$P(F|c_k) = P(f_1 | f_2,f_3,...,f_d,c_k) P(f_1, f_2 | f_3,...,f_d,c_k) ... P(f_d| f_1,f_2,...,f_{d-1},c_k)$ \n",
    "\n",
    "which its complexity is $O(2^d)$. \n",
    "In Naive Bayes method, it is assumed that the features are independent. Under this assumption:\n",
    "\n",
    "$ P(F|c_k) = P(f_1 |c_k) P(f_2 |c_k) ... P(f_d|c_k) = \\prod\\limits_{i=1}^{d}P(f_i|c_k)$ \n",
    "\n",
    "This assumption is not correct in all cases, specially for data in which features are correlated. For example, in images, each pixel is similar to its neighbors so actually they are not independent. An intermediate approach which asumes that some features are dependent and some features are independent is semi-naive bayes method. In semi-naive bayes, features are partitioned into $L$ non-overlapping subsets. features in each subset are dependent and features of a subset are independent from features of other subsets. In this case:\n",
    "\n",
    "$P(F|c_k) = \\prod\\limits_{l=1}^{L} P(F|c_k)$                         \n",
    "$H(F) = \\underset{k}{argmax} P(c_k) \\prod\\limits_{l=1}^{L}P(F_l|c_k)$\n",
    "\n",
    "<b>R</b>andom <b>F</b>erns(RFs) is an estimation for semi-naive bayes method. In RFs, estimation of $P(F_l|c_k)$ is done by a set of binary tests. Binary tests are used to estimate feature dependencies. Each fern estimates the conditional probability $P(F_l|c_k)$ based on the features in subset $F_l$. Binary test is simply as follows:\n",
    "\n",
    "$T(f_i,f_j) = \\begin{cases} 1, & \\text{if} & f_i > f_j \\\\ 0, & \\text{otherwise} \\end{cases} $\n",
    "\n",
    "applying $\\text{S}$ different binary tests leads to a $\\text{S}$ bit binary code which can be converted to a decimal number. So, each fern takes an input and produces a number in range $[0, 2^S - 1]$:\n",
    "<img src=\"data/temp1.png\" width=\"400\" height=\"200\">\n",
    "\n",
    "In other words, a Fern converts a subset of features to an integer number. For a set of training data from class $c_k$, we feed the input training samples to a fern and get the result. We can make a Histogram by the outputs of the fern.\n",
    "<img src=\"data/temp2.png\" width=\"500\" height=\"400\">\n",
    "The obtained histogram is an estimation for $P(F_l | c_k)$.\n",
    "\n",
    "Here is an implementation of <b>Random Ferns</b> method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "\n",
    "class Fern(object):\n",
    "    def __init__(self, features, n_class, code_length):            \n",
    "        self.F = features\n",
    "        self.S = code_length\n",
    "        self.C = n_class\n",
    "        \n",
    "        self.Tests = []\n",
    "        self.Histogram = None \n",
    "        self.FITTED = False\n",
    "        \n",
    "        passes = 0\n",
    "        cntr = 0\n",
    "        while cntr < self.S:\n",
    "            passes += 1\n",
    "            if passes > 2**self.S:\n",
    "                print(\"Error! It's not possible to make \" , 2**self.S , \n",
    "                      \" different random tests on \" , len(self.F) , \" smaples...\")\n",
    "                return\n",
    "            b = random.choices(self.F, k=2)\n",
    "            if ((b[0],b[1]) in self.Tests) or ((b[1],b[0]) in self.Tests) or (b[0]==b[1]):\n",
    "                continue\n",
    "            self.Tests.append((b[0],b[1]))\n",
    "            cntr += 1\n",
    "            \n",
    "    def __str__(self):\n",
    "        return(str(self.Tests))\n",
    "    \n",
    "    def forward(self, x):\n",
    "        if type(x)==np.ndarray:\n",
    "            if x.ndim >= 2:\n",
    "                x = x.flatten()\n",
    "        \n",
    "        code = []\n",
    "        for test in self.Tests:\n",
    "            if x[test[0]] > x[test[1]]:\n",
    "                code.append(1)\n",
    "            else:\n",
    "                code.append(0)\n",
    "        \n",
    "        exp = 1\n",
    "        decimal = 0\n",
    "        for b in code:\n",
    "            decimal = decimal + (exp*b)\n",
    "            exp = exp*2\n",
    "        \n",
    "        return decimal, code\n",
    "    \n",
    "    def pdf(self, x):\n",
    "        if x>2**self.S or x<0:\n",
    "            print(\"Error. Invalid Input.\")\n",
    "            return 0\n",
    "        res = []\n",
    "        for i in range(self.C):\n",
    "            res.append(self.Histogram[i,x])\n",
    "        return res\n",
    "    \n",
    "    def fit(self, X, Y):\n",
    "        self.Histogram = np.zeros((self.C, 2**self.S))\n",
    "        for (sample,label) in zip(X, Y):\n",
    "            decimal, _ = self.forward(sample)\n",
    "            self.Histogram[label, decimal] += 1\n",
    "        self.FITTED = True\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we will train some Random Ferns on MNIST dataset.\n",
    "First of all, we need a method to read <a href=\"http://yann.lecun.com/exdb/mnist/\">MNIST data</a>.\n",
    "<i>read_mnist</i> function gets paths for MNIST ubyte image file and ubyte label file and returns an array of size $n \\times 28 \\times 28$, and a list of size $n$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Labels:  [7 2 1 ... 4 5 6]\n",
      "Images:  (10000, 28, 28)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import struct\n",
    "\n",
    "def read_mnist(images_file_path, labels_file_path):\n",
    "    try:\n",
    "        with open(labels_file_path , 'rb') as f:\n",
    "            struct.unpack(\">II\", f.read(8))\n",
    "            labels = np.fromfile(f, dtype=np.int8)\n",
    "\n",
    "        with open(images_file_path, 'rb') as f:\n",
    "            _,_, rows, cols = struct.unpack(\">IIII\", f.read(16))\n",
    "            images = np.fromfile(f, dtype=np.uint8).reshape(len(labels), rows, cols)\n",
    "\n",
    "        return images, labels    \n",
    "    except IOError:\n",
    "        print(\"Error in reading files.\")\n",
    "            \n",
    "\n",
    "imgs,lbls = read_mnist(\"MNIST/t10k-images-idx3-ubyte\", \"MNIST/t10k-labels-idx1-ubyte\")\n",
    "print(\"Labels: \", lbls)\n",
    "print(\"Images: \" , imgs.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label of the sample image:  0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAANq0lEQVR4nO3db6xU9Z3H8c9HFp6gRsRowJotEGNcjesfYkjERW3auEpUHlQhcXUj5vqnJm1ckjUssSSmCW62bnyEuUSE3bA2jdBIaiM1iLqIMeCfBRRb0bDthRuQoHKJJl3kuw/uobnFO2cuM2fmDHzfr2QyM+c7Z843Ez6cM/M75/4cEQJw+juj7gYAdAdhB5Ig7EAShB1IgrADSfxVNzdmm5/+gQ6LCI+2vK09u+2bbf/O9m7bj7XzXgA6y62Os9seJ+n3kr4vaUDSVkkLIuLDknXYswMd1ok9+7WSdkfEpxHxJ0m/kHR7G+8HoIPaCfuFkv444vlAsewv2O6zvc32tja2BaBN7fxAN9qhwrcO0yOiX1K/xGE8UKd29uwDki4a8fw7kva11w6ATmkn7FslXWx7mu0JkuZLWl9NWwCq1vJhfEQctf2IpA2SxklaGREfVNYZgEq1PPTW0sb4zg50XEdOqgFw6iDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IImuTtmMzpg9e3bD2ltvvVW67iWXXFJanzt3bmn91ltvLa2/9NJLpfUyW7ZsKa1v3ry55ffOiD07kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBLK494Oyzzy6tr1mzprR+0003Nax9/fXXpetOmDChtH7mmWeW1jupWe9fffVVaf2hhx5qWHvhhRda6ulU0GgW17ZOqrG9R9KQpG8kHY2Ime28H4DOqeIMuhsj4mAF7wOgg/jODiTRbthD0m9tv2O7b7QX2O6zvc32tja3BaAN7R7GXxcR+2yfL+kV2x9FxBsjXxAR/ZL6JX6gA+rU1p49IvYV9wck/UrStVU0BaB6LYfd9kTbZx1/LOkHknZW1RiAarU8zm57uob35tLw14H/ioifNVmHw/hRLF++vLT+wAMPdGzbu3btKq1/9tlnpfXDhw+3vG171OHgP2t2rXwzQ0NDDWvXX3996brbt29va9t1qnycPSI+lfS3LXcEoKsYegOSIOxAEoQdSIKwA0kQdiAJLnHtgssuu6y0/tprr5XWJ0+eXFofGBhoWLvnnntK1929e3dp/YsvviitHzlypLRe5owzyvc1jz/+eGl9yZIlpfVx48Y1rK1bt6503fvvv7+0/vnnn5fW69Ro6I09O5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwZTNXXDWWWeV1puNozc7F+LJJ59sWGs2hl+nY8eOldaXLl1aWm/2Z7AXLVrUsDZv3rzSdVeuXFlab2cq6rqwZweSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJLievQvmzJlTWt+0aVNpfdWqVaX1++6772RbSuGTTz5pWJs2bVrpus8991xpfeHChS311A1czw4kR9iBJAg7kARhB5Ig7EAShB1IgrADSXA9exc88cQTba3/9ttvV9RJLhs2bGhYe/DBB0vXnTVrVtXt1K7pnt32StsHbO8csexc26/Y/ri4n9TZNgG0ayyH8ask3XzCssckbYyIiyVtLJ4D6GFNwx4Rb0g6dMLi2yWtLh6vlnRHxX0BqFir39kviIhBSYqIQdvnN3qh7T5JfS1uB0BFOv4DXUT0S+qX8l4IA/SCVofe9tueIknF/YHqWgLQCa2Gfb2ke4vH90p6sZp2AHRK08N4289LukHSebYHJP1U0jJJv7S9UNIfJP2wk032uunTp5fWp06dWlr/8ssvS+s7duw46Z4gvfrqqw1rzcbZT0dNwx4RCxqUvldxLwA6iNNlgSQIO5AEYQeSIOxAEoQdSIJLXCtw9913l9abDc2tXbu2tL5ly5aT7gk4EXt2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCcfYKzJ8/v7Te7BLWp59+usp2gFGxZweSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJBhn74KPPvqotL558+YudYLM2LMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMs4/RxIkTG9bGjx/fxU6A1jTds9teafuA7Z0jli21vdf2+8Xtls62CaBdYzmMXyXp5lGW/3tEXFncflNtWwCq1jTsEfGGpENd6AVAB7XzA90jtrcXh/mTGr3Idp/tbba3tbEtAG1qNezLJc2QdKWkQUk/b/TCiOiPiJkRMbPFbQGoQEthj4j9EfFNRByTtELStdW2BaBqLYXd9pQRT+dJ2tnotQB6Q9NxdtvPS7pB0nm2ByT9VNINtq+UFJL2SHqggz32hDvvvLNhbcaMGaXrHjx4sOp2MAa33XZby+sePXq0wk56Q9OwR8SCURY/24FeAHQQp8sCSRB2IAnCDiRB2IEkCDuQBJe44pR1zTXXlNbnzp3b8nsvXry45XV7FXt2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCcXb0rGbj6I8++mhp/ZxzzmlYe/PNN0vX3bBhQ2n9VMSeHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSYJx9jPbs2dOwNjQ01L1GTiPjxo0rrS9atKi0ftddd5XW9+7d2/J7n45/Spo9O5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4k4Yjo3sbs7m2siz788MPSerPPeM6cOaX1Xp7y+YorriitP/zwww1rV199dem6M2fObKmn42688caGtddff72t9+5lEeHRljfds9u+yPYm27tsf2D7x8Xyc22/Yvvj4n5S1U0DqM5YDuOPSvqniLhU0ixJP7L9N5Iek7QxIi6WtLF4DqBHNQ17RAxGxLvF4yFJuyRdKOl2SauLl62WdEenmgTQvpM6N972dyVdJeltSRdExKA0/B+C7fMbrNMnqa+9NgG0a8xht32mpLWSfhIRh+1RfwP4lojol9RfvMdp+QMdcCoY09Cb7fEaDvqaiFhXLN5ve0pRnyLpQGdaBFCFpnt2D+/Cn5W0KyKeGlFaL+leScuK+xc70uFp4NJLLy2tv/zyy6X1wcHBKtup1KxZs0rrkydPbvm9mw05rl+/vrS+devWlrd9OhrLYfx1kv5B0g7b7xfLFms45L+0vVDSHyT9sDMtAqhC07BHxGZJjb6gf6/adgB0CqfLAkkQdiAJwg4kQdiBJAg7kASXuFZg3rx5pfUlS5aU1q+66qoq2+kpx44da1g7dOhQ6bpPPfVUaX3ZsmUt9XS6a/kSVwCnB8IOJEHYgSQIO5AEYQeSIOxAEoQdSIJx9i6YOnVqab3Z9eyXX355le1UasWKFaX19957r2HtmWeeqbodiHF2ID3CDiRB2IEkCDuQBGEHkiDsQBKEHUiCcXbgNMM4O5AcYQeSIOxAEoQdSIKwA0kQdiAJwg4k0TTsti+yvcn2Ltsf2P5xsXyp7b223y9ut3S+XQCtanpSje0pkqZExLu2z5L0jqQ7JN0p6UhE/NuYN8ZJNUDHNTqpZizzsw9KGiweD9neJenCatsD0Gkn9Z3d9nclXSXp7WLRI7a3215pe1KDdfpsb7O9ra1OAbRlzOfG2z5T0uuSfhYR62xfIOmgpJD0hIYP9e9r8h4cxgMd1ugwfkxhtz1e0q8lbYiIb822V+zxfx0RpX8ZkbADndfyhTC2LelZSbtGBr344e64eZJ2ttskgM4Zy6/xsyX9t6Qdko7Pv7tY0gJJV2r4MH6PpAeKH/PK3os9O9BhbR3GV4WwA53H9exAcoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkmv7ByYodlPS/I56fVyzrRb3aW6/2JdFbq6rs7a8bFbp6Pfu3Nm5vi4iZtTVQold769W+JHprVbd64zAeSIKwA0nUHfb+mrdfpld769W+JHprVVd6q/U7O4DuqXvPDqBLCDuQRC1ht32z7d/Z3m37sTp6aMT2Hts7immoa52frphD74DtnSOWnWv7FdsfF/ejzrFXU289MY13yTTjtX52dU9/3vXv7LbHSfq9pO9LGpC0VdKCiPiwq400YHuPpJkRUfsJGLb/TtIRSf9xfGot2/8q6VBELCv+o5wUEf/cI70t1UlO492h3hpNM/6PqvGzq3L681bUsWe/VtLuiPg0Iv4k6ReSbq+hj54XEW9IOnTC4tslrS4er9bwP5aua9BbT4iIwYh4t3g8JOn4NOO1fnYlfXVFHWG/UNIfRzwfUG/N9x6Sfmv7Hdt9dTcziguOT7NV3J9fcz8najqNdzedMM14z3x2rUx/3q46wj7a1DS9NP53XURcLenvJf2oOFzF2CyXNEPDcwAOSvp5nc0U04yvlfSTiDhcZy8jjdJXVz63OsI+IOmiEc+/I2lfDX2MKiL2FfcHJP1Kw187esn+4zPoFvcHau7nzyJif0R8ExHHJK1QjZ9dMc34WklrImJdsbj2z260vrr1udUR9q2SLrY9zfYESfMlra+hj2+xPbH44US2J0r6gXpvKur1ku4tHt8r6cUae/kLvTKNd6NpxlXzZ1f79OcR0fWbpFs0/Iv8J5L+pY4eGvQ1XdL/FLcP6u5N0vMaPqz7Pw0fES2UNFnSRkkfF/fn9lBv/6nhqb23azhYU2rqbbaGvxpul/R+cbul7s+upK+ufG6cLgskwRl0QBKEHUiCsANJEHYgCcIOJEHYgSQIO5DE/wMI00LClW/+dgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "sample_index = 10\n",
    "im = imgs[sample_index,:,:]\n",
    "print(\"Label of the sample image: \", lbls[sample_index])\n",
    "plt.imshow(im, cmap='gray')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In next step, we will create 20 random ferns and train them on MNIST samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 20 ferns for each class\n",
    "\n",
    "d = imgs.shape[1]*imgs.shape[2]   # number of features(pixels)\n",
    "n_classes = 10\n",
    "n_ferns = 20\n",
    "code_length = 8   # number of binary tests, or S in formulas\n",
    "\n",
    "ferns = []\n",
    "for i in range(n_ferns):\n",
    "    fern = Fern(features=range(d), n_class=n_classes, code_length=code_length)\n",
    "    fern.fit(imgs,lbls)\n",
    "    ferns.append(fern)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a test sample, we will feed it to all the ferns and get the outputs. Then, a majority voting will be used for the classification:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input sample:  4\n",
      "Prediction:  4\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEWCAYAAAB/tMx4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxcdbn48c+TPU3aNGnTNF3TlrKUUoqErSiyVakFiwsIIqKXa+UK4sLVH+6Kl9+PexUVN7RKBWSXTbTIchGBKltboAuFFpLuSZp0mezrPL8/zkk7yUySmclsZ+Z5v155TeY7Z855huqT73zP8/1+RVUxxhjjPVnJDsAYY0x0LIEbY4xHWQI3xhiPsgRujDEeZQncGGM8yhK4McZ4lCVwY4zxKEvgxpiIiMiTInJDiPZlIlIvIjlDvO9MEdkV/wgzhyVwY0ykbgcuFxEZ1H45cLeq9iY+pMxkCdwYE6lHgTLgff0NIlIKnA/cKSI/E5E97s/PRCRfRIqAvwFTRKTV/ZkiIlkicr2IvCsi+0TkAREpc89ZICJ3ue0HReRVEalIxgdOVZbAjTERUdUO4AHg0wHNFwNvAR8DTgUWAscDJwPfVtU2YAmwR1WL3Z89wLXAhcD7gSnAAeBX7jmvAEqA6cAE4CqgI76fzlssgRtjonEHcJGIFLrPP+22XQbcoKp7VbUR+AHO0MpQPg98S1V3qWoX8H3g4+44eg9O4j5CVftUda2qNsfp83iSJXBjTMRUdTXQCCwTkdnAScA9OL3o7QGHbnfbhjITeMQdIjkIbAb6gArgj8CTwH3ucMz/iEhu7D+Nd1kCN8ZE606cnvflwFOq2gDswUnK/Wa4bQChlj7dCSxR1fEBPwWqultVe1T1B6o6D1iEM8b+6RDnyFiWwI0x0boTOBf4HM7wCcC9wLdFpFxEJgLfBe5yX2sAJohIScA5fgPcKCIzAdz3LXN/P0tEjhORbKAZZ0ilL94fykssgRtjoqKq24B/AUXAY27zfwFrgPXABmCd24aqvoWT4GvcIZMpwC3ue58SkRbgJeAU91yTgQdxkvdm4DkO/zEwgNiGDsYY403WAzfGGI+yBG6MMR5lCdwYYzzKErgxxnhUyFXDjDGRmThxolZVVSU7DJOm1q5d26Sq5YPbLYGbtCciK3EmgexV1flu20U407aPAU5W1TUBx38DuBKn5vhaVX1ypGtUVVWxZs2akQ4zJioisj1Uuw2hmExwO3DeoLaNwEeB5wMbRWQecAlwrPueX7sTSYxJOZbATdpT1eeB/YPaNqvq2yEOXwbcp6pdqloLvIOzop4xKccSuDEDTcVZn6PfLrfNmJRjCdyYgQbvMgOhF2FCRJaLyBoRWdPY2BjnsIwJZgncmIF24Wwg0G8ah1fTG0BVV6hqtapWl5cHFQgYE3eWwI0Z6DHgEncbsFnAXOCVJMfkHXffDVVVkJXlPN59d7IjSmtWRmjSnojcC5wJTHR3Rf8ezk3NXwDlwCoReV1VP6iqm0TkAeBNoBe4WlVtCdNw3H03LF8O7e3O8+3bnecAl12WvLjSmK1GaEwMVFdXa8bXgVdVOUl7sJkzYdu2REeTVkRkrapWD263IRRjTGzs2BFZuxk1S+DGmNiYMSOydjNqlsCNMbFx4434CwsHto0ZAzfemJx4MoAlcGNMbFx2GW/dcDO7xpXjR2ieNAVWrLAbmHFkVSjGmJh56dTzuOE/ZlBZUsDs8iLuvuzUZIeU1qwHboyJmdqmNsYW5LB4XgWv7ThIb58/2SGlNUvgxpiYqWlqZXZ5MdVVZbR397G5riXZIaU1S+DGmJipaWxjzsQiTqoqBeDVbftHeIcZDUvgxpiYaO/upc7XyayJRVSWFDJ1fCFrtx9IdlhpzRL4ICKyTUTOTcB1vi8id6VCLMbEQm1TGwCzy4sBqK4q5dVt+7HZ3vFjCdwYExM1jf0JvAiA6pml7G3pYteBjmSGldYsgQ9DRD4jIqtF5McickBEakVkScDr/xCR/ycir4iIT0T+LCJl7mtnugsnBZ5vm4icKyLnAd8EPiEirSLyRpix/FNEfioiB0WkRkQWue07RWSviFwRcPxSEXlNRJrd178/6HyfFpHtIrJPRL4T2NsXkSwRuV5E3nVff6D/cxkzlP4eeNUEN4FXOf+TsXHw+LEEPrJTgLeBicD/ALeJSOCi/58G/g2YgrN63c9HOqGqPgH8X+B+VS1W1eMjiGU9MAG4B7gPOAk4AvgU8EsRKXaPbXNjGw8sBf5DRC6EQ/s+/hq4DKgEShi468y1wIXA+93PdQD4VZgxmgxV09jK1PGFFOY5W4geWTGWsQU5vLrNxsHjxRL4yLar6u/cJUXvwEl4FQGv/1FVN6pqG/Ad4OI4boJbq6p/cGO5H2fjgRvc/RufArpxkjmq+g9V3aCqflVdD9yLk5ABPg78RVVXq2o38F0G7jrzeeBbqrpLVbtwdm//uIjYxC8zpJqmtkPDJwDZWcJ7ZpSydrv1wOPFEvjI6vt/UVV3oWOKA14P3D9xO5CL01uPh4aA3zvcmAa3FQOIyCki8qyINIqID7gqIK4pgXG7n2tfwHlmAo+4QzUHgc1AHwP/cBlziKpS29jGrIlFA9pPqiplS0MrB9u7kxRZerMEPnqB22/NAHqAJpwhjDH9L7i98sB9t+J9a/4enN1lpqtqCfAbDu/3WIezVVh/bIU4wzL9dgJLVHV8wE+Bqu6Oc8zGoxpbu2jp6mX2oATePw5u5YTDe3xDHTv3t4984CCWwEfvUyIyT0TGADcAD7pDHFuAAvdmYi7wbSA/4H0NQJWIxOvfYCywX1U7ReRk4JMBrz0IXODeBM0DfsDAzXx/A9woIjMBRKRcRJbFKU6TBg5XoBQPaD9+2nhysoQ1lsCHtL+tm68+8Do/f2ZrxO+1BD56fwRuxxlqKcC5AYiq+oAvAL8HduP0yAOrUv7kPu4TkXVxiOsLwA0i0oIzxv1A/wuqugn4Is5N0DqgBdgLdLmH3ILTe3/Kff9LODdQjQmpvwJl8BBKYV4286eWsMYqUYZ054vb6Ozxs/yM2RG/125KDaKqVQG/346TnANfl4Hv4F1V/cYQ5xr8/h8HvLYPeG+0sajqOwzsNaOq0wJ+fxCnpz3UuQ+dz61c+R7uHxhV9QM/cX+MGVFNYyv5OVlMHV8Y9Fr1zFLufGk7Xb195OfE6/6+N3V093Hni9s5++hJzK0YG/H7rQeeoUTkAhEZIyJFOH9YNgDbkhuV8aoa9wZmVtbg/o0zDt7d62fjbl8SIkttD67bxf627qh632AJPJMtA/a4P3OBS9TmPJso1TYFV6D0qz60sJWNgwfq8yu/f6GG46eVcMqs6ObJWQIfBVU9U1V/n+w4oqGq/+5Wl5So6jmq+nayYzLe1NPnZ8f+9gE14IEmFucza2IRayyBD/DUpnq272tn+RlzGDg3MHyWwI0xo7Jjfzu9fmX2xOIhj6me6Uzo8fvtSx44dfO/eb6GGWVjOG/+5KjPM6qbmO6aHrcA2cDvVfWmEY63fz0TVyFuMps4q3VLCGcN0QMHOKmqjD+t3UVNUytHTIr8Zl26eaV2P2/sPMgPlx1Ldoj7BuGKugfuTkz5FbAEmAdc6q6xYYzJIDVNrQDMGaYHfqI7Dm7DKI4Vz9dQVpTHx0+cPvLBwxjNEMrJwDuqWuOup3Efzo0xY1KKiKx0V2vcGNBWJiJPi8hW97E04LVviMg7IvK2iHwwOVF7R01jGxOK8igZkzvkMbMnFlFWlGc3MoGtDS0889ZeLj915qGFv6I1mgQ+lYHrgOxi4Ip2AIjIchFZIyJrRnEtY0bjduC8QW3XA8+o6lzgGfd5/0qNlwDHuu/5dRwXJ0sLNcNUoPQTEU6cWcoaW9iK371QQ35OFp8+beaozzWaBB5q4CZojFtVV6hqtapWj+JaxkRNVZ8HBmeOZTirS+I+XhjQfp+7wmMt8A7Ot00zhJrGtiErUAKdVFXK9n3t7G3pTEBUqWlvcyePvraHi6qnMaE4f+Q3jGA0CXwXAxdymoZTU2yMF1Soah2A+zjJbQ/rmyUM/HbZ2NgY12BTVXNnD02tXUFroIRyaGGrDB5G+cO/ttHr9/Pv741u4s5go0ngrwJzRWSWuyDSJTjrZxjjZWF9s4SB3y7Ly8tDHZL2DlWgjDCEAjB/Sgn5OVkZu7BVa1cvd720nfPmT6YqjP9e4Yg6gatqL3AN8CTOetEPuIskGeMFDSJSCeA+7nXb7ZtlBA5VoIQxhJKXk8Xx08dn7MJW972yg5bOXpafMSdm5xzVRB5VfVxVj1TVOap6Y6yCMiYBHgP69xC9AvhzQPslIpIvIrNwlhl4JQnxeUJNYxtZAjPKwutRnlRVysY9zbR398Y5stTS0+dn5epaTp5VxsLp42N2XpuJadKeiNwLvAgcJSK7RORK4CZgsYhsBRa7z/uX2n0AeBN4ArjaXd/dhFDT1Mb0sjHk5YSXSqpnltHnV17feTDOkaWWv67fwx5fJ5+PctGqodhysibtqeqlQ7x0zhDH3wjYN8ow1DS2Be3CM5z3zChFxJnQs2hOvHYeTC2qym+fq+GIScWcddSkkd8QAeuBG2Oi4vcrtU2tYVWg9CsZk8tRFWN5NYPGwV/Y2sRb9S0sf9/skMvtjoYlcGNMVOqbO+ns8YdVgRLoxJmlvLbjIH0ptrBVn1/jEtNvn3+XSWPzWXbClJif2xK4MSYqh/fBjCyBn1RVRmtXL2/VN8cjrKj955/e4Iz/eTamcW3c7eOf7+zjs6fPistuRJbAjTFROVxCGP4QCjg9cEitneo37vbxyGu7qW/u5OO3vsgLW2MzMWvF8zUU5WXzyVNmxOR8g1kCN8ZEpaaxjaK8bCaNjWxK+LTSQiaPK0ipha1+8vQWSgpz+duX3se00kI++4dXeeDVnSO/cRi7DrSzakMdl548g5LCoRf6Gg1L4MaYqNQ0tTGrvCji3WREhOqq0pSZ0LNuxwH+/tZelp8xmyMrxvKnq07jtDkT+PpD67n5qbeJdqfB21bXIsC/vXdWbAMOYAncGBOVmsZWZg2zBvhwqmeWUufrZPfBjhhHFbmfPr2FCUV5fGZRFQBjC3JZ+ZmT+ET1dH7x93f4yv2v09Ub2VQAX3sP97+6kwuOn8KU8YVxiNphCdwYE7HOnj52H+yIqAY8UP/CVsnuhb9Su58XtjZx1fvnUJR/eFpMbnYWN33sOL72waN49PU9fPq2V/C194R93rte3k57d1/Uu82HyxK4MSZi2/e1oxp5BUq/oyePpTg/J6n14KrKzU+9TfnYfD51avDa3CLC1WcdwS2XLOS1HQf56K3/ZOf+9hHP29nTxx/+uY0zjiznmMpx8Qj9EEvgxpiI1TQ6FSjDbWQ8nJzsLE6YMT6pW6z96919vFy7n6vPnDPszjjLFk7lj1eeTFNrNx/59T9HXAbgkdd209TaFfNp86FYAjfGRKymaeSNjEdSPbOMtxta8HWEPzQRK/2978qSAi45eeQSv1NmT+Ch/1hEYV42l6x4kSc31Yc8zu9XfvdCDcdOGceiORNiHXYQS+DGmIjVNLZRMS6f4vzol1OqripF1akCSbR/bGlk3Y6DXHP2ERTkhjfB5ohJxTzyhdM5avI4rrprLStX1wYd87+bG6hpbGP5GbMjrs6JhiVwY0zEappaI55CP9jC6ePJzpKE79Cjqvz06S1MKy3kogh3hZ9YnM99nzuVD8yr4Ia/vsn3H9s0YPr9iudrmDq+kKXHVcY67JAsgRtjIlbb1BbRIlahFOXncOyUcQm/kfn0mw2s3+Xj2nPmhr0MbqDCvGx+fdmJXPneWdz+r21cdddaOrr7WLv9AGu2H+Df3zeLnOzEpFZbTtYYE5H9bd0cbO+JuoQw0IkzS7n3lR109/qjSqaR8vuVnzy9hVkTi/joCSG3Og1LdpbwnfPnMb20kBv++iaXrHiRsQW5lBTmcnF1ZL360bAEbqLyrW99K6jthhtuCHnskUceGdT27rvvxjwmkxiHKlBGcQOz30lVZfzhn9vYtMfHCTNKR32+kTy+sY636lv42ScWxqSX/JnTZzG1dAzX3vsaHT19XHPWEQPqyeNtVJ9ARLaJyAYReV1E1sQqKGNM6uqvQIm2hDBQtbuwVSLKCfv8ys/+dytzJxVzwfGxW9p18bwK7v/8qXzsPdPiOm0+lFh8ZzlLVReqanUMzmWMSXE1jW3kZgvTSkc/RXzSuAJmlI1hzfb4j4M/9sZu3tnbylcWH0l2jDdWWDBtPDdffDxlRXkxPe9I7CamMSYiNY2tzCgbE7Mbdc7CVgeiXjQqHL19fm75360cUzmO846dHLfrJNpo/wUUeEpE1orI8lAHiMhyEVljQyzGpIdYVKAEqp5Zxr62bmrdoZl4eHjdbrbta+eri4+M+bZmyTTaBH66qr4HWAJcLSJnDD5AVVeoarUNsRjjfX1+Zfu+9pjcwOx3UpU7Dh6nDR66e/3c8sxWjp9WwrnHxHZT4WQb1e1SVd3jPu4VkUeAk4HnYxFYOhgzZkzI9vPPPz+orbu7O6jt0UcfjXlMkVq6dGnI9lAVJy+88ELIY63iJPHaunrZ29I16sk2g+060E53nz8mJYT95pQXM35MLmu27Y9LCd4Da3ay+2AHN35kfkJmRyZS1D1wESkSkbH9vwMfADbGKjBjTPR+90INS255PubrjByqQInhEEpWlnDijNK4VKJ09vTxy7+/w4kzS3n/keUxP3+yjWYIpQJYLSJvAK8Aq1T1idiEZYwZjW1NbXT2+HlqiEWXonVoI+MY9+yrq8qoaWpjX2tXTM977ys7qG/u5LrFR6Zd7xtGkcBVtUZVj3d/jlXVG2MZmDEmenW+TgBWbaiL6XlrGlsZV5AT83K5eIyDd3T38atn3+XU2WUsOmJizM6bSqyM0GQsEfmSiGwUkU0i8mW3rUxEnhaRre5j/KcHxkF9s5PAV29t4mB78P2VaPVXoMS6Nzt/agl52Vkx3an+jy9to6m1i+s+cFTMzplqbCp9HF1wwQUh2++7776w2hJ9E3PevHlBbStXrgx57J49e4Lali1bFvOY4kVE5gOfw7nx3g08ISKr3LZnVPUmEbkeuB74P8mLNHKqSr2vk9NmT+DFmn08uameT5w08prX4ahpbGPREbFf57ogN5sF00pitrBVa1cvv3muhvfNnchJ7vZt6ch64CZTHQO8pKrtqtoLPAd8BFgG3OEecwdwYZLii9rB9h66ev2cO6+CGWVj+Ov62AyjtHX1Ut/cGfPx737VVWVs3O2jsyeyDYRDueNf29jf1p3WvW+wBG4y10bgDBGZICJjgA8B04EKVa0DcB+HLBwOnKTW2NiYkKDD0T/+PaWkgKULKvnXu/vY3zb6YZTaOFSgBKqeWUpPn/LGCFuWjcTX0cNvn3uXc4+ZxMLp42MUXWqyBG4ykqpuBv4beBp4AngD6I3wHIcmqZWXp06JWn1zBwCTSwpYelwlfX7liY2jr0Y5XEIYnx74iTNjcyPzttW1NHf28pXFwatgphtL4CZjqeptqvoeVT0D2A9sBRpEpBLAfdybzBij0d8Drywp5Ngp45g1sYhVG4LvWUSqprEVEaiaEJ8EXlqUxxGTikc1Dn6grZuVq2tZMn8yx04piWF0qcluYpqMJSKT3FnEM4CPAqcBs4ArgJvcxz8nMcSo1Ps6yc4SysfmIyIsPa6SX//jHZpau5hYnB/1eWub2phSUhj2HpLROKmqlL+8Uccv/76VvJws8nOyycvJIi87y33uPPa/duh5dhb5uVmsXF1LW3dm9L7BEnjMlJQE/7X/0Y9+FPLYxx9/PKjtk5/8ZMxjilSoTRr27g3dAV20aFFQW0tLS8xjirOHRGQC0ANcraoHROQm4AERuRLYAVyU1AijUO/rpLw4/9CSqUsXVPLLZ9/hbxvrufzUmVGft6axLW7DJ/0+MG8yD67dxY+f2hL1OT58/BSOrBgbw6hSlyVwk7FU9X0h2vYB5yQhnJipb+5kcknBoedHTx7LnPIiVq3fE3UCV1VqGlv5+InTYhVmSGcdPYkt/7WEXr/S1eun2/3p6u1zH/109/np6nEeA1/r7vXT61eWzE+f5WJHYgncmDRT5+tk7qTDlSIiwtIFU/jF37eyt6WTSWMLhnl3aI0tXbR198WtAiWQiJCbLeRmZ0H0Iz4ZwW5iGpNm6n0De+AA5y+oRBX+tiG6apR3G+NbgWKiYwncmDTS0tlDa1cvlYMS+JEVYzmyophVUU7qqWlyNjKO9fK0ZnRsCCVGQq39PX166LWNQ92wjOd2UoMNVbN8xhlB+3FQVBT6/7C5ubkxjcnERoO7BkrFuOBhkqXHTeFnz2wJ2UMfSW1jGwW5WUwpGf0+mCZ2rAduTBoJrAEfbKk7jPJ4FCsU1jS1UTWhKK22I0sHlsCNSSOHE3hwD/uIScUcPXlsVEvM1jS22vh3CrIEbkwaqXcT+KRxocs3zl9QydrtB9hzsCPsc3b3+tl5oIPZE+NfgWIiYwncmDRS5+tkYnEe+TmhZ0suXTAFiGwYZcf+dvr8aj3wFGQJ3Jg0Uu/rGPYG5ayJRRw7ZVxES8zWNFoFSqoasQpFRFYC5wN7VXW+21YG3A9UAduAi1U19juSesisWbPCPnbt2rVxjGRkFRUVIdunTQueZXfrrbeGPHb//tgsvG9iq765i6njh68wOX/BFP77ibfYub+d6WXB1VODxXsZWRO9cHrgtwPnDWq7HmfXkrnAM+5zY0ySjdQDB1h6XCUQ/jBKTWMbE4vzKCm00tFUM2ICV9XncZbaDOT5XUuMSTedPX0caO8JWUIYaMaEMSyYVhJ2NUpNU6sNn6SoaMfAo9q1JMprGWPC0F+BMjnEJJ7Bzl9QyfpdPnbsax/x2NqmNqtASVFxv4kZuGtJvK9lTCYbrgZ8sA+5wyh/HWGjB19HD02t3VaBkqKinUrfICKVqlrn1V1LojV2bOh1hu+6666gtqFuVvb09MQ0puGUlpYGta1YsSLksR0dwbXBX/ziF2Mek4mPQ9Pow0jg00rHsHD6eFatr+MLZx4x5HFWgZLaou2BP4azWwl4dNcSY9JNXQRDKOAMo2za03yoyiQUq0BJbSMmcBG5F3gROEpEdrk7ldwELBaRrcBi97kxJonqfR2MK8ihKD+8L9b9wyir1g89jFLT2EZ2ljAjjHJDk3gj/kur6qVDvOTpXUuMSTd1vs4RK1ACTRlfSPXMUv66vo5rzp4b8piaplamlxaSl2Nz/lKR/asYkyYGb6UWjqULKnmrvoV39raGfN3ZB9OGT1KVJXBj0oTTA48sgX/ouEpECLnRg9+vbNvXZjcwU5ht6BCh6urQ1ZChpqcPtdN8b29vTGMazrnnnhvUdtppp4U89uDBg0FtfX19MY/JxF5Pn5+m1q6QGzkMp2JcASdVlbFqwx6+dO7AYZQ9vg46e/xWQpjCrAduTBrY29KFang14IOdv6CSLQ2tbGloGdB+qALFJvGkLEvgxqSBep9Twx/pGDjAefMnkyUErVBYYxsZpzxL4MakgeG2UhvJpLEFnDJrAqvW7xmwN2tNYytFedlMGht6cwiTfJbATcYSka+IyCYR2Sgi94pIgYiUicjTIrLVfQyeypqCDq2DEkUPHJxqlHcb23ir/vAwSk2TU4EiYvtgpiq7iRmhb3zjGyHbm5ubg9peeumleIdzyFC7xF933XVBbS0tLSGOhI9+9KMxjSmVichU4Fpgnqp2iMgDwCXAPJylkm8Sketxlkr+P0kMNSz1vk4Kc7MZVxDd/6WXzJ/Md/+8kVXr6zimchzgDKGcONMTf78ylvXATSbLAQpFJAcYA+zBo0sl1zU7JYTR9pYnFOezaM5EVm2oQ1Xp7Oljj6/Dxr9TnCVwk5FUdTfwY2AHUAf4VPUpIlgqOZXU+yKfxDPY0gWV1Da1sWlPM9v2taFqa6CkOkvgJiO5Y9vLgFnAFKBIRD4V4TkOrXXf2NgYjzDDFosEft6xk8nOElZtqDtcgWKTeFKaJXCTqc4FalW1UVV7gIeBRbhLJQOMtFRy4Fr35eXlCQk6FL9faWiOfBbmYKVFeZx+xERWra+zZWQ9wm5iDuMTn/hEUNvixYtDHnvxxRfHO5xhnXXWWSHbTznllKC2F154IeSxzz77bExjSnE7gFNFZAzQgbM42xqgDWeJ5JvwyFLJTW1d9PqVyVGUEA52/nGVfP2h9Tz2xh4mjysIe2VDkxzWAzcZSVVfBh4E1gEbcP6/sAIPLpUcyVZqI/ngsZPJzRa2NNg+mF5gf15NxlLV7wHfG9TchceWSo5kK7WRlIzJ5b1HTOTZtxutAsUDrAdujMeNdhLPYOcvmAJYBYoXWA/cGI+r83WSl51F2Zi8mJzvvPmTeW5LI+ce44kKyoxmCdwYj6v3dVBRkk9WVmymvBfl5/DzS0+IyblMfI2YwEVkJXA+sFdV57tt3wc+B/QXv35TVR+PV5DJMtT09FDeeOONhF3/Yx/7WFDbzTffHPLY2traoLbLLrtsdIGZlFLf3EnluNFXoBjvCWcM/HbgvBDtP1XVhe5P2iVvY7yi3tdJRYzGv423jJjAVfV5YH8CYjHGREhVo9pKzaSH0VShXCMi60Vk5XBLbgZONx7FtYwxIRxs76Gr1x+TGnDjPdEm8FuBOcBCnIWAQg/AMnC6cZTXMsYMIZY14MZ7oqpCUdWG/t9F5HfAX2MWkUd9/vOfD2p77rnnQh67Y8eOoLa5c+cGtYVayxtCT48fyl/+8pegtp07d4b9fpPaGppjWwNuvCWqHnj/Yj+ujwAbYxOOMSYSdTGexGO8JZwywnuBM4GJIrILZ+rxmSKyEFBgGxDc/TTGxF29r4MsgfJi27cyE42YwFX10hDNt8UhFmNMhOp8nUwaW0BOtq2KkYnsX90YD6tvHv1GDsa7LIEb42FWA57ZbC2UYaxbty6obf/+0HOavvrVr4bVBtDd3R3UFmqn+Jdffjnk+1evXh3UtmjRopDH/va3vw3ZbtJDg6+T982dmOwwTJJYD9wYj2rp7KGlq9cm8WQwS+DGeJTVgBtL4MZ41OFZmLYSYaayBG6MR+PP+mwAABA6SURBVNk0emM3MYfx5ptvBrUtWbIk5LFXXHFF2Od96aWXgtr+8Y9/BLXV1dWFfP/jjwev3nvPPfeEPHbVqlVhx2W8pcFN4JPG2SSeTGU9cGM8qq65kwlFeeTnZCc7FJMklsCN8ah6n03iyXSWwI3xKJvEYyyBG+NR9b4O64FnOEvgxnhQZ08fB9p7rIQww1kVSoReeeWViNpH4xe/+EXI9sWLFwe1ff3rX4/59dOdiBwF3B/QNBv4LnCn216Fs1zyxap6INHxDefQJB6bhZnRrAduMpaqvq2qC1V1IXAi0A48AlwPPKOqc4Fn3OcpxTZyMGAJ3Jh+5wDvqup2YBlwh9t+B3Bh0qIaQr0lcIMlcGP6XQLc6/5eoap1AO7jpFBvEJHlIrJGRNY0NjYmKEzHoR64DaFkNEvgJuOJSB7wYeBPkbxPVVeoarWqVpeXl8cnuCHU+zoYV5BDUb7dxspk4eyJOR3nps5kwA+sUNVbRKSMFL/R4yWVlZVBbUNN2w81xf7OO++MeUwZZAmwTlUb3OcNIlKpqnXuBt57kxhbSPXNnVaBYsLqgfcC16nqMcCpwNUiMg8P3OgxJkyXcnj4BOAxoH9xmyuAPyc8ohHU+zqpsPHvjDdiAlfVOlVd5/7eAmwGpuKBGz3GjERExgCLgYcDmm8CFovIVve1m5IR23DqfJ1U2vh3xotoAE1EqoATgJcZdKNHRIa80QMsH12YxsSHqrYDEwa17cOpSklJPX1+Glu7rALFhJ/ARaQYeAj4sqo2i0hY71PVFcAK9xwaTZDGmMP2tnShauuAmzCrUEQkFyd5362q/V81G9wbPKTqjR5j0lG9rwOwGnATXhWKALcBm1X1JwEv9d/ouYkUvdGTqrKygv9u3nvvvUFtc+bMCfn+H/7wh0FtDQ0NIY406aje1wXYVmomvCGU04HLgQ0i8rrb9k2cxP2AiFwJ7AAuik+IxphAdf09cLuJmfFGTOCquhoYasA7ZW/0GJOu6n2dFOZmM67QJvFkOpuJaYzH1DU7GzmEW0hg0pclcGM8xrZSM/3sO1gSXHDBBUFt73//+4PaHnvssZDvD3UT02SOel8np8wuS3YYJgVYD9wYD/H7lYbmTruBaQBL4MZ4SlNbF71+tUk8BrAEboynHN7IwWrAjSVwYzylfyMH64EbsARujKfYVmomkFWhxFFxcXHI9pUrVwa17d0bvJTM1772tZDv7+npGV1gxrPqmzvJy86ibExeskMxKcB64MZ4SL2vk0nj8snKskk8xhK4MZ5S5+uw8W9ziCVwYzzEmYVpFSjGYQncGI9QVWcrNeuBG5fdxIyjpUuXhmz3+/1BbaF2oN+yZUvMYzLe5evooavXb7MwzSHWAzfGI+qshNAMYgncGI+wGnAzmCVwYzzCZmGawSyBG+MR9b4OsgTKi/OTHYpJESMmcBGZLiLPishmEdkkIl9y278vIrtF5HX350PxD9eYzFXn62TS2AJysq3fZRzhVKH0Atep6joRGQusFZGn3dd+qqo/jl943nb//fdH1G7McOqbbSceM9CIf8pVtU5V17m/twCbganxDsyYRBCR8SLyoIi85X7LPE1EykTkaRHZ6j6WJjtOcCfxWAmhCRDRdzERqQJOAF52m64RkfUisjJV/kduTIRuAZ5Q1aOB43E6KNcDz6jqXOAZ93nS2V6YZrCwE7iIFAMPAV9W1WbgVmAOsBCoA24e4n3LRWSNiKyJQbzGxIyIjAPOAG4DUNVuVT0ILAPucA+7A7gwOREe1tLZQ0tXr1WgmAHCSuAikouTvO9W1YcBVLVBVftU1Q/8Djg51HtVdYWqVqtqdayCNiZGZgONwB9E5DUR+b2IFAEVqloHzhAiMCnUmwM7J42NjXENtKHZasBNsHCqUASnh7JZVX8S0F4ZcNhHgI2xD8+YuMoB3gPcqqonAG1EMFwS2DkpLy+PV4wA1Pu6AKi0haxMgHCqUE4HLgc2iMjrbts3gUtFZCGgwDbg83GJ0Jj42QXsUtX+ezoP4iTwBhGpVNU6t6MSvNtGgtX5OgDsJqYZYMQErqqrgVCrxz8e+3CMSRxVrReRnSJylKq+DZwDvOn+XAHc5D7+OYlhAoen0U8aZ5N4zGG2GqHJdF8E7haRPKAG+CzO0OIDInIlsAO4KInxAVDX3MmEojwKcrOTHYpJIZbATUZT1deBUDfYz0l0LMOxEkITis3JNcYD6m0jBxOCJXBjPMCm0ZtQLIEbk+I6e/rY39ZtFSgmiCVwY1Lc4Uk8VgNuBrIEbkyKs40czFAsgRuT4mwrNTOURJcRNgHb3d8nus/TjX2u5JmZ7ADiob5/CMXGwM0gCU3gqnpowQgRWZOOC1zZ5zKxVu/rZGxBDkX5Nm3DDGRDKMakuDpfh41/m5AsgRuT4pxZmFaBYoIlM4GvSOK148k+l4mpOl8nlTb+bUJIWgJX1bRMCPa5TCz19PlpbO2yChQTkg2hGJPCGlu6ULUacBOaJXBjUlj/JJ4KS+AmhIQncBE5T0TeFpF3RCQldvuOloisFJG9IrIxoK1MRJ4Wka3uY2kyY4yGiEwXkWdFZLOIbBKRL7ntnv9sXlNvszDNMBKawEUkG/gVsASYh7Mt27xExhBjtwPnDWq7HnhGVecCzxDBHosppBe4TlWPAU4Frnb/ndLhs3lK/1ZqleOsCsUES3QP/GTgHVWtUdVu4D5gWYJjiBlVfR7YP6h5GXCH+/sdwIUJDSoGVLVOVde5v7cAm4GppMFn85p6XyeFudmMK7RJPCZYohP4VGBnwPNdbls6qVDVOnASITApyfGMiohUAScAL5Nmn80L6pudjRxEQm1LazJdohN4qP8VaoJjMGESkWLgIeDLqtqc7HgyUb2vkwqrATdDSHQC3wVMD3g+DdiT4BjirUFEKgHcx71JjicqIpKLk7zvVtWH3ea0+GxeUmdbqZlhJDqBvwrMFZFZ7i7glwCPJTiGeHsMuML9/Qrgz0mMJSrifF+/Ddisqj8JeMnzn81L/H6lwbZSM8NI9GqEvSJyDfAkkA2sVNVNiYwhlkTkXuBMYKKI7AK+B9wEPCAiVwI7gIuSF2HUTgcuBzaIyOtu2zdJj8/mGU1tXfT61XrgZkgJv7Wtqo8Djyf6uvGgqpcO8dI5CQ0kxlR1NaHvV4DHP5uXNPi6ANtKzQzNZmIak6IO1YBbD9wMwYpLTUYTkW1AC9AH9KpqtYiUAfcDVcA24GJVPZDo2Pp34rEqFDMU64EbA2ep6sKAHYdSYsZpna+T3GxhQlFeMi5vPMASuDHBUmLGaX8NeFaWTeIxodkQisl0CjwlIgr81l33fMCMUxGJasZpc2cPtY1t5GZnkZcj7mMWudnOT152FrnZQnaWhJxpaVupmZFYAjeZ7nRV3eMm6adF5K1w3ygiy4HlADNmzAh6ff1OH5+67eUwzgN5/Qk9x0nqudlZ7G3u4gPHVoT/SUzGsQRuMpqq7nEf94rIIzgLrjWISKXb+x5yxqnbW18BUF1dHbQkxLwp41j5mWq6e/109yk9vX56+pyfrl4/PX166Hl3n5/u/td79VDbJ08J/sNgTD9L4CZjiUgRkKWqLe7vHwBu4PCM05sYxYzTsqI8zj7aetAmfiyBm0xWATzijj/nAPeo6hMi8io249R4gCVwk7FUtQY4PkT7PmzGqfEAKyM0xhiPsgRujDEeZQncGGM8yhK4McZ4lCVwY4zxKEvgxhjjUaJqewobM1oi0ghsD/HSRKApweHYtZNz7Xhed6aqlg9utARuTByJyJqAZWrt2ml87WRc14ZQjDHGoyyBG2OMR1kCNya+Vti1M+baCb+ujYEbY4xHWQ/cGGM8yhK4McZ4lCVwY+JERM4TkbdF5B0RSdjO9iIyXUSeFZHNIrJJRL6UqGu7188WkddE5K8Jvu54EXlQRN5yP/tpCbz2V9z/1htF5F4RSchmppbAjYkDEckGfgUsAeYBl4rIvARdvhe4TlWPAU4Frk7gtQG+BGxO4PX63QI8oapH46zznpAYRGQqcC1QrarzgWzgkkRc2xK4MfFxMvCOqtaoajdwH7AsERdW1TpVXef+3oKTyKYm4toiMg1YCvw+EdcLuO444AzgNgBV7VbVgwkMIQcoFJEcYAywJxEXtQRuTHxMBXYGPN9FgpJoIBGpAk4AXk7QJX8GfB3wJ+h6/WYDjcAf3OGb37v7nMadqu4Gfoyz/V4d4FPVpxJxbUvgxsSHhGhLaM2uiBQDDwFfVtXmBFzvfGCvqq6N97VCyAHeA9yqqicAbUBC7juISCnOt6tZwBSgSEQ+lYhrWwI3Jj52AdMDnk8jQV+rAUQkFyd5362qDyfosqcDHxaRbThDRmeLyF0JuvYuYJeq9n/TeBAnoSfCuUCtqjaqag/wMLAoERe2BG5MfLwKzBWRWSKSh3NT67FEXFhEBGcseLOq/iQR1wRQ1W+o6jRVrcL5vH9X1YT0RFW1HtgpIke5TecAbybi2jhDJ6eKyBj3v/05JOgGqu1Kb0wcqGqviFwDPIlTlbBSVTcl6PKnA5cDG0Tkdbftm6r6eIKunyxfBO52/2DWAJ9NxEVV9WUReRBYh1MB9BoJmlZvU+mNMcajbAjFGGM8yhK4McZ4lCVwY4zxKEvgxhjjUZbAjTHGoyyBG2NCEpHvi8h/xunc20Rk4gjHtEZ4zrjFm6osgRtjjEdZAjfGICKfFpH1IvKGiPwxxOufE5FX3dcfEpExbvtF7hrYb4jI827bsSLyioi87p5z7gjXflRE1rrraS8f9NrNIrJORJ4RkXK3bY6IPOG+5wUROTrEOa8VkTfd6983mv82qcwSuDEZTkSOBb4FnK2qx+Os5z3Yw6p6kvv6ZuBKt/27wAfd9g+7bVcBt6jqQqAaZ52S4fybqp7oHnutiExw24uAdar6HuA54Htu+wrgi+57/hP4dYhzXg+coKoL3HjSkk2lN8acDTyoqk0Aqro/xDHzReS/gPFAMc4SAQD/BG4XkQdwFnECeBH4lrs2+MOqunWE618rIh9xf58OzAX24SxJe7/bfhfwsLvC4iLgT86yIwDkhzjnepxp9Y8Cj45wfc+yHrgxRhh5qdvbgWtU9TjgB0ABgKpeBXwbJ/G+LiITVPUenN54B/CkiJw95IVFzsRZze80txf/Wv+5Q1CcnHVQVRcG/BwT4tilODsinQisdTdaSDuWwI0xzwAX9w9diEhZiGPGAnXuMrWX9TeKyBxVfVlVvws0AdNFZDZQo6o/x1mBccEw1y4BDqhquzuWfWrAa1nAx93fPwmsdtc1rxWRi9zri4gcH3hCEckCpqvqszibS/R/a0g7aflXyRgTPlXdJCI3As+JSB9OL/gzgw77Ds6uPtuBDTgJHeBH7k1KwflD8AbO+POnRKQHqAduGObyTwBXich64G3gpYDX2oBjRWQt4AM+4bZfBtwqIt8GcnHWHn8j4H3ZwF0iUuLG9dMEb6+WMLYaoTHGeJQNoRhjjEdZAjfGGI+yBG6MMR5lCdwYYzzKErgxxniUJXBjjPEoS+DGGONR/x9z9sywepkmHwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "index = 555\n",
    "\n",
    "print(\"Input sample: \",lbls[index])\n",
    "results = []\n",
    "for fern in ferns:\n",
    "    x, _ = fern.forward(imgs[index])\n",
    "    results.append(fern.pdf(x))\n",
    "\n",
    "results = np.asarray(results)\n",
    "log_results = np.log(results+1e-6)\n",
    "p = log_results.sum(axis=0)\n",
    "a = np.argmax(p)\n",
    "print(\"Prediction: \" , a)\n",
    "\n",
    "plt.subplot(1,2,1)\n",
    "plt.title(\"Input Image\")\n",
    "plt.imshow(np.asarray(imgs[index,:,:]), cmap='gray')\n",
    "plt.subplot(1,2,2)\n",
    "plt.title(\"Votes\")\n",
    "plt.plot(p)\n",
    "plt.plot(a,p[a],'ro')\n",
    "plt.xlabel(\"class labels\")\n",
    "plt.show()\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}