Part 4 - Model Training & Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "67511341",
   "metadata": {},
   "source": [
    "# Python for Data Science\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AI-Core/Python-for-Data-Science/blob/main/Part%204%20-%20Model%20Training%20%26%20Evaluation.ipynb)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f77141fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv(\n",
    "    \"https://raw.githubusercontent.com/AI-Core/Python-for-Data-Science/main/part-3-output.csv\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1368617",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dropping non significant columns from X and the target column\n",
    "X = df\n",
    "y = df.drop(labels=['m13'], axis= 1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efa6658b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# apply one-hot encoding to nominal variables\n",
    "X = pd.get_dummies(data=X, columns=['source', 'loan_purpose'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6f7d1f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# apply normalisation using sklearn\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "normalisation = MinMaxScaler()\n",
    "X = normalisation.fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3f961b1",
   "metadata": {},
   "source": [
    "## The Problem\n",
    "\n",
    "__Loan default prediction__ is one of the most critical and crucial problems faced by financial institutions and organizations as it has a noteworthy effect on the profitability of these institutions. In recent years, there is a tremendous increase in the volume of _non–performing_ loans which results in a jeopardizing effect on the growth of these institutions. \n",
    "\n",
    "Therefore, to maintain a healthy portfolio, banks put stringent monitoring and evaluation measures in place to ensure the timely repayment of loans by borrowers. Despite these measures, a major proportion of loans become delinquent. _Delinquency_ occurs when a borrower misses a payment against his/her loan.\n",
    "\n",
    "Given the information like mortgage details, borrowers-related details, and payment details, your objective is to build a system that can predict the defaulter status of loans for the next month given the defaulter status for the previous 12 months (in the number of months).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f73320e4",
   "metadata": {},
   "source": [
    "# Part 4 - Model Training and Evaluation\n",
    "\n",
    "Now, it's time to do some machine learning.\n",
    "You have your final training and testing data, and you're ready to start training different some machine learning models to model the relationships between the features and labels."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94ce3a3d",
   "metadata": {},
   "source": [
    "## Train Test Split\n",
    "\n",
    "The final thing required before training a machine learning model on our data is to split the data into different sets.\n",
    "training and testing for model training and evaluation. \n",
    "\n",
    "You can learn more about why we perform train and test splitting [here](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "c0269597",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y) # stratify to have same ratio in train and test split for target column"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1dadc4d",
   "metadata": {},
   "source": [
    "Alright, so we are getting closer to training our mdodel. This notebook was just focussed around preparing the data before we actually train the model. We did Feature Selection, Feature Encoding and then Feature Scaling. It is important to notice the flow of the preprocessing steps. It should most likely happen in this manner if not always."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3f66a7f",
   "metadata": {},
   "source": [
    "### Model Training \n",
    "\n",
    "We will be training different models and see which works best. \n",
    "1. Logistic Regression : For this model, how to prepare the data w.r.t outliers, feature scaling and feature encoding ? \n",
    "2. Decision Tree : How to prepare the data for this model ? Is it robust to outliers ? \n",
    "3. Random Forest : [Read More Here](https://towardsdatascience.com/understanding-random-forest-58381e0602d2)\n",
    "4. SVM : [Read More Here](https://towardsdatascience.com/https-medium-com-pupalerushikesh-svm-f4b42800e989#:~:text=SVM%20or%20Support%20Vector%20Machine,separates%20the%20data%20into%20classes.)\n",
    "5. Boosting Models: GradientBoostingClassifier(), ExtraTreesClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "76cab4ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier, ExtraTreesClassifier\n",
    "from sklearn.metrics import classification_report, accuracy_score, precision_score, recall_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "8ba64e99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train Logistic Model\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8dcf348",
   "metadata": {},
   "source": [
    "### Model Evaluation\n",
    "\n",
    "These metrics help us understand which model works better over others. You can learn more about the metrics here: [Link](https://neptune.ai/blog/performance-metrics-in-machine-learning-complete-guide#:~:text=They're%20used%20to%20train,t%20need%20to%20be%20differentiable.)\n",
    "1. Accuracy : Is Accuracy as a metric relavant here ?\n",
    "2. Precision : [Read More](https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall)\n",
    "3. Recall : [Read More](https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall)\n",
    "4. Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "75464824",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy is 0.9952897926359929\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score, confusion_matrix, classification_report, plot_confusion_matrix, precision_score, recall_score\n",
    "pred = model.predict(X_test)\n",
    "print (f\"The accuracy is {accuracy_score(y_test, pred)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "96c42733",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x29b37c310>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgwAAAGwCAYAAADFZj2cAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA7OUlEQVR4nO3dfXxT9f3//2d6kfQyaQu0paMgjgl0XGnB0k0RRqVcTGXgTVGmBQG/uNYJnVw4FUE28YNzCANljml1P5iiG2yCgrUIiFSUahUQuoE4QGhBS5u20KRN8vujazTj4jTQkgKP++2W242c88r7vE6tybPnvM+JyePxeAQAAHAWQYFuAAAAtH4EBgAAYIjAAAAADBEYAACAIQIDAAAwRGAAAACGCAwAAMBQSKAbOB9ut1uHDx9WdHS0TCZToNsBAPjJ4/GoqqpKSUlJCgpqub9ha2tr5XQ6z3scs9mssLCwZujo4nNRB4bDhw8rOTk50G0AAM7TwYMH1aFDhxYZu7a2VnFxcTp58uR5j5WYmKj9+/dflqHhog4M0dHRkqQ777xTZrM5wN0ALSPxrsRAtwC0GEeNQwtuXuB9P28JTqdTJ0+ePO/PCqfTqRUrVsjpdBIYLjaNpyHMZjOBAZcsS5Ql0C0ALe5CnFbms+L8MOkRAAAYIjAAAABDBAYAAGCIwAAAAAwRGAAAgCECAwAAMERgAAAAhggMAADAEIEBAAAYIjAAAABDBAYAAGCIwAAAAAwRGAAAgCECAwAAMERgAAAAhggMAADAEIEBAAAYIjAAAABDBAYAAGCIwAAAAAwRGAAAgCECAwAAMERgAAAAhggMAADAEIEBAAAYIjAAAABDBAYAAGCIwAAAAAwRGAAAgCECAwAAMERgAAAAhggMAADAEIEBAAAYIjAAAABDBAYAAGCIwAAAAAwRGAAAgCECAwAAMERgAAAAhggMAADAEIEBAAAYIjAAAABDBAYAAGCIwAAAuCzcemKt7qj5xzk/bj2x1q/tPffcc+rVq5esVqusVqvS09P11ltvedfX1tYqOztbbdq0UVRUlEaPHq2ysjKfMQ4cOKARI0YoIiJC8fHxmjZtmurr631qNm7cqGuuuUYWi0VdunRRXl7eKb0sWbJEV1xxhcLCwpSWlqYPP/zQr32RCAwAALSIDh066Mknn1RRUZG2b9+un/zkJ7rlllu0a9cuSdLUqVP1xhtv6LXXXtOmTZt0+PBhjRo1yvt6l8ulESNGyOl0auvWrXrppZeUl5enWbNmeWv279+vESNGaNCgQSouLtaUKVM0ceJErV+/3lvz6quvKjc3V4899pg+/vhj9e7dW5mZmTp69Khf+2PyeDye8/yZBIzdbpfNZtO4ceNkNpsD3Q7QItpPaB/oFoAW46h26MnBT6qyslJWq7VFttH4WbHmtnaKDD33v5Nr6tz66cpjOnjwoE+vFotFFoulSWPExcXpqaee0q233qp27dppxYoVuvXWWyVJe/bsUffu3VVYWKj+/fvrrbfe0k9/+lMdPnxYCQkJkqSlS5dqxowZOnbsmMxms2bMmKG1a9dq586d3m2MGTNGFRUVWrdunSQpLS1N/fr10+LFiyVJbrdbycnJuv/++zVz5swm7z9HGAAA8ENycrJsNpv3MW/ePMPXuFwuvfLKK6qpqVF6erqKiopUV1enjIwMb023bt3UsWNHFRYWSpIKCwvVs2dPb1iQpMzMTNntdu9RisLCQp8xGmsax3A6nSoqKvKpCQoKUkZGhremqUL8qgYA4DJ3uiMMZ7Jjxw6lp6ertrZWUVFRWrVqlVJSUlRcXCyz2ayYmBif+oSEBJWWlkqSSktLfcJC4/rGdWersdvtOnnypI4fPy6Xy3Xamj179vi13wQGAAD80DiJsSm6du2q4uJiVVZW6vXXX1dWVpY2bdrUwh22DAIDAAAtxGw2q0uXLpKk1NRUffTRR1q4cKFuv/12OZ1OVVRU+BxlKCsrU2JioiQpMTHxlKsZGq+i+G7N/15ZUVZWJqvVqvDwcAUHBys4OPi0NY1jNBVzGAAAuEDcbrccDodSU1MVGhqqgoIC77qSkhIdOHBA6enpkqT09HTt2LHD52qG/Px8Wa1WpaSkeGu+O0ZjTeMYZrNZqampPjVut1sFBQXemqbiCAMAAC3goYce0rBhw9SxY0dVVVVpxYoV2rhxo9avXy+bzaYJEyYoNzdXcXFxslqtuv/++5Wenq7+/ftLkoYMGaKUlBTdddddmj9/vkpLS/XII48oOzvbO29i8uTJWrx4saZPn6577rlHGzZs0MqVK7V27bf3jMjNzVVWVpb69u2ra6+9Vs8884xqamo0fvx4v/aHwAAAQAs4evSo7r77bh05ckQ2m029evXS+vXrdeONN0qSFixYoKCgII0ePVoOh0OZmZl69tlnva8PDg7WmjVrdN999yk9PV2RkZHKysrS448/7q3p3Lmz1q5dq6lTp2rhwoXq0KGDli1bpszMTG/N7bffrmPHjmnWrFkqLS1Vnz59tG7dulMmQhrhPgxAK8d9GHApuxjvw9CSvbZmzGEAAACGCAwAAMAQgQEAABgiMAAAAEMEBgAAYIjAAAAADBEYAACAIQIDAAAwRGAAAACGCAwAAMAQgQEAABgiMAAAAEMEBgAAYIjAAAAADBEYAACAIQIDAAAwRGAAAACGCAwAAMAQgQEAABgiMAAAAEMEBgAAYIjAAAAADBEYAACAIQIDAAAwRGAAAACGCAwAAMAQgQEAABgiMAAAAEMEBgAAYIjAAAAADIUEugG0rIhuEYroHqHgqGBJUn1Fvao/qZbjkOOU2tghsQpLDlP5O+Vy/Md3ffgPwhXZI1Ih1hC569yq3V8re6FdkmRONCuyR6RC24XKFGqSy+5S9Y5q1e6r9Xl9zIAYnzE99R6VvlTazHsM+K/602rVflmr+sp6mYJNCo0PlbWfVSExvEUCjfi/4RLnqnGp6qMq1dvrJUkRP4hQbEasvl79teor6r11kT+MPOMYkT0iFdkjUvYP7ao7VidTiEnB0cHe9aEJoaorr1P1Z9Vyn3TL0tGimAExOu48LsfBb4OH2+nWsdePfTuwpxl3FDgPziNORXSPUGi7UMktVW2vUvm6crUd3VZBoRyIBaRWckpiyZIluuKKKxQWFqa0tDR9+OGHgW7pkuE46JDjkEMuu0suu0tVRVXy1HsUGh/qrQmJC1Fkz0hVvld5yutNZpOiU6NVsalCtV/UylXlUv3xejkOfBsEaj6tUfXH1ao7WidXlUsndp2Q4yuHwq4I8x3MI7lPur991LpbbL8Bf8QNjVPEVREKjQ1VaJtQ2QbY5Kpxqe7rukC3BrQaAQ8Mr776qnJzc/XYY4/p448/Vu/evZWZmamjR48GurVLj0kKuzJMphCT6o7+940wWIoZGKPKrZVynzz1A9zyPUtDWWSw2o1up/gx8YoZFKOgyLP/6gSFBsnt8B3PFGpSu9vbKf72eMVmxHK4F62Wp67h8FeQJeBvkUCrEfD/G37/+99r0qRJGj9+vFJSUrR06VJFRETohRdeCHRrl4yQ2BAl3J2gxHGJsv3IpuPvHPeejrD2t6ruaJ3PEYPvCo4OlkxSVO8oVX5QqeMFxxVkCVKboW3O+NsT1jlMoe1CdfJfJ73L6ivrVflepY7nH1fFpgrJJLW5qY2CIgL+Kwj48Hg8sn9gV2hCqELjQo1fAFwmAvpu7XQ6VVRUpIyMDO+yoKAgZWRkqLCw8JR6h8Mhu93u84Cx+sp6fb3qa339z691Ys8J2QbYFBITIktHiyztLbJ/cJafo0kyBZtU+UGlnF85VXesThUbKxRsDZa5vfmUcnN7s2zX21S5pdJnjkTd0Tqd3HtS9eX1cpY6dfyd43LXuhXRLaIldhk4Z/atdtUfr1fsoNhAtwK0KgE9Jvz111/L5XIpISHBZ3lCQoL27NlzSv28efM0Z86cC9XepcMtuapckqSqb6oU2jZUET+MkOqlYGuwEu7y/fnH/iRWzjKnyt8sl/tEw2mF+uPffvi7a91yO9zeKy8amRPNir0xVvZtdp3ce1Jn5ZHqvqlTiJXTEmg9KrdWqvZgrdqMaKPgyGDjF+Ci0uu1CbKaLOf8ervHIenJ5mvoInNRvVs/9NBDys3N9T632+1KTk4OYEcXKZNkCjKp6rMqnfjXCZ9V7Ua1k32b3XuKwlnmlCSF2ELkPNHwb5PZpCBLkFzVLu/rzIlmxQ6JVdVHVTpZYhAW/ttDaGyoag/VGtcCLczj8cheaFftf2rVZngbhURfVG+NwAUR0P8r2rZtq+DgYJWVlfksLysrU2Ji4in1FotFFsu5p8PLUXTf6IarJKpdMoWaFP79cJnbm1W+rtx7tcL/ctW4vGHAZXep9j+1sqZbVbmlUp46j6L7Rqu+sl7Oww0Bwty+4cjCiV0nVPtlrYLCG850eVweeZwNk8ei+kTJecwpl90lk9mkqJ5RCo4Kblq4AFqYfatdJ784qdiM2IZ7iZxo+P0PMgfJFGIKcHdA6xDQwGA2m5WamqqCggKNHDlSkuR2u1VQUKCcnJxAtnbJCAoLkm2ATcERwXI73aovr1f5unLvh31TVGyqkDXNqrghcZJHcpY6Vb6+3HsfhfAu4QoKDVJUnyhF9Ynyvs5xxKHyN8slSSaLSbbrbAoOD5bb4VbdN3X6eo3vvSCAQDmxp+FIW+PvayPb9TZFXMU8G0BqBackcnNzlZWVpb59++raa6/VM888o5qaGo0fPz7QrV0SKrecem+Fszny5yOnLPPUeVS5pfKMY1W+V3naezh8V9W2KlVtq/KrF+BCaT+hfaBbAFq9gAeG22+/XceOHdOsWbNUWlqqPn36aN26dadMhAQAAIHTKi6Cz8nJ0X/+8x85HA5t27ZNaWlpgW4JAIDzMm/ePPXr10/R0dGKj4/XyJEjVVJS4lMzcOBAmUwmn8fkyZN9ag4cOKARI0YoIiJC8fHxmjZtmurrfU/nbty4Uddcc40sFou6dOmivLy8U/o537sqt4rAAADApWbTpk3Kzs7WBx98oPz8fNXV1WnIkCGqqanxqZs0aZKOHDnifcyfP9+7zuVyacSIEXI6ndq6dateeukl5eXladasWd6a/fv3a8SIERo0aJCKi4s1ZcoUTZw4UevXr/fWNMddlQN+SgIAgEvRunXrfJ7n5eUpPj5eRUVFGjBggHd5RETEaa8MlKS3335bn3/+ud555x0lJCSoT58+mjt3rmbMmKHZs2fLbDZr6dKl6ty5s55++mlJUvfu3bVlyxYtWLBAmZmZknzvqixJS5cu1dq1a/XCCy9o5syZTdofjjAAAOCH/73jsMNx+lvr/6/KyobJ4XFxcT7Lly9frrZt26pHjx566KGHdOLEt/fHKSwsVM+ePX3m9WVmZsput2vXrl3emu/eMbmxpvGOyf7eVflMOMIAAIAf/veGgY899phmz5591te43W5NmTJFP/7xj9WjRw/v8jvvvFOdOnVSUlKSPvvsM82YMUMlJSX6+9//LkkqLS097d2QG9edrcZut+vkyZM6fvy4X3dVPhMCAwAAfjh48KCsVqv3eVNuKJidna2dO3dqy5YtPsvvvfde77979uyp9u3ba/Dgwdq3b5++//3vN1/TzYBTEgAA+MFqtfo8jAJDTk6O1qxZo3fffVcdOnQ4a23jVYJ79+6VJCUmJp72bsiN685WY7VaFR4e7vddlc+EwAAAQAvweDzKycnRqlWrtGHDBnXu3NnwNcXFxZKk9u0bbiaWnp6uHTt2+FzNkJ+fL6vVqpSUFG9NQUGBzzj5+flKT0+X5HtX5UaNd1VurGkKTkkAANACsrOztWLFCv3jH/9QdHS0d86BzWZTeHi49u3bpxUrVmj48OFq06aNPvvsM02dOlUDBgxQr169JElDhgxRSkqK7rrrLs2fP1+lpaV65JFHlJ2d7T2yMXnyZC1evFjTp0/XPffcow0bNmjlypVau3att5fmuKsygQEAgBbw3HPPSWq4OdN3vfjiixo3bpzMZrPeeecd74d3cnKyRo8erUceecRbGxwcrDVr1ui+++5Tenq6IiMjlZWVpccff9xb07lzZ61du1ZTp07VwoUL1aFDBy1btsx7SaXUPHdVNnk8Hs85/iwCzm63y2azeX/wwKWI7znApcxR7dCTg59UZWWlz0TC5tT4WXHANFNW07l/47Hd41BHT8v22poxhwEAABgiMAAAAEMEBgAAYIjAAAAADBEYAACAIQIDAAAwRGAAAACGCAwAAMAQgQEAABgiMAAAAEMEBgAAYIjAAAAADBEYAACAIQIDAAAwRGAAAACGCAwAAMAQgQEAABgiMAAAAEMEBgAAYIjAAAAADBEYAACAIQIDAAAwRGAAAACGCAwAAMAQgQEAABgiMAAAAEMEBgAAYIjAAAAADBEYAACAoZBANwAAwIUwalIvhZgjzvn19c4T0vPN2NBFhiMMAADAEIEBAAAYIjAAAABDBAYAAGCIwAAAAAwRGAAAgCECAwAAMERgAAAAhggMAADAEIEBAAAYIjAAAABDBAYAAGCIwAAAAAwRGAAAaAHz5s1Tv379FB0drfj4eI0cOVIlJSU+NbW1tcrOzlabNm0UFRWl0aNHq6yszKfmwIEDGjFihCIiIhQfH69p06apvr7ep2bjxo265pprZLFY1KVLF+Xl5Z3Sz5IlS3TFFVcoLCxMaWlp+vDDD/3anyZ9vfU///nPJg948803+9UAAACXok2bNik7O1v9+vVTfX29fv3rX2vIkCH6/PPPFRkZKUmaOnWq1q5dq9dee002m005OTkaNWqU3n//fUmSy+XSiBEjlJiYqK1bt+rIkSO6++67FRoaqieeeEKStH//fo0YMUKTJ0/W8uXLVVBQoIkTJ6p9+/bKzMyUJL366qvKzc3V0qVLlZaWpmeeeUaZmZkqKSlRfHx8k/bH5PF4PEZFQUFNOxBhMpnkcrmaVNsc7Ha7bDabxo0bJ7PZfMG2C1xI7Se0D3QLQItxVDv05OAnVVlZKavV2iLbaPys6HvvCoWYI855nHrnCW1//k4dPHjQp1eLxSKLxWL4+mPHjik+Pl6bNm3SgAEDVFlZqXbt2mnFihW69dZbJUl79uxR9+7dVVhYqP79++utt97ST3/6Ux0+fFgJCQmSpKVLl2rGjBk6duyYzGazZsyYobVr12rnzp3ebY0ZM0YVFRVat26dJCktLU39+vXT4sWLJUlut1vJycm6//77NXPmzCbtf5OSgNvtbtLjQoYFAAACITk5WTabzfuYN29ek15XWVkpSYqLi5MkFRUVqa6uThkZGd6abt26qWPHjiosLJQkFRYWqmfPnt6wIEmZmZmy2+3atWuXt+a7YzTWNI7hdDpVVFTkUxMUFKSMjAxvTVM06ZTEmdTW1iosLOx8hgAA4KJyuiMMRtxut6ZMmaIf//jH6tGjhySptLRUZrNZMTExPrUJCQkqLS311nw3LDSub1x3thq73a6TJ0/q+PHjcrlcp63Zs2dPE/a4gd+THl0ul+bOnavvfe97ioqK0hdffCFJevTRR/XnP//Z3+EAALioWK1Wn0dTAkN2drZ27typV1555QJ02DL8Dgy//e1vlZeXp/nz5/vMG+jRo4eWLVvWrM0BAHCxy8nJ0Zo1a/Tuu++qQ4cO3uWJiYlyOp2qqKjwqS8rK1NiYqK35n+vmmh8blRjtVoVHh6utm3bKjg4+LQ1jWM0hd+B4eWXX9bzzz+vsWPHKjg42Lu8d+/efh3aAADgUubxeJSTk6NVq1Zpw4YN6ty5s8/61NRUhYaGqqCgwLuspKREBw4cUHp6uiQpPT1dO3bs0NGjR701+fn5slqtSklJ8dZ8d4zGmsYxzGazUlNTfWrcbrcKCgq8NU3h9xyGr776Sl26dDlludvtVl1dnb/DAQBwScrOztaKFSv0j3/8Q9HR0d45BzabTeHh4bLZbJowYYJyc3MVFxcnq9Wq+++/X+np6erfv78kaciQIUpJSdFdd92l+fPnq7S0VI888oiys7O9p0ImT56sxYsXa/r06brnnnu0YcMGrVy5UmvXrvX2kpubq6ysLPXt21fXXnutnnnmGdXU1Gj8+PFN3h+/A0NKSoree+89derUyWf566+/rquvvtrf4QAAuCQ999xzkqSBAwf6LH/xxRc1btw4SdKCBQsUFBSk0aNHy+FwKDMzU88++6y3Njg4WGvWrNF9992n9PR0RUZGKisrS48//ri3pnPnzlq7dq2mTp2qhQsXqkOHDlq2bJn3HgySdPvtt+vYsWOaNWuWSktL1adPH61bt+6UiZBn43dgmDVrlrKysvTVV1/J7Xbr73//u0pKSvTyyy9rzZo1/g4HAMAlqQm3OVJYWJiWLFmiJUuWnLGmU6dOevPNN886zsCBA/XJJ5+ctSYnJ0c5OTmGPZ2J33MYbrnlFr3xxht65513FBkZqVmzZmn37t164403dOONN55zIwAAoPU6p/swXH/99crPz2/uXgAAQCt1zjdu2r59u3bv3i2pYV5DampqszUFAABaF78Dw6FDh3THHXfo/fff996dqqKiQj/60Y/0yiuv+FxjCgAALg1+z2GYOHGi6urqtHv3bpWXl6u8vFy7d++W2+3WxIkTW6JHAAAQYH4fYdi0aZO2bt2qrl27epd17dpVf/jDH3T99dc3a3MAAKB18PsIQ3Jy8mlv0ORyuZSUlNQsTQEAgNbF78Dw1FNP6f7779f27du9y7Zv364HHnhAv/vd75q1OQAA0Do06ZREbGysTCaT93lNTY3S0tIUEtLw8vr6eoWEhOiee+7RyJEjW6RRAAAQOE0KDM8880wLtwEAAFqzJgWGrKyslu4DAAC0Yud84yZJqq2tldPp9FlmtVrPqyEAAND6+D3psaamRjk5OYqPj1dkZKRiY2N9HgAA4NLjd2CYPn26NmzYoOeee04Wi0XLli3TnDlzlJSUpJdffrklegQAAAHm9ymJN954Qy+//LIGDhyo8ePH6/rrr1eXLl3UqVMnLV++XGPHjm2JPgEAQAD5fYShvLxcV155paSG+Qrl5eWSpOuuu06bN29u3u4AAECr4HdguPLKK7V//35JUrdu3bRy5UpJDUceGr+MCgAAXFr8Dgzjx4/Xp59+KkmaOXOmlixZorCwME2dOlXTpk1r9gYBAEDg+T2HYerUqd5/Z2RkaM+ePSoqKlKXLl3Uq1evZm0OAAC0Dud1HwZJ6tSpkzp16tQcvQAAgFaqSYFh0aJFTR7wl7/85Tk3AwAAWqcmBYYFCxY0aTCTyRSQwJB4V6IsUZYLvl0AwMWj4KmjslrDz/n1dvtJ2Z5vxoYuMk0KDI1XRQAAgMuT31dJAACAyw+BAQAAGCIwAAAAQwQGAABgiMAAAAAMnVNgeO+99/Tzn/9c6enp+uqrryRJf/nLX7Rly5ZmbQ4AALQOfgeGv/3tb8rMzFR4eLg++eQTORwOSVJlZaWeeOKJZm8QAAAEnt+B4Te/+Y2WLl2qP/3pTwoNDfUu//GPf6yPP/64WZsDAACtg9+BoaSkRAMGDDhluc1mU0VFRXP0BAAAWhm/A0NiYqL27t17yvItW7boyiuvbJamAABA6+J3YJg0aZIeeOABbdu2TSaTSYcPH9by5cv14IMP6r777muJHgEAQID5/fXWM2fOlNvt1uDBg3XixAkNGDBAFotFDz74oO6///6W6BEAAASY34HBZDLp4Ycf1rRp07R3715VV1crJSVFUVFRLdEfAABoBfwODI3MZrNSUlKasxcAANBK+R0YBg0aJJPJdMb1GzZsOK+GAABA6+N3YOjTp4/P87q6OhUXF2vnzp3Kyspqrr4AAEAr4ndgWLBgwWmXz549W9XV1efdEAAAaH2a7cunfv7zn+uFF15oruEAAEAr0myBobCwUGFhYc01HAAAaEX8DgyjRo3yefzsZz9T//79NX78eP2///f/WqJHAAAuOps3b9ZNN92kpKQkmUwmrV692mf9uHHjZDKZfB5Dhw71qSkvL9fYsWNltVoVExOjCRMmnHL6/7PPPtP111+vsLAwJScna/78+af08tprr6lbt24KCwtTz5499eabb/q9P34HBpvN5vOIi4vTwIED9eabb+qxxx7zuwEAAC5FNTU16t27t5YsWXLGmqFDh+rIkSPex1//+lef9WPHjtWuXbuUn5+vNWvWaPPmzbr33nu96+12u4YMGaJOnTqpqKhITz31lGbPnq3nn3/eW7N161bdcccdmjBhgj755BONHDlSI0eO1M6dO/3aH78mPbpcLo0fP149e/ZUbGysXxsCAOByMmzYMA0bNuysNRaLRYmJiaddt3v3bq1bt04fffSR+vbtK0n6wx/+oOHDh+t3v/udkpKStHz5cjmdTr3wwgsym8364Q9/qOLiYv3+97/3BouFCxdq6NChmjZtmiRp7ty5ys/P1+LFi7V06dIm749fRxiCg4M1ZMgQvpUSAHDZstvtPg+Hw3HOY23cuFHx8fHq2rWr7rvvPn3zzTfedYWFhYqJifGGBUnKyMhQUFCQtm3b5q0ZMGCAzGaztyYzM1MlJSU6fvy4tyYjI8Nnu5mZmSosLPSrV79PSfTo0UNffPGFvy8DAOCSkJyc7HNqft68eec0ztChQ/Xyyy+roKBA//d//6dNmzZp2LBhcrlckqTS0lLFx8f7vCYkJERxcXEqLS311iQkJPjUND43qmlc31R+34fhN7/5jR588EHNnTtXqampioyM9FlvtVr9HRIAgIvGwYMHfT7rLBbLOY0zZswY77979uypXr166fvf/742btyowYMHn3efza3JRxgef/xx1dTUaPjw4fr000918803q0OHDoqNjVVsbKxiYmKY1wAAuORZrVafx7kGhv915ZVXqm3bttq7d68kKTExUUePHvWpqa+vV3l5uXfeQ2JiosrKynxqGp8b1Zxp7sSZNPkIw5w5czR58mS9++67fm0AAAAYO3TokL755hu1b99ekpSenq6KigoVFRUpNTVVUsP3NbndbqWlpXlrHn74YdXV1Sk0NFSSlJ+fr65du3r/iE9PT1dBQYGmTJni3VZ+fr7S09P96q/JgcHj8UiSbrjhBr82AADA5ai6utp7tECS9u/fr+LiYsXFxSkuLk5z5szR6NGjlZiYqH379mn69Onq0qWLMjMzJUndu3fX0KFDNWnSJC1dulR1dXXKycnRmDFjlJSUJEm68847NWfOHE2YMEEzZszQzp07tXDhQp+vcXjggQd0ww036Omnn9aIESP0yiuvaPv27T6XXjaFX5Mez/YtlQAA4Fvbt2/X1VdfrauvvlqSlJubq6uvvlqzZs1ScHCwPvvsM91888266qqrNGHCBKWmpuq9997zOcWxfPlydevWTYMHD9bw4cN13XXX+XzQ22w2vf3229q/f79SU1P1q1/9SrNmzfK5V8OPfvQjrVixQs8//7x69+6t119/XatXr1aPHj382h+Tp/HQgYGgoCDZbDbD0FBeXu5XA+fDbrfLZrNpZsFMWaKa5xwSAODCcVQ79OTgJ1VZWdlik+YbPysqK5+R1Rp+HuOclM02pUV7bc38ukpizpw5stlsLdULAABopfwKDGPGjDnlmlAAAHDpa/IcBuYvAABw+WpyYGjiVAcAAHAJavIpCbfb3ZJ9AACAVszv75IAAACXHwIDAAAwRGAAAACGCAwAAMAQgQEAABgiMAAAAEMEBgAAYIjAAAAADBEYAACAIb++fAoAgIuVe0u53JFh5/76mtpm7ObiwxEGAABgiMAAAAAMERgAAIAhAgMAADBEYAAAAIYIDAAAwBCBAQAAGCIwAAAAQwQGAABgiMAAAAAMERgAAIAhAgMAADBEYAAAAIYIDAAAwBCBAQAAGCIwAAAAQwQGAABgiMAAAAAMERgAAIAhAgMAADBEYAAAAIYIDAAAwBCBAQAAGCIwAAAAQwQGAABgiMAAAAAMERgAAIAhAgMAAC1g8+bNuummm5SUlCSTyaTVq1f7rPd4PJo1a5bat2+v8PBwZWRk6N///rdPTXl5ucaOHSur1aqYmBhNmDBB1dXVPjWfffaZrr/+eoWFhSk5OVnz588/pZfXXntN3bp1U1hYmHr27Kk333zT7/0hMAAA0AJqamrUu3dvLVmy5LTr58+fr0WLFmnp0qXatm2bIiMjlZmZqdraWm/N2LFjtWvXLuXn52vNmjXavHmz7r33Xu96u92uIUOGqFOnTioqKtJTTz2l2bNn6/nnn/fWbN26VXfccYcmTJigTz75RCNHjtTIkSO1c+dOv/bH5PF4PH7+DFoNu90um82mmQUzZYmyBLodAICfHNUOPTn4SVVWVspqtbbINho/K46vnSVrZNi5j1NTq9gRj59TryaTSatWrdLIkSMlNRxdSEpK0q9+9Ss9+OCDkqTKykolJCQoLy9PY8aM0e7du5WSkqKPPvpIffv2lSStW7dOw4cP16FDh5SUlKTnnntODz/8sEpLS2U2myVJM2fO1OrVq7Vnzx5J0u23366amhqtWbPG20///v3Vp08fLV26tMn7wBEGAAD8YLfbfR4Oh8PvMfbv36/S0lJlZGR4l9lsNqWlpamwsFCSVFhYqJiYGG9YkKSMjAwFBQVp27Zt3poBAwZ4w4IkZWZmqqSkRMePH/fWfHc7jTWN22kqAgMAAH5ITk6WzWbzPubNm+f3GKWlpZKkhIQEn+UJCQnedaWlpYqPj/dZHxISori4OJ+a043x3W2cqaZxfVOF+FUNAMBl7uDBgz6nJCyWy+OUOEcYAADwg9Vq9XmcS2BITEyUJJWVlfksLysr865LTEzU0aNHfdbX19ervLzcp+Z0Y3x3G2eqaVzfVAQGAAAusM6dOysxMVEFBQXeZXa7Xdu2bVN6erokKT09XRUVFSoqKvLWbNiwQW63W2lpad6azZs3q66uzluTn5+vrl27KjY21lvz3e001jRup6kIDAAAtIDq6moVFxeruLhYUsNEx+LiYh04cEAmk0lTpkzRb37zG/3zn//Ujh07dPfddyspKcl7JUX37t01dOhQTZo0SR9++KHef/995eTkaMyYMUpKSpIk3XnnnTKbzZowYYJ27dqlV199VQsXLlRubq63jwceeEDr1q3T008/rT179mj27Nnavn27cnJy/Nof5jAAANACtm/frkGDBnmfN36IZ2VlKS8vT9OnT1dNTY3uvfdeVVRU6LrrrtO6desUFvbtpZ/Lly9XTk6OBg8erKCgII0ePVqLFi3yrrfZbHr77beVnZ2t1NRUtW3bVrNmzfK5V8OPfvQjrVixQo888oh+/etf6wc/+IFWr16tHj16+LU/3IcBABAwl8t9GC4FnJIAAACGCAwAAMAQcxggxxGHanbUqO6bOrlPuBU7OFZhV/getqurqFPVR1VyHnFKHikkJkSxg2MVHBUst8Otqo+r5PjKIVe1S0FhQQrrFKbo1GgFmcmkaH2qP61W7Ze1qq+slynYpND4UFn7WRUS8+1bYr29XvYP7aorq5PH5ZGlg0XWdKuCw4MD2DkQOAQGyFPvUWhcqCKuitDxguOnrK+31+ubNd8o4qoIRV8dLZPZpPrj9dJ/3zddNS65T7hlvbbhDddV7VLl+5Xe8AG0Ns4jTkV0j1Bou1DJLVVtr1L5unK1Hd1WQaFBcte5Vb6uXCFxIYobFidJqiqq0vG3j6vNzW1kMpkCvAfAhRfQP/+MvvoTF0ZYcpii+0afclShUdX2KoV1CJP1WqtC24YqxBqisE5h3r+0QuNCG45KdAxTiDVEliSLovtGq/ZArTzui3ZOLS5hcUPjFHFVhEJjQxXaJlS2ATa5alyq+7rhWva6sjq5ql2KGRCj0LhQhcaFKuaGGNV9XSfnYWeAuwcCI6CBweirPxF4Ho9HjkMOBduC9c26b1S2vExf//Nr1X5Ze/bXOT0KMgfJFMRfYmj9PHUNwTbI0vCW2Bh0TcHf/v6agk2SSXKWERhweQroKYlhw4Zp2LBhTa53OBw+3wpmt9tboi18h/ukW546j2o+q1FUapSs/axyHHLoeMFxxQ2Pk6X9qZezumvdqv6kWuFdwwPQMeAfj8cj+wd2hSY0HEmQpNB2oTKFmGT/yC5rX6s8Ho+qtldJHsl9wh3gjoHAuKjmMMybN09z5swJdBuXl/+eUbB0tCiqR5QkKbRNqJxHnTqx58QpgcHtdKv87XKFxIYo+proC90t4Df7Vrvqj9erzU/beJcFhwcr9iexqtxaqdJdpZJJCr8yXCFtQiQOml20thwfrkhH1Dm/vuZEtaTHm6+hi8xFNYX9oYceUmVlpfdx8ODBQLd0yQsKC5JM8pk9LkkhtobJjd/ldrpVvr5cplCTYgfHcjoCrV7l1krVHqxV3PA4BUf6Xv1g6WBR/G3xShiboISxCYoZGCP3CbeCo7lKApeni+oIg8ViuWy+RrS1MAWbFNouVK5K33BQb69XcNS3b5xuZ8OsclOwSXE3xskUQlhA6+XxeGQvtKv2P7VqM7yNQqLP/FYYFNbwd5XjsEPuk26FdTz3OwUCF7OLKjCgZbjr3HLZvw0E9dX1qvumTkGWIAVHBSuqZ5SOv3tc5kSzzElmOQ455DjgUJvhDYdwG8OCp97T8FeY0y39d15YUBgTH9H62LfadfKLk4rNiJUp1CTXiYbf/yBzkDfsnvjXCYXEhCgoLEjOo07ZP7ArskfkKUfbgMsFv/lQ3dd1Kn+z3Pu8aluVqlSl8B+EK2ZAjMKuCJPtxzZVf1ot1wcuhdgabtpkTjQ3vP6bOtUda7gc7dhrx3zGbndbu7P+9QYEwok9JyTJ5/dekmzX2xRxVYQkqb6yXlXbq+R2uBuCc+8oRfaIvOC9Aq1FQN/Jq6urtXfvXu/zxq/+jIuLU8eOHQPY2eXF0t6i9hPan7Um4qoI7xvpubweaE2a8vtq7WeVtd/l9wVDwJkENDAYffUnAABoHQIaGAYOHKiL+Nu1AQC4bFxUl1UCAIDAIDAAAABDBAYAAGCIwAAAAAwRGAAAgCECAwAAMERgAAAAhggMAADAEIEBAAAYIjAAAABDBAYAAGCIwAAAAAwRGAAAgCECAwAAMERgAAAAhggMAADAEIEBAAAYIjAAAABDBAYAAGCIwAAAAAwRGAAAgCECAwAAMERgAAAAhggMAADAEIEBAAAYIjAAAABDBAYAAGCIwAAAAAwRGAAAaAGzZ8+WyWTyeXTr1s27vra2VtnZ2WrTpo2ioqI0evRolZWV+Yxx4MABjRgxQhEREYqPj9e0adNUX1/vU7Nx40Zdc801slgs6tKli/Ly8lpkfwgMAAC0kB/+8Ic6cuSI97FlyxbvuqlTp+qNN97Qa6+9pk2bNunw4cMaNWqUd73L5dKIESPkdDq1detWvfTSS8rLy9OsWbO8Nfv379eIESM0aNAgFRcXa8qUKZo4caLWr1/f7PsS0uwjAgBwCbPb7T7PLRaLLBbLaWtDQkKUmJh4yvLKykr9+c9/1ooVK/STn/xEkvTiiy+qe/fu+uCDD9S/f3+9/fbb+vzzz/XOO+8oISFBffr00dy5czVjxgzNnj1bZrNZS5cuVefOnfX0009Lkrp3764tW7ZowYIFyszMbNb95ggDAAB+SE5Ols1m8z7mzZt3xtp///vfSkpK0pVXXqmxY8fqwIEDkqSioiLV1dUpIyPDW9utWzd17NhRhYWFkqTCwkL17NlTCQkJ3prMzEzZ7Xbt2rXLW/PdMRprGsdoThxhAADADwcPHpTVavU+P9PRhbS0NOXl5alr1646cuSI5syZo+uvv147d+5UaWmpzGazYmJifF6TkJCg0tJSSVJpaalPWGhc37jubDV2u10nT55UeHj4ee3rdxEYAADwg9Vq9QkMZzJs2DDvv3v16qW0tDR16tRJK1eubNYP8guFUxIAAFwAMTExuuqqq7R3714lJibK6XSqoqLCp6asrMw75yExMfGUqyYanxvVWK3WZg8lBAYAAC6A6upq7du3T+3bt1dqaqpCQ0NVUFDgXV9SUqIDBw4oPT1dkpSenq4dO3bo6NGj3pr8/HxZrValpKR4a747RmNN4xjNicAAAEALePDBB7Vp0yZ9+eWX2rp1q372s58pODhYd9xxh2w2myZMmKDc3Fy9++67Kioq0vjx45Wenq7+/ftLkoYMGaKUlBTddddd+vTTT7V+/Xo98sgjys7O9s6bmDx5sr744gtNnz5de/bs0bPPPquVK1dq6tSpzb4/zGEAAKAFHDp0SHfccYe++eYbtWvXTtddd50++OADtWvXTpK0YMECBQUFafTo0XI4HMrMzNSzzz7rfX1wcLDWrFmj++67T+np6YqMjFRWVpYef/xxb03nzp21du1aTZ06VQsXLlSHDh20bNmyZr+kUpJMHo/H0+yjXiB2u102m00zC2bKEnX6WaoAgNbLUe3Qk4OfVGVlZZMmEp6Lxs+KN/6/DxQZEXXO49ScqNZNP+/for22ZhxhAABcFt5vv/q8/rh0VDuasZuLD3MYAACAIQIDAAAwRGAAAACGCAwAAMAQgQEAABgiMAAAAEMEBgAAYIjAAAAADBEYAACAIQIDAAAwRGAAAACGCAwAAMAQgQEAABgiMAAAAEMEBgAAYIjAAAAADBEYAACAIQIDAAAwRGAAAACGCAwAAMAQgQEAABgiMAAAAEMEBgAAYIjAAAAADBEYAACAIQIDAAAwRGAAAACGCAwAAMAQgQEAABgiMAAAAEMEBgAAYIjAAAAADBEYAACAIQIDAAAwRGAAAACGCAwAAMAQgQEAABgiMAAAAEMEBgAAYIjAAAAADBEYAACAIQIDAAAwRGAAAACGCAwAAMBQSKAbOB8ej0eS5KhxBLgTAMC5aHz/bnw/vxDbCtTrL3Ymz4X4r9RCDh06pOTk5EC3AQA4TwcPHlSHDh1aZOza2lp17txZpaWl5z1WYmKi9u/fr7CwsGbo7OJyUQcGt9utw4cPKzo6WiaTKdDtXBbsdruSk5N18OBBWa3WQLcDNCt+vy88j8ejqqoqJSUlKSio5c6S19bWyul0nvc4ZrP5sgwL0kV+SiIoKKjFEinOzmq18oaKSxa/3xeWzWZr8W2EhYVdth/0zYVJjwAAwBCBAQAAGCIwwC8Wi0WPPfaYLBZLoFsBmh2/38CZXdSTHgEAwIXBEQYAAGCIwAAAAAwRGAAAgCECAwAAMERgQJMtWbJEV1xxhcLCwpSWlqYPP/ww0C0BzWLz5s266aablJSUJJPJpNWrVwe6JaDVITCgSV599VXl5ubqscce08cff6zevXsrMzNTR48eDXRrwHmrqalR7969tWTJkkC3ArRaXFaJJklLS1O/fv20ePFiSQ3f45GcnKz7779fM2fODHB3QPMxmUxatWqVRo4cGehWgFaFIwww5HQ6VVRUpIyMDO+yoKAgZWRkqLCwMICdAQAuFAIDDH399ddyuVxKSEjwWZ6QkNAsXxcLAGj9CAwAAMAQgQGG2rZtq+DgYJWVlfksLysrU2JiYoC6AgBcSAQGGDKbzUpNTVVBQYF3mdvtVkFBgdLT0wPYGQDgQgkJdAO4OOTm5iorK0t9+/bVtddeq2eeeUY1NTUaP358oFsDzlt1dbX27t3rfb5//34VFxcrLi5OHTt2DGBnQOvBZZVossWLF+upp55SaWmp+vTpo0WLFiktLS3QbQHnbePGjRo0aNApy7OyspSXl3fhGwJaIQIDAAAwxBwGAABgiMAAAAAMERgAAIAhAgMAADBEYAAAAIYIDAAAwBCBAQAAGCIwAAAAQwQG4DyNGzdOI0eO9D4fOHCgpkyZcsH72Lhxo0wmkyoqKs5YYzKZtHr16iaPOXv2bPXp0+e8+vryyy9lMplUXFx8XuMACCwCAy5J48aNk8lkkslkktlsVpcuXfT444+rvr6+xbf997//XXPnzm1SbVM+5AGgNeDLp3DJGjp0qF588UU5HA69+eabys7OVmhoqB566KFTap1Op8xmc7NsNy4urlnGAYDWhCMMuGRZLBYlJiaqU6dOuu+++5SRkaF//vOfkr49jfDb3/5WSUlJ6tq1qyTp4MGDuu222xQTE6O4uDjdcsst+vLLL71julwu5ebmKiYmRm3atNH06dP1v1/H8r+nJBwOh2bMmKHk5GRZLBZ16dJFf/7zn/Xll196v/AoNjZWJpNJ48aNk9Tw9eHz5s1T586dFR4ert69e+v111/32c6bb76pq666SuHh4Ro0aJBPn001Y8YMXXXVVYqIiNCVV16pRx99VHV1dafU/fGPf1RycrIiIiJ02223qbKy0mf9smXL1L17d4WFhalbt2569tln/e4FQOtGYMBlIzw8XE6n0/u8oKBAJSUlys/P15o1a1RXV6fMzExFR0frvffe0/vvv6+oqCgNHTrU+7qnn35aeXl5euGFF7RlyxaVl5dr1apVZ93u3Xffrb/+9a9atGiRdu/erT/+8Y+KiopScnKy/va3v0mSSkpKdOTIES1cuFCSNG/ePL388staunSpdu3apalTp+rnP/+5Nm3aJKkh2IwaNUo33XSTiouLNXHiRM2cOdPvn0l0dLTy8vL0+eefa+HChfrTn/6kBQsW+NTs3btXK1eu1BtvvKF169bpk08+0S9+8Qvv+uXLl2vWrFn67W9/q927d+uJJ57Qo48+qpdeesnvfgC0Yh7gEpSVleW55ZZbPB6Px+N2uz35+fkei8XiefDBB73rExISPA6Hw/uav/zlL56uXbt63G63d5nD4fCEh4d71q9f7/F4PJ727dt75s+f711fV1fn6dChg3dbHo/Hc8MNN3geeOABj8fj8ZSUlHgkefLz80/b57vvvuuR5Dl+/Lh3WW1trSciIsKzdetWn9oJEyZ47rjjDo/H4/E89NBDnpSUFJ/1M2bMOGWs/yXJs2rVqjOuf+qppzypqane54899pgnODjYc+jQIe+yt956yxMUFOQ5cuSIx+PxeL7//e97VqxY4TPO3LlzPenp6R6Px+PZv3+/R5Lnk08+OeN2AbR+zGHAJWvNmjWKiopSXV2d3G637rzzTs2ePdu7vmfPnj7zFj799FPt3btX0dHRPuPU1tZq3759qqys1JEjR5SWluZdFxISor59+55yWqJRcXGxgoODdcMNNzS577179+rEiRO68cYbfZY7nU5dffXVkqTdu3f79CFJ6enpTd5Go1dffVWLFi3Svn37VF1drfr6elmtVp+ajh076nvf+57Pdtxut0pKShQdHa19+/ZpwoQJmjRpkremvr5eNpvN734AtF4EBlyyBg0apOeee05ms1lJSUkKCfH9dY+MjPR5Xl1drdTUVC1fvvyUsdq1a3dOPYSHh/v9murqaknS2rVrfT6opYZ5Gc2lsLBQY8eO1Zw5c5SZmSmbzaZXXnlFTz/9tN+9/ulPfzolwAQHBzdbrwACj8CAS1ZkZKS6dOnS5PprrrlGr776quLj40/5K7tR+/bttW3bNg0YMEBSw1/SRUVFuuaaa05b37NnT7ndbm3atEkZGRmnrG88wuFyubzLUlJSZLFYdODAgTMemejevbt3AmejDz74wHgnv2Pr1q3q1KmTHn74Ye+y//znP6fUHThwQIcPH1ZSUpJ3O0FBQeratasSEhKUlJSkL774QmPHjvVr+wAuLkx6BP5r7Nixatu2rW655Ra999572r9/vzZu3Khf/vKXOnTokCTpgQce0JNPPqnVq1drz549+sUvfnHWeyhcccUVysrK0j333KPVq1d7x1y5cqUkqVOnTjKZTFqzZo2OHTum6upqRUdH68EHH9TUqVP10ksvad++ffr444/1hz/8wTuRcPLkyfr3v/+tadOmqaSkRCtWrFBeXp5f+/uDH/xABw4c0CuvvKJ9+/Zp0aJFp53AGRYWpqysLH366ad677339Mtf/lK33XabEhMTJUlz5szRvHnztGjRIv3rX//Sjh079OKLL+r3v/+9X/0AaN0IDMB/RUREaPPmzerYsaNGjRql7t27a8KECaqtrfUecfjVr36lu+66S1lZWUpPT1d0dLR+9rOfnXXc5557Trfeeqt+8YtfqFu3bpo0aZJqamokSd/73vc0Z84czZw5UwkJCcrJyZEkzZ07V48++qjmzZun7t27a+jQoVq7dq06d+4sqWFewd/+9jetXr1avXv31tKlS/XEE0/4tb8333yzpk6dqpycHPXp00dbt27Vo48+ekpdly5dNGrUKA0fPlxDhgxRr169fC6bnDhxopYtW6YXX3xRPXv21A033KC8vDxvrwAuDSbPmWZrAQAA/BdHGAAAgCECAwAAMERgAAAAhggMAADAEIEBAAAYIjAAAABDBAYAAGCIwAAAAAwRGAAAgCECAwAAMERgAAAAhv5/1TGUGAXSpsgAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_confusion_matrix(model,X_test, y_test, cmap='Accent')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "f75457bd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00     34787\n",
      "           1       0.15      0.94      0.26        31\n",
      "\n",
      "    accuracy                           1.00     34818\n",
      "   macro avg       0.58      0.97      0.63     34818\n",
      "weighted avg       1.00      1.00      1.00     34818\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"Classification report:\")\n",
    "print(classification_report(pred, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c41e406",
   "metadata": {},
   "source": [
    "Implement a __evaluation()__ function to evaluate the metrics mentioned above "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "0c248054",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, verbose=False):\n",
    "    \"\"\"\n",
    "    Set verbose = True to print accuracy, classification report and plot\n",
    "    confusion matrix\n",
    "    \"\"\"\n",
    "    metrics = {}\n",
    "    pred = model.predict(X_test)\n",
    "    metrics['accuracy'] = accuracy_score(y_test, pred)\n",
    "    verbose and print (f\"The accuracy is {metrics['accuracy']}\")\n",
    "    verbose and plot_confusion_matrix(model,X_test, y_test)\n",
    "    metrics['precision'] = precision_score(y_test, pred)\n",
    "    metrics['recall'] = recall_score(y_test, pred)\n",
    "    verbose and print (f\"The classification report looks like : \\n {classification_report(pred, y_test)}\")\n",
    "    return metrics\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "f6083f58",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy is 0.9957493250617496\n",
      "The classification report looks like : \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00     34725\n",
      "           1       0.36      0.73      0.48        93\n",
      "\n",
      "    accuracy                           1.00     34818\n",
      "   macro avg       0.68      0.86      0.74     34818\n",
      "weighted avg       1.00      1.00      1.00     34818\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.9957493250617496,\n",
       " 'precision': 0.7311827956989247,\n",
       " 'recall': 0.35602094240837695}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgwAAAGwCAYAAADFZj2cAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAABCRklEQVR4nO3de1hVdfr//9cG3YDIRlEBUcRTqYynQkNqMi1GNKcy7VeWFZnaaOAklKcOHrJyRqeDpmllRc1XJ+2glZZFmlpJNlJUmlKapqagkwqCctzr94exP+08bLYLRFjPx3WtK1nrvda+lxdX+/Z+3+u9bIZhGAIAADgLn5oOAAAAXPhIGAAAgEckDAAAwCMSBgAA4BEJAwAA8IiEAQAAeETCAAAAPKpX0wGY4XQ6tX//fgUFBclms9V0OAAALxmGoWPHjikiIkI+PtX3b9iioiKVlJSYvo7dbpe/v38VRFT71OqEYf/+/YqMjKzpMAAAJu3du1ctW7aslmsXFRWpTVRD5RwsN32t8PBw7dq1y5JJQ61OGIKCgiRJP3/VWo6GzK6gbrrx4i41HQJQbcpUqs/0vuv/59WhpKREOQfL9XNmazmCzv27Iv+YU1Exu1VSUkLCUNtUTEM4GvqY+iUALmT1bPVrOgSg+vz2coLzMa3cMMimhkHn/jlOWXvqu1YnDAAAVFa54VS5ibcnlRvOqgumFiJhAABYglOGnDr3jMHMuXUBdXwAAOARFQYAgCU45ZSZSQVzZ9d+JAwAAEsoNwyVG+c+rWDm3LqAKQkAAOARFQYAgCXQ9GgOCQMAwBKcMlROwnDOmJIAAAAeUWEAAFgCUxLmkDAAACyBpyTMYUoCAAB4RIUBAGAJzt82M+dbGQkDAMASyk0+JWHm3LqAhAEAYAnlhky+rbLqYqmN6GEAAAAeUWEAAFgCPQzmkDAAACzBKZvKZTN1vpUxJQEAADyiwgAAsASncXIzc76VkTAAACyh3OSUhJlz6wKmJAAAgEdUGAAAlkCFwRwSBgCAJTgNm5yGiackTJxbFzAlAQAAPKLCAACwBKYkzCFhAABYQrl8VG6isF5ehbHURiQMAABLMEz2MBj0MAAAAJwdFQYAgCXQw2AOCQMAwBLKDR+VGyZ6GCy+NDRTEgAAwCMqDAAAS3DKJqeJfyc7Ze0SAwkDAMAS6GEwhykJAADgERUGAIAlmG96ZEoCAIA672QPg4mXTzElAQAAcHZUGAAAluA0+S4JnpIAAMAC6GEwh4QBAGAJTvmwDoMJ9DAAAACPqDAAACyh3LCp3MQrqs2cWxdQYQAAWEL5b02PZjZvLFiwQF27dpXD4ZDD4VBcXJw++OAD1/GioiIlJSWpSZMmatiwoYYMGaLc3Fy3a+zZs0cDBw5UgwYNFBoaqvHjx6usrMxtzLp163TppZfKz89P7du3V1pa2imxzJ8/X61bt5a/v79iY2P15ZdfenUvEgkDAADVomXLlvrHP/6hzMxMbd68WVdffbVuuOEGbd26VZKUkpKi9957T2+88YbWr1+v/fv3a/Dgwa7zy8vLNXDgQJWUlGjjxo169dVXlZaWpilTprjG7Nq1SwMHDlTfvn2VlZWlcePGaeTIkfrwww9dY5YuXarU1FRNnTpVX331lbp166aEhAQdPHjQq/uxGUbtbfvMz89XcHCwjvzQVo4gch/UTQkR3Ws6BKDalBmlWqd3lJeXJ4fDUS2fUfFd8fJXl6hBkO85X+f4sXLdfenX2rt3r1usfn5+8vPzq9Q1QkJCNHv2bN10001q1qyZlixZoptuukmStH37dnXq1EkZGRnq1auXPvjgA/31r3/V/v37FRYWJklauHChJk6cqEOHDslut2vixIlatWqVtmzZ4vqMoUOH6ujRo1q9erUkKTY2Vj179tS8efMkSU6nU5GRkRo7dqwmTZpU6fvnWxYAYAlVNSURGRmp4OBg1zZz5kzPn11ertdff12FhYWKi4tTZmamSktLFR8f7xrTsWNHtWrVShkZGZKkjIwMdenSxZUsSFJCQoLy8/NdVYqMjAy3a1SMqbhGSUmJMjMz3cb4+PgoPj7eNaayaHoEAMALp6swnMl3332nuLg4FRUVqWHDhlq+fLmio6OVlZUlu92uRo0auY0PCwtTTk6OJCknJ8ctWag4XnHsbGPy8/N14sQJHTlyROXl5acds337dq/um4QBAGAJTpl70sH5238rmhgro0OHDsrKylJeXp7efPNNJSYmav369eccQ00iYQAAWIL5hZu8P9dut6t9+/aSpJiYGP33v//VnDlzdMstt6ikpERHjx51qzLk5uYqPDxckhQeHn7K0wwVT1H8fswfn6zIzc2Vw+FQQECAfH195evre9oxFdeoLHoYAAA4T5xOp4qLixUTE6P69etrzZo1rmPZ2dnas2eP4uLiJElxcXH67rvv3J5mSE9Pl8PhUHR0tGvM769RMabiGna7XTExMW5jnE6n1qxZ4xpTWVQYAACWYP5dEt6dO3nyZA0YMECtWrXSsWPHtGTJEq1bt04ffvihgoODNWLECKWmpiokJEQOh0Njx45VXFycevXqJUnq16+foqOjdccdd2jWrFnKycnRww8/rKSkJFffxOjRozVv3jxNmDBBd999t9auXatly5Zp1apVrjhSU1OVmJioHj166LLLLtMzzzyjwsJCDR8+3Kv7IWEAAFiCUzY5ZaaHwbtzDx48qDvvvFMHDhxQcHCwunbtqg8//FB/+ctfJElPP/20fHx8NGTIEBUXFyshIUHPPfec63xfX1+tXLlSY8aMUVxcnAIDA5WYmKhHH33UNaZNmzZatWqVUlJSNGfOHLVs2VKLFi1SQkKCa8wtt9yiQ4cOacqUKcrJyVH37t21evXqUxohPWEdBuACxzoMqMvO5zoMT2++XAENz/3fyScKypTSY2O1xnoh41sWAAB4xJQEAMASzuV9EH8838pIGAAAluA0bHKaWYeBt1UCAACcHRUGAIAlOE1OSZhZ9KkuIGEAAFiC0/CR08Q6DGbOrQusffcAAKBSqDAAACyhXDaVm1i4ycy5dQEJAwDAEpiSMMfadw8AACqFCgMAwBLKZW5aobzqQqmVSBgAAJbAlIQ5JAwAAEs436+3rmusffcAAKBSqDAAACzBkE1OEz0MBo9VAgBQ9zElYY617x4AAFQKFQYAgCXwemtzSBgAAJZQbvJtlWbOrQusffcAAKBSqDAAACyBKQlzSBgAAJbglI+cJgrrZs6tC6x99wAAoFKoMAAALKHcsKncxLSCmXPrAhIGAIAl0MNgDgkDAMASDJNvqzRY6REAAODsqDAAACyhXDaVm3iBlJlz6wISBgCAJTgNc30ITqMKg6mFmJIAAAAeUWGo4957tYlWvdZUuXvtkqSoDkUalpKjnlcfcxtnGNLDt7fV5k8cmvrSLl0+IM/t+EdLQ/T2C8207yc/NWhYrt5/Parkmb+4jv/0vb/mPdhSP3zTQMEhZbrh7v/p5qSDruPvLw7Rx2+E6Odsf0lS+y4nNHzyAXW85Hh13TpwRrck5+qKa/MU2b5YJUU++n5zA730eHPt2+nvGjPrzR3qdnmh23mrXmuiuZNanu9wUUWcJpsezZxbF5Aw1HHNmpfq7gf3q0WbYhmGTelvNNa04W00/6Mf1LpDkWvc8hebyXaGSt1bzzfTW88308iH96vjpcdVdNzHlYBIUuExHz14aztdcuUx/f2f+7R7m7+eSm2lhsHluvb2XyVJ325sqL6Djii6x3HV93Nq2fxQPXhrO73wyXY1bV5arX8HwB91jSvUe2lN9UNWA/nWM3TXpAN64j8/adRVHVR8wtc17v3/F6LXZoe7fi4+Ye0vjNrOKZucJvoQzJxbF1wQv/3z589X69at5e/vr9jYWH355Zc1HVKd0atfvi675phatC1Ry3bFGj4pR/6BTm3PbOAas3NLgN56vplSn9pzyvnHjvrq1X821/g5e3T14KOKaF2ittFFikvId41Z+3ZjlZbalPrUXrXuUKQ+g47qhhGH9NbzzVxjJs3fo+vu+lXtOp9Qq4uKlfLkXhlO6evPGlbvXwBwGg8Na6v0ZSH6+Qd//fR9gJ4c10phLUt1UdcTbuOKT/joyKH6ru14ge8ZrgjUfTWeMCxdulSpqamaOnWqvvrqK3Xr1k0JCQk6ePCg55PhlfJyad2KRio+7qNOPU6WWouO2/SPpCglPb5PIaFlp5zz1YYgOQ3pfzn1NbJ3Rw2LidZjf4vSwV/qu8ZsywxUl9hC1bf/X0dQTJ9j2rfTX8eOnv5/sMUnfFRWZlNQo/IqvkvAe4GOk7+Hf/x97Tv4iJZt2aLn12Zr+OQD8gtw1kR4qCIVKz2a2aysxqcknnrqKY0aNUrDhw+XJC1cuFCrVq3Syy+/rEmTJtVwdHXDrm3+GnfdRSop9lFAoFNTXtqlqIuLJUnPT2uh6B6Furx//mnPzfnZLsMpvT43TGNm/KLAoHKl/bO5Jg9tp4VrslXfbujIwXoKb1Xidl7jZienGY4cqnfapOClxyPUJKxUl1557JRjwPlksxkaPf0XbfmygX7ODnDt/2R5Yx3cV1+/5tZXm05FGvHQAbVsV6wZI1vXXLAwhR4Gc2o0YSgpKVFmZqYmT57s2ufj46P4+HhlZGScMr64uFjFxcWun/PzT/8lB3ct2xXrufRsHT/mq09XNtK/7ovS7Ld/1P5dfsr6PEjPfZR9xnOdhlRW6qN7Z/yimD4nv9wnL9itW7t11jcbG6pHH++/8Jc+G6p17zTS7Dd3yO5v8eeUUOOSn/hFUR2LdP+g9m77P1jcxPXn3dsDdPhgPc164yc1jyrWgZ/9zneYQI2r0YThf//7n8rLyxUWFua2PywsTNu3bz9l/MyZMzV9+vTzFV6dUd9uqEWbkxWAi7qeUHZWA61Y1Ex2f0MHdts1uGMXt/EzRrVW59hCzX5rh2uaotXF/9cg2ahJuRwhZa5picahZTpyqL7bNSp+btzMfZrjjQXNtHR+mP6xdIfaRhcJqElJj+9T7F/ydf+N7fS/A/azjt3+1cm+n4jWJAy1lVMm3yVh8abHGp+S8MbkyZOVmprq+jk/P1+RkZE1GFHtZBhSaYmP7nhgvwbc9qvbsb9d3VF/m/aLevU7Wb35U8+TvQ77dvqpWcTJaYb8I77KP1xPYS1O/twpplBp/2yuslKp3m95w1cbgtSyXZHbdMSy+aH6z9wwPbFkpy7u5t5cBpxfhpIe/0WX98/T+JvaK3ev5wSgXeeTCe7hg/U9jMSFyjD5lIRBwlBzmjZtKl9fX+Xm5rrtz83NVXh4+Cnj/fz85OdHZu+Nl59orp5X56tZi1KdKPDRJ8sb69uNDfX4kp0KCS07baNjaItSV09Cy3bFikvI04IpLXTfrL0KDHLq5Seaq2X7InW74uR0xNU3HtHip8L11P2tdHPSQe3e7q8Vi5pq9PT9rmsunReqf/8rXBPn/6ywyBIdPnjyVy8g0KmAQBrJcH4lP/GL+t54RNOGt9GJAh9Xz03hMV+VFPmoeVSx+t54VF+uCdKxI/XUJvqE/jZtv77NCNSubQEero4LFW+rNKdGEwa73a6YmBitWbNGgwYNkiQ5nU6tWbNGycnJNRlanXH0f/U0++9ROnywnhoElatNpyI9vmSnYq4qqPQ1xs/9Wc9PbaEpd7aVzUfq2qtAjy/+yVVNCHQ49cR/dmregy2V3P9iBYeUaVhKrmsNBkla9VpTlZb46LFRbdyufXtqju54IKdK7hWorOvuOvm7+a+3d7rt/9e4SKUvC1FZqU2XXHlMN448JP8GTh3aX1+fvR+s/zwTdrrLAZZgMwyjRrvOli5dqsTERD3//PO67LLL9Mwzz2jZsmXavn37Kb0Nf5Sfn6/g4GAd+aGtHEHW7l5F3ZUQ0b2mQwCqTZlRqnV6R3l5eXI4HNXyGRXfFTemD1f9wLP3qpxNaWGJlv/llWqN9UJW4z0Mt9xyiw4dOqQpU6YoJydH3bt31+rVqz0mCwAAeIMpCXMuiH+WJycn6+eff1ZxcbE2bdqk2NjYmg4JAABTZs6cqZ49eyooKEihoaEaNGiQsrPdH2Pv06ePbDab2zZ69Gi3MXv27NHAgQPVoEEDhYaGavz48Sorc+8/W7dunS699FL5+fmpffv2SktLOyUes6sqXxAJAwAA1a3iXRJmNm+sX79eSUlJ+uKLL5Senq7S0lL169dPhYXuLzUbNWqUDhw44NpmzZrlOlZeXq6BAweqpKREGzdu1Kuvvqq0tDRNmTLFNWbXrl0aOHCg+vbtq6ysLI0bN04jR47Uhx9+6BpTFasq13gPgxn0MMAK6GFAXXY+exgGfjjSdA/DqoRF5xzroUOHFBoaqvXr16t3796STlYYunfvrmeeeea053zwwQf661//qv3797um6hcuXKiJEyfq0KFDstvtmjhxolatWqUtW7a4zhs6dKiOHj2q1atXS5JiY2PVs2dPzZs3T9LJBwwiIyM1duzYSq+qzLcsAABeyM/Pd9t+vwLx2eTl5UmSQkJC3PYvXrxYTZs2VefOnTV58mQdP37cdSwjI0NdunRx6+tLSEhQfn6+tm7d6hoTHx/vds2EhATXiskVqyr/fszZVlU+kxpvegQA4HyoqqbHPy4YOHXqVE2bNu3s5zqdGjdunK644gp17tzZtf+2225TVFSUIiIi9O2332rixInKzs7W22+/LUnKyck57WrIFcfONiY/P18nTpzQkSNHvFpV+UxIGAAAllBVCcPevXvdpiQqs6BgUlKStmzZos8++8xt/z333OP6c5cuXdS8eXNdc8012rlzp9q1a3fOsVYHpiQAAPCCw+Fw2zwlDMnJyVq5cqU++eQTtWzZ8qxjK54S3LFjhyQpPDz8tKshVxw72xiHw6GAgACvV1U+ExIGAIAlVFQYzGzeMAxDycnJWr58udauXas2bdp4PCcrK0uS1Lx5c0lSXFycvvvuO7enGdLT0+VwOBQdHe0as2bNGrfrpKenKy4uTpL7qsquv4vfVlWuGFMZTEkAACzBkLk3Tnr7SGFSUpKWLFmid955R0FBQa6eg+DgYAUEBGjnzp1asmSJrr32WjVp0kTffvutUlJS1Lt3b3Xt2lWS1K9fP0VHR+uOO+7QrFmzlJOTo4cfflhJSUmuysbo0aM1b948TZgwQXfffbfWrl2rZcuWadWqVa5YUlNTlZiYqB49erhWVS4sLNTw4cMrfT8kDAAASzjfKz0uWLBA0slHJ3/vlVde0V133SW73a6PP/7Y9eUdGRmpIUOG6OGHH3aN9fX11cqVKzVmzBjFxcUpMDBQiYmJevTRR11j2rRpo1WrViklJUVz5sxRy5YttWjRIiUkJLjGVMWqyqzDAFzgWIcBddn5XIfh6lWjVS/w3N94XFZYrLUDF/IuCQAA6jLeJWEOCQMAwBJIGMyhjg8AADyiwgAAsAQqDOaQMAAALMEwbDJMfOmbObcuYEoCAAB4RIUBAGAJTtlMLdxk5ty6gIQBAGAJ9DCYw5QEAADwiAoDAMASaHo0h4QBAGAJTEmYQ8IAALAEKgzm0MMAAAA8osIAALAEw+SUhNUrDCQMAABLMCQZhrnzrYwpCQAA4BEVBgCAJThlk42VHs8ZCQMAwBJ4SsIcpiQAAIBHVBgAAJbgNGyysXDTOSNhAABYgmGYfErC4o9JMCUBAAA8osIAALAEmh7NIWEAAFgCCYM5JAwAAEug6dEcehgAAIBHVBgAAJbAUxLmkDAAACzhZMJgpoehCoOphZiSAAAAHlFhAABYAk9JmEPCAACwBOO3zcz5VsaUBAAA8IgKAwDAEpiSMIeEAQBgDcxJmELCAACwBpMVBlm8wkAPAwAA8IgKAwDAEljp0RwSBgCAJdD0aA5TEgAAwCMqDAAAazBs5hoXLV5hIGEAAFgCPQzmMCUBAAA8osIAALAGFm4yhQoDAMASKp6SMLN5Y+bMmerZs6eCgoIUGhqqQYMGKTs7221MUVGRkpKS1KRJEzVs2FBDhgxRbm6u25g9e/Zo4MCBatCggUJDQzV+/HiVlZW5jVm3bp0uvfRS+fn5qX379kpLSzslnvnz56t169by9/dXbGysvvzyS6/up1IVhnfffbfSF7z++uu9CgAAgLpo/fr1SkpKUs+ePVVWVqYHH3xQ/fr10/fff6/AwEBJUkpKilatWqU33nhDwcHBSk5O1uDBg/X5559LksrLyzVw4ECFh4dr48aNOnDggO68807Vr19fTzzxhCRp165dGjhwoEaPHq3FixdrzZo1GjlypJo3b66EhARJ0tKlS5WamqqFCxcqNjZWzzzzjBISEpSdna3Q0NBK3Y/NMDy3cfj4VK4QYbPZVF5eXqmxVSE/P1/BwcE68kNbOYIolqBuSojoXtMhANWmzCjVOr2jvLw8ORyOavmMiu+KVi9MkU+A/zlfx3miSHvueVR79+51i9XPz09+fn4ezz906JBCQ0O1fv169e7dW3l5eWrWrJmWLFmim266SZK0fft2derUSRkZGerVq5c++OAD/fWvf9X+/fsVFhYmSVq4cKEmTpyoQ4cOyW63a+LEiVq1apW2bNni+qyhQ4fq6NGjWr16tSQpNjZWPXv21Lx5807ei9OpyMhIjR07VpMmTarU/VfqW9bpdFZqO5/JAgAA3qiqKYnIyEgFBwe7tpkzZ1bq8/Py8iRJISEhkqTMzEyVlpYqPj7eNaZjx45q1aqVMjIyJEkZGRnq0qWLK1mQpISEBOXn52vr1q2uMb+/RsWYimuUlJQoMzPTbYyPj4/i4+NdYyrDVNNjUVGR/P3PPVsDAOC8qaKmx9NVGDxxOp0aN26crrjiCnXu3FmSlJOTI7vdrkaNGrmNDQsLU05OjmvM75OFiuMVx842Jj8/XydOnNCRI0dUXl5+2jHbt2/3GHsFr+v45eXlmjFjhlq0aKGGDRvqp59+kiQ98sgjeumll7y9HAAAtYrD4XDbKpMwJCUlacuWLXr99dfPQ4TVw+uE4fHHH1daWppmzZolu93u2t+5c2ctWrSoSoMDAKDq2Kpg815ycrJWrlypTz75RC1btnTtDw8PV0lJiY4ePeo2Pjc3V+Hh4a4xf3xqouJnT2McDocCAgLUtGlT+fr6nnZMxTUqw+uE4bXXXtMLL7ygYcOGydfX17W/W7duXpU2AAA4r4wq2Lz5OMNQcnKyli9frrVr16pNmzZux2NiYlS/fn2tWbPGtS87O1t79uxRXFycJCkuLk7fffedDh486BqTnp4uh8Oh6Oho15jfX6NiTMU17Ha7YmJi3MY4nU6tWbPGNaYyvO5h+OWXX9S+fftT9judTpWWlnp7OQAA6qSkpCQtWbJE77zzjoKCglw9B8HBwQoICFBwcLBGjBih1NRUhYSEyOFwaOzYsYqLi1OvXr0kSf369VN0dLTuuOMOzZo1Szk5OXr44YeVlJTkmgoZPXq05s2bpwkTJujuu+/W2rVrtWzZMq1atcoVS2pqqhITE9WjRw9ddtlleuaZZ1RYWKjhw4dX+n68Thiio6P16aefKioqym3/m2++qUsuucTbywEAcH6c55UeFyxYIEnq06eP2/5XXnlFd911lyTp6aeflo+Pj4YMGaLi4mIlJCToueeec4319fXVypUrNWbMGMXFxSkwMFCJiYl69NFHXWPatGmjVatWKSUlRXPmzFHLli21aNEi1xoMknTLLbfo0KFDmjJlinJyctS9e3etXr36lEbIs/E6YZgyZYoSExP1yy+/yOl06u2331Z2drZee+01rVy50tvLAQBwfpznt1VWYpkj+fv7a/78+Zo/f/4Zx0RFRen9998/63X69Omjr7/++qxjkpOTlZyc7DGmM/G6h+GGG27Qe++9p48//liBgYGaMmWKtm3bpvfee09/+ctfzjkQAABw4TqndRiuvPJKpaenV3UsAABUG15vbc45L9y0efNmbdu2TdLJvoaYmJgqCwoAgCrH2ypN8Tph2Ldvn2699VZ9/vnnrtWpjh49qssvv1yvv/662zOmAACgbvC6h2HkyJEqLS3Vtm3bdPjwYR0+fFjbtm2T0+nUyJEjqyNGAADMq2h6NLNZmNcVhvXr12vjxo3q0KGDa1+HDh307LPP6sorr6zS4AAAqCo24+Rm5nwr8zphiIyMPO0CTeXl5YqIiKiSoAAAqHL0MJji9ZTE7NmzNXbsWG3evNm1b/Pmzbrvvvv0r3/9q0qDAwAAF4ZKVRgaN24sm+3/5m4KCwsVGxurevVOnl5WVqZ69erp7rvv1qBBg6olUAAATDnPCzfVNZVKGJ555plqDgMAgGrGlIQplUoYEhMTqzsOAABwATvnhZskqaioSCUlJW77HA6HqYAAAKgWVBhM8brpsbCwUMnJyQoNDVVgYKAaN27stgEAcEEyqmCzMK8ThgkTJmjt2rVasGCB/Pz8tGjRIk2fPl0RERF67bXXqiNGAABQw7yeknjvvff02muvqU+fPho+fLiuvPJKtW/fXlFRUVq8eLGGDRtWHXECAGAOT0mY4nWF4fDhw2rbtq2kk/0Khw8fliT9+c9/1oYNG6o2OgAAqkjFSo9mNivzOmFo27atdu3aJUnq2LGjli1bJulk5aHiZVQAAKBu8TphGD58uL755htJ0qRJkzR//nz5+/srJSVF48ePr/IAAQCoEjQ9muJ1D0NKSorrz/Hx8dq+fbsyMzPVvn17de3atUqDAwAAFwZT6zBIUlRUlKKioqoiFgAAqo1NJt9WWWWR1E6VShjmzp1b6Qv+/e9/P+dgAADAhalSCcPTTz9dqYvZbLYaSRhuvLiL6tnqn/fPBQDUIjxWaUqlEoaKpyIAAKi1WBraFK+fkgAAANZjuukRAIBagQqDKSQMAABLMLtaIys9AgAAeECFAQBgDUxJmHJOFYZPP/1Ut99+u+Li4vTLL79Ikv7973/rs88+q9LgAACoMiwNbYrXCcNbb72lhIQEBQQE6Ouvv1ZxcbEkKS8vT0888USVBwgAAGqe1wnDY489poULF+rFF19U/fr/t1jSFVdcoa+++qpKgwMAoKrwemtzvO5hyM7OVu/evU/ZHxwcrKNHj1ZFTAAAVD1WejTF6wpDeHi4duzYccr+zz77TG3btq2SoAAAqHL0MJjidcIwatQo3Xfffdq0aZNsNpv279+vxYsX64EHHtCYMWOqI0YAAFDDvJ6SmDRpkpxOp6655hodP35cvXv3lp+fnx544AGNHTu2OmIEAMA0Fm4yx+uEwWaz6aGHHtL48eO1Y8cOFRQUKDo6Wg0bNqyO+AAAqBqsw2DKOS/cZLfbFR0dXZWxAACAC5TXCUPfvn1ls525U3Tt2rWmAgIAoFqYfTSSCoN3unfv7vZzaWmpsrKytGXLFiUmJlZVXAAAVC2mJEzxOmF4+umnT7t/2rRpKigoMB0QAAC48FTZ2ypvv/12vfzyy1V1OQAAqhbrMJhSZW+rzMjIkL+/f1VdDgCAKsVjleZ4XWEYPHiw23bjjTeqV69eGj58uP72t79VR4wAANQ6GzZs0HXXXaeIiAjZbDatWLHC7fhdd90lm83mtvXv399tzOHDhzVs2DA5HA41atRII0aMOGX6/9tvv9WVV14pf39/RUZGatasWafE8sYbb6hjx47y9/dXly5d9P7773t9P14nDMHBwW5bSEiI+vTpo/fff19Tp071OgAAAOqiwsJCdevWTfPnzz/jmP79++vAgQOu7T//+Y/b8WHDhmnr1q1KT0/XypUrtWHDBt1zzz2u4/n5+erXr5+ioqKUmZmp2bNna9q0aXrhhRdcYzZu3Khbb71VI0aM0Ndff61BgwZp0KBB2rJli1f349WURHl5uYYPH64uXbqocePGXn0QAAA16jw/JTFgwAANGDDgrGP8/PwUHh5+2mPbtm3T6tWr9d///lc9evSQJD377LO69tpr9a9//UsRERFavHixSkpK9PLLL8tut+tPf/qTsrKy9NRTT7kSizlz5qh///4aP368JGnGjBlKT0/XvHnztHDhwkrfj1cVBl9fX/Xr14+3UgIAap2qer11fn6+21ZcXHzOMa1bt06hoaHq0KGDxowZo19//dV1LCMjQ40aNXIlC5IUHx8vHx8fbdq0yTWmd+/estvtrjEJCQnKzs7WkSNHXGPi4+PdPjchIUEZGRlexer1lETnzp31008/eXsaAAB1QmRkpNvU/MyZM8/pOv3799drr72mNWvW6J///KfWr1+vAQMGqLy8XJKUk5Oj0NBQt3Pq1aunkJAQ5eTkuMaEhYW5jan42dOYiuOV5fVTEo899pgeeOABzZgxQzExMQoMDHQ77nA4vL0kAADnRxU86bB371637zo/P79zus7QoUNdf+7SpYu6du2qdu3aad26dbrmmmtMx1nVKp0wPProo7r//vt17bXXSpKuv/56tyWiDcOQzWZzZUYAAFxQqqiHweFwVMs/jtu2baumTZtqx44duuaaaxQeHq6DBw+6jSkrK9Phw4ddfQ/h4eHKzc11G1Pxs6cxZ+qdOJNKJwzTp0/X6NGj9cknn3j1AQAAwLN9+/bp119/VfPmzSVJcXFxOnr0qDIzMxUTEyPp5PuanE6nYmNjXWMeeughlZaWqn79+pKk9PR0dejQwfVwQlxcnNasWaNx48a5Pis9PV1xcXFexVfphMEwTqZWV111lVcfAADAheB8L9xUUFCgHTt2uH7etWuXsrKyFBISopCQEE2fPl1DhgxReHi4du7cqQkTJqh9+/ZKSEiQJHXq1En9+/fXqFGjtHDhQpWWlio5OVlDhw5VRESEJOm2227T9OnTNWLECE2cOFFbtmzRnDlz3F7jcN999+mqq67Sk08+qYEDB+r111/X5s2b3R69rAyvmh7P9pZKAAAuaOd5aejNmzfrkksu0SWXXCJJSk1N1SWXXKIpU6bI19dX3377ra6//npdfPHFGjFihGJiYvTpp5+69UQsXrxYHTt21DXXXKNrr71Wf/7zn92+6IODg/XRRx9p165diomJ0f33368pU6a4rdVw+eWXa8mSJXrhhRfUrVs3vfnmm1qxYoU6d+7s1f3YjIrSgQc+Pj4KDg72mDQcPnzYqwDMyM/PV3BwsProBtWz1T9vnwsAqBplRqnW6R3l5eVVW9N8xXfFReOfkK/fub/CoLy4SD/OfrBaY72QefWUxPTp0xUcHFxdsQAAUG14l4Q5XiUMQ4cOPeWZUAAAaoXzvNJjXVPpHgb6FwAAsC6vn5IAAKBWosJgSqUTBqfTWZ1xAABQrehhMMfrpaEBAKiVqDCY4vXLpwAAgPVQYQAAWAMVBlNIGAAAlkAPgzlMSQAAAI+oMAAArIEpCVNIGAAAlsCUhDlMSQAAAI+oMAAArIEpCVNIGAAA1kDCYApTEgAAwCMqDAAAS7D9tpk538pIGAAA1sCUhCkkDAAAS+CxSnPoYQAAAB5RYQAAWANTEqaQMAAArMPiX/pmMCUBAAA8osIAALAEmh7NIWEAAFgDPQymMCUBAAA8osIAALAEpiTMIWEAAFgDUxKmMCUBAAA8osIAALAEpiTMIWEAAFgDUxKmkDAAAKyBhMEUehgAAIBHVBgAAJZAD4M5JAwAAGtgSsIUpiQAAIBHVBgAAJZgMwzZjHMvE5g5ty4gYQAAWANTEqYwJQEAADyiwgAAsASekjCHhAEAYA1MSZjClAQAAPCIhAEAYAkVUxJmNm9s2LBB1113nSIiImSz2bRixQq344ZhaMqUKWrevLkCAgIUHx+vH3/80W3M4cOHNWzYMDkcDjVq1EgjRoxQQUGB25hvv/1WV155pfz9/RUZGalZs2adEssbb7yhjh07yt/fX126dNH777/v3c2IhAEAYBVGFWxeKCwsVLdu3TR//vzTHp81a5bmzp2rhQsXatOmTQoMDFRCQoKKiopcY4YNG6atW7cqPT1dK1eu1IYNG3TPPfe4jufn56tfv36KiopSZmamZs+erWnTpumFF15wjdm4caNuvfVWjRgxQl9//bUGDRqkQYMGacuWLV7dj80wau+Dpfn5+QoODlYf3aB6tvo1HQ4AwEtlRqnW6R3l5eXJ4XBUy2dUfFfE3PK4fO3+53yd8pIiZS596JxitdlsWr58uQYNGiTpZHUhIiJC999/vx544AFJUl5ensLCwpSWlqahQ4dq27Ztio6O1n//+1/16NFDkrR69Wpde+212rdvnyIiIrRgwQI99NBDysnJkd1ulyRNmjRJK1as0Pbt2yVJt9xyiwoLC7Vy5UpXPL169VL37t21cOHCSt8DFQYAALyQn5/vthUXF3t9jV27diknJ0fx8fGufcHBwYqNjVVGRoYkKSMjQ40aNXIlC5IUHx8vHx8fbdq0yTWmd+/ermRBkhISEpSdna0jR464xvz+cyrGVHxOZZEwAACsoYqmJCIjIxUcHOzaZs6c6XUoOTk5kqSwsDC3/WFhYa5jOTk5Cg0NdTter149hYSEuI053TV+/xlnGlNxvLJ4rBIAYBlVsZbC3r173aYk/Pz8zF+0FqDCAACAFxwOh9t2LglDeHi4JCk3N9dtf25urutYeHi4Dh486Ha8rKxMhw8fdhtzumv8/jPONKbieGWRMAAArMEwzG9VpE2bNgoPD9eaNWtc+/Lz87Vp0ybFxcVJkuLi4nT06FFlZma6xqxdu1ZOp1OxsbGuMRs2bFBpaalrTHp6ujp06KDGjRu7xvz+cyrGVHxOZZEwAAAs4Xyvw1BQUKCsrCxlZWVJOtnomJWVpT179shms2ncuHF67LHH9O677+q7777TnXfeqYiICNeTFJ06dVL//v01atQoffnll/r888+VnJysoUOHKiIiQpJ02223yW63a8SIEdq6dauWLl2qOXPmKDU11RXHfffdp9WrV+vJJ5/U9u3bNW3aNG3evFnJycle3Q89DAAAVIPNmzerb9++rp8rvsQTExOVlpamCRMmqLCwUPfcc4+OHj2qP//5z1q9erX8/f/v0c/FixcrOTlZ11xzjXx8fDRkyBDNnTvXdTw4OFgfffSRkpKSFBMTo6ZNm2rKlCluazVcfvnlWrJkiR5++GE9+OCDuuiii7RixQp17tzZq/thHQYAQI05n+sw9BjymOrVP/d1GMpKi7T5rYerNdYLGRUGAIAl2JwnNzPnWxk9DAAAwCMqDFDn2AL9f/ce0kVdjqtJeJmm3d1aGauDJUm+9QzdNfGAel59TM2jSlSY76OvPw3SS0801+Hc/5sGmpa2S+3+dEKNmpTpWJ7vyTGPu48BLhRNwks14qH96tn3mPwCnNq/209PpkTqx28bSJL8G5RrxEMHFJeQL0fjMuXsteudl5pq1b+b1nDkMIXXW5tCwgD5N3Dqp63++vA/IZr68m63Y34BTrXvckJLngnTT9/7q2FwucY8ul/T03Zp7ICLXeO++byhXp8bqsO59dW0ealGTdmvR17crZTrLzrPdwOcXcPgMj31zo/6dmNDPXx7Wx391Vct2paoIM/XNeZv0/ar+xUFmjW2lXL32nXpVcc0duY+/ZpbX198FFyD0cOMc3nS4Y/nW1mNJgwbNmzQ7NmzlZmZqQMHDri9mAPnz+ZPHNr8yekbeI4f89Xkoe3c9s1/qIWe/eBHNWtRokO/nFy/fPmLzVzHD/5i19J5oZr68m751jNUXmarvuABL92cdFD/22/XkymtXPty97ovvBPd47jS3wjRtxkNJUkfLG6igXf8qg7dj5Mw1GZm11Kovc8IVIka7WHw9OpPXJgCHeVyOqXC3/2L7PeCGpXp6sFH9P3mBiQLuOD06pevH74J0EPP79bSb7dq/kfZGnDbr25jvt/cQL365alJeKkkQ90uL1CLtsXKXB9UM0EDF4AarTAMGDBAAwYMqPT44uJit7eC5efnV0dYOIv6fk6NeOiA1q1opOMF7gnDiIf26/rhv8q/gVPfb26gKYltaihK4MyatyrRX+/8VW+/0EyvPxuqi7ud0JgZv6i01KaP3wiRJD33cAvdN2uflnz1vcpKJafTpjnjW2rLpoY1HD3MYErCnFr1lMTMmTPd3hAWGRlZ0yFZim89Qw89/7Nkk56d1PKU428sCNW9/S7W5KFt5XRK4+fskeW7hHDBsflIO7YE6JV/NNfOLQ30weIm+mDJySmHCjfc/T91jDmuKYmtldz/Yr34aISSnvhFl1x5rAYjh2lV9LZKq6pVCcPkyZOVl5fn2vbu3VvTIVnGyWRht8JalGjy0LanVBckKf9wPf3yk5++2hCkmWOiFBt/TJ1ijtdAtMCZHT5YTz//4L54z94f/RTaokSSZPd36q5JOXphWoQ2pQdr17YAvftKU61/t5FuGn2oJkIGLgi16ikJPz8/y7xG9EJSkSy0aFOiCTe107Ejnn9tbL+lovXtFk/JccH5/r+BimxX7LavRdtiHfytgbdePUP17Yacf1ikx1ku2Xz4fa7NmJIwp1YlDKge/g3KFdGmxPVzeGSJ2v7phI4d9dXh3Pp65MXdat/lhKbc2UY+voYaNzv5VrRjR31VVuqjDpcUqkP3E9ryZaAKjvqqeetiJU7I0f5ddm3LbFBTtwWc1tsvNNPT7/6ooWNzteG9RupwyXFde/thPTP+5DTb8QJffbMxUKMeOaCSIh/l7quvrnGFir/piF6YHlHD0cMUnpIwhYQBurjbCc1+a6fr59HT90uSPlraWP/vyXDFJZxsLl3w8Q9u540f0k7fZjRU8QkfXTEgT3fcnyP/Bk4dPlhfmz8J0uNzwlRaUqtmvWABP3zTQI+OaKPhkw9oWEqucvbatXBKhD5Z3tg1ZuaYKN394AFNnPezghqV6+AvdqX9s7lWvtakBiMHalaNJgwFBQXasWOH6+eKV3+GhISoVatWZzkTVenbjIZKiOh2xuNnOyZJu7cHaOLN7c46BriQbPrYoU0fn/nlQUcO1XdbpwF1A1MS5tRowuDp1Z8AAFQZloY2pUYThj59+qgWv10bAADLoIcBAGAJTEmYQ8IAALAGp3FyM3O+hZEwAACsgR4GU3jmDQAAeESFAQBgCTaZ7GGoskhqJxIGAIA1sNKjKUxJAAAAj6gwAAAsgccqzSFhAABYA09JmMKUBAAA8IgKAwDAEmyGIZuJxkUz59YFJAwAAGtw/raZOd/CmJIAAAAeUWEAAFgCUxLmkDAAAKyBpyRMIWEAAFgDKz2aQg8DAADwiAoDAMASWOnRHBIGAIA1MCVhClMSAADAIyoMAABLsDlPbmbOtzISBgCANTAlYQpTEgAAwCMqDAAAa2DhJlNIGAAAlsDS0OYwJQEAADyiwgAAsAaaHk0hYQAAWIMhycyjkdbOF5iSAABYQ0UPg5nNG9OmTZPNZnPbOnbs6DpeVFSkpKQkNWnSRA0bNtSQIUOUm5vrdo09e/Zo4MCBatCggUJDQzV+/HiVlZW5jVm3bp0uvfRS+fn5qX379kpLSzvnv6OzIWEAAKCa/OlPf9KBAwdc22effeY6lpKSovfee09vvPGG1q9fr/3792vw4MGu4+Xl5Ro4cKBKSkq0ceNGvfrqq0pLS9OUKVNcY3bt2qWBAweqb9++ysrK0rhx4zRy5Eh9+OGHVX4vTEkAAKzBkMkeBu9PqVevnsLDw0/Zn5eXp5deeklLlizR1VdfLUl65ZVX1KlTJ33xxRfq1auXPvroI33//ff6+OOPFRYWpu7du2vGjBmaOHGipk2bJrvdroULF6pNmzZ68sknJUmdOnXSZ599pqeffloJCQnnfq+nQYUBAGANFU2PZjZJ+fn5bltxcfEZP/LHH39URESE2rZtq2HDhmnPnj2SpMzMTJWWlio+Pt41tmPHjmrVqpUyMjIkSRkZGerSpYvCwsJcYxISEpSfn6+tW7e6xvz+GhVjKq5RlUgYAADwQmRkpIKDg13bzJkzTzsuNjZWaWlpWr16tRYsWKBdu3bpyiuv1LFjx5STkyO73a5GjRq5nRMWFqacnBxJUk5OjluyUHG84tjZxuTn5+vEiRNVcbsuTEkAAKzBKclm8nxJe/fulcPhcO328/M77fABAwa4/ty1a1fFxsYqKipKy5YtU0BAgIlAagYVBgCAJVTVUxIOh8NtO1PC8EeNGjXSxRdfrB07dig8PFwlJSU6evSo25jc3FxXz0N4ePgpT01U/OxpjMPhqPKkhIQBAIDzoKCgQDt37lTz5s0VExOj+vXra82aNa7j2dnZ2rNnj+Li4iRJcXFx+u6773Tw4EHXmPT0dDkcDkVHR7vG/P4aFWMqrlGVSBgAANZQRU2PlfXAAw9o/fr12r17tzZu3Kgbb7xRvr6+uvXWWxUcHKwRI0YoNTVVn3zyiTIzMzV8+HDFxcWpV69ekqR+/fopOjpad9xxh7755ht9+OGHevjhh5WUlOSqaowePVo//fSTJkyYoO3bt+u5557TsmXLlJKSUuV/ffQwAACs4TwvDb1v3z7deuut+vXXX9WsWTP9+c9/1hdffKFmzZpJkp5++mn5+PhoyJAhKi4uVkJCgp577jnX+b6+vlq5cqXGjBmjuLg4BQYGKjExUY8++qhrTJs2bbRq1SqlpKRozpw5atmypRYtWlTlj1RKks0wau/i2Pn5+QoODlYf3aB6tvo1HQ4AwEtlRqnW6R3l5eW5NRJWpYrvimuiH1A938r1G5xOWXmx1nz/r2qN9UJGhQEAYA28fMoUEgYAgDVU0WOVVkXCAACwhHN5gdQfz7cynpIAAAAeUWEAAFgDPQymkDAAAKzBaUg2E1/6TmsnDExJAAAAj6gwAACsgSkJU0gYAAAWYTJhkLUTBqYkAACAR1QYAADWwJSEKSQMAABrcBoyNa3AUxIAAABnR4UBAGANhvPkZuZ8CyNhAABYAz0MppAwAACsgR4GU+hhAAAAHlFhAABYA1MSppAwAACswZDJhKHKIqmVmJIAAAAeUWEAAFgDUxKmkDAAAKzB6ZRkYi0Fp7XXYWBKAgAAeESFAQBgDUxJmELCAACwBhIGU5iSAAAAHlFhAABYA0tDm0LCAACwBMNwyjDxxkkz59YFJAwAAGswDHNVAnoYAAAAzo4KAwDAGgyTPQwWrzCQMAAArMHplGwm+hAs3sPAlAQAAPCICgMAwBqYkjCFhAEAYAmG0ynDxJSE1R+rZEoCAAB4RIUBAGANTEmYQsIAALAGpyHZSBjOFVMSAADAIyoMAABrMAxJZtZhsHaFgYQBAGAJhtOQYWJKwiBhAADAAgynzFUYeKwSAADgrKgwAAAsgSkJc0gYAADWwJSEKbU6YajI9spUamotDgBAzShTqaTz8693s98VFbFaVa1OGI4dOyZJ+kzv13AkAAAzjh07puDg4Gq5tt1uV3h4uD7LMf9dER4eLrvdXgVR1T42oxZPyjidTu3fv19BQUGy2Ww1HY4l5OfnKzIyUnv37pXD4ajpcIAqxe/3+WcYho4dO6aIiAj5+FRfH35RUZFKSkpMX8dut8vf378KIqp9anWFwcfHRy1btqzpMCzJ4XDwP1TUWfx+n1/VVVn4PX9/f8t+0VcVHqsEAAAekTAAAACPSBjgFT8/P02dOlV+fn41HQpQ5fj9Bs6sVjc9AgCA84MKAwAA8IiEAQAAeETCAAAAPCJhAAAAHpEwoNLmz5+v1q1by9/fX7Gxsfryyy9rOiSgSmzYsEHXXXedIiIiZLPZtGLFipoOCbjgkDCgUpYuXarU1FRNnTpVX331lbp166aEhAQdPHiwpkMDTCssLFS3bt00f/78mg4FuGDxWCUqJTY2Vj179tS8efMknXyPR2RkpMaOHatJkybVcHRA1bHZbFq+fLkGDRpU06EAFxQqDPCopKREmZmZio+Pd+3z8fFRfHy8MjIyajAyAMD5QsIAj/73v/+pvLxcYWFhbvvDwsKUk5NTQ1EBAM4nEgYAAOARCQM8atq0qXx9fZWbm+u2Pzc3V+Hh4TUUFQDgfCJhgEd2u10xMTFas2aNa5/T6dSaNWsUFxdXg5EBAM6XejUdAGqH1NRUJSYmqkePHrrsssv0zDPPqLCwUMOHD6/p0ADTCgoKtGPHDtfPu3btUlZWlkJCQtSqVasajAy4cPBYJSpt3rx5mj17tnJyctS9e3fNnTtXsbGxNR0WYNq6devUt2/fU/YnJiYqLS3t/AcEXIBIGAAAgEf0MAAAAI9IGAAAgEckDAAAwCMSBgAA4BEJAwAA8IiEAQAAeETCAAAAPCJhAAAAHpEwACbdddddGjRokOvnPn36aNy4cec9jnXr1slms+no0aNnHGOz2bRixYpKX3PatGnq3r27qbh2794tm82mrKwsU9cBULNIGFAn3XXXXbLZbLLZbLLb7Wrfvr0effRRlZWVVftnv/3225oxY0alxlbmSx4ALgS8fAp1Vv/+/fXKK6+ouLhY77//vpKSklS/fn1Nnjz5lLElJSWy2+1V8rkhISFVch0AuJBQYUCd5efnp/DwcEVFRWnMmDGKj4/Xu+++K+n/phEef/xxRUREqEOHDpKkvXv36uabb1ajRo0UEhKiG264Qbt373Zds7y8XKmpqWrUqJGaNGmiCRMm6I+vY/njlERxcbEmTpyoyMhI+fn5qX379nrppZe0e/du1wuPGjduLJvNprvuukvSydeHz5w5U23atFFAQIC6deumN9980+1z3n//fV188cUKCAhQ37593eKsrIkTJ+riiy9WgwYN1LZtWz3yyCMqLS09Zdzzzz+vyMhINWjQQDfffLPy8vLcji9atEidOnWSv7+/OnbsqOeee87rWABc2EgYYBkBAQEqKSlx/bxmzRplZ2crPT1dK1euVGlpqRISEhQUFKRPP/1Un3/+uRo2bKj+/fu7znvyySeVlpaml19+WZ999pkOHz6s5cuXn/Vz77zzTv3nP//R3LlztW3bNj3//PNq2LChIiMj9dZbb0mSsrOzdeDAAc2ZM0eSNHPmTL322mtauHChtm7dqpSUFN1+++1av369pJOJzeDBg3XdddcpKytLI0eO1KRJk7z+OwkKClJaWpq+//57zZkzRy+++KKefvpptzE7duzQsmXL9N5772n16tX6+uuvde+997qOL168WFOmTNHjjz+ubdu26YknntAjjzyiV1991et4AFzADKAOSkxMNG644QbDMAzD6XQa6enphp+fn/HAAw+4joeFhRnFxcWuc/79738bHTp0MJxOp2tfcXGxERAQYHz44YeGYRhG8+bNjVmzZrmOl5aWGi1btnR9lmEYxlVXXWXcd999hmEYRnZ2tiHJSE9PP22cn3zyiSHJOHLkiGtfUVGR0aBBA2Pjxo1uY0eMGGHceuuthmEYxuTJk43o6Gi34xMnTjzlWn8kyVi+fPkZj8+ePduIiYlx/Tx16lTD19fX2Ldvn2vfBx98YPj4+BgHDhwwDMMw2rVrZyxZssTtOjNmzDDi4uIMwzCMXbt2GZKMr7/++oyfC+DCRw8D6qyVK1eqYcOGKi0tldPp1G233aZp06a5jnfp0sWtb+Gbb77Rjh07FBQU5HadoqIi7dy5U3l5eTpw4IBiY2Ndx+rVq6cePXqcMi1RISsrS76+vrrqqqsqHfeOHTt0/Phx/eUvf3HbX1JSoksuuUSStG3bNrc4JCkuLq7Sn1Fh6dKlmjt3rnbu3KmCggKVlZXJ4XC4jWnVqpVatGjh9jlOp1PZ2dkKCgrSzp07NWLECI0aNco1pqysTMHBwV7HA+DCRcKAOqtv375asGCB7Ha7IiIiVK+e+697YGCg288FBQWKiYnR4sWLT7lWs2bNzimGgIAAr88pKCiQJK1atcrti1o62ZdRVTIyMjRs2DBNnz5dCQkJCg4O1uuvv64nn3zS61hffPHFUxIYX1/fKosVQM0jYUCdFRgYqPbt21d6/KWXXqqlS5cqNDT0lH9lV2jevLk2bdqk3r17Szr5L+nMzExdeumlpx3fpUsXOZ1OrV+/XvHx8accr6hwlJeXu/ZFR0fLz89Pe/bsOWNlolOnTq4GzgpffPGF55v8nY0bNyoqKkoPPfSQa9/PP/98yrg9e/Zo//79ioiIcH2Oj4+POnTooLCwMEVEROinn37SsGHDvPp8ALULTY/Ab4YNG6amTZvqhhtu0Keffqpdu3Zp3bp1+vvf/659+/ZJku677z794x//0IoVK7R9+3bde++9Z11DoXXr1kpMTNTdd9+tFStWuK65bNkySVJUVJRsNptWrlypQ4cOqaCgQEFBQXrggQeUkpKiV199VTt37tRXX32lZ5991tVIOHr0aP34448aP368srOztWTJEqWlpXl1vxdddJH27Nmj119/XTt37tTcuXNP28Dp7++vxMREffPNN/r000/197//XTfffLPCw8MlSdOnT9fMmTM1d+5c/fDDD/ruu+/0yiuv6KmnnvIqHgAXNhIG4DcNGjTQhg0b1KpVKw0ePFidOnXSiBEjVFRU5Ko43H///brjjjuUmJiouLg4BQUF6cYbbzzrdRcsWKCbbrpJ9957rzp27KhRo0apsLBQktSiRQtNnz5dkyZNUlhYmJKTkyVJM2bM0COPPKKZM2eqU6dO6t+/v1atWqU2bdpIOtlX8NZbb2nFihXq1q2bFi5cqCeeeMKr+73++uuVkpKi5ORkde/eXRs3btQjjzxyyrj27dtr8ODBuvbaa9WvXz917drV7bHJkSNHatGiRXrllVfUpUsXXXXVVUpLS3PFCqBusBln6tYCAAD4DRUGAADgEQkDAADwiIQBAAB4RMIAAAA8ImEAAAAekTAAAACPSBgAAIBHJAwAAMAjEgYAAOARCQMAAPCIhAEAAHj0/wNQxBcpkMw97QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "random_forest = RandomForestClassifier()\n",
    "random_forest.fit(X_train, y_train)\n",
    "evaluate(random_forest, verbose = True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "70b8e026",
   "metadata": {},
   "outputs": [],
   "source": [
    "output = pd.DataFrame(columns=['model', 'accuracy', 'precision', 'recall'])\n",
    "\n",
    "def train_model(model):\n",
    "    global output\n",
    "    model.fit( X_train, y_train) #X_train and y_train are used globally\n",
    "    metrics = evaluate(model)\n",
    "    metrics[\"model\"] = f\"{model}\"\n",
    "    output = output.append(metrics, ignore_index=True)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "e06af131",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model: RandomForestClassifier()\n",
      "Training model: GradientBoostingClassifier()\n",
      "Training model: ExtraTreesClassifier()\n",
      "Training model: KNeighborsClassifier()\n"
     ]
    }
   ],
   "source": [
    "for model in [RandomForestClassifier(), GradientBoostingClassifier(), ExtraTreesClassifier(), KNeighborsClassifier()]:\n",
    "    print (f\"Training model: {model}\" )\n",
    "    train_model(model)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "e17583e5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RandomForestClassifier()</td>\n",
       "      <td>0.995807</td>\n",
       "      <td>0.741935</td>\n",
       "      <td>0.361257</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GradientBoostingClassifier()</td>\n",
       "      <td>0.995778</td>\n",
       "      <td>0.729167</td>\n",
       "      <td>0.366492</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ExtraTreesClassifier()</td>\n",
       "      <td>0.995749</td>\n",
       "      <td>0.721649</td>\n",
       "      <td>0.366492</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>KNeighborsClassifier()</td>\n",
       "      <td>0.994802</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>0.078534</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          model  accuracy  precision    recall\n",
       "0      RandomForestClassifier()  0.995807   0.741935  0.361257\n",
       "1  GradientBoostingClassifier()  0.995778   0.729167  0.366492\n",
       "2        ExtraTreesClassifier()  0.995749   0.721649  0.366492\n",
       "3        KNeighborsClassifier()  0.994802   0.750000  0.078534"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b824bf4a",
   "metadata": {},
   "source": [
    "If we consider __Recall__ `GradientBoostinClassifier` is working better than other models in default configuration. Lets improve the function to automatically determine the best model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "fb3b5edc",
   "metadata": {},
   "outputs": [],
   "source": [
    "output = pd.DataFrame(columns=['model','accuracy','precision','recall'])\n",
    "\n",
    "def train_model(model):\n",
    "    global output\n",
    "    model.fit( X_train, y_train) #X_train and y_train are used globally\n",
    "    metrics = evaluate(model)\n",
    "    metrics[\"model\"] = f\"{model}\"\n",
    "    output = output.append(metrics, ignore_index=True)\n",
    "\n",
    "def select_best(model_list):\n",
    "    for model in model_list:\n",
    "        print (\"Training Model: \", model)\n",
    "        train_model(model)\n",
    "    output.recall = output.recall.astype('float64')\n",
    "    print (f\"The best model is \\n { output.loc[[output.recall.argmax()]]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "5326b374",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Model:  RandomForestClassifier()\n",
      "Training Model:  GradientBoostingClassifier()\n",
      "Training Model:  ExtraTreesClassifier()\n",
      "Training Model:  KNeighborsClassifier()\n",
      "Training Model:  LogisticRegression()\n",
      "Training Model:  SVC()\n",
      "The best model is \n",
      "                     model  accuracy  precision    recall\n",
      "2  ExtraTreesClassifier()  0.995749   0.704762  0.387435\n"
     ]
    }
   ],
   "source": [
    "select_best([\n",
    "    RandomForestClassifier(),\n",
    "    GradientBoostingClassifier(), \n",
    "    ExtraTreesClassifier(), \n",
    "    KNeighborsClassifier(),\n",
    "    LogisticRegression(),\n",
    "    SVC() \n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "da7822d8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RandomForestClassifier()</td>\n",
       "      <td>0.995864</td>\n",
       "      <td>0.752688</td>\n",
       "      <td>0.366492</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GradientBoostingClassifier()</td>\n",
       "      <td>0.995778</td>\n",
       "      <td>0.729167</td>\n",
       "      <td>0.366492</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ExtraTreesClassifier()</td>\n",
       "      <td>0.995749</td>\n",
       "      <td>0.704762</td>\n",
       "      <td>0.387435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>KNeighborsClassifier()</td>\n",
       "      <td>0.994802</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>0.078534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>LogisticRegression()</td>\n",
       "      <td>0.995290</td>\n",
       "      <td>0.935484</td>\n",
       "      <td>0.151832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>SVC()</td>\n",
       "      <td>0.994974</td>\n",
       "      <td>0.944444</td>\n",
       "      <td>0.089005</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          model  accuracy  precision    recall\n",
       "0      RandomForestClassifier()  0.995864   0.752688  0.366492\n",
       "1  GradientBoostingClassifier()  0.995778   0.729167  0.366492\n",
       "2        ExtraTreesClassifier()  0.995749   0.704762  0.387435\n",
       "3        KNeighborsClassifier()  0.994802   0.750000  0.078534\n",
       "4          LogisticRegression()  0.995290   0.935484  0.151832\n",
       "5                         SVC()  0.994974   0.944444  0.089005"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "5e8117cc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ExtraTreesClassifier()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ExtraTreesClassifier</label><div class=\"sk-toggleable__content\"><pre>ExtraTreesClassifier()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "ExtraTreesClassifier()"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = ExtraTreesClassifier()\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2ccc6c2",
   "metadata": {},
   "source": [
    "### Saving the Model\n",
    "\n",
    "So you're happy with your model? Now you need to save it.\n",
    "\n",
    "Once it's saved, we can share it with our team, load it back in to make predictions in some other system, or load it back in to compare with new models that you train, for example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "4caa50f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['finalized_model.sav']"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import joblib \n",
    "filename = 'finalized_model.pkl'\n",
    "joblib.dump(model, filename)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72c99f5c",
   "metadata": {},
   "source": [
    "## Key Takeaways\n",
    "- The process for model training and evaluation goes as follows\n",
    "    - Train a model on the training set\n",
    "    - Evaluate the model on the validation set\n",
    "    - Repeat for as many variations of model as you can\n",
    "    - Select the best model based on its performance on the validation set\n",
    "    - Report the performance of the model using the test set, but do not use this to make any decisions about the model!\n",
    "- Never use the test set to choose between models. The moment you pick a model based on test set performance, you're showing it the answers!\n",
    "- Both the validatoin and test set are a measure of how well the model will generalise to unseen examples\n",
    "- The validation set is used to compare different models and choose between them once they've been trained\n",
    "- The difference \n",
    "- This demo barely scratches the surface. Going forward, you could explore:\n",
    "    - the settings of the different models we've experimented with, which are known as hyperparameters\n",
    "    - other machine learning models, such as deep learning models like neural networks\n",
    "    - other machine learning libraries, like Apache Spark's MLLib\n",
    "    - other preprocessing techniques\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.5 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "vscode": {
   "interpreter": {
    "hash": "ffef1fb3247e42ae9cf3614f3519d4998b3b95643236a5d32641564963f5a3b8"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}