assignment4 - Classifier Model Evaluation and Selection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "69d305e9",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e532a7b0bbaf722cc8e5496bfde02472",
     "grade": false,
     "grade_id": "cell-24e3b5e89f5669c5",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "---\n",
    "\n",
    "_You are currently looking at **version 1.0** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the Jupyter Notebook FAQ course resource._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0be71ec1",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "2ddd869ec642dbe6404672b445fd4dd1",
     "grade": false,
     "grade_id": "cell-6757b7ecd16ad875",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Assignment 4 - Predicting and understanding viewer engagement with educational videos \n",
    "\n",
    "With the accelerating popularity of online educational experiences, the role of online lectures and other educational video continues to increase in scope and importance. Open access educational repositories such as <a href=\"http://videolectures.net/\">videolectures.net</a>, as well as Massive Open Online Courses (MOOCs) on platforms like Coursera, have made access to many thousands of lectures and tutorials an accessible option for millions of people around the world. Yet this impressive volume of content has also led to a challenge in how to find, filter, and match these videos with learners. This assignment gives you an example of how machine learning can be used to address part of that challenge.\n",
    "\n",
    "## About the prediction problem\n",
    "\n",
    "One critical property of a video is engagement: how interesting or \"engaging\" it is for viewers, so that they decide to keep watching. Engagement is critical for learning, whether the instruction is coming from a video or any other source. There are many ways to define engagement with video, but one common approach is to estimate it by measuring how much of the video a user watches. If the video is not interesting and does not engage a viewer, they will typically abandon it quickly, e.g. only watch 5 or 10% of the total. \n",
    "\n",
    "A first step towards providing the best-matching educational content is to understand which features of educational material make it engaging for learners in general. This is where predictive modeling can be applied, via supervised machine learning. For this assignment, your task is to predict how engaging an educational video is likely to be for viewers, based on a set of features extracted from the video's transcript, audio track, hosting site, and other sources.\n",
    "\n",
    "We chose this prediction problem for several reasons:\n",
    "\n",
    "* It combines a variety of features derived from a rich set of resources connected to the original data;\n",
    "* The manageable dataset size means the dataset and supervised models for it can be easily explored on a wide variety of computing platforms;\n",
    "* Predicting popularity or engagement for a media item, especially combined with understanding which features contribute to its success with viewers, is a fun problem but also a practical representative application of machine learning in a number of business and educational sectors.\n",
    "\n",
    "\n",
    "## About the dataset\n",
    "\n",
    "We extracted training and test datasets of educational video features from the VLE Dataset put together by researcher Sahan Bulathwela at University College London. \n",
    "\n",
    "We provide you with two data files for use in training and validating your models: train.csv and test.csv. Each row in these two files corresponds to a single educational video, and includes information about diverse properties of the video content as described further below. The target variable is `engagement` which was defined as True if the median percentage of the video watched across all viewers was at least 30%, and False otherwise.\n",
    "\n",
    "Note: Any extra variables that may be included in the training set are simply for your interest if you want an additional source of data for visualization, or to enable unsupervised and semi-supervised approaches. However, they are not included in the test set and thus cannot be used for prediction. **Only the data already included in your Coursera directory can be used for training the model for this assignment.**\n",
    "\n",
    "For this final assignment, you will bring together what you've learned across all four weeks of this course, by exploring different prediction models for this new dataset. In addition, we encourage you to apply what you've learned about model selection to do hyperparameter tuning using training/validation splits of the training data, to optimize the model and further increase its performance. In addition to a basic evaluation of model accuracy, we've also provided a utility function to visualize which features are most and least contributing to the overall model performance.\n",
    "\n",
    "**File descriptions** \n",
    "    assets/train.csv - the training set (Use only this data for training your model!)\n",
    "    assets/test.csv - the test set\n",
    "<br>\n",
    "\n",
    "**Data fields**\n",
    "\n",
    "train.csv & test.csv:\n",
    "\n",
    "    title_word_count - the number of words in the title of the video.\n",
    "    \n",
    "    document_entropy - a score indicating how varied the topics are covered in the video, based on the transcript. Videos with smaller entropy scores will tend to be more cohesive and more focused on a single topic.\n",
    "    \n",
    "    freshness - The number of days elapsed between 01/01/1970 and the lecture published date. Videos that are more recent will have higher freshness values.\n",
    "    \n",
    "    easiness - A text difficulty measure applied to the transcript. A lower score indicates more complex language used by the presenter.\n",
    "    \n",
    "    fraction_stopword_presence - A stopword is a very common word like 'the' or 'and'. This feature computes the fraction of all words that are stopwords in the video lecture transcript.\n",
    "    \n",
    "    speaker_speed - The average speaking rate in words per minute of the presenter in the video.\n",
    "    \n",
    "    silent_period_rate - The fraction of time in the lecture video that is silence (no speaking).\n",
    "    \n",
    "train.csv only:\n",
    "    \n",
    "    engagement - Target label for training. True if learners watched a substantial portion of the video (see description), or False otherwise.\n",
    "    \n",
    "\n",
    "## Evaluation\n",
    "\n",
    "Your predictions will be given as the probability that the corresponding video will be engaging to learners.\n",
    "\n",
    "The evaluation metric for this assignment is the Area Under the ROC Curve (AUC). \n",
    "\n",
    "Your grade will be based on the AUC score computed for your classifier. A model with an AUC (area under ROC curve) of at least 0.8 passes this assignment, and over 0.85 will receive full points.\n",
    "___\n",
    "\n",
    "For this assignment, create a function that trains a model to predict significant learner engagement with a video using `asset/train.csv`. Using this model, return a Pandas Series object of length 2309 with the data being the probability that each corresponding video from `readonly/test.csv` will be engaging (according to a model learned from the 'engagement' label in the training set), and the video index being in the `id` field.\n",
    "\n",
    "Example:\n",
    "\n",
    "    id\n",
    "       9240    0.401958\n",
    "       9241    0.105928\n",
    "       9242    0.018572\n",
    "                 ...\n",
    "       9243    0.208567\n",
    "       9244    0.818759\n",
    "       9245    0.018528\n",
    "             ...\n",
    "       Name: engagement, dtype: float32\n",
    "       \n",
    "### Hints\n",
    "\n",
    "* Make sure your code is working before submitting it to the autograder.\n",
    "\n",
    "* Print out and check your result to see whether there is anything weird (e.g., all probabilities are the same).\n",
    "\n",
    "* Generally the total runtime should be less than 10 mins. \n",
    "\n",
    "* Try to avoid global variables. If you have other functions besides engagement_model, you should move those functions inside the scope of engagement_model.\n",
    "\n",
    "* Be sure to first check the pinned threads in Week 4's discussion forum if you run into a problem you can't figure out.\n",
    "\n",
    "### Extensions\n",
    "\n",
    "* If this prediction task motivates you to explore further, you can find more details here on the original VLE dataset and others related to video engagement: https://github.com/sahanbull/VLE-Dataset\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1218318f",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b7dc38d94db79fb7160854a629c825a8",
     "grade": false,
     "grade_id": "cell-2c0cf4e0ffe5f447",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "np.random.seed(0)   # Do not change this value: required to be compatible with solutions generated by the autograder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "895aa1ca-8965-4ce8-a4c3-9deb700d4319",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures, MinMaxScaler, MaxAbsScaler, StandardScaler\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score, validation_curve\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score, precision_score, recall_score, f1_score\n",
    "from sklearn.metrics import classification_report, precision_recall_curve, roc_curve, auc, mean_squared_error\n",
    "from sklearn.metrics import roc_auc_score, r2_score, SCORERS\n",
    "from sklearn.dummy import DummyClassifier, DummyRegressor\n",
    "from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor\n",
    "from sklearn.linear_model import LinearRegression, Ridge, RidgeClassifier, Lasso, LogisticRegression\n",
    "from sklearn.svm import LinearSVC, SVC\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier\n",
    "from sklearn.neural_network import MLPClassifier, MLPRegressor\n",
    "from math import sqrt\n",
    "from adspy_shared_utilities import plot_feature_importances\n",
    "\n",
    "Scalers = [MinMaxScaler() ,MaxAbsScaler(), StandardScaler()]\n",
    "DummyClass_Strategies = ['most_frequent', 'prior', 'stratified', 'uniform', 'constant']\n",
    "DummyClass_Strategies.pop()\n",
    "LogReg_Solvers = ['lbfgs', 'liblinear', 'newton-cg', 'newton-cholesky', 'sag', 'saga']\n",
    "Ridge_Solver = ['lbfgs', 'cholesky', 'sparse_cg']\n",
    "SVM_Kernels = ['linear','polynomial','rbf','sigmoid']\n",
    "SCORERS = sorted(list(SCORERS.keys()))\n",
    "SCORERS_subset = ['precision','recall','f1','roc_auc']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "678c1b3e-2218-4d06-ac99-40da1d998fd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Explore the data a bit\n",
    "# Input data, both training/validation and test sets\n",
    "trainval_df = pd.read_csv('assets/train.csv')\n",
    "test_df = pd.read_csv('assets/test.csv')\n",
    "# Break training/validation sets into targer and features\n",
    "trainval_y = trainval_df['engagement']\n",
    "trainval_X = trainval_df.loc[:, trainval_df.columns != 'engagement'].set_index('id')\n",
    "features = list(trainval_X.columns)\n",
    "X_test = test_df.set_index('id')\n",
    "X_test_ids = X_test.index.tolist()\n",
    "# Binarize the target values for the training and validation sets\n",
    "trainval_y = trainval_y.astype(str)\n",
    "trainval_y.replace(to_replace={'False':0,'True':1}, inplace=True)\n",
    "\n",
    "# Create training and validation sets for both features and targets\n",
    "X_train, X_val, y_train, y_val = train_test_split(trainval_X, trainval_y)\n",
    "\n",
    "# Define the scaler, then fit to X_train and transform X_train, X_val, and X_test\n",
    "scaler = MinMaxScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_val = scaler.transform(X_val)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bec2045e-d83c-4824-80ab-d1b89675fbe0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tree Validation set AUC:  0.8526580503412106\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAy4AAAEmCAYAAACEf8TvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAAxOAAAMTgF/d4wjAABGP0lEQVR4nO3deVxV5d7//9cGNHKqFKckRdM0hs0WCJQASVGPmphDZqGRVsaxbr/mkJZmSlZa5sn0a1aeY6mYDU6pHe8TKWk5pCdxnhWTnHAiEJTp+v3Rz/WNnLYpsNX38/Hgce+1rrWu9Vn7uive51rXwmaMMYiIiIiIiLgwt7IuQERERERE5EoUXERERERExOUpuIiIiIiIiMtTcBEREREREZen4CIiIiIiIi5PwUVERERERFyeR1kXIFIabrvtNqpXr17WZYiIiIjIJWRkZHDu3LlLtiu4yC2hevXqpKenl3UZIiIiInIJ3t7el23Xo2IiIiIiIuLyFFxERERERMTlKbiIiIiIiIjLU3ARERERERGXp+AiIiIiIiIuT8FFRERERERcnoKLiIiIiIi4PAUXERERERFxeQouIiIiIiLi8hRcRERERETE5Sm4yC3hSOZZfIYtwWfYkrIuRURERET+AgUXERERERFxeQouIiIiIiLi8hRcRERERETE5Sm4iIiIiIiIy1NwERERERERl6fgIiIiIiIiLk/BRUREREREXJ6Ci4iIiIiIuDwFlxKwfv164uLiAEhLS8PLy+ua+xw1ahR5eXnX3I+zRo4cyeeff37V50VHR7N48eLrVsd7773HsWPHrlt/IiIiInJjUnApASEhISQlJV3XPkePHl1qwaWgoIDExEQee+yxUrnW5Si4iIiIiAgouFyz3NxcHnvsMXx9fQkMDKRNmzakpKQQEhJy0ePXrVtHy5YtCQkJISgoiLlz5wL/b2Zm5MiRBAcH07BhQ7755hsAEhISAAgPD8fhcFzyF/nzfQwePJiwsDD8/PxYtmyZ1f6///u/REREEBwcTFhYGCtWrAAgJSUFh8NB//79ad68OfPnz+epp55i8uTJAGRnZ9OnTx/8/f3x9/dn9OjRVp/btm0jLCyMoKAg4uLiOHv27GW/r4tda/bs2YSFhdG0aVMcDod134mJiRw6dIhu3brhcDhITU0lPz+fYcOGERoaisPhoEePHpw+ffqC60yYMAFvb2/rpyg/97J1iYiIiIiLM3JN5s2bZ1q3bm1tnzhxwixfvtwEBwcbY4zZv3+/qVatmjHGmFOnTpmmTZuaQ4cOGWOMycjIMHXr1jWHDx82+/fvN4BZsGCBMcaYf//73+a+++6z+gVMVlbWZWs538cnn3xijDFm9erVpmbNmiY7O9vs3bvXNG/e3GRmZhpjjNm9e7e5++67TV5enlm+fLmx2Wxm5cqVVl/x8fFm0qRJxhhjXnrpJRMXF2cKCwtNdna2cTgc5osvvjDGGBMUFFTsem5ubmbRokWXrPFi1zp+/LgpKiqy7qF27domLy/PGGNMvXr1zObNm61j33jjDfP6669b24mJiaZ///6X/V6MMca9UjVTb+hiU2/o4iseKyIiIiKlr06dOpdt9yjj3HTDCwwMZMeOHfTr148WLVrQvn37Sx67atUq9u3bR7t27ax9xhh27txJvXr1qFixIp06dQKgefPm7N2796rrKV++PL169QKgWbNm1KpVi40bN5KamsqePXuIiooqdvzBgwcBuO+++4iIiLhon8nJyUycOBE3NzcqVqzIk08+SXJyMm3btmXLli3FrhcQEHDFGv98rf379xMXF0d6ejoeHh4cP36cAwcO0LBhwwvOXbBgAb/99htfffUVAHl5edx7771OfDMiIiIiciNTcLlGDRo0YNu2bSxbtozk5GReeukl3nvvvYsea4zBbrdbj2j9UVpaGp6enta2u7s7hYWF16VGm82GMYa//e1vzJgx44L2X375hUqVKl3yfGMMNpvtgj7/+H+vxp+v1aNHD8aPH88jjzwCQNWqVS/5yJkxhilTptCyZcurvq6IiIiI3Li0xuUapaenY7PZiI2NZfz48RhjrFmMPwsPD2f37t3F1p2kpqY6tei+cuXKZGZmXvG4vLw868UAP/30E0eOHMFut9OmTRuWLl3Kli1brGN/+umnK/YH0Lp1az7++GOMMZw5c4ZZs2YRExNDlSpV8Pf3L3a9zZs3O9XnH506dQofHx8AZs2axalTp6y2KlWqFLvv2NhYJkyYQE5ODgA5OTls3br1qq8pIiIiIjcWBZdrtHnzZsLDw7Hb7QQFBdGrVy/sdvtFj73rrrtYtGgRr7/+OoGBgfj6+jJs2DCKioqueJ1BgwbRsmXLyy7OB6hWrRp79uwhLCyM3r17M3v2bCpWrEijRo2YNWsWzzzzDIGBgdx///1MnDjRqXt89dVXsdlsBAQEEBYWRmxsLN26dQNgxowZTJ48maCgID766CPCwsKc6vOPJk6cSOfOnYmIiGDjxo3UrVvXauvfvz+9e/e2FucPGzYMh8NBWFgYdrudZs2akZqaetXXFBEREZEbi80YY8q6CLk+0tLSCAkJ4fjx42VdisvxqOyF9/OfApA2tkMZVyMiIiIif+bt7U16evol2zXjIiIiIiIiLk+L829ACQkJrFmz5oL9q1evdpnZltjYWH755Zdi++666y6WL19eRhWJiIiIyI1MweUGNHXq1LIu4Yq+/vrrsi5BRERERG4ielRMRERERERcnoKLiIiIiIi4PD0qJreEWnd46m1iIiIiIjcwzbiIiIiIiIjLU3ARERERERGXp+AiIiIiIiIuT8FFRERERERcnhbnyy3hSOZZfIYtKesyrhu9aEBERERuNZpxERERERERl6fgIiIiIiIiLk/BRUREREREXJ6Ci4iIiIiIuDwFFxERERERcXkKLiIiIiIi4vIUXERERERExOUpuIiIiIiIiMtTcLlFPPXUU0yePLmsy7jubDYb2dnZZV2GiIiIiJQwBRdxSkFBQVmXICIiIiK3MAWXMpabm8tjjz2Gr68vgYGBtGnThpSUFAIDA+nduzfBwcGEhISwceNG65yZM2cSFhZGUFAQLVq0YMuWLQBs3ryZyMhIgoKC8PX15a233rroNVeuXImfnx/r168nPz+fYcOGERoaisPhoEePHpw+fRr4fZamf//+/O1vfyMwMPCS97Bo0SLsdjsOhwN/f38WLlwIQHR0NAMGDCA6OppGjRoxZMgQjDEAHDlyhO7duxMaGordbmfkyJFWf7t376ZDhw488MADBAYGMmXKFKtt3rx5NGnShObNm/P666//tS9dRERERG44HmVdwK1u6dKlnDp1im3btgFw8uRJNm3axKZNm5g4cSLR0dF88cUXPPHEE2zdupUff/yROXPmsGLFCm677TZWrlxJXFwcGzduxMfHh+TkZG677TZyc3MJDw+ndevWhISEWNf77LPPePvtt1myZAk+Pj68+eabVKpUiZ9++gmA119/nddee42JEycC8MMPP7BixQoqVap0yXsYMWIEU6dOJTw8nKKiIn777Terbdu2bXz77bfk5+cTFRXFl19+Sffu3YmPj2f48OFERUVRUFDAww8/zPz584mNjeWJJ55g5syZNGnShJycHJo1a0azZs3w9vbm2WefZdWqVTRu3Ji33377kjVNmDCBCRMmWNtF+bl/bYBERERExCUouJSxwMBAduzYQb9+/WjRogXt27cHoGHDhkRHRwPQvXt3+vbty6FDh1i4cCEbN24kLCzM6iMjI4O8vDxyc3Pp168fqampuLm5cfDgQVJTU63gMn36dMqVK8fy5cu58847AViwYAG//fYbX331FQB5eXnce++9Vt/du3e/bGgBaNWqFQMGDKBbt260adMGh8NhtcXHx1OuXDnKlStHz549SU5OpkOHDixbtoyjR49ax2VnZ7Njxw4aN27M1q1b6dGjh9WWlZXFtm3bSE9PJygoiMaNGwPQt29fhg4detGaBg4cyMCBA61tj8pel70HEREREXFtCi5lrEGDBmzbto1ly5aRnJzMSy+9xHvvvXfRY202G8YY+vTpQ2Ji4gXtr7zyCjVr1mTDhg14eHjQpUsXzp49a7UHBgayYsUKtmzZQkREBADGGKZMmULLli0ves0rhRb4fXZj69atLF++nPj4eOLi4njppZcueQ9FRUXYbDbWrVtHuXLlirVv3boVLy8vUlNTLzj3/CNoIiIiInLr0RqXMpaeno7NZiM2Npbx48djjOHgwYPs2bOHFStWAPDVV19Rp04dateuTceOHZkxYwYHDx4EoKioiPXr1wNw6tQpvL298fDwYOfOnXz77bfFrhUUFMTXX39N7969rbbY2FgmTJhATk4OADk5OWzduvWq7mHHjh34+fnxwgsv8Pe//501a9ZYbTNnzqSgoIDc3Fxmz55NTEwMlStXJjIykrFjx1rHHTp0iPT0dBo3bkyFChWYMWOG1bZnzx5OnjxJ8+bN2bBhA7t27QJg2rRpV1WniIiIiNy4NONSxjZv3sywYcMwxlBUVESvXr2she5z5sxh4MCBGGOYPXs2AFFRUbz55pt06tSJwsJC8vPz6dChAyEhIYwYMYJevXqRlJSEj4/PRWdRfH19Wbp0KR06dGDs2LEMGzaM0aNHExYWhs1mA2Do0KH4+fk5fQ8vv/wyu3btonz58lSoUIEPPvjAagsKCiImJoZff/2VRx55hG7dugGQlJTEwIEDCQgIAH6f2Zk6dSre3t4sWrSIF198kfHjx1NYWEj16tVJSkqiTp06fPTRR3Ts2JFq1apZfYmIiIjIzc9mzr/mSVxGSkoKgwcPtmZSblTR0dEMHjyYhx9+uKxLwaOyF97Pf1rWZVw3aWM7lHUJIiIiIteVt7c36enpl2zXo2IiIiIiIuLy9KiYC4qOjna52ZZjx47Rpk2bC/a3bt2ad95556LnpKSklHBVIiIiInKrUHARp9SoUeOib/oSERERESkNelRMRERERERcnoKLiIiIiIi4PD0qJreEWnd46k1cIiIiIjcwzbiIiIiIiIjLU3ARERERERGXp+AiIiIiIiIuT8FFRERERERcnoKLiIiIiIi4PL1VTG4JRzLP4jNsyQX79aYxERERkRuDZlxERERERMTlKbiIiIiIiIjLU3ARERERERGXp+AiIiIiIiIuT8FFRERERERcnoKLiIiIiIi4PAUXERERERFxeQouIiIiIiLi8hRcblApKSmEhIQAcOjQIR566KG/3FdqaipffPFFsX0Oh4Pc3NxrqvFanT59mrfffrtMaxARERER16DgUsoKCgque5933303y5cv/8vnXyy4pKamcvvtt19raZdVVFREUVHRJdsVXERERETkPAWXq2Cz2Rg3bhxhYWHUr1+f6dOnW23r16+nefPm2O12QkND+fHHHwFIS0vDy8uLxMREIiMjmTRpEtHR0QwZMoSoqCjuuece3nnnHebMmUN4eDj16tVjzpw5Vr89e/YkJCQEu93Oww8/zLFjxy6o6/w1AL755hscDof14+npyaeffkpBQQFt27YlJCQEPz8/4uLiyMnJ4dixY4wcOZLk5GQcDgcJCQnWvWZnZzt1byNHjiQ4OJiGDRvyzTffXPY7HDVqFL169aJLly44HA4OHz7MkCFDeOCBB3A4HLRo0YLdu3cDkJCQwOnTp3E4HNbs0pEjR+jevTuhoaHY7XZGjhz5l8ZSRERERG4wRpwGmPfee88YY8y2bdtMpUqVTH5+vjl37py55557zNKlS40xxqxcudLUqlXLZGdnm/379xvAJCUlWf20aNHCdO/e3RQWFppff/3VeHp6muHDhxtjjFm7dq2pXbu2dWxGRob1+a233jLPP/+8McaY5cuXm+DgYGOMMfv37zfVqlW7oN6ZM2cah8NhfvvtN1NUVGSOHz9ujDGmqKjIJCQkmHfeeccYY8z06dNN165dL7jXrKwsp+5twYIFxhhj/v3vf5v77rvvst/ha6+9ZurUqWOOHj160Xv87LPPTIcOHS55X23atDHff/+9McaY/Px807ZtWzNv3rwLrvPuu++aOnXqWD+22yqYekMXX/AjIiIiIq6hTp06l233KNvYdOOJi4sD4P7778fDw4MjR45w6tQpypcvT9u2bQGIiIigRo0abNq0idq1a+Pp6cnjjz9erJ9HH30UNzc37r77bry8vHjkkUcACA4O5vDhw5w9exZPT0+SkpKYOXMm586dIzc3l1q1ajlV5/Llyxk1ahQrVqygcuXKFBUV8Y9//IMlS5ZQUFBAZmYmUVFRV+xn586dl723ihUr0qlTJwCaN2/O3r17r9jnww8/TI0aNazt//znP0yaNImsrCyKior47bffLnremTNnWLZsGUePHrX2ZWdns2PHjguOHThwIAMHDrS2PSp7XbEuEREREXFdCi5XydPT0/rs7u5OQUEBxhhsNtsFx57fV7FixQva/9zP+W13d3fg97UwP/zwA5MnT2bVqlVUr16dr7/+msTExCvWuGXLFnr37s2SJUu4++67AZg9ezbff/+9FWTef/99VqxYccW+rnRvf76PwsLCK/ZZqVIl6/Mvv/xC//79+emnn2jQoAGbNm2iZcuWFz2vqKgIm83GunXrKFeu3BWvIyIiIiI3D61xuQ6aNGnCuXPnWLZsGQCrVq3i2LFjBAQEXFO/p06dokqVKlStWpW8vDw+/PDDK57z66+/8sgjjzB9+nT8/PyK9VWtWjUqV65MVlYWn3zyidVWpUoVMjMzS/XezsvMzKR8+fLUqlULYwyTJ08uVldOTo71QoPKlSsTGRnJ2LFjrWMOHTpEenr6dalFRERERFyXgst1UL58eebOncvw4cOx2+0MGDCAL7/8kooVK15Tv+3ataNhw4Y0adKEtm3b4nA4rnjOtGnTyMjI4MUXX7QW6H/99dc8+eSTZGdn4+vrS5cuXYiMjLTOadWqFWfOnCEwMNBanF/S93ZeQEAAjz76KH5+fkRHR1O3bl2rrWrVqsTFxREQEGAtzk9KSmL79u0EBAQQEBBA165dOXHixHWpRURERERcl80YY8q6CJGS5lHZC+/nP71gf9rYDmVQjYiIiIj8mbe392WfpHF6xmXhwoWMGzcO+P1xpM2bN197dSIiIiIiIk5wanH+qFGjWLt2LXv37mXo0KG4ubmRkJBg/T0PkT86duwYbdq0uWB/69ateeedd8qgIhERERG50TkVXBYsWMB///tfa51B7dq1ycrKKtHC5MZVo0YNUlNTy7oMEREREbmJOPWomKenp/WaXhERERERkdLm1IxLvXr1+OGHH7DZbBQVFfHmm29et9fhioiIiIiIXIlTbxU7evQo8fHxLFu2DDc3NyIjI0lKSir2189FXNmV3lIhIiIiImXrSr+vOTXjUrNmTZYuXUpOTg5FRUXF/vK5iIiIiIhISXMquADs2bOHffv2WX/FHKB9+/YlUpSIiIiIiMgfORVcBg0axKxZs2jcuLG1SN9msym4iIiIiIhIqXAquCxcuJD9+/dToUKFkq5HRERERETkAk69DrlevXqUL1++pGsRERERERG5KKdmXN599106duxI69at8fT0tPb369evxAoTuZ6OZJ7FZ9iSsi6jmLSxHcq6BBEREZEbhlPB5a233uLw4cOkpqYWW+MiIiIiIiJSGpwKLj///DO7du1SWBERERERkTLh1BqXJk2acObMmZKuRURERERE5KKcmnGpXLkywcHBtG3bttgal7fffrvEChMRERERETnPqeDSuHFjGjduXNK1iIiIiIiIXJRTweW1114r6TpEREREREQuyangAjBv3jxSU1M5e/astU+PiomIiIiISGlwanH+gAEDmD59OtOmTaOwsJA5c+Zw4sSJkq5NREREREQEcDK4fPfddyxcuJDq1avz7rvvsm7dOo4dO+bUBRYuXMj999+Pw+Fg8+bN11QsQEpKCv/5z3+s7UOHDvHQQw9dc79X47333nP6/svSqFGjGDx4cFmXISIiIiJyzZwKLp6enri5uWGz2cjPz6dmzZr8+uuvTl1g6tSpJCYmkpqaSkBAgLW/oKDgLxX85+By9913s3z58r/U11/lisHlr36frtK/iIiIiMjlOBVcKleuTE5ODhEREcTHxzNw4EDKlSt3xfP69+/PypUrGTp0KOHh4dhsNt59912io6N5+eWX2bx5M5GRkQQFBeHr68tbb71lnZuZmckzzzxDQEAAgYGB9OnTh9TUVKZOncqMGTNwOBwkJiaSlpaGl5eXdd7SpUsJCgrCbrfTokULtm3bBvweeBwOB/369SMwMBA/Pz/Wr19/2fqnTZuGr68vDoeDgIAA1q5dS2JiIocOHaJbt244HA5SU1PJzs6mT58++Pv74+/vz+jRo60+oqOjGTBgANHR0TRq1IghQ4ZgjGHnzp3Wm9qMMXh5eTF8+HDg9xmuVq1aAXD06FE6d+5MQEAA/v7+fPTRR1bfPj4+vPHGGzz00EPEx8eTmZlJt27d8PX1pW3btuzZs+eKY2Sz2Rg1ahQPPvgg9913H5999lmxtj+OV1ZWFs8++yyhoaHY7XYSEhLIz88HYMyYMdbMmsPh4MCBAwCsW7eOli1bEhISQlBQEHPnzgWwxm3kyJEEBwfTsGFDvvnmG+vaq1evJjIyksDAQOx2OwsXLgRg9+7ddOjQgQceeIDAwECmTJly0fuaMGEC3t7e1k9Rfu4VvwsRERERcV1OLc7/7LPP8PDw4J133mHChAmcOnWKL7/88ornvf/++2zatInBgwfz8MMPY7PZOHfuHCkpKQBkZWWRnJzMbbfdRm5uLuHh4bRu3ZqQkBAGDBhApUqV2LhxI25ubmRkZFC9enUSEhLIzs5m/PjxwO+/AJ937NgxevbsyfLlywkICCApKYnu3buzZcsWALZu3cq0adOYMmUKU6dOZfjw4fzv//7vJesfNGgQ27dv5+677yY/P59z584RFhbGv/71L7766iv8/f0BGDp0KHl5eWzatInc3FwiIiLw9fXl0UcfBWDbtm18++235OfnExUVxZdffkn37t3Jzc3ll19+4cSJE9x3330sW7YMgOTkZGJiYoDfw1+TJk2YP38+x44dIzg4GIfDQWhoKAC//PILy5Ytw2azMWjQIKpUqcK2bds4fvw4QUFBdO/e/YrjZLPZ+PHHH9m3bx+hoaFERERwzz33ABQbr759+xIVFcXHH3+MMYZnn32WyZMn89RTTzF+/HgOHz7M7bffTk5ODm5ubpw+fZrnnnuOJUuWULt2bY4fP05wcDAPPvggACdOnCA4OJjExESWLl3K//k//4f27dtz8uRJOnfuzLx58wgPD6eoqIjTp09TWFjIE088wcyZM2nSpAk5OTk0a9aMZs2aERQUVOyeBg4cyMCBA61tj8peiIiIiMiNy6kZl5o1a1K+fHluv/12hg8fzvjx46lbt+5fumCfPn2sz7m5udasSrNmzThw4ACpqakALF68mCFDhuDm9nuJ1atXv2Lfa9eutWZHAOLi4khPT+fw4cPA73+PJiQkBIDmzZuzd+/ey/bXsmVLnnzySSZOnMj+/fupVKnSRY9LTk4mISEBNzc3KlasyJNPPklycrLVHh8fT7ly5ahQoQI9e/a02lq1akVycjLJycn07NmT3NxcMjMziwWX5ORknn/+eQBq1KhBly5d+O6776y+e/fujc1mA2D58uU8/fTTAHh5edGlS5crfmcAzzzzDAANGjQgIiKClStXWm1/HK8FCxbwzjvv4HA4aNq0KStXrmT37t1UqVKFRo0a0bNnTz788ENOnjyJp6cnq1atYt++fbRr1w6Hw0FMTIw12wRQsWJFOnXqBBQfj9WrV+Pr60t4eDgAbm5uVK1alZ07d7J161Z69OiBw+EgPDycrKwsa1ZNRERERG5eTs24pKWlMW7cOPbu3VtsrcP5GYKr8cdf/l955RVq1qzJhg0b8PDwoEuXLsVet3y1jDHWL/F/dH6fp6entc/d3f2K6zbmzZvHf//7X1JSUmjfvj1jxoyhR48eTl33YnX8uS0mJoYlS5Zw8uRJ3n//fXbv3s28efPYv38/wcHBl+zrj9t//D6NMZe9H2ddrv8FCxbQoEGDC85Zs2YNq1atIiUlhWbNmvHZZ59hjMFut7NixYoLjk9LS7tgPAoLCy9b1/lH6s6HWxERERG5dTg149K9e3fuvPNOXnjhBQYNGmT9XKtTp07h7e2Nh4cHO3fu5Ntvv7XaYmNjeeeddygqKgIgIyMDgCpVqpCZmXnR/po3b05qairbt28HYM6cOXh7e1OrVq2rrq2goIC9e/cSEhLC4MGD6datGz/99NNFa2jdurX1+NSZM2eYNWuWNWMCMHPmTAoKCsjNzWX27NlWW0xMDN999x0HDhzgvvvuIyYmhtGjR9OiRQtrpikmJsZa15KRkcH8+fNp2bLlRWtu1aoV06dPB+DkyZPMnz/fqXv917/+BfweJn744QciIiIuelxsbCxjx461At+pU6fYs2cPWVlZHD16lMjISF599VUiIiLYsGED4eHh7N69u1jATU1NJS8v77L1hIeHs337dlatWgVAUVERJ0+epHHjxlSoUIEZM2ZYx+7Zs4eTJ086dZ8iIiIicuNyasbl7NmzxRbOXy8jRoygV69eJCUl4ePjU+wX8n/84x+8+OKL+Pv7U758eR544AE+/vhjOnfuzMyZM3E4HHTp0oUnn3zSOqd69erMnDmTuLg4CgsLufPOO/niiy/+Um2FhYX07t2bU6dO4eHhQfXq1a1Q0L9/f3r37k2FChX45JNPePXVV/mf//kf6xG1Rx99lG7dull9BQUFERMTw6+//sojjzxitdWsWZOaNWtaj6+1aNGCQ4cOMXToUOvc999/n4SEBOx2O0VFRQwfPtxa3/Jnr776Kn369MHX15d69erRunVrp+71tttu48EHHyQjI4NJkyZZ61v+7L333mPo0KE4HA7c3NwoV64c48aNw9PTk27dunHmzBlsNhuNGjUiPj6eO+64g0WLFjFkyBBefPFF8vPzqVu3LgsWLLhsPXfddRfz589n0KBBZGVlYbPZeP3114mNjWXRokW8+OKLjB8/nsLCQqpXr05SUpJT9ykiIiIiNy6bceL5oieeeIK3334bb2/v0qjpphIdHW29nMAV2Ww2srKyLrl+52bhUdkL7+c/Lesyikkb26GsSxARERFxGd7e3qSnp1+y3akZl5MnT2K324mIiCi2LuGvzmaIiIiIiIhcDaeCy+OPP87jjz9e0rWUidTUVJ566qkL9sfHx/Piiy9ec//nXyVclhITE5k3b94F++fOnXvdFvSLiIiIiJQkpx4VE7nR6VExEREREdd2pUfFnHqrmIiIiIiISFlScBEREREREZfn1BoXkRtdrTs89WiWiIiIyA3M6RmXhQsXMm7cOAB+/fVXNm/eXGJFiYiIiIiI/JFTwWXUqFFMnTqVf/7zn7+f5OZGQkJCiRYmIiIiIiJynlPBZcGCBSxevJiKFSsCULt2bbKyskq0MBERERERkfOcCi6enp64u7uXdC0iIiIiIiIX5dTi/Hr16vHDDz9gs9koKirizTffJCAgoKRrE7lujmSexWfYkrIuQ+SmphdgiIhISXIquLz//vvEx8ezZcsWKlSoQGRkJElJSSVdm4iIiIiICOBEcCksLGTOnDksXbqUnJwcioqKqFSpUmnUJiIiIiIiAjixxsXd3Z25c+cCUKFCBYUWEREREREpdU4tzm/dujWff/55SdciIiIiIiJyUU6vcTlx4gR9+vShYsWKGGOw2WwcO3aspOsTERERERFxLrisX7++pOsQERERERG5JKdfhywiIiIiIlJWnAou9evXx2azXbB/3759170gERERERGRP3MquCxevNj6fPbsWWbOnMldd91VYkXJjW/kyJH4+fnx2GOPlXUpIiIiInITcCq4+Pn5FdsODg6mVatWJVKQ3BwSExPLugQRERERuYk49TrkPzt16hT79++/3rWIC1i3bh0tW7YkJCSEoKAg5s6dS0FBAW3btiUkJAQ/Pz/i4uLIyckBYM2aNQQHB+NwOPD39+eDDz4A4KmnnmLy5MkAjBo1iieeeIKOHTvi6+tLy5YtOXnypHXN8ePHExoaSlBQEO3bt+fgwYMALFq0CLvdbvW9cOFCAMaMGcP999+Pw+HA4XBw4MCB0vyKRERERKQMODXj8sADD1hrXAoLC9m/fz+DBg0q0cKk9J0+fZrnnnuOJUuWULt2bY4fP05wcDDh4eHMnj2batWqYYyhX79+TJkyhcGDB/PWW28xaNAgnnjiCeD3UHsxa9euZd26dVStWpUePXrw4Ycf8vLLLzN79mx27drF6tWrcXd3Z+bMmbzwwgssXLiQESNGMHXqVMLDwykqKuK3337j1KlTjB8/nsOHD3P77beTk5ODm9uF+XvChAlMmDDB2i7Kzy2ZL01ERERESoVTwWX8+PH/7wQPD+rXr8/dd99dYkVJ2Vi1ahX79u2jXbt21j5jDLt27eLbb79lyZIlFBQUkJmZSVRUFAAPPfQQY8aMYc+ePbRs2ZKIiIiL9t2uXTuqVq0KQPPmzdm8eTMACxYsYP369QQHBwO/B2N3d3cAWrVqxYABA+jWrRtt2rTB4XBQWFhIo0aN6NmzJ23atKFDhw54e3tfcL2BAwcycOBAa9ujstd1+IZEREREpKw49ajY3r17adGiBS1atODBBx/k7rvv5l//+ldJ1yalzBiD3W4nNTXV+vnll184ePAg33//PStWrGDz5s0MHjyYs2fPAjBgwAAWL15M7dq1eeWVV+jXr99F+/b09LQ+u7u7U1BQYF1zxIgR1vU2b95Mamoq8PusyfTp06lQoQLx8fG8/fbbuLu7s2bNGgYMGMCxY8do1qwZK1euLNkvRkRERETKnFPB5fxahSvtkxtbeHg4u3fvZtmyZda+1NRUTpw4QbVq1ahcuTJZWVl88sknVvvOnTtp0KABzz77LK+88gpr1qy5qmvGxsYyZcoUa81Lfn4+GzZsAGDHjh34+fnxwgsv8Pe//501a9aQlZXF0aNHiYyM5NVXXyUiIsI6XkRERERuXpd9VGz9+vWsXbuW48ePM2XKFGt/ZmYmeXl5JV6clK677rqLRYsWMWTIEF588UXy8/OpW7cun3/+OYsWLcLX15c6deoQGRnJr7/+CsCkSZNYvnw55cuXx93dnXffffeqrtmrVy9OnDhBdHQ0NpuNgoICnn76aZo2bcrLL7/Mrl27KF++PBUqVOCDDz4gMzOTbt26cebMGWw2G40aNSI+Pr4kvg4RERERcSE2Y4y5VOPChQtZsGABX3/9NbGxsdb+KlWq0KtXL0JCQkqlSJFr5VHZC+/nPy3rMkRuamljO5R1CSIicgPz9vYmPT39ku2XnXHp1KkTnTp14t///nexBdsiIiIiIiKlyam3irVr146ff/6Z1NRUa1E2cMmF2CIiIiIiIteTU8Fl3LhxfP755/zyyy+0aNGCb7/9llatWim4iIiIiIhIqXDqrWIzZ85k1apVeHt7M3fuXNatW0f58uVLujYRERERERHAyeDi6emJp6cnRUVFGGNo3LgxaWlpJVyaiIiIiIjI75x6VKxChQrk5+fjcDgYOnQo3t7e5OTklHRtItdNrTs89cYjERERkRuYUzMuU6ZMIS8vj3fffZdTp06xYsUKZs6cWdK1iYiIiIiIAFf4Oy5/VlBQgIeHU5M0Ii7lSu8FFxEREZGydaXf15yacdm6dSsOh4P69esD8N///pehQ4denwpFRERERESuwKng8sILLzB58mS8vLwACAoKYsmSJSVamIiIiIiIyHlOBZesrCwiIiKsbZvNRrly5UqsKBERERERkT9yasGKh4cH+fn52Gw2ANLT03FzcyrziLiEI5ln8RmmWUL5nd4wJyIicuNx+lGxzp07c/z4cUaNGkVUVBRDhgwp6dpEREREREQAJ2dcevbsSYMGDVi4cCE5OTl8+umnREZGlnRtIiIiIiIiwBWCS79+/ZgyZQoAGRkZjBs3rlSKEhERERER+aPLPiq2Zs0a6/Po0aNLvBgREREREZGLuWxw+ePfpryKv1MpIiIiIiJyXV32UbFz586xfft2jDHFPp/n6+tb4gWKiIiIiIhcNrjk5OTQvn17a/uPn202G/v27Su5ykRERERERP5/lw0uaWlppVSGlLaFCxcybNgwbrvtNmbOnElAQMA19Wez2cjKyqJSpUrXqUIRERERkf/Hqdchy81n6tSpJCYm8uijjxbbX1BQgIeH/t9CRERERFyLU3+AUm4u/fv3Z+XKlQwdOpTw8HBsNhvvvvsu0dHRvPzyy2RlZfHss88SGhqK3W4nISGB/Px8AMaMGcP999+Pw+HA4XBw4MABq9//+3//L2FhYdSvX5/p06db+318fBg9ejTh4eHUr1+fMWPGWG1Hjhyhe/fu1rVGjhwJQFFRES+88AJNmjQhMDCQ4OBgzp49S0ZGBm3atCEgIAC73U7v3r1L6VsTERERkbKk/2n9FvT++++zadMmBg8ezMMPP4zNZuPcuXOkpKQA0LdvX6Kiovj4448xxvDss88yefJknnrqKcaPH8/hw4e5/fbbycnJwc3t/2VfT09P1q5dy/bt2wkNDaVXr17W7M3p06dZtWoVGRkZNGzYkN69e1OnTh3i4+MZPnw4UVFRFBQU8PDDDzN//nx8fHz47rvv2LZtG25ubmRmZlK+fHlmzZqFj48P//nPfwA4efJkqX9/IiIiIlL6FFwEgD59+lifFyxYwJo1a3j33XcByM3NpXz58lSpUoVGjRrRs2dP2rRpQ4cOHfD29rbOi4uLA+D+++/Hw8ODI0eOWO3n26pXr06DBg3Yv38/d955J8uWLePo0aNWH9nZ2ezYsYOWLVuSn59Pnz59eOihh+jQoQNubm40a9aMf/zjHwwaNIgWLVrQtm3bi97PhAkTmDBhgrVdlJ97nb4pERERESkLCi4CUGxRvTGGBQsW0KBBgwuOW7NmDatWrSIlJYVmzZrx2WefERkZCfw+43Keu7s7BQUF1vbF2oqKirDZbKxbt45y5cpdcK2tW7fy/fffs3z5cl5++WVWrFhB8+bNSU1NJTk5mblz5zJixAg2bNiAu7t7sXMHDhzIwIEDrW2Pyl5/4VsREREREVehNS5ygdjYWMaOHWsFj1OnTrFnzx6ysrI4evQokZGRvPrqq0RERLBhw4a/fJ3KlSsTGRnJ2LFjrX2HDh0iPT2djIwMzpw5Q5s2bXjzzTfx8fFh27Zt7N+/n0qVKtG9e3cmTZrErl27yM7OvuZ7FhERERHXpuAiF3jvvffw8PDA4XBgt9uJiYkhLS2NzMxMunTpYi2Mz8/PJz4+/pqulZSUxPbt2wkICCAgIICuXbty4sQJDh48SOvWrbHb7QQEBODv70+7du1ISUkhODgYh8PBgw8+yDvvvMMdd9xxne5cRERERFyVzRhjyroIkZLmUdkL7+c/LesyxEWkje1Q1iWIiIjIn3h7e5Oenn7Jds24iIiIiIiIy1NwERERERERl6fgIiIiIiIiLk/BRUREREREXJ6Ci4iIiIiIuDwFFxERERERcXkeZV2ASGmodYenXoErIiIicgPTjIuIiIiIiLg8BRcREREREXF5Ci4iIiIiIuLyFFxERERERMTlKbiIiIiIiIjL01vF5JZwJPMsPsOWlHUZIiIiIjcEV3wbq2ZcRERERETE5Sm4iIiIiIiIy1NwERERERERl6fgIiIiIiIiLk/BRUREREREXJ6Ci4iIiIiIuDwFFxERERERcXkKLiIiIiIi4vIUXK6BzWYjOzu7rMu4amlpaXz00Ucl1n9qaipffPFFifUvIiIiIrceBZdbUFkHl4KCghK7toiIiIjcnBRcrsK8efNo0qQJzZs35/XXX7f2L126lKCgIOx2Oy1atGDbtm1W2/Tp03E4HAQGBhISEkJaWhppaWl4eXlZx2RnZ2Oz2axtm83GW2+9RWhoKA0aNCA5OZmXX36Zpk2b4ufnx9atW61jZ86cSVhYGEFBQbRo0YItW7YA8Mknn9C2bVsef/xxAgICCAkJYd++fQAkJCSwbds2HA4HsbGxl73n8ePHExoaSlBQEO3bt+fgwYMAjBo1iieeeIKOHTvi6+tLy5YtOXnyJMeOHWPkyJEkJyfjcDhISEiw7undd98lOjqal19+maNHj9K5c2cCAgLw9/cvFqR8fHx4+eWXiYqKomHDhkyYMAGAL7/8krZt21rHFRYWUq9evWLf93kTJkzA29vb+inKz73sfYqIiIiIa1NwcdKxY8d49tlnWbhwIatXr+a2224D4OTJk/Ts2ZNPP/2UTZs20bdvX7p37w5ASkoKb7zxBv/+97/ZuHEjK1asoEaNGk5dr0qVKvz000+MGzeOTp06ERERwYYNG4iPj+eNN94A4Mcff2TOnDmsWLGCn3/+mTFjxhAXF2f1sXbtWsaOHcvmzZuJiYlh3LhxAEydOhVfX19SU1P5+uuvL1nD7Nmz2bVrF6tXr+bnn3/m8ccf54UXXijW/6effsq2bduoUaMGH374ITVq1CAxMZGYmBhSU1OZOnWqdfy5c+dISUnhnXfeoX///jRp0oTNmzezbNkyXn/9dX766Sfr2KNHj7JixQrWrFnDxIkTWbt2LV26dGHnzp3s3r0bgAULFtCwYUN8fX0vqH3gwIGkp6dbP27lbnfqexcRERER16Tg4qQ1a9YQFBRE48aNAejbty8AGzduxOFwEBAQAEBcXBzp6ekcPnyYJUuW8OSTT1K7dm0AKlSoQIUKFZy63mOPPQZAUFAQbm5udOjQAYDg4GBr5mThwoVs3LiRsLAwHA4H//M//0NGRgZ5eXkAREREUK9ePQCaN2/O3r17r+qeFyxYQHJyMsHBwTgcDt5++20OHDhgtbdr146qVas63X+fPn2sz8nJyTz//PMA1KhRgy5duvDdd99Z7U8//TQAXl5edO7cme+++w53d3f69evHlClTAJg8eXKxICUiIiIiNy+Psi7gRmGMueT+Pz7mdd7F9p3n4eFBYWGhtX327NkLjvH09ATA3d3dmt05v31+jYgxhj59+pCYmHjR65zv48/nOcsYw4gRI4oFjmvpv1KlSsW2//wdXe47O9/27LPP4u/vz+OPP86+ffuu+KibiIiIiNwcNOPipObNm7NhwwZ27doFwLRp0wBo2rQpqampbN++HYA5c+bg7e1NrVq16NixIzNmzODIkSMA5OTkkJOTQ61atSgoKGDnzp0AzJgx4y/VdL7/8+tOioqKWL9+/RXPq1KlCpmZmVc8LjY2lilTpnDy5EkA8vPz2bBhw3XpPyYmxlrXkpGRwfz582nZsqXVPn36dOD3R/EWLFhAq1atALjrrrvo2LEjXbt2JSEhAXd39yvWIyIiIiI3PgUXJ9WoUYOPPvqIjh07Eh4ejpvb71/dXXfdxcyZM4mLiyMwMJAPPvjAeqNWVFQUI0aMoE2bNgQGBtKiRQsyMjLw8PDg/fffp127dkRFRXHu3Lm/VFNUVBRvvvkmnTp1IjAwEH9/fz7//PMrnme322ncuDH+/v6XnbHo1asXPXv2JDo6msDAQBwOB8uXL79i/61ateLMmTMEBgZai/P/7P3332fTpk3Y7XYeeughhg8fTmhoqNVer149IiMjCQ0NpX///sXann32WTIyMnjmmWeuWIuIiIiI3Bxs5lLPQImUER8fHxYvXoy/v/9F299++2127tzJP//5T6f79Kjshffzn16vEkVERERuamljO5T6Nb29vUlPT79ku9a4yA3Fz88Pm83G0qVLy7oUERERESlFCi5CSEjIBQvr/fz8SEpKKpN60tLSLtn2x79hIyIiIiK3DgUXcWpBv4iIiIhIWdLifBERERERcXkKLiIiIiIi4vL0qJjcEmrd4Vkmb8cQERERketDMy4iIiIiIuLyFFxERERERMTlKbiIiIiIiIjLU3ARERERERGXp+AiIiIiIiIuT28Vk1vCkcyz+Axb4vTxegOZiIiIiGvRjIuIiIiIiLg8BRcREREREXF5Ci4iIiIiIuLyFFxERERERMTlKbiIiIiIiIjLU3ARERERERGXp+AiIiIiIiIuT8FFRERERERcnoJLCRs1ahR5eXkAjBw5ks8//xyAlJQU/vOf/1jHpaWl4eXlVSY1OuOTTz6hW7duZV2G5ZNPPmHXrl1lXYaIiIiIlBIFlxI2evRoK7gkJiby2GOPARcGF1dSUFBQ1iVckYKLiIiIyK1FwaUEJSQkABAeHo7D4aB9+/ZMnjyZ1NRUpk6dyowZM3A4HCQmJl5w7rp162jZsiUhISEEBQUxd+7cS14nOzubqlWrkp+fD0BQUBBxcXEA7Nu3j3vvvdc6rk+fPvj7++Pv78/o0aOtPqKjoxk+fDitWrWibdu25OXl8dxzz3Hffffx0EMPsXbt2ive7/bt22nbti12ux273c7UqVMB2LNnDzExMdjtdhwOBwsWLLDOsdlsZGdnW9teXl6kpaUB4OPjw+jRowkPD6d+/fqMGTMGgGnTprF+/Xr69++Pw+Hgm2++uaCWCRMm4O3tbf0U5edesX4RERERcV0eZV3AzWzq1Kl8+OGHrFq1ikqVKvHUU08B4HA4SEhIIDs7m/HjxwNYv6wDnD59mueee44lS5ZQu3Ztjh8/TnBwMA8++CC1atW64DqVKlXCz8+P1atX4+fnR2FhIevWrQPg22+/JSYmBoDXX3+dvLw8Nm3aRG5uLhEREfj6+vLoo48CkJqaytKlSylXrhyTJk1i//79bN26lfz8fKKiovDx8bnkvRYUFNCpUyfGjBlD9+7dATh+/DgAcXFxPP300/Tt25fdu3fTrFkzgoODueeee674HZ4+fZpVq1aRkZFBw4YN6d27N8888wyzZs1i8ODBPPzwwxc9b+DAgQwcONDa9qjsuo/hiYiIiMiVacbFBa1atYp9+/bRrl07HA4HMTExGGPYuXPnJc+JiYkhOTmZZcuW0bZtWxo3bsyWLVtITk62gktycjIJCQm4ublRsWJFnnzySZKTk60+evXqRbly5QBYvnw58fHxlCtXjgoVKtCzZ8/L1rxz504KCgqs0AK/z55kZWWRmprK008/DUCjRo2IiIjghx9+cOq7OD9zVL16dRo0aMD+/fudOk9EREREbi6acXFBxhjsdjsrVqxw+pyYmBiGDBnCsWPH6NKlC3Xq1OHbb7/l+++/tx7ZMsZgs9mKnffH7UqVKhWr4Xo438+lruvu7k5hYaG1/+zZs8WO8/T0tD67u7vfEOtvREREROT604xLCatcuTKZmZkX7K9SpcpF98Pva2J2797NsmXLrH2pqanWIv+LCQsLY8eOHSQnJxMZGUlMTAwTJ07knnvuoVq1agC0bt2ajz/+GGMMZ86cYdasWdZszJ+1atWKmTNnUlBQQG5uLrNnz77sfTZu3Jjy5cvz5ZdfWvuOHz9OlSpVcDgcfPrppwDs3buXH3/8kQcffBCAe++911o/M2/ePM6cOXPZ65x3ue9PRERERG4+Ci4lbNCgQbRs2RKHw8GxY8es/Z07d2b9+vUXXZx/1113sWjRIl5//XUCAwPx9fVl2LBhFBUVXfI6Hh4eREREULduXW6//Xb8/PzIz88vFkxeffVVbDYbAQEBhIWFERsbe8lXHPft25e6devi6+tLhw4diIyMvOx9enh4sHDhQj766CMCAgKw2+3WCwWSkpKYNWsWgYGBdO3alWnTplnrW9577z2ef/55HnzwQX7++WcrZF1J3759SUxMvOTifBERERG5udjM9XomSMSFeVT2wvv5T50+Pm1shxKsRkRERET+zNvbm/T09Eu2a8ZFRERERERcnhbn30BSU1OtVyr/UXx8PC+++GKp1DBt2jQmT558wf5JkyZd8XEyEREREZG/So+KyS1Bj4qJiIiIuDY9KiYiIiIiIjc8BRcREREREXF5WuMit4Rad3jq8S8RERGRG5hmXERERERExOUpuIiIiIiIiMtTcBEREREREZen4CIiIiIiIi5PwUVERERERFyegouIiIiIiLg8BRcREREREXF5Ci4iIiIiIuLyFFxERERERMTlKbiIiIiIiIjLsxljTFkXIVLSPDw8qFWrVlmXIZeRnZ1NpUqVyroMuQyNkevTGLk+jZFr0/iUrYyMDM6dO3fJdo9SrEWkzNSqVYv09PSyLkMuw9vbW2Pk4jRGrk9j5Po0Rq5N4+Pa9KiYiIiIiIi4PAUXERERERFxeQoucksYOHBgWZcgV6Axcn0aI9enMXJ9GiPXpvFxbVqcLyIiIiIiLk8zLiIiIiIi4vIUXERERERExOUpuMhNY/fu3YSHh3PfffcRGhrKtm3bLnrcP//5Txo1asS9995L3759KSgoKOVKb13OjFFaWhrR0dHccccdhISElEGVtzZnxmjZsmWEhYXh6+uLv78/w4cPR08dlx5nxmj16tU4HA4cDgd+fn4899xzl/3bCHL9OPvfIoCzZ8/i6+urf9eVMmfGKCUlhQoVKlj/HDkcDnJzc8ugWinGiNwkHnroITN9+nRjjDFffvmladas2QXH7Nu3z9SuXdscOXLEFBUVmY4dO5qpU6eWcqW3LmfG6MSJE2blypVm8eLFJjg4uJQrFGfG6OeffzZ79+41xhiTm5trHnzwQZOUlFSaZd7SnBmjM2fOmLy8PGOMMYWFhaZz585m4sSJpVnmLcuZ8Tlv4MCBpk+fPvp3XSlzZoyWL1+ucXFBmnGRm8KxY8f4+eef6dmzJwBdu3Zl//79pKWlFTvuq6++onPnztSsWRObzUZCQgKfffZZGVR863F2jKpWrUpERAQVK1Ysgypvbc6OUdOmTWnQoAEAnp6eOBwO9u3bV9rl3pKcHaMKFSpQrlw5APLy8sjNzcXNTf/JL2nOjg/AypUr2b17N7169SrlKm9tVzNG4nr0bzG5KRw8eJC7774bDw8PAGw2G3Xr1uWXX34pdtwvv/xCvXr1rG0fH58LjpGS4ewYSdn5K2N05MgRvvrqK9q3b19aZd7SrmaM0tLScDgceHl5UaVKFfr27Vva5d5ynB2fM2fOMGDAAD744IOyKPOWdjX/DO3cuZOgoCAeeOABpkyZUtqlykUouMhNw2azFds2l3jm/o/HXeoYKRnOjpGUnasZo99++42OHTvy0ksvERQUVNKlyf/P2THy8fEhNTWVI0eOcO7cOebNm1ca5d3ynBmfIUOG8Pzzz1OnTp3SKkv+wJkxCgoKIj09nZ9//pn58+czdepUvvjii9IqUS5BwUVuCvfccw/p6enWQntjDAcPHqRu3brFjqtbt26x6eADBw5ccIyUDGfHSMrO1YxRVlYWf/vb34iNjdUfbCtFf+Wfo0qVKtGjRw+SkpJKq8xblrPj88MPP5CYmIiPjw89evRg8+bN+Pn5lUXJtxxnx6hKlSrccccdAHh7e/P444+zcuXKUq9XilNwkZtCjRo1aNq0KbNmzQJg7ty5+Pj44OPjU+y4rl27Mn/+fI4ePYoxhqlTp9KjR48yqPjW4+wYSdlxdoyys7P529/+Rtu2bXn11VfLoNJbl7NjtHfvXvLz84Hf17jMmzcPu91e2uXecpwdn02bNpGWlkZaWhpz5swhICCArVu3lkHFtx5nx+jw4cMUFRUBv/8PNYsXL6Zp06alXa78WVm8EUCkJOzYscM0a9bMNGrUyAQHB5stW7YYY4x5+umnzcKFC63jPvroI3Pvvfea+vXrm6efftp6846UPGfG6OzZs6ZOnTrGy8vLlCtXztSpU8cMGzasLMu+pTgzRmPGjDEeHh4mMDDQ+hkzZkxZln1LcWaMpk2bZvz8/Izdbje+vr7mhRdeMLm5uWVZ9i3D2f8Wnae3V5U+Z8Zo0qRJxtfX1/pn6LXXXjNFRUVlWbYYY2zG6CFzERERERFxbXpUTEREREREXJ6Ci4iIiIiIuDwFFxERERERcXkKLiIiIiIi4vIUXERERERExOUpuIiIiIiIiMtTcBERkZuej48PTZo0weFw4HA4SEhI+Mt9LViwgJ9++uk6VndxX3/9NUOGDCnx61zK6dOnefvtt8vs+iIif6a/4yIiIjc9Hx8fFi9ejL+//zX39dRTTxESEsILL7xw1ecWFBTg4eFxzTWUtIKCAtLT0wkJCeH48eNlXY6ICKAZFxERuYXNnDmTsLAwgoKCaNGiBVu2bAFg8+bNREZGEhQUhK+vL2+99RYA33zzDV9//TVjx47F4XAwbdo0UlJSCAkJsfrcsmULPj4+AKSlpeHl5UViYiKRkZFMmjSJI0eO0L17d0JDQ7Hb7YwcOfKitX3yySd069YNgJSUFGumKCAggKCgILZs2cJjjz2Gr68vrVu3Jjs7G4BRo0bRvXt32rdvj7+/P7GxsZw6dQqA7Oxs+vTpg7+/P/7+/owePdq6XnR0NMOHD6dVq1a0bduWhIQETp8+jcPhsO5vwoQJPPDAAzRt2pTQ0FDWrl1rnW+z2Rg3bhxhYWHUr1+f6dOnW23bt2+nbdu22O127HY7U6dOBXD6uxARAcCIiIjc5OrVq2caN25sAgMDTWBgoJk3b5754YcfTPv27c3Zs2eNMcasWLHC2O12Y4wxv/32m7U/JyfHOBwOs27dOmOMMfHx8WbSpElW38uXLzfBwcHW9ubNm029evWMMcbs37/fACYpKclqb9Omjfn++++NMcbk5+ebtm3bmnnz5l1Q8/Tp003Xrl2ta3h4eJgNGzYYY4zp16+fqVOnjjl48KAxxph27dqZDz/80BhjzGuvvWZq1apljhw5Yowx5u9//7v5+9//bowx5qWXXjJxcXGmsLDQZGdnG4fDYb744gtjjDEtWrQw7du3N3l5eVbt1apVK1bTsWPHrM+rV682fn5+1jZg3nvvPWOMMdu2bTOVKlUy+fn5Jj8/3zRq1Mh8/vnn1rEZGRlX9V2IiBhjjOvPV4uIiFwHX331VbFHxV566SU2btxIWFiYtS8jI4O8vDxyc3Pp168fqampuLm5cfDgQVJTU4vNrDjL09OTxx9/HIAzZ86wbNkyjh49arVnZ2ezY8eOK/bTuHFjHA4HAEFBQRw4cABvb28AgoOD2bdvn3Xsww8/TM2aNQHo27cv3bt3ByA5OZmJEyfi5uZGxYoVefLJJ0lOTubRRx8FoFevXpQrV+6SNWzYsIE33niDEydO4OHhwbZt28jLy6N8+fIAxMXFAXD//ffj4eHBkSNHyMzMpKCgwKoBwMvL65q+CxG5NSm4iIjILckYQ58+fUhMTLyg7ZVXXqFmzZps2LABDw8PunTpwtmzZy/aj4eHB4WFhdb2n4+rWLEiNpsNgKKiImw2G+vWrbtsQLgYT09P67O7u/sF27m5uZc89/z1jTHW5z+3AVSqVOmSfeTl5dG1a1dSUlIIDg7mt99+44477igWXP5cU0FBwSX7u5bvQkRuTVrjIiIit6SOHTsyY8YMDh48CPz+i/T69esBOHXqFN7e3nh4eLBz506+/fZb67wqVaqQmZlpbdevX5/9+/dz4sQJ4Pd1M5dSuXJlIiMjGTt2rLXv0KFDpKenX9d7W7JkCceOHQPgn//8JzExMQC0bt2ajz/+GGMMZ86cYdasWVbbn1WpUoWcnBwrfJw9e5b8/HzuueceACZNmuRULY0bN6Z8+fJ8+eWX1r7jx4+X2nchIjcPBRcREbklRUVF8eabb9KpUycCAwPx9/fn888/B2DEiBFMmzaNBx54gBEjRtCyZUvrvF69ejF79mxrcX6dOnUYPHgwISEhPPTQQ9x5552XvW5SUhLbt28nICCAgIAAunbtaoWe66VVq1Y8/fTT+Pv7c+DAAcaMGQPAq6++is1mIyAggLCwMGJjY60XAPxZ1apViYuLIyAggJCQEKpUqUJiYiKhoaFERUVx2223OVWLh4cHCxcu5KOPPiIgIAC73c7cuXOB0vkuROTmodchi4iI3ERGjRpFdnY248ePL+tSRESuK824iIiIiIiIy9OMi4iIiIiIuDzNuIiIiIiIiMtTcBEREREREZen4CIiIiIiIi5PwUVERERERFyegouIiIiIiLg8BRcREREREXF5/x+HXJGEUFmpQAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 800x320 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Decision Tree Classifier\n",
    "# tree = DecisionTreeClassifier().fit(X_train, y_train)\n",
    "# print(\"Accuracy Score for Decision Tree on Training Data:\",tree.score(X_train, y_train))\n",
    "# print(\"Accuracy Score for Decision Tree on Validation Data:\",tree.score(X_val, y_val))\n",
    "## Decision Tree Classifier with GridSearchCV\n",
    "tree = DecisionTreeClassifier()\n",
    "grid_values = {'max_depth': [2, 3, 4, 5, 6]}\n",
    "cv = 3\n",
    "tree_grid = GridSearchCV(tree, param_grid=grid_values, scoring='roc_auc', cv=cv)\n",
    "tree_grid.fit(X_train, y_train)\n",
    "y_val_scores_tree_grid = tree_grid.predict_proba(X_val)\n",
    "y_val_scores_tree_grid = y_val_scores_tree_grid[:,1]\n",
    "tree_grid_fpr, tree_grid_tpr, _ = roc_curve(y_val, y_val_scores_tree_grid)\n",
    "tree_grid_roc_auc = auc(tree_grid_fpr, tree_grid_tpr)\n",
    "# print('Tree Grid best parameter (max. AUC): ', tree_grid.best_params_)\n",
    "# print('Tree Grid best score (AUC): ', tree_grid.best_score_)\n",
    "print('Tree Validation set AUC: ', roc_auc_score(y_val, y_val_scores_tree_grid))\n",
    "tree = DecisionTreeClassifier(max_depth=tree_grid.best_params_['max_depth'])\n",
    "tree.fit(X_train, y_train)\n",
    "tree_fig = plt.figure(figsize=(10,4), dpi=80)\n",
    "plot_feature_importances(tree, features)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d03a42bc-200b-4f89-a7b4-0784736702a9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random Forest Grid best parameter (max. AUC):  {'max_features': 2, 'n_estimators': 200}\n",
      "Random Forest Validation set AUC:  0.8692199238604378\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAy8AAAEmCAYAAABrva/RAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAAxOAAAMTgF/d4wjAABGuklEQVR4nO3deVxV1f7/8deBo5FTZTglKZqmMRxOQKAESIqamlhmNqCRVkbm9ZeoaWleRW+pmTfTn5fKrqViljlQ2ddvkZKWc3mcZ0UlU3EiEJRp//7o4f6F4zFBOPp+Ph7n0dl7rb3WZ+91941Pa699LIZhGIiIiIiIiFRwbuUdgIiIiIiIiDOUvIiIiIiIiEtQ8iIiIiIiIi5ByYuIiIiIiLgEJS8iIiIiIuISlLyIiIiIiIhLsJZ3ACLXwy233EKtWrXKOwwRERERuYTMzEzOnj172TpKXuSmUKtWLTIyMso7DBERERG5BC8vryvW0WNjIiIiIiLiEpS8iIiIiIiIS1DyIiIiIiIiLkHJi4iIiIiIuAQlLyIiIiIi4hKUvIiIiIiIiEtQ8iIiIiIiIi5ByYuIiIiIiLgEJS8iIiIiIuISlLyIiIiIiIhLsJZ3ACLXw+GsM3gPXVTeYYgLSB/bqbxDEBERkUvQzIuIiIiIiLgEJS8iIiIiIuISlLyIiIiIiIhLUPIiIiIiIiIuQcmLiIiIiIi4BCUvIiIiIiLiEpS8iIiIiIiIS1DyIiIiIiIiLkHJSxlYt24dsbGxAKSnp+Pp6XnNbY4cOZL8/PxrbsdZI0aM4PPPP7/q46Kiovjmm29KLY733nuPo0ePllp7IiIiIuK6lLyUgeDgYJKTk0u1zVGjRl235KWwsJDExESefPLJ69LX5Sh5EREREZFzlLxco7y8PJ588kl8fHwICAigXbt2pKWlERwcfNH6a9eupXXr1gQHBxMYGMi8efOA/z9DM2LECIKCgmjSpAnffvstAPHx8QCEhYVht9sv+cf8uTYGDRpEaGgovr6+LFmyxCz/3//9X8LDwwkKCiI0NJRly5YBkJaWht1up3///rRs2ZIFCxbw3HPPMWXKFABycnLo3bs3fn5++Pn5MWrUKLPNrVu3EhoaSmBgILGxsZw5c+ay1+tifc2ePZvQ0FDuv/9+7Ha7ed6JiYkcOnSIbt26YbfbcTgcFBQUMHToUEJCQrDb7Tz11FOcOnXqgn4mTpyIl5eX+SkuyLtsXCIiIiJS8Sl5uUaLFy/m5MmTbN26lQ0bNjBnzpxL1j116hQvvfQSycnJrFu3ju+++46EhAQOHz4MwPHjxwkKCuKXX35hypQpDBgwAICkpCQAVqxYgcPhoHbt2pfs4/jx4/j7+7N69Wo+/vhjnnnmGU6fPs3evXsZNWoU3377Lb/88gvJyck8/fTTFBQUALBx40a6d+/OypUreeKJJ0q0OXr0aPLz89m4cSOrV69m4cKFzJ07F4CePXvSt29ffv31V/7xj3+wdu3aK16z8/tq3749q1atYv369SxcuJAXXniBgoICRowYwV133cWXX36Jw+HAbrfzzjvvUK1aNdasWYPD4cDX15d//vOfF/SRkJBARkaG+XGrdOsV4xIRERGRis1a3gG4uoCAALZv307fvn1p1aoVHTt2vGTdFStWsHfvXjp06GDuMwyDHTt20LBhQ6pWrUqXLl0AaNmyJXv27LnqeCpXrkzPnj0BaNGiBXXr1mXDhg04HA52795NZGRkifoHDx4E4N577yU8PPyibaampjJp0iTc3NyoWrUqzz77LKmpqbRv357NmzeX6M/f3/+KMZ7f1759+4iNjSUjIwOr1cqxY8fYv38/TZo0ueDYhQsX8scff/Dll18CkJ+fzz333OPElRERERERV6fk5Ro1btyYrVu3smTJElJTU3nttdd47733LlrXMAxsNpv5uNZfpaen4+HhYW67u7tTVFRUKjFaLBYMw+Dhhx9mxowZF5QfOHCAatWqXfJ4wzCwWCwXtPnXf16N8/t66qmnmDBhAo8++igANWvWvOTjZ4ZhMHXqVFq3bn3V/YqIiIiIa9NjY9coIyMDi8VCTEwMEyZMwDAMczbjfGFhYezatavEOhSHw+HUQvzq1auTlZV1xXr5+fnmywLWrFnD4cOHsdlstGvXjsWLF7N582az7po1a67YHkDbtm356KOPMAyD06dPM2vWLKKjo6lRowZ+fn4l+tu0aZNTbf7VyZMn8fb2BmDWrFmcPHnSLKtRo0aJ846JiWHixInk5uYCkJuby5YtW666TxERERFxPUpertGmTZsICwvDZrMRGBhIz549sdlsF617xx138PXXXzN69GgCAgLw8fFh6NChFBcXX7GfgQMH0rp168su2Ae488472b17N6GhofTq1YvZs2dTtWpVmjZtyqxZs3jhhRcICAjgvvvuY9KkSU6d45tvvonFYsHf35/Q0FBiYmLo1q0bADNmzGDKlCkEBgby4YcfEhoa6lSbfzVp0iQee+wxwsPD2bBhAw0aNDDL+vfvT69evcwF+0OHDsVutxMaGorNZqNFixY4HI6r7lNEREREXI/FMAyjvIOQ0pGenk5wcDDHjh0r71AqHGt1T7xe+bS8wxAXkD62U3mHICIiclPy8vIiIyPjsnU08yIiIiIiIi5BC/ZdUHx8PKtWrbpg/8qVKyvMrEtMTAwHDhwose+OO+5g6dKl5RSRiIiIiLg6JS8u6NzvvlRkX331VXmHICIiIiI3GD02JiIiIiIiLkHJi4iIiIiIuAQ9NiY3hbq3eegtUiIiIiIuTjMvIiIiIiLiEpS8iIiIiIiIS1DyIiIiIiIiLkHJi4iIiIiIuAQt2JebwuGsM3gPXVTeYYhIGdELOUREbg6aeREREREREZeg5EVERERERFyCkhcREREREXEJSl5ERERERMQlKHkRERERERGXoORFRERERERcgpIXERERERFxCUpeRERERETEJSh5uUk899xzTJkypbzDKHUWi4WcnJzyDkNERERErgMlL+KUwsLC8g5BRERERG5ySl7KWV5eHk8++SQ+Pj4EBATQrl070tLSCAgIoFevXgQFBREcHMyGDRvMY2bOnEloaCiBgYG0atWKzZs3A7Bp0yYiIiIIDAzEx8eHt99++6J9Ll++HF9fX9atW0dBQQFDhw4lJCQEu93OU089xalTp4A/Z2v69+/Pww8/TEBAwCXP4euvv8Zms2G32/Hz8yMlJQWAqKgoXn31VaKiomjatCmDBw/GMAwADh8+TPfu3QkJCcFmszFixAizvV27dtGpUyceeOABAgICmDp1qlk2f/58mjdvTsuWLRk9evTfu+giIiIi4pKs5R3AzW7x4sWcPHmSrVu3AnDixAk2btzIxo0bmTRpElFRUXzxxRc888wzbNmyhZ9//pk5c+awbNkybrnlFpYvX05sbCwbNmzA29ub1NRUbrnlFvLy8ggLC6Nt27YEBweb/X322WeMHz+eRYsW4e3tzVtvvUW1atVYs2YNAKNHj+af//wnkyZNAuCnn35i2bJlVKtW7ZLnMHz4cJKSkggLC6O4uJg//vjDLNu6dSvff/89BQUFREZGMnfuXLp3705cXBzDhg0jMjKSwsJCHnnkERYsWEBMTAzPPPMMM2fOpHnz5uTm5tKiRQtatGiBl5cXL774IitWrKBZs2aMHz/+kjFNnDiRiRMnmtvFBXl/b4BEREREpMJQ8lLOAgIC2L59O3379qVVq1Z07NgRgCZNmhAVFQVA9+7d6dOnD4cOHSIlJYUNGzYQGhpqtpGZmUl+fj55eXn07dsXh8OBm5sbBw8exOFwmMnL9OnTqVSpEkuXLuX2228HYOHChfzxxx98+eWXAOTn53PPPfeYbXfv3v2yiQtAmzZtePXVV+nWrRvt2rXDbrebZXFxcVSqVIlKlSrRo0cPUlNT6dSpE0uWLOHIkSNmvZycHLZv306zZs3YsmULTz31lFmWnZ3N1q1bycjIIDAwkGbNmgHQp08fhgwZctGYEhISSEhIMLet1T0vew4iIiIiUvEpeSlnjRs3ZuvWrSxZsoTU1FRee+013nvvvYvWtVgsGIZB7969SUxMvKD8jTfeoE6dOqxfvx6r1UrXrl05c+aMWR4QEMCyZcvYvHkz4eHhABiGwdSpU2nduvVF+7xS4gJ/znJs2bKFpUuXEhcXR2xsLK+99tolz6G4uBiLxcLatWupVKlSifItW7bg6emJw+G44Nhzj6OJiIiIyM1Ja17KWUZGBhaLhZiYGCZMmIBhGBw8eJDdu3ezbNkyAL788kvq169PvXr16Ny5MzNmzODgwYMAFBcXs27dOgBOnjyJl5cXVquVHTt28P3335foKzAwkK+++opevXqZZTExMUycOJHc3FwAcnNz2bJly1Wdw/bt2/H19aVfv368/PLLrFq1yiybOXMmhYWF5OXlMXv2bKKjo6levToRERGMHTvWrHfo0CEyMjJo1qwZVapUYcaMGWbZ7t27OXHiBC1btmT9+vXs3LkTgGnTpl1VnCIiIiLi2jTzUs42bdrE0KFDMQyD4uJievbsaS5+nzNnDgkJCRiGwezZswGIjIzkrbfeokuXLhQVFVFQUECnTp0IDg5m+PDh9OzZk+TkZLy9vS86m+Lj48PixYvp1KkTY8eOZejQoYwaNYrQ0FAsFgsAQ4YMwdfX1+lzeP3119m5cyeVK1emSpUq/Oc//zHLAgMDiY6O5rfffuPRRx+lW7duACQnJ5OQkIC/vz/w5wxPUlISXl5efP311wwYMIAJEyZQVFRErVq1SE5Opn79+nz44Yd07tyZO++802xLRERERG4OFuPc65+kwkhLS2PQoEHmjIqrioqKYtCgQTzyyCPlHQrW6p54vfJpeYchImUkfWyn8g5BRESukZeXFxkZGZeto8fGRERERETEJeixsQooKiqqws26HD16lHbt2l2wv23btrzzzjsXPSYtLa2MoxIRERGRm4mSF3FK7dq1L/oGMBERERGR60WPjYmIiIiIiEtQ8iIiIiIiIi5Bj43JTaHubR56G5GIiIiIi9PMi4iIiIiIuAQlLyIiIiIi4hKUvIiIiIiIiEtQ8iIiIiIiIi5ByYuIiIiIiLgEvW1MbgqHs87gPXRReYchct3pLXsiInIj0cyLiIiIiIi4BCUvIiIiIiLiEpS8iIiIiIiIS1DyIiIiIiIiLkHJi4iIiIiIuAQlLyIiIiIi4hKUvIiIiIiIiEtQ8iIiIiIiIi5ByYuLSktLIzg4GIBDhw7x0EMP/e22HA4HX3zxRYl9drudvLy8a4rxWp06dYrx48eXawwiIiIiUnEoebnOCgsLS73Nu+66i6VLl/7t4y+WvDgcDm699dZrDe2yiouLKS4uvmS5khcRERER+SslL1fBYrEwbtw4QkNDadSoEdOnTzfL1q1bR8uWLbHZbISEhPDzzz8DkJ6ejqenJ4mJiURERDB58mSioqIYPHgwkZGR3H333bzzzjvMmTOHsLAwGjZsyJw5c8x2e/ToQXBwMDabjUceeYSjR49eENe5PgC+/fZb7Ha7+fHw8ODTTz+lsLCQ9u3bExwcjK+vL7GxseTm5nL06FFGjBhBamoqdrud+Ph481xzcnKcOrcRI0YQFBREkyZN+Pbbby97DUeOHEnPnj3p2rUrdrud33//ncGDB/PAAw9gt9tp1aoVu3btAiA+Pp5Tp05ht9vNWabDhw/TvXt3QkJCsNlsjBgx4m+NpYiIiIi4HiUvV8nDw4PVq1fz7bff0r9/fwoLC8nPz6dr166MHDmSjRs3MnHiRLp168bp06cBOH78OE2aNGH58uUMGDAAgAMHDpCWlsbq1asZMWIEmzdvZsWKFcydO5eEhASzv/fee49169axceNGwsPDSUxMvGx8HTt2xOFw4HA4GDRoEPfddx9du3bF3d2d2bNns27dOjZv3kyNGjWYOnUqtWvXJjExkejoaBwOB0lJSSXac+bcgoKC+OWXX5gyZYp5fpezdOlSkpKS2LhxI/Xr12fIkCGsXbsWh8PByy+/bLaRlJTE7bffjsPhYN26dQDExcXRr18/1qxZw6+//sqaNWtYsGDBBX1MnDgRLy8v81NcUL6PwImIiIjItbOWdwCuJjY2FoD77rsPq9XK4cOHOXnyJJUrV6Z9+/YAhIeHU7t2bTZu3Ei9evXw8PDg6aefLtHOE088gZubG3fddReenp48+uijAAQFBfH7779z5swZPDw8SE5OZubMmZw9e5a8vDzq1q3rVJxLly5l5MiRLFu2jOrVq1NcXMy///1vFi1aRGFhIVlZWURGRl6xnR07dlz23KpWrUqXLl0AaNmyJXv27Llim4888gi1a9c2t7/77jsmT55MdnY2xcXF/PHHHxc97vTp0yxZsoQjR46Y+3Jycti+ffsFdRMSEkokgdbqnleMS0REREQqNiUvV8nDw8P87u7uTmFhIYZhYLFYLqh7bl/VqlUvKD+/nXPb7u7uwJ9rY3766SemTJnCihUrqFWrFl999dUVZ14ANm/eTK9evVi0aBF33XUXALNnz+bHH380k5n333+fZcuWXbGtK53b+edRVFR0xTarVatmfj9w4AD9+/dnzZo1NG7cmI0bN9K6deuLHldcXIzFYmHt2rVUqlTpiv2IiIiIyI1Fj42VgubNm3P27FmWLFkCwIoVKzh69Cj+/v7X1O7JkyepUaMGNWvWJD8/nw8++OCKx/z22288+uijTJ8+HV9f3xJt3XnnnVSvXp3s7Gw++eQTs6xGjRpkZWVd13M7Jysri8qVK1O3bl0Mw2DKlCkl4srNzTVfclC9enUiIiIYO3asWefQoUNkZGSUSiwiIiIiUrEpeSkFlStXZt68eQwbNgybzcarr77K3LlzqVq16jW126FDB5o0aULz5s1p3749drv9isdMmzaNzMxMBgwYYC7a/+qrr3j22WfJycnBx8eHrl27EhERYR7Tpk0bTp8+TUBAgLlgv6zP7Rx/f3+eeOIJfH19iYqKokGDBmZZzZo1iY2Nxd/f31ywn5yczLZt2/D398ff35/HH3+c48ePl0osIiIiIlKxWQzDMMo7CJGyZq3uidcrn5Z3GCLXXfrYTuUdgoiIiFO8vLyu+ESN0zMvKSkpjBs3Dvjz0aRNmzZdW3QiIiIiIiJXwakF+yNHjmT16tXs2bOHIUOG4ObmRnx8vPl7HyJ/dfToUdq1a3fB/rZt2/LOO++UQ0QiIiIiciNwKnlZuHAhv/zyi7nuoF69emRnZ5dpYOK6ateujcPhKO8wREREROQG49RjYx4eHuYrfEVERERERMqDUzMvDRs25KeffsJisVBcXMxbb71Vaq/KFRERERERcYZTbxs7cuQIcXFxLFmyBDc3NyIiIkhOTi7xK+kiFZkzb68QERERkfLjzN9rTs281KlTh8WLF5Obm0txcXGJX0gXERERERG5HpxKXgB2797N3r17zV87B+jYsWOZBCUiIiIiInI+p5KXgQMHMmvWLJo1a2Yu3LdYLEpeRERERETkunEqeUlJSWHfvn1UqVKlrOMRERERERG5KKdeldywYUMqV65c1rGIiIiIiIhcklMzL++++y6dO3embdu2eHh4mPv79u1bZoGJlKbDWWfwHrqovMMQkTKWPrZTeYcgIiJlyKnk5e233+b333/H4XCUWPMiIiIiIiJyvTiVvPz666/s3LlTCYuIiIiIiJQbp9a8NG/enNOnT5d1LCIiIiIiIpfk1MxL9erVCQoKon379iXWvIwfP77MAhMREREREfkrp5KXZs2a0axZs7KORURERERE5JKcSl7++c9/lnUcIiIiIiIil+VU8gIwf/58HA4HZ86cMffpsTEREREREblenFqw/+qrrzJ9+nSmTZtGUVERc+bM4fjx42Udm4iIiIiIiMmp5OWHH34gJSWFWrVq8e6777J27VqOHj3qVAcpKSncd9992O12Nm3adE3BAqSlpfHdd9+Z24cOHeKhhx665navxnvvvef0+ZenkSNHMmjQoPIOQ0RERESkVDiVvHh4eODm5obFYqGgoIA6derw22+/OdVBUlISiYmJOBwO/P39zf2FhYV/K+Dzk5e77rqLpUuX/q22/q6KmLz83etZUdoXEREREbkSp5KX6tWrk5ubS3h4OHFxcSQkJFCpUqUrHte/f3+WL1/OkCFDCAsLw2Kx8O677xIVFcXrr7/Opk2biIiIIDAwEB8fH95++23z2KysLF544QX8/f0JCAigd+/eOBwOkpKSmDFjBna7ncTERNLT0/H09DSPW7x4MYGBgdhsNlq1asXWrVuBP5Meu91O3759CQgIwNfXl3Xr1l02/mnTpuHj44Pdbsff35/Vq1eTmJjIoUOH6NatG3a7HYfDQU5ODr1798bPzw8/Pz9GjRplthEVFcWrr75KVFQUTZs2ZfDgwRiGwY4dO8w3uBmGgaenJ8OGDQP+nOlq06YNAEeOHOGxxx7D398fPz8/PvzwQ7Ntb29v/vWvf/HQQw8RFxdHVlYW3bp1w8fHh/bt27N79+4rjpHFYmHkyJE8+OCD3HvvvXz22Wclyv46XtnZ2bz44ouEhIRgs9mIj4+noKAAgDFjxpgzbHa7nf379wOwdu1aWrduTXBwMIGBgcybNw/AHLcRI0YQFBREkyZN+Pbbb82+V65cSUREBAEBAdhsNlJSUgDYtWsXnTp14oEHHiAgIICpU6de9LwmTpyIl5eX+SkuyLvitRARERGRis2pBfufffYZVquVd955h4kTJ3Ly5Enmzp17xePef/99Nm7cyKBBg3jkkUewWCycPXuWtLQ0ALKzs0lNTeWWW24hLy+PsLAw2rZtS3BwMK+++irVqlVjw4YNuLm5kZmZSa1atYiPjycnJ4cJEyYAf/4RfM7Ro0fp0aMHS5cuxd/fn+TkZLp3787mzZsB2LJlC9OmTWPq1KkkJSUxbNgw/vd///eS8Q8cOJBt27Zx1113UVBQwNmzZwkNDeW///0vX375JX5+fgAMGTKE/Px8Nm7cSF5eHuHh4fj4+PDEE08AsHXrVr7//nsKCgqIjIxk7ty5dO/enby8PA4cOMDx48e59957WbJkCQCpqalER0cDfyaAzZs3Z8GCBRw9epSgoCDsdjshISEAHDhwgCVLlmCxWBg4cCA1atRg69atHDt2jMDAQLp3737FcbJYLPz888/s3buXkJAQwsPDufvuuwFKjFefPn2IjIzko48+wjAMXnzxRaZMmcJzzz3HhAkT+P3337n11lvJzc3Fzc2NU6dO8dJLL7Fo0SLq1avHsWPHCAoK4sEHHwTg+PHjBAUFkZiYyOLFi/k//+f/0LFjR06cOMFjjz3G/PnzCQsLo7i4mFOnTlFUVMQzzzzDzJkzad68Obm5ubRo0YIWLVoQGBhY4pwSEhJISEgwt63VPRERERER1+bUzEudOnWoXLkyt956K8OGDWPChAk0aNDgb3XYu3dv83teXp45u9KiRQv279+Pw+EA4JtvvmHw4MG4uf0ZYq1ata7Y9urVq81ZEoDY2FgyMjL4/fffgT9/ryY4OBiAli1bsmfPnsu217p1a5599lkmTZrEvn37qFat2kXrpaamEh8fj5ubG1WrVuXZZ58lNTXVLI+Li6NSpUpUqVKFHj16mGVt2rQhNTWV1NRUevToQV5eHllZWSWSl9TUVF555RUAateuTdeuXfnhhx/Mtnv16oXFYgFg6dKlPP/88wB4enrStWvXK14zgBdeeAGAxo0bEx4ezvLly82yv47XwoULeeedd7Db7dx///0sX76cXbt2UaNGDZo2bUqPHj344IMPOHHiBB4eHqxYsYK9e/fSoUMH7HY70dHR5qwTQNWqVenSpQtQcjxWrlyJj48PYWFhALi5uVGzZk127NjBli1beOqpp7Db7YSFhZGdnW3OromIiIjIjc2pmZf09HTGjRvHnj17Sqx9ODdTcDX+mgC88cYb1KlTh/Xr12O1WunatWuJVzFfLcMwzD/k/+rcPg8PD3Ofu7v7FddxzJ8/n19++YW0tDQ6duzImDFjeOqpp5zq92JxnF8WHR3NokWLOHHiBO+//z67du1i/vz57Nu3j6CgoEu29dftv15PwzAuez7Oulz7CxcupHHjxhccs2rVKlasWEFaWhotWrTgs88+wzAMbDYby5Ytu6B+enr6BeNRVFR02bjOPV53LsEVERERkZuLUzMv3bt35/bbb6dfv34MHDjQ/FyrkydP4uXlhdVqZceOHXz//fdmWUxMDO+88w7FxcUAZGZmAlCjRg2ysrIu2l7Lli1xOBxs27YNgDlz5uDl5UXdunWvOrbCwkL27NlDcHAwgwYNolu3bqxZs+aiMbRt29Z8lOr06dPMmjXLnDkBmDlzJoWFheTl5TF79myzLDo6mh9++IH9+/dz7733Eh0dzahRo2jVqpU54xQdHW2uc8nMzGTBggW0bt36ojG3adOG6dOnA3DixAkWLFjg1Ln+97//Bf5MKH766SfCw8MvWi8mJoaxY8eaSd/JkyfZvXs32dnZHDlyhIiICN58803Cw8NZv349YWFh7Nq1q0SS63A4yM/Pv2w8YWFhbNu2jRUrVgBQXFzMiRMnaNasGVWqVGHGjBlm3d27d3PixAmnzlNEREREXJtTMy9nzpwpsZi+tAwfPpyePXuSnJyMt7d3iT/K//3vfzNgwAD8/PyoXLkyDzzwAB999BGPPfYYM2fOxG6307VrV5599lnzmFq1ajFz5kxiY2MpKiri9ttv54svvvhbsRUVFdGrVy9OnjyJ1WqlVq1aZmLQv39/evXqRZUqVfjkk0948803+cc//mE+rvbEE0/QrVs3s63AwECio6P57bffePTRR82yOnXqUKdOHfNRtlatWnHo0CGGDBliHvv+++8THx+PzWajuLiYYcOGmetdzvfmm2/Su3dvfHx8aNiwIW3btnXqXG+55RYefPBBMjMzmTx5srne5XzvvfceQ4YMwW634+bmRqVKlRg3bhweHh5069aN06dPY7FYaNq0KXFxcdx22218/fXXDB48mAEDBlBQUECDBg1YuHDhZeO54447WLBgAQMHDiQ7OxuLxcLo0aOJiYnh66+/ZsCAAUyYMIGioiJq1apFcnKyU+cpIiIiIq7NYjjxrNEzzzzD+PHj8fLyuh4x3VCioqLMFxZURBaLhezs7Euu57lRWKt74vXKp+UdhoiUsfSxnco7BBER+Zu8vLzIyMi4bB2nZl5OnDiBzWYjPDy8xDqFvzurISIiIiIicrWcSl6efvppnn766bKOpVw4HA6ee+65C/bHxcUxYMCAa27/3GuGy1NiYiLz58+/YP+8efNKbZG/iIiIiEhZc+qxMRFXp8fGRG4OemxMRMR1OfPYmFNvGxMRERERESlvSl5ERERERMQlOLXmRcTV1b3NQ4+TiIiIiLg4p2deUlJSGDduHAC//fYbmzZtKrOgREREREREzudU8jJy5EiSkpL4+OOP/zzIzY34+PgyDUxEREREROSvnEpeFi5cyDfffEPVqlUBqFevHtnZ2WUamIiIiIiIyF85lbx4eHjg7u5e1rGIiIiIiIhcklML9hs2bMhPP/2ExWKhuLiYt956C39//7KOTaTUHM46g/fQReUdhoiUIr2EQ0Tk5uNU8vL+++8TFxfH5s2bqVKlChERESQnJ5d1bCIiIiIiIqYrJi9FRUXMmTOHxYsXk5ubS3FxMdWqVbsesYmIiIiIiJiuuObF3d2defPmAVClShUlLiIiIiIiUi6cWrDftm1bPv/887KORURERERE5JKcXvNy/PhxevfuTdWqVTEMA4vFwtGjR8s6PhEREREREcDJ5GXdunVlHYeIiIiIiMhlOf2qZBERERERkfLkVPLSqFEjLBbLBfv37t1b6gGJiIiIiIhcjFPJyzfffGN+P3PmDDNnzuSOO+4os6DE9Y0YMQJfX1+efPLJ8g5FRERERG4QTiUvvr6+JbaDgoJo06ZNmQQkN4bExMTyDkFEREREbjBOvSr5fCdPnmTfvn2lHYtUAGvXrqV169YEBwcTGBjIvHnzKCwspH379gQHB+Pr60tsbCy5ubkArFq1iqCgIOx2O35+fvznP/8B4LnnnmPKlCkAjBw5kmeeeYbOnTvj4+ND69atOXHihNnnhAkTCAkJITAwkI4dO3Lw4EEAvv76a2w2m9l2SkoKAGPGjOG+++7Dbrdjt9vZv3//9bxEIiIiIlJOnJp5eeCBB8w1L0VFRezbt4+BAweWaWBy/Z06dYqXXnqJRYsWUa9ePY4dO0ZQUBBhYWHMnj2bO++8E8Mw6Nu3L1OnTmXQoEG8/fbbDBw4kGeeeQb4M7G9mNWrV7N27Vpq1qzJU089xQcffMDrr7/O7Nmz2blzJytXrsTd3Z2ZM2fSr18/UlJSGD58OElJSYSFhVFcXMwff/zByZMnmTBhAr///ju33norubm5uLldmINPnDiRiRMnmtvFBXllc9FERERE5LpxKnmZMGHC/z/AaqVRo0bcddddZRaUlI8VK1awd+9eOnToYO4zDIOdO3fy/fffs2jRIgoLC8nKyiIyMhKAhx56iDFjxrB7925at25NeHj4Rdvu0KEDNWvWBKBly5Zs2rQJgIULF7Ju3TqCgoKAP5Njd3d3ANq0acOrr75Kt27daNeuHXa7naKiIpo2bUqPHj1o164dnTp1wsvL64L+EhISSEhIMLet1T1L4QqJiIiISHly6rGxPXv20KpVK1q1asWDDz7IXXfdxX//+9+yjk2uM8MwsNlsOBwO83PgwAEOHjzIjz/+yLJly9i0aRODBg3izJkzALz66qt888031KtXjzfeeIO+fftetG0PDw/zu7u7O4WFhWafw4cPN/vbtGkTDocD+HP2ZPr06VSpUoW4uDjGjx+Pu7s7q1at4tVXX+Xo0aO0aNGC5cuXl+2FEREREZEKwank5dzahSvtE9cWFhbGrl27WLJkibnP4XBw/Phx7rzzTqpXr052djaffPKJWb5jxw4aN27Miy++yBtvvMGqVauuqs+YmBimTp1qroEpKChg/fr1AGzfvh1fX1/69evHyy+/zKpVq8jOzubIkSNERETw5ptvEh4ebtYXERERkRvbZR8bW7duHatXr+bYsWNMnTrV3J+VlUV+fn6ZByfX1x133MHXX3/N4MGDGTBgAAUFBTRo0IDPP/+cr7/+Gh8fH+rXr09ERAS//fYbAJMnT2bp0qVUrlwZd3d33n333avqs2fPnhw/fpyoqCgsFguFhYU8//zz3H///bz++uvs3LmTypUrU6VKFf7zn/+QlZVFt27dOH36NBaLhaZNmxIXF1cWl0NEREREKhiLYRjGpQpTUlJYuHAhX331FTExMeb+GjVq0LNnT4KDg69LkCLXylrdE69XPi3vMESkFKWP7VTeIYiISCny8vIiIyPjsnUuO/PSpUsXunTpwv/8z/+UWMQtIiIiIiJyvTn1trEOHTrw66+/4nA4zIXawCUXZ4uIiIiIiJQ2p5KXcePG8fnnn3PgwAFatWrF999/T5s2bZS8iIiIiIjIdePU28ZmzpzJihUr8PLyYt68eaxdu5bKlSuXdWwiIiIiIiImp5IXDw8PPDw8KC4uxjAMmjVrRnp6ehmHJiIiIiIi8v859dhYlSpVKCgowG63M2TIELy8vMjNzS3r2ERKTd3bPPRmIhEREREX59TMy9SpU8nPz+fdd9/l5MmTLFu2jJkzZ5Z1bCIiIiIiIqbL/s7L+QoLC7FanZqsEalQnHlvuIiIiIiUH2f+XnNq5mXLli3Y7XYaNWoEwC+//MKQIUOuPUIREREREREnOZW89OvXjylTpuDp6QlAYGAgixYtKtPARERERERE/sqp5CU7O5vw8HBz22KxUKlSpTILSkRERERE5HxOLWCxWq0UFBRgsVgAyMjIwM3NqbxHpEI4nHUG76GaLRS5megNgyIiNx6nHxt77LHHOHbsGCNHjiQyMpLBgweXdWwiIiIiIiImp2ZeevToQePGjUlJSSE3N5dPP/2UiIiIso5NRERERETEdNnkpW/fvkydOhWAzMxMxo0bd12CEhEREREROd9lHxtbtWqV+X3UqFFlHoyIiIiIiMilXDZ5+evvV17Fb1mKiIiIiIiUuss+Nnb27Fm2bduGYRglvp/j4+NT5gGKiIiIiIjAFZKX3NxcOnbsaG7/9bvFYmHv3r1lF5mIiIiIiMhfXDZ5SU9Pv05hyPWWkpLC0KFDueWWW5g5cyb+/v7X1J7FYiE7O5tq1aqVUoQiIiIiIiU59apkufEkJSWRmJjIE088UWJ/YWEhVqv+ZyEiIiIiFY9TP1IpN5b+/fuzfPlyhgwZQlhYGBaLhXfffZeoqChef/11srOzefHFFwkJCcFmsxEfH09BQQEAY8aM4b777sNut2O329m/f7/Z7v/9v/+X0NBQGjVqxPTp08393t7ejBo1irCwMBo1asSYMWPMssOHD9O9e3ezrxEjRgBQXFxMv379aN68OQEBAQQFBXHmzBkyMzNp164d/v7+2Gw2evXqdZ2umoiIiIiUN/0n9pvQ+++/z8aNGxk0aBCPPPIIFouFs2fPkpaWBkCfPn2IjIzko48+wjAMXnzxRaZMmcJzzz3HhAkT+P3337n11lvJzc3Fze3/578eHh6sXr2abdu2ERISQs+ePc1ZnFOnTrFixQoyMzNp0qQJvXr1on79+sTFxTFs2DAiIyMpLCzkkUceYcGCBXh7e/PDDz+wdetW3NzcyMrKonLlysyaNQtvb2++++47AE6cOHHdr5+IiIiIlA8lLwJA7969ze8LFy5k1apVvPvuuwDk5eVRuXJlatSoQdOmTenRowft2rWjU6dOeHl5mcfFxsYCcN9992G1Wjl8+LBZfq6sVq1aNG7cmH379nH77bezZMkSjhw5YraRk5PD9u3bad26NQUFBfTu3ZuHHnqITp064ebmRosWLfj3v//NwIEDadWqFe3bt7/o+UycOJGJEyea28UFeaV0pURERESkvCh5EYASC+0Nw2DhwoU0btz4gnqrVq1ixYoVpKWl0aJFCz777DMiIiKAP2deznF3d6ewsNDcvlhZcXExFouFtWvXUqlSpQv62rJlCz/++CNLly7l9ddfZ9myZbRs2RKHw0Fqairz5s1j+PDhrF+/Hnd39xLHJiQkkJCQYG5bq3v+jasiIiIiIhWJ1rzIBWJiYhg7dqyZfJw8eZLdu3eTnZ3NkSNHiIiI4M033yQ8PJz169f/7X6qV69OREQEY8eONfcdOnSIjIwMMjMzOX36NO3ateOtt97C29ubrVu3sm/fPqpVq0b37t2ZPHkyO3fuJCcn55rPWUREREQqPiUvcoH33nsPq9WK3W7HZrMRHR1Neno6WVlZdO3a1VwsX1BQQFxc3DX1lZyczLZt2/D398ff35/HH3+c48ePc/DgQdq2bYvNZsPf3x8/Pz86dOhAWloaQUFB2O12HnzwQd555x1uu+22UjpzEREREanILIZhGOUdhEhZs1b3xOuVT8s7DBG5jtLHdirvEERE5Cp4eXmRkZFx2TqaeREREREREZeg5EVERERERFyCkhcREREREXEJSl5ERERERMQlKHkRERERERGXoORFRERERERcgrW8AxC5Hure5qHXpoqIiIi4OM28iIiIiIiIS1DyIiIiIiIiLkHJi4iIiIiIuAQlLyIiIiIi4hKUvIiIiIiIiEvQ28bkpnA46wzeQxeVdxgiIiIiLqGivqVVMy8iIiIiIuISlLyIiIiIiIhLUPIiIiIiIiIuQcmLiIiIiIi4BCUvIiIiIiLiEpS8iIiIiIiIS1DyIiIiIiIiLkHJi4iIiIiIuAQlL9fAYrGQk5NT3mFctfT0dD788MMya9/hcPDFF1+UWfsiIiIicnNS8nITKu/kpbCwsMz6FhEREZEbl5KXqzB//nyaN29Oy5YtGT16tLl/8eLFBAYGYrPZaNWqFVu3bjXLpk+fjt1uJyAggODgYNLT00lPT8fT09Osk5OTg8ViMbctFgtvv/02ISEhNG7cmNTUVF5//XXuv/9+fH192bJli1l35syZhIaGEhgYSKtWrdi8eTMAn3zyCe3bt+fpp5/G39+f4OBg9u7dC0B8fDxbt27FbrcTExNz2XOeMGECISEhBAYG0rFjRw4ePAjAyJEjeeaZZ+jcuTM+Pj60bt2aEydOcPToUUaMGEFqaip2u534+HjznN59912ioqJ4/fXXOXLkCI899hj+/v74+fmVSKa8vb15/fXXiYyMpEmTJkycOBGAuXPn0r59e7NeUVERDRs2LHG9z5k4cSJeXl7mp7gg77LnKSIiIiIVn5IXJx09epQXX3yRlJQUVq5cyS233ALAiRMn6NGjB59++ikbN26kT58+dO/eHYC0tDT+9a9/8T//8z9s2LCBZcuWUbt2baf6q1GjBmvWrGHcuHF06dKF8PBw1q9fT1xcHP/6178A+Pnnn5kzZw7Lli3j119/ZcyYMcTGxpptrF69mrFjx7Jp0yaio6MZN24cAElJSfj4+OBwOPjqq68uGcPs2bPZuXMnK1eu5Ndff+Xpp5+mX79+Jdr/9NNP2bp1K7Vr1+aDDz6gdu3aJCYmEh0djcPhICkpyax/9uxZ0tLSeOedd+jfvz/Nmzdn06ZNLFmyhNGjR7NmzRqz7pEjR1i2bBmrVq1i0qRJrF69mq5du7Jjxw527doFwMKFC2nSpAk+Pj4XxJ6QkEBGRob5cat0q1PXXUREREQqLiUvTlq1ahWBgYE0a9YMgD59+gCwYcMG7HY7/v7+AMTGxpKRkcHvv//OokWLePbZZ6lXrx4AVapUoUqVKk719+STTwIQGBiIm5sbnTp1AiAoKMicQUlJSWHDhg2EhoZit9v5xz/+QWZmJvn5+QCEh4fTsGFDAFq2bMmePXuu6pwXLlxIamoqQUFB2O12xo8fz/79+83yDh06ULNmTafb7927t/k9NTWVV155BYDatWvTtWtXfvjhB7P8+eefB8DT05PHHnuMH374AXd3d/r27cvUqVMBmDJlSolkSkRERERubNbyDsBVGIZxyf1/feTrnIvtO8dqtVJUVGRunzlz5oI6Hh4eALi7u5uzPOe2z60ZMQyD3r17k5iYeNF+zrVx/nHOMgyD4cOHl0g6rqX9atWqldg+/xpd7pqdK3vxxRfx8/Pj6aefZu/evVd87E1EREREbhyaeXFSy5YtWb9+PTt37gRg2rRpANx///04HA62bdsGwJw5c/Dy8qJu3bp07tyZGTNmcPjwYQByc3PJzc2lbt26FBYWsmPHDgBmzJjxt2I61/65dSjFxcWsW7fuisfVqFGDrKysK9aLiYlh6tSpnDhxAoCCggLWr19fKu1HR0eb61wyMzNZsGABrVu3NsunT58O/PlY3sKFC2nTpg0Ad9xxB507d+bxxx8nPj4ed3f3K8YjIiIiIjcGJS9Oql27Nh9++CGdO3cmLCwMN7c/L90dd9zBzJkziY2NJSAggP/85z/mm7YiIyMZPnw47dq1IyAggFatWpGZmYnVauX999+nQ4cOREZGcvbs2b8VU2RkJG+99RZdunQhICAAPz8/Pv/88yseZ7PZaNasGX5+fpeduejZsyc9evQgKiqKgIAA7HY7S5cuvWL7bdq04fTp0wQEBJgL9s/3/vvvs3HjRmw2Gw899BDDhg0jJCTELG/YsCERERGEhITQv3//EmUvvvgimZmZvPDCC1eMRURERERuHBbjUs9DiZQTb29vvvnmG/z8/C5aPn78eHbs2MHHH3/sdJvW6p54vfJpaYUoIiIickNLH9vpuvfp5eVFRkbGZetozYu4FF9fXywWC4sXLy7vUERERETkOlPyIgQHB1+w2N7X15fk5ORyiSc9Pf2SZX/9jRsRERERubkoeRGnFvmLiIiIiJQ3LdgXERERERGXoORFRERERERcgh4bk5tC3ds8yuWtGSIiIiJSejTzIiIiIiIiLkHJi4iIiIiIuAQlLyIiIiIi4hKUvIiIiIiIiEtQ8iIiIiIiIi5BbxuTm8LhrDN4D11U3mHIVdIb4kREROSvNPMiIiIiIiIuQcmLiIiIiIi4BCUvIiIiIiLiEpS8iIiIiIiIS1DyIiIiIiIiLkHJi4iIiIiIuAQlLyIiIiIi4hKUvIiIiIiIiEtQ8lLGRo4cSX5+PgAjRozg888/ByAtLY3vvvvOrJeeno6np2e5xOiMTz75hG7dupV3GKZPPvmEnTt3lncYIiIiInIdKXkpY6NGjTKTl8TERJ588kngwuSlIiksLCzvEK5IyYuIiIjIzUfJSxmKj48HICwsDLvdTseOHZkyZQoOh4OkpCRmzJiB3W4nMTHxgmPXrl1L69atCQ4OJjAwkHnz5l2yn5ycHGrWrElBQQEAgYGBxMbGArB3717uueces17v3r3x8/PDz8+PUaNGmW1ERUUxbNgw2rRpQ/v27cnPz+ell17i3nvv5aGHHmL16tVXPN9t27bRvn17bDYbNpuNpKQkAHbv3k10dDQ2mw273c7ChQvNYywWCzk5Oea2p6cn6enpAHh7ezNq1CjCwsJo1KgRY8aMAWDatGmsW7eO/v37Y7fb+fbbby+IZeLEiXh5eZmf4oK8K8YvIiIiIhWbtbwDuJElJSXxwQcfsGLFCqpVq8Zzzz0HgN1uJz4+npycHCZMmABg/sEOcOrUKV566SUWLVpEvXr1OHbsGEFBQTz44IPUrVv3gn6qVauGr68vK1euxNfXl6KiItauXQvA999/T3R0NACjR48mPz+fjRs3kpeXR3h4OD4+PjzxxBMAOBwOFi9eTKVKlZg8eTL79u1jy5YtFBQUEBkZibe39yXPtbCwkC5dujBmzBi6d+8OwLFjxwCIjY3l+eefp0+fPuzatYsWLVoQFBTE3XfffcVreOrUKVasWEFmZiZNmjShV69evPDCC8yaNYtBgwbxyCOPXPS4hIQEEhISzG1r9Yr7SJ6IiIiIOEczLxXQihUr2Lt3Lx06dMButxMdHY1hGOzYseOSx0RHR5OamsqSJUto3749zZo1Y/PmzaSmpprJS2pqKvHx8bi5uVG1alWeffZZUlNTzTZ69uxJpUqVAFi6dClxcXFUqlSJKlWq0KNHj8vGvGPHDgoLC83EBf6cRcnOzsbhcPD8888D0LRpU8LDw/npp5+cuhbnZpBq1apF48aN2bdvn1PHiYiIiMiNRzMvFZBhGNhsNpYtW+b0MdHR0QwePJijR4/StWtX6tevz/fff8+PP/5oPr5lGAYWi6XEcX/drlatWokYSsO5di7Vr7u7O0VFReb+M2fOlKjn4eFhfnd3d3eJ9TgiIiIiUjY081LGqlevTlZW1gX7a9SocdH98OcamV27drFkyRJzn8PhMBf+X0xoaCjbt28nNTWViIgIoqOjmTRpEnfffTd33nknAG3btuWjjz7CMAxOnz7NrFmzzFmZ87Vp04aZM2dSWFhIXl4es2fPvux5NmvWjMqVKzN37lxz37Fjx6hRowZ2u51PP/0UgD179vDzzz/z4IMPAnDPPfeY62nmz5/P6dOnL9vPOZe7fiIiIiJyY1LyUsYGDhxI69atsdvtHD161Nz/2GOPsW7duosu2L/jjjv4+uuvGT16NAEBAfj4+DB06FCKi4sv2Y/VaiU8PJwGDRpw66234uvrS0FBQYnk5M0338RiseDv709oaCgxMTGXfP1xnz59aNCgAT4+PnTq1ImIiIjLnqfVaiUlJYUPP/wQf39/bDab+ZKB5ORkZs2aRUBAAI8//jjTpk0z17u89957vPLKKzz44IP8+uuvZqJ1JX369CExMfGSC/ZFRERE5MZjMUrr+SCRCsxa3ROvVz4t7zDkKqWP7VTeIYiIiMh14uXlRUZGxmXraOZFRERERERcghbsuxCHw2G+bvmv4uLiGDBgwHWJYdq0aUyZMuWC/ZMnT77io2UiIiIiItdCj43JTUGPjbkmPTYmIiJy89BjYyIiIiIicsNQ8iIiIiIiIi5Ba17kplD3Ng89giQiIiLi4jTzIiIiIiIiLkHJi4iIiIiIuAQlLyIiIiIi4hKUvIiIiIiIiEtQ8iIiIiIiIi5ByYuIiIiIiLgEJS8iIiIiIuISlLyIiIiIiIhLUPIiIiIiIiIuQcmLiIiIiIi4BIthGEZ5ByFS1qxWK3Xr1i3vMOQ8OTk5VKtWrbzDkL/QmFRMGpeKR2NS8WhMKqarGZfMzEzOnj172TrW0ghKpKKrW7cuGRkZ5R2GnMfLy0vjUsFoTComjUvFozGpeDQmFVNpj4seGxMREREREZeg5EVERERERFyCkhe5KSQkJJR3CHIRGpeKR2NSMWlcKh6NScWjMamYSntctGBfRERERERcgmZeRERERETEJSh5ERERERERl6DkRVzarl27CAsL49577yUkJIStW7detN7HH39M06ZNueeee+jTpw+FhYVm2TfffEPz5s1p0qQJjz/+ODk5Odcr/BvStY5Jeno6VqsVu91ufvbs2XM9T+GG48yYpKenExUVxW233UZwcPAF5bpPSt+1jovuldLnzJgsWbKE0NBQfHx88PPzY9iwYfz1CXzdK6XvWsdF90rpc2ZMVq5caV5vX19fXnrppRK/4fK37xVDxIU99NBDxvTp0w3DMIy5c+caLVq0uKDO3r17jXr16hmHDx82iouLjc6dOxtJSUmGYRhGdna2Ubt2bWPbtm2GYRjGK6+8YgwdOvS6xX8jutYx2bdvn3HnnXdez5BveM6MyfHjx43ly5cb33zzjREUFFSiTPdJ2bjWcdG9UvqcGZNff/3V2LNnj2EYhpGXl2c8+OCDRnJysmEYulfKyrWOi+6V0ufMmJw+fdrIz883DMMwioqKjMcee8yYNGmSYRjXdq8oeRGXdeTIEeO2224zCgoKDMMwjOLiYqNOnTrGvn37StQbP3680bdvX3N70aJFRqtWrQzDMIwvvvjC6Nixo1m2ZcsWo2HDhmUd+g2rNMZE/5IpXc6OyTlLly694I9k3SelrzTGRfdK6braMTnnlVdeMUaPHm0Yhu6VslAa46J7pXT9nTHJy8szHn74YWPy5MmGYVzbvaLHxsRlHTx4kLvuugur1QqAxWKhQYMGHDhwoES9AwcO0LBhQ3Pb29vbrHOxst9++43i4uLrcAY3ntIYE4A//viDBx54gMDAQBITEykqKro+J3ADcnZMLkf3SekrjXEB3Sul6e+MyeHDh/nyyy/p2LEjoHulLJTGuIDuldJ0NWOSnp6O3W7H09OTGjVq0KdPH+Da7hUlL+LSLBZLiW3jEm/+/mu98+uc34Zcm2sdk3r16pGRkcHatWtJTU1l+fLlvPvuu2UT7E3C2TG5mjbk2l3ruOheKX1XMyZ//PEHnTt35rXXXiMwMPCSbci1u9Zx0b1S+pwdE29vbxwOB4cPH+bs2bPMnz//km04S8mLuKy7776bjIwMc6G3YRgcPHiQBg0alKjXoEED0tPTze39+/ebdc4vS09Pp379+ri56db4O0pjTG655RZq164NQM2aNenduzfLly+/PidwA3J2TC5H90npK41x0b1Suq5mTLKzs3n44YeJiYkp8QN8uldKX2mMi+6V0vV3/v+rWrVqPPXUUyQnJwPXdq/obhKXVbt2be6//35mzZoFwLx58/D29sbb27tEvccff5wFCxZw5MgRDMMgKSmJp556CoCHH36YtWvXsn37dgCmTp1qlsnVK40xOXr0KAUFBQDmf6W5//77r+t53EicHZPL0X1S+kpjXHSvlC5nxyQnJ4eHH36Y9u3b8+abb5Yo071S+kpjXHSvlC5nx2TPnj3mdc/Pz2f+/PnYbDbgGu8VJ9fmiFRI27dvN1q0aGE0bdrUCAoKMjZv3mwYhmE8//zzRkpKilnvww8/NO655x6jUaNGxvPPP2++/cIwDCMlJcVo1qyZcc899xiPPvqokZWVdd3P40ZyrWMyb948w9fX17DZbIaPj4/Rr18/48yZM+VyLjcKZ8bkzJkzRv369Q1PT0+jUqVKRv369Uu8+UX3Sem71nHRvVL6nBmTMWPGGFar1QgICDA/Y8aMMdvQvVL6rnVcdK+UPmfGZNq0aRdc97y8PLONv3uvWAzjbzz8LCIiIiIicp3psTEREREREXEJSl5ERERERMQlKHkRERERERGXoORFRERERERcgpIXERERERFxCUpeRERERETEJSh5ERGRG563tzfNmzfHbrdjt9uJj4//220tXLiQNWvWlGJ0F/fVV18xePDgMu/nUk6dOsX48ePLrX8RkYvR77yIiMgNz9vbm2+++QY/P79rbuu5554jODiYfv36XfWxhYWFWK3Wa46hrBUWFpKRkUFwcDDHjh0r73BEREyaeRERkZvWzJkzCQ0NJTAwkFatWrF582YANm3aREREBIGBgfj4+PD2228D8O233/LVV18xduxY7HY706ZNIy0tjeDgYLPNzZs34+3tDUB6ejqenp4kJiYSERHB5MmTOXz4MN27dyckJASbzcaIESMuGtsnn3xCt27dAEhLSzNnjPz9/QkMDGTz5s08+eST+Pj40LZtW3JycgAYOXIk3bt3p2PHjvj5+RETE8PJkycByMnJoXfv3vj5+eHn58eoUaPM/qKiohg2bBht2rShffv2xMfHc+rUKex2u3l+EydO5IEHHuD+++8nJCSE1atXm8dbLBbGjRtHaGgojRo1Yvr06WbZtm3baN++PTabDZvNRlJSEoDT10JExGSIiIjc4Bo2bGg0a9bMCAgIMAICAoz58+cbP/30k9GxY0fjzJkzhmEYxrJlywybzWYYhmH88ccf5v7c3FzDbrcba9euNQzDMOLi4ozJkyebbS9dutQICgoytzdt2mQ0bNjQMAzD2LdvnwEYycnJZnm7du2MH3/80TAMwygoKDDat29vzJ8//4KYp0+fbjz++ONmH1ar1Vi/fr1hGIbRt29fo379+sbBgwcNwzCMDh06GB988IFhGIbxz3/+06hbt65x+PBhwzAM4+WXXzZefvllwzAM47XXXjNiY2ONoqIiIycnx7Db7cYXX3xhGIZhtGrVyujYsaORn59vxn7nnXeWiOno0aPm95UrVxq+vr7mNmC89957hmEYxtatW41q1aoZBQUFRkFBgdG0aVPj888/N+tmZmZe1bUQETmn4s9di4iIlIIvv/yyxGNjr732Ghs2bCA0NNTcl5mZSX5+Pnl5efTt2xeHw4GbmxsHDx7E4XCUmGFxloeHB08//TQAp0+fZsmSJRw5csQsz8nJYfv27Vdsp1mzZtjtdgACAwPZv38/Xl5eAAQFBbF3716z7iOPPEKdOnUA6NOnD927dwcgNTWVSZMm4ebmRtWqVXn22WdJTU3liSeeAKBnz55UqlTpkjGsX7+ef/3rXxw/fhyr1crWrVvJz8+ncuXKAMTGxgJw3333YbVaOXz4MFlZWRQWFpoxAHh6el7TtRCRm5eSFxERuSkZhkHv3r1JTEy8oOyNN96gTp06rF+/HqvVSteuXTlz5sxF27FarRQVFZnb59erWrUqFosFgOLiYiwWC2vXrr1sknAxHh4e5nd3d/cLtvPy8i557Ln+DcMwv59fBlCtWrVLtpGfn8/jjz9OWloaQUFB/PHHH9x2220lkpfzYyosLLxke9dyLUTk5qU1LyIiclPq3LkzM2bM4ODBg8Cff0yvW7cOgJMnT+Ll5YXVamXHjh18//335nE1atQgKyvL3G7UqBH79u3j+PHjwJ/raC6levXqREREMHbsWHPfoUOHyMjIKNVzW7RoEUePHgXg448/Jjo6GoC2bdvy0UcfYRgGp0+fZtasWWbZ+WrUqEFubq6ZgJw5c4aCggLuvvtuACZPnuxULM2aNaNy5crMnTvX3Hfs2LHrdi1E5Mai5EVERG5KkZGRvPXWW3Tp0oWAgAD8/Pz4/PPPARg+fDjTpk3jgQceYPjw4bRu3do8rmfPnsyePdtcsF+/fn0GDRpEcHAwDz30ELfffvtl+01OTmbbtm34+/vj7+/P448/biY+paVNmzY8//zz+Pn5sX//fsaMGQPAm2++icViwd/fn9DQUGJiYsyXApyvZs2axMbG4u/vT3BwMDVq1CAxMZGQkBAiIyO55ZZbnIrFarWSkpLChx9+iL+/PzabjXnz5gHX51qIyI1Fr0oWERG5gYwcOZKcnBwmTJhQ3qGIiJQ6zbyIiIiIiIhL0MyLiIiIiIi4BM28iIiIiIiIS1DyIiIiIiIiLkHJi4iIiIiIuAQlLyIiIiIi4hKUvIiIiIiIiEtQ8iIiIiIiIi7h/wHPzJTsBKPcBgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 800x320 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Random Forest\n",
    "randfor = RandomForestClassifier()\n",
    "sqrt_features = int(sqrt(len(features)))\n",
    "num_features_list = list(range(sqrt_features-1,sqrt_features+2,1))\n",
    "grid_values = {'max_features':num_features_list, \"n_estimators\":[10,15,25,50,100,200]}\n",
    "cv = 3\n",
    "randfor_grid = GridSearchCV(randfor, param_grid=grid_values, scoring='roc_auc', cv=cv)\n",
    "randfor_grid.fit(X_train, y_train)\n",
    "y_val_scores_randfor_grid = randfor_grid.predict_proba(X_val)\n",
    "y_val_scores_randfor_grid = y_val_scores_randfor_grid[:,1]\n",
    "randfor_grid_fpr, randfor_grid_tpr, _ = roc_curve(y_val, y_val_scores_randfor_grid)\n",
    "randfor_grid_roc_auc = auc(randfor_grid_fpr, randfor_grid_tpr)\n",
    "print('Random Forest Grid best parameter (max. AUC): ', randfor_grid.best_params_)\n",
    "# print('Random Forest Grid best score (AUC): ', randfor_grid.best_score_)\n",
    "print('Random Forest Validation set AUC: ', roc_auc_score(y_val, y_val_scores_randfor_grid))\n",
    "randfor = RandomForestClassifier(max_features=randfor_grid.best_params_['max_features'], n_estimators=randfor_grid.best_params_['n_estimators'])\n",
    "randfor.fit(X_train, y_train)\n",
    "randfor_fig = plt.figure(figsize=(10,4), dpi=80)\n",
    "plot_feature_importances(randfor, features)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7f8dffd6-9c77-49b4-bf50-16e33eb0d5a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Random Forest Grid best parameter (max. AUC):  {'learning_rate': 0.1, 'max_depth': 5}\n",
      "Gradient-Boosted Decision Trees Validation set AUC:  0.873926756171152\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAy4AAAEmCAYAAACEf8TvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAAxOAAAMTgF/d4wjAABHT0lEQVR4nO3deVhVVf///+cBNHKqDKckRdM0hsMJCJQASVFTE8vMLDTSysi8/SpqWpq3onepmXemH6Oy21Ixyxyo7PZTpKTlXB7nWVHJVJwIBGTavz/6uT+R0zEZjvp6XBfXffZee6/13ouVF+97rbWPxTAMAxERERERESfmUtEBiIiIiIiIXIkSFxERERERcXpKXERERERExOkpcREREREREaenxEVERERERJyeEhcREREREXF6bhUdgEh5uOWWW6hVq1ZFhyEiIiIil5CRkcG5c+cuWa7ERW4KtWrVIj09vaLDEBEREZFL8PT0vGy5loqJiIiIiIjTU+IiIiIiIiJOT4mLiIiIiIg4PSUuIiIiIiLi9JS4iIiIiIiI01PiIiIiIiIiTk+Ji4iIiIiIOD0lLiIiIiIi4vSUuIiIiIiIiNNT4iIiIiIiIk7PraIDECkPRzPz8Bq+pKLDACBtfKeKDkFERETkuqMZFxERERERcXpKXERERERExOkpcREREREREaenxEVERERERJyeEhcREREREXF6SlxERERERMTpKXERERERERGnp8RFREREREScnhKXMrBhwwZiYmIASEtLw8PD45rrHD16NPn5+ddcj6NGjRrFZ599dtX3RUZG8vXXX5daHO+88w7Hjx8vtfpERERE5PqkxKUMBAUFkZSUVKp1jhkzptwSl8LCQhISEnjyySfLpa3LUeIiIiIiIqDE5Zrl5uby5JNP4u3tjb+/P+3atSM1NZWgoKCLXr9+/Xpat25NUFAQAQEBLFiwAPi/mZlRo0YRGBhIkyZN+OabbwCIi4sDIDQ0FJvNdsk/5M/XMWTIEEJCQvDx8WHZsmVm+f/+7/8SFhZGYGAgISEhrFixAoDU1FRsNhsDBgygZcuWLFq0iGeffZZp06YBkJ2dTZ8+ffD19cXX15cxY8aYdW7fvp2QkBACAgKIiYkhLy/vsv11sbbmzp1LSEgI999/PzabzXzuhIQEjhw5Qrdu3bDZbNjtdgoKChg+fDjBwcHYbDZ69OjBmTNnLmhn8uTJeHp6mj/FBbmXjUtEREREnJsSl2u0dOlSTp8+zfbt29m0aRPz5s275LVnzpzhxRdfJCkpiQ0bNvDtt98SHx/P0aNHATh58iSBgYH8/PPPTJs2jUGDBgGQmJgIwKpVq7Db7dSuXfuSbZw8eRI/Pz/Wrl3LRx99xNNPP83Zs2fZv38/Y8aM4ZtvvuHnn38mKSmJp556ioKCAgA2b95M9+7dWb16NU888USJOseOHUt+fj6bN29m7dq1LF68mPnz5wPQq1cv+vXrxy+//MI//vEP1q9ff8U++2tb7du3Z82aNWzcuJHFixfz/PPPU1BQwKhRo7jrrrv44osvsNvt2Gw23nrrLapVq8a6deuw2+34+Pjwz3/+84I24uPjSU9PN39cKt16xbhERERExHm5VXQA1zt/f3927txJv379aNWqFR07drzktatWrWL//v106NDBPGcYBrt27aJhw4ZUrVqVLl26ANCyZUv27dt31fFUrlyZXr16AdCiRQvq1q3Lpk2bsNvt7N27l4iIiBLXHz58GIB7772XsLCwi9aZkpLClClTcHFxoWrVqjzzzDOkpKTQvn17tm7dWqI9Pz+/K8b417YOHDhATEwM6enpuLm5ceLECQ4ePEiTJk0uuHfx4sX8/vvvfPHFFwDk5+dzzz33ONAzIiIiInI9U+JyjRo3bsz27dtZtmwZKSkpvPLKK7zzzjsXvdYwDKxWq7lE68/S0tJwd3c3j11dXSkqKiqVGC0WC4Zh8PDDDzNr1qwLyg8dOkS1atUueb9hGFgslgvq/PP/Xo2/ttWjRw8mTZrEo48+CkDNmjUvueTMMAymT59O69atr7pdEREREbl+aanYNUpPT8disRAdHc2kSZMwDMOcxfir0NBQ9uzZU2Lfid1ud2jTffXq1cnMzLzidfn5+eaLAdatW8fRo0exWq20a9eOpUuXsnXrVvPadevWXbE+gLZt2/Lhhx9iGAZnz55lzpw5REVFUaNGDXx9fUu0t2XLFofq/LPTp0/j5eUFwJw5czh9+rRZVqNGjRLPHR0dzeTJk8nJyQEgJyeHbdu2XXWbIiIiInJ9UeJyjbZs2UJoaChWq5WAgAB69eqF1Wq96LV33HEHX331FWPHjsXf3x9vb2+GDx9OcXHxFdsZPHgwrVu3vuzmfIA777yTvXv3EhISQu/evZk7dy5Vq1aladOmzJkzh+effx5/f3/uu+8+pkyZ4tAzvv7661gsFvz8/AgJCSE6Oppu3boBMGvWLKZNm0ZAQAAffPABISEhDtX5Z1OmTOGxxx4jLCyMTZs20aBBA7NswIAB9O7d29ycP3z4cGw2GyEhIVitVlq0aIHdbr/qNkVERETk+mIxDMOo6CCkdKSlpREUFMSJEycqOhSn41bdA8+XP6noMABIG9+pokMQERERcTqenp6kp6dfslwzLiIiIiIi4vS0Of86FBcXx5o1ay44v3r1aqeZbYmOjubQoUMlzt1xxx0sX768giISERERkeuZEpfr0PnvdXFmX375ZUWHICIiIiI3EC0VExERERERp6fERUREREREnJ6WislNoe5t7nqbl4iIiMh1TDMuIiIiIiLi9JS4iIiIiIiI01PiIiIiIiIiTk+Ji4iIiIiIOD1tzpebwtHMPLyGL6noMEScml5gISIizkwzLiIiIiIi4vSUuIiIiIiIiNNT4iIiIiIiIk5PiYuIiIiIiDg9JS4iIiIiIuL0lLiIiIiIiIjTU+IiIiIiIiJOT4mLiIiIiIg4PSUuN4lnn32WadOmVXQYpc5isZCdnV3RYYiIiIhIGVPiIg4pLCys6BBERERE5CamxKWC5ebm8uSTT+Lt7Y2/vz/t2rUjNTUVf39/evfuTWBgIEFBQWzatMm8Z/bs2YSEhBAQEECrVq3YunUrAFu2bCE8PJyAgAC8vb158803L9rmypUr8fHxYcOGDRQUFDB8+HCCg4Ox2Wz06NGDM2fOAH/M0gwYMICHH34Yf3//Sz7DV199hdVqxWaz4evrS3JyMgCRkZEMHDiQyMhImjZtytChQzEMA4CjR4/SvXt3goODsVqtjBo1yqxvz549dOrUiQceeAB/f3+mT59uli1cuJDmzZvTsmVLxo4d+/c6XURERESuO24VHcDNbunSpZw+fZrt27cDcOrUKTZv3szmzZuZMmUKkZGRfP755zz99NNs27aNn376iXnz5rFixQpuueUWVq5cSUxMDJs2bcLLy4uUlBRuueUWcnNzCQ0NpW3btgQFBZntffrpp0ycOJElS5bg5eXFG2+8QbVq1Vi3bh0AY8eO5Z///CdTpkwB4Mcff2TFihVUq1btks8wcuRIEhMTCQ0Npbi4mN9//90s2759O9999x0FBQVEREQwf/58unfvTmxsLCNGjCAiIoLCwkIeeeQRFi1aRHR0NE8//TSzZ8+mefPm5OTk0KJFC1q0aIGnpycvvPACq1atolmzZkycOPGSMU2ePJnJkyebx8UFuX/vFyQiIiIiTkGJSwXz9/dn586d9OvXj1atWtGxY0cAmjRpQmRkJADdu3enb9++HDlyhOTkZDZt2kRISIhZR0ZGBvn5+eTm5tKvXz/sdjsuLi4cPnwYu91uJi4zZ86kUqVKLF++nNtvvx2AxYsX8/vvv/PFF18AkJ+fzz333GPW3b1798smLQBt2rRh4MCBdOvWjXbt2mGz2cyy2NhYKlWqRKVKlejZsycpKSl06tSJZcuWcezYMfO67Oxsdu7cSbNmzdi2bRs9evQwy7Kysti+fTvp6ekEBATQrFkzAPr27cuwYcMuGlN8fDzx8fHmsVt1j8s+g4iIiIg4NyUuFaxx48Zs376dZcuWkZKSwiuvvMI777xz0WstFguGYdCnTx8SEhIuKH/ttdeoU6cOGzduxM3Nja5du5KXl2eW+/v7s2LFCrZu3UpYWBgAhmEwffp0WrdufdE2r5S0wB+zG9u2bWP58uXExsYSExPDK6+8cslnKC4uxmKxsH79eipVqlSifNu2bXh4eGC32y+49/wSNBERERG5+WiPSwVLT0/HYrEQHR3NpEmTMAyDw4cPs3fvXlasWAHAF198Qf369alXrx6dO3dm1qxZHD58GIDi4mI2bNgAwOnTp/H09MTNzY1du3bx3XfflWgrICCAL7/8kt69e5tl0dHRTJ48mZycHABycnLYtm3bVT3Dzp078fHxoX///rz00kusWbPGLJs9ezaFhYXk5uYyd+5coqKiqF69OuHh4YwfP9687siRI6Snp9OsWTOqVKnCrFmzzLK9e/dy6tQpWrZsycaNG9m9ezcAM2bMuKo4RUREROT6pRmXCrZlyxaGDx+OYRgUFxfTq1cvc6P7vHnziI+PxzAM5s6dC0BERARvvPEGXbp0oaioiIKCAjp16kRQUBAjR46kV69eJCUl4eXlddFZFG9vb5YuXUqnTp0YP348w4cPZ8yYMYSEhGCxWAAYNmwYPj4+Dj/Dq6++yu7du6lcuTJVqlThvffeM8sCAgKIiori119/5dFHH6Vbt24AJCUlER8fj5+fH/DHzE5iYiKenp589dVXDBo0iEmTJlFUVEStWrVISkqifv36fPDBB3Tu3Jk777zTrEtEREREbnwW4/xrnsRppKamMmTIEHMm5XoVGRnJkCFDeOSRRyo6FNyqe+D58icVHYaIU0sb36miQxARkZuYp6cn6enplyzXUjEREREREXF6WirmhCIjI51utuX48eO0a9fugvNt27blrbfeuug9qampZRyViIiIiNwslLiIQ2rXrn3RN32JiIiIiJQHLRUTERERERGnp8RFREREREScnpaKyU2h7m3uemOSiIiIyHVMMy4iIiIiIuL0lLiIiIiIiIjTU+IiIiIiIiJOT4mLiIiIiIg4PSUuIiIiIiLi9PRWMbkpHM3Mw2v4kooOwyno7WoiIiJyPdKMi4iIiIiIOD0lLiIiIiIi4vSUuIiIiIiIiNNT4iIiIiIiIk5PiYuIiIiIiDg9JS4iIiIiIuL0lLiIiIiIiIjTU+IiIiIiIiJOT4nLdSo1NZWgoCAAjhw5wkMPPfS367Lb7Xz++eclztlsNnJzc68pxmt15swZJk6cWKExiIiIiIhzUOJSzgoLC0u9zrvuuovly5f/7fsvlrjY7XZuvfXWaw3tsoqLiykuLr5kuRIXERERETlPictVsFgsTJgwgZCQEBo1asTMmTPNsg0bNtCyZUusVivBwcH89NNPAKSlpeHh4UFCQgLh4eFMnTqVyMhIhg4dSkREBHfffTdvvfUW8+bNIzQ0lIYNGzJv3jyz3p49exIUFITVauWRRx7h+PHjF8R1vg2Ab775BpvNZv64u7vzySefUFhYSPv27QkKCsLHx4eYmBhycnI4fvw4o0aNIiUlBZvNRlxcnPms2dnZDj3bqFGjCAwMpEmTJnzzzTeX7cPRo0fTq1cvunbtis1m47fffmPo0KE88MAD2Gw2WrVqxZ49ewCIi4vjzJkz2Gw2c3bp6NGjdO/eneDgYKxWK6NGjfpbv0sRERERub4ocblK7u7urF27lm+++YYBAwZQWFhIfn4+Xbt2ZfTo0WzevJnJkyfTrVs3zp49C8DJkydp0qQJK1euZNCgQQAcOnSI1NRU1q5dy6hRo9i6dSurVq1i/vz5xMfHm+298847bNiwgc2bNxMWFkZCQsJl4+vYsSN2ux273c6QIUO477776Nq1K66ursydO5cNGzawdetWatSowfTp06lduzYJCQlERUVht9tJTEwsUZ8jzxYYGMjPP//MtGnTzOe7nOXLl5OYmMjmzZupX78+w4YNY/369djtdl566SWzjsTERG6//XbsdjsbNmwAIDY2lv79+7Nu3Tp++eUX1q1bx6JFiy5oY/LkyXh6epo/xQUVu+xNRERERK6NW0UHcL2JiYkB4L777sPNzY2jR49y+vRpKleuTPv27QEICwujdu3abN68mXr16uHu7s5TTz1Vop4nnngCFxcX7rrrLjw8PHj00UcBCAwM5LfffiMvLw93d3eSkpKYPXs2586dIzc3l7p16zoU5/Llyxk9ejQrVqygevXqFBcX8+9//5slS5ZQWFhIZmYmERERV6xn165dl322qlWr0qVLFwBatmzJvn37rljnI488Qu3atc3jb7/9lqlTp5KVlUVxcTG///77Re87e/Ysy5Yt49ixY+a57Oxsdu7cecG18fHxJRJAt+oeV4xLRERERJyXEper5O7ubn52dXWlsLAQwzCwWCwXXHv+XNWqVS8o/2s9549dXV2BP/bC/Pjjj0ybNo1Vq1ZRq1YtvvzyyyvOuABs3bqV3r17s2TJEu666y4A5s6dyw8//GAmMu+++y4rVqy4Yl1Xera/PkdRUdEV66xWrZr5+dChQwwYMIB169bRuHFjNm/eTOvWrS96X3FxMRaLhfXr11OpUqUrtiMiIiIiNw4tFSsFzZs359y5cyxbtgyAVatWcfz4cfz8/K6p3tOnT1OjRg1q1qxJfn4+77///hXv+fXXX3n00UeZOXMmPj4+Jeq68847qV69OllZWXz88cdmWY0aNcjMzCzXZzsvMzOTypUrU7duXQzDYNq0aSXiysnJMV9oUL16dcLDwxk/frx5zZEjR0hPTy+VWERERETEeSlxKQWVK1dmwYIFjBgxAqvVysCBA5k/fz5Vq1a9pno7dOhAkyZNaN68Oe3bt8dms13xnhkzZpCRkcGgQYPMDfpffvklzzzzDNnZ2Xh7e9O1a1fCw8PNe9q0acPZs2fx9/c3N+eX9bOd5+fnxxNPPIGPjw+RkZE0aNDALKtZsyYxMTH4+fmZm/OTkpLYsWMHfn5++Pn58fjjj3Py5MlSiUVEREREnJfFMAyjooMQKWtu1T3wfPmTig7DKaSN71TRIYiIiIhcwNPT87IraRyecUlOTmbChAnAH8uRtmzZcu3RiYiIiIiIOMChzfmjR49m7dq17Nu3j2HDhuHi4kJcXJz5fR4if3b8+HHatWt3wfm2bdvy1ltvVUBEIiIiInK9cyhxWbx4MT///LO5z6BevXpkZWWVaWBy/apduzZ2u72iwxARERGRG4hDS8Xc3d3N1/SKiIiIiIiUN4dmXBo2bMiPP/6IxWKhuLiYN954o9RehysiIiIiInIlDr1V7NixY8TGxrJs2TJcXFwIDw8nKSmpxLefizizK72lQkREREQq1pX+XnNoxqVOnTosXbqUnJwciouLS3zzuYiIiIiISFlzKHEB2Lt3L/v37ze/xRygY8eOZRKUiIiIiIjInzmUuAwePJg5c+bQrFkzc5O+xWJR4iIiIiIiIuXCocQlOTmZAwcOUKVKlbKOR0RERERE5AIOvQ65YcOGVK5cuaxjERERERERuSiHZlzefvttOnfuTNu2bXF3dzfP9+vXr8wCEylNRzPz8Bq+pKLDkAqWNr5TRYcgIiIif5NDicubb77Jb7/9ht1uL7HHRUREREREpDw4lLj88ssv7N69W8mKiIiIiIhUCIf2uDRv3pyzZ8+WdSwiIiIiIiIX5dCMS/Xq1QkMDKR9+/Yl9rhMnDixzAITERERERE5z6HEpVmzZjRr1qysYxEREREREbkohxKXf/7zn2Udh4iIiIiIyCU5lLgALFy4ELvdTl5ennlOS8VERERERKQ8OLQ5f+DAgcycOZMZM2ZQVFTEvHnzOHnyZFnHJiIiIiIiAjiYuHz//fckJydTq1Yt3n77bdavX8/x48cdaiA5OZn77rsPm83Gli1brilYgNTUVL799lvz+MiRIzz00EPXXO/VeOeddxx+/oo0evRohgwZUtFhiIiIiIhcM4cSF3d3d1xcXLBYLBQUFFCnTh1+/fVXhxpITEwkISEBu92On5+feb6wsPBvBfzXxOWuu+5i+fLlf6uuv8sZE5e/25/OUr+IiIiIyOU4lLhUr16dnJwcwsLCiI2NJT4+nkqVKl3xvgEDBrBy5UqGDRtGaGgoFouFt99+m8jISF599VW2bNlCeHg4AQEBeHt78+abb5r3ZmZm8vzzz+Pn54e/vz99+vTBbreTmJjIrFmzsNlsJCQkkJaWhoeHh3nf0qVLCQgIwGq10qpVK7Zv3w78kfDYbDb69euHv78/Pj4+bNiw4bLxz5gxA29vb2w2G35+fqxdu5aEhASOHDlCt27dsNls2O12srOz6dOnD76+vvj6+jJmzBizjsjISAYOHEhkZCRNmzZl6NChGIbBrl27zDe1GYaBh4cHI0aMAP6Y4WrTpg0Ax44d47HHHsPPzw9fX18++OADs24vLy/+9a9/8dBDDxEbG0tmZibdunXD29ub9u3bs3fv3iv+jiwWC6NHj+bBBx/k3nvv5dNPPy1R9uffV1ZWFi+88ALBwcFYrVbi4uIoKCgAYNy4cebMms1m4+DBgwCsX7+e1q1bExQUREBAAAsWLAAwf2+jRo0iMDCQJk2a8M0335htr169mvDwcPz9/bFarSQnJwOwZ88eOnXqxAMPPIC/vz/Tp0+/6HNNnjwZT09P86e4IPeKfSEiIiIizsuhzfmffvopbm5uvPXWW0yePJnTp08zf/78K9737rvvsnnzZoYMGcIjjzyCxWLh3LlzpKamApCVlUVKSgq33HILubm5hIaG0rZtW4KCghg4cCDVqlVj06ZNuLi4kJGRQa1atYiLiyM7O5tJkyYBf/wBfN7x48fp2bMny5cvx8/Pj6SkJLp3787WrVsB2LZtGzNmzGD69OkkJiYyYsQI/vd///eS8Q8ePJgdO3Zw1113UVBQwLlz5wgJCeE///kPX3zxBb6+vgAMGzaM/Px8Nm/eTG5uLmFhYXh7e/PEE08AsH37dr777jsKCgqIiIhg/vz5dO/endzcXA4dOsTJkye59957WbZsGQApKSlERUUBfyR/zZs3Z9GiRRw/fpzAwEBsNhvBwcEAHDp0iGXLlmGxWBg8eDA1atRg+/btnDhxgoCAALp3737F35PFYuGnn35i//79BAcHExYWxt133w1Q4vfVt29fIiIi+PDDDzEMgxdeeIFp06bx7LPPMmnSJH777TduvfVWcnJycHFx4cyZM7z44ossWbKEevXqceLECQIDA3nwwQcBOHnyJIGBgSQkJLB06VL+3//7f3Ts2JFTp07x2GOPsXDhQkJDQykuLubMmTMUFRXx9NNPM3v2bJo3b05OTg4tWrSgRYsWBAQElHim+Ph44uPjzWO36h6IiIiIyPXLoRmXOnXqULlyZW699VZGjBjBpEmTaNCgwd9qsE+fPubn3Nxcc1alRYsWHDx4ELvdDsDXX3/N0KFDcXH5I8RatWpdse61a9easyMAMTExpKen89tvvwF/fB9NUFAQAC1btmTfvn2Xra9169Y888wzTJkyhQMHDlCtWrWLXpeSkkJcXBwuLi5UrVqVZ555hpSUFLM8NjaWSpUqUaVKFXr27GmWtWnThpSUFFJSUujZsye5ublkZmaWSFxSUlJ4+eWXAahduzZdu3bl+++/N+vu3bs3FosFgOXLl/Pcc88B4OHhQdeuXa/YZwDPP/88AI0bNyYsLIyVK1eaZX/+fS1evJi33noLm83G/fffz8qVK9mzZw81atSgadOm9OzZk/fff59Tp07h7u7OqlWr2L9/Px06dMBmsxEVFWXONgFUrVqVLl26ACV/H6tXr8bb25vQ0FAAXFxcqFmzJrt27WLbtm306NEDm81GaGgoWVlZ5qyaiIiIiNy4HJpxSUtLY8KECezbt6/EXofzMwRX489//L/22mvUqVOHjRs34ubmRteuXUu8bvlqGYZh/hH/Z+fPubu7m+dcXV2vuG9j4cKF/Pzzz6SmptKxY0fGjRtHjx49HGr3YnH8tSwqKoolS5Zw6tQp3n33Xfbs2cPChQs5cOAAgYGBl6zrz8d/7k/DMC77PI66XP2LFy+mcePGF9yzZs0aVq1aRWpqKi1atODTTz/FMAysVisrVqy44Pq0tLQLfh9FRUWXjev8krrzya2IiIiI3DwcmnHp3r07t99+O/3792fw4MHmz7U6ffo0np6euLm5sWvXLr777juzLDo6mrfeeovi4mIAMjIyAKhRowaZmZkXra9ly5bY7XZ27NgBwLx58/D09KRu3bpXHVthYSH79u0jKCiIIUOG0K1bN9atW3fRGNq2bWsunzp79ixz5swxZ0wAZs+eTWFhIbm5ucydO9csi4qK4vvvv+fgwYPce++9REVFMWbMGFq1amXONEVFRZn7WjIyMli0aBGtW7e+aMxt2rRh5syZAJw6dYpFixY59Kz/+c9/gD+SiR9//JGwsLCLXhcdHc348ePNhO/06dPs3buXrKwsjh07Rnh4OK+//jphYWFs3LiR0NBQ9uzZUyLBtdvt5OfnXzae0NBQduzYwapVqwAoLi7m1KlTNGvWjCpVqjBr1izz2r1793Lq1CmHnlNERERErl8Ozbjk5eWV2DhfWkaOHEmvXr1ISkrCy8urxB/k//73vxk0aBC+vr5UrlyZBx54gA8//JDHHnuM2bNnY7PZ6Nq1K88884x5T61atZg9ezYxMTEUFRVx++238/nnn/+t2IqKiujduzenT5/Gzc2NWrVqmUnBgAED6N27N1WqVOHjjz/m9ddf5x//+Ie5RO2JJ56gW7duZl0BAQFERUXx66+/8uijj5plderUoU6dOubytVatWnHkyBGGDRtm3vvuu+8SFxeH1WqluLiYESNGmPtb/ur111+nT58+eHt707BhQ9q2bevQs95yyy08+OCDZGRkMHXqVHN/y1+98847DBs2DJvNhouLC5UqVWLChAm4u7vTrVs3zp49i8VioWnTpsTGxnLbbbfx1VdfMXToUAYNGkRBQQENGjRg8eLFl43njjvuYNGiRQwePJisrCwsFgtjx44lOjqar776ikGDBjFp0iSKioqoVasWSUlJDj2niIiIiFy/LIYD64uefvppJk6ciKenZ3nEdEOJjIw0X07gjCwWC1lZWZfcv3OjcKvugefLn1R0GFLB0sZ3qugQRERE5BI8PT1JT0+/ZLlDMy6nTp3CarUSFhZWYl/C353NEBERERERuRoOJS5PPfUUTz31VFnHUiHsdjvPPvvsBedjY2MZNGjQNdd//lXCFSkhIYGFCxdecH7BggWltqFfRERERKQsObRUTOR6p6ViAloqJiIi4syutFTMobeKiYiIiIiIVCQlLiIiIiIi4vQc2uMicr2re5u7lgmJiIiIXMccnnFJTk5mwoQJAPz6669s2bKlzIISERERERH5M4cSl9GjR5OYmMhHH330x00uLsTFxZVpYCIiIiIiIuc5lLgsXryYr7/+mqpVqwJQr149srKyyjQwERERERGR8xxKXNzd3XF1dS3rWERERERERC7Koc35DRs25Mcff8RisVBcXMwbb7yBn59fWccmUmqOZubhNXxJRYchIqVEL9sQEbn5OJS4vPvuu8TGxrJ161aqVKlCeHg4SUlJZR2biIiIiIgI4EDiUlRUxLx581i6dCk5OTkUFxdTrVq18ohNREREREQEcGCPi6urKwsWLACgSpUqSlpERERERKTcObQ5v23btnz22WdlHYuIiIiIiMhFObzH5eTJk/Tp04eqVatiGAYWi4Xjx4+XdXwiIiIiIiKOJS4bNmwo6zhEREREREQuyeHXIYuIiIiIiFQUhxKXRo0aYbFYLji/f//+Ug9IRERERETkrxxKXL7++mvzc15eHrNnz+aOO+4os6Dk+jdq1Ch8fHx48sknKzoUEREREbkBOJS4+Pj4lDgODAykTZs2ZRKQ3BgSEhIqOgQRERERuYE49Drkvzp9+jQHDhwo7VjECaxfv57WrVsTFBREQEAACxYsoLCwkPbt2xMUFISPjw8xMTHk5OQAsGbNGgIDA7HZbPj6+vLee+8B8OyzzzJt2jQARo8ezdNPP03nzp3x9vamdevWnDp1ymxz0qRJBAcHExAQQMeOHTl8+DAAX331FVar1aw7OTkZgHHjxnHfffdhs9mw2WwcPHiwPLtIRERERCqAQzMuDzzwgLnHpaioiAMHDjB48OAyDUzK35kzZ3jxxRdZsmQJ9erV48SJEwQGBhIaGsrcuXO58847MQyDfv36MX36dIYMGcKbb77J4MGDefrpp4E/ktqLWbt2LevXr6dmzZr06NGD999/n1dffZW5c+eye/duVq9ejaurK7Nnz6Z///4kJyczcuRIEhMTCQ0Npbi4mN9//53Tp08zadIkfvvtN2699VZycnJwcbkw/548eTKTJ082j4sLcsum00RERESkXDiUuEyaNOn/bnBzo1GjRtx1111lFpRUjFWrVrF//346dOhgnjMMg927d/Pdd9+xZMkSCgsLyczMJCIiAoCHHnqIcePGsXfvXlq3bk1YWNhF6+7QoQM1a9YEoGXLlmzZsgWAxYsXs2HDBgIDA4E/EmNXV1cA2rRpw8CBA+nWrRvt2rXDZrNRVFRE06ZN6dmzJ+3ataNTp054enpe0F58fDzx8fHmsVt1j1LoIRERERGpKA4tFdu3bx+tWrWiVatWPPjgg9x111385z//KevYpJwZhoHVasVut5s/hw4d4vDhw/zwww+sWLGCLVu2MGTIEPLy8gAYOHAgX3/9NfXq1eO1116jX79+F63b3d3d/Ozq6kphYaHZ5siRI832tmzZgt1uB/6YNZk5cyZVqlQhNjaWiRMn4urqypo1axg4cCDHjx+nRYsWrFy5smw7RkREREQqnEOJy/m9Clc6J9e30NBQ9uzZw7Jly8xzdrudkydPcuedd1K9enWysrL4+OOPzfJdu3bRuHFjXnjhBV577TXWrFlzVW1GR0czffp0c89LQUEBGzduBGDnzp34+PjQv39/XnrpJdasWUNWVhbHjh0jPDyc119/nbCwMPN6EREREblxXXap2IYNG1i7di0nTpxg+vTp5vnMzEzy8/PLPDgpX3fccQdfffUVQ4cOZdCgQRQUFNCgQQM+++wzvvrqK7y9valfvz7h4eH8+uuvAEydOpXly5dTuXJlXF1defvtt6+qzV69enHy5EkiIyOxWCwUFhby3HPPcf/99/Pqq6+ye/duKleuTJUqVXjvvffIzMykW7dunD17FovFQtOmTYmNjS2L7hARERERJ2IxDMO4VGFycjKLFy/myy+/JDo62jxfo0YNevXqRVBQULkEKXKt3Kp74PnyJxUdhoiUkrTxnSo6BBERKWWenp6kp6dfsvyyMy5dunShS5cu/Pe//y2xYVtERERERKQ8OfRWsQ4dOvDLL79gt9vNTdnAJTdii4iIiIiIlCaHEpcJEybw2WefcejQIVq1asV3331HmzZtlLiIiIiIiEi5cOitYrNnz2bVqlV4enqyYMEC1q9fT+XKlcs6NhEREREREcDBxMXd3R13d3eKi4sxDINmzZqRlpZWxqGJiIiIiIj8waGlYlWqVKGgoACbzcawYcPw9PQkJyenrGMTKTV1b3PXW4hERERErmMOzbhMnz6d/Px83n77bU6fPs2KFSuYPXt2WccmIiIiIiICXOF7XP6qsLAQNzeHJmlEnMqV3gsuIiIiIhXrSn+vOTTjsm3bNmw2G40aNQLg559/ZtiwYaUToYiIiIiIyBU4lLj079+fadOm4eHhAUBAQABLliwp08BERERERETOcyhxycrKIiwszDy2WCxUqlSpzIISERERERH5M4c2rLi5uVFQUIDFYgEgPT0dFxeHch4Rp3A0Mw+v4ZolFLkSvX1PRESclcNLxR577DFOnDjB6NGjiYiIYOjQoWUdm4iIiIiICODgjEvPnj1p3LgxycnJ5OTk8MknnxAeHl7WsYmIiIiIiABXSFz69evH9OnTAcjIyGDChAnlEpSIiIiIiMifXXap2Jo1a8zPY8aMKfNgRERERERELuayicufv5vyKr6nUkREREREpFRddqnYuXPn2LFjB4ZhlPh8nre3d5kHKCIiIiIictnEJScnh44dO5rHf/5ssVjYv39/2UUmIiIiIiLy/7ts4pKWllZOYUh5S05OZvjw4dxyyy3Mnj0bPz+/a6rPYrGQlZVFtWrVSilCEREREZH/49DrkOXGk5iYSEJCAk888USJ84WFhbi5aViIiIiIiHNx6Aso5cYyYMAAVq5cybBhwwgNDcVisfD2228TGRnJq6++SlZWFi+88ALBwcFYrVbi4uIoKCgAYNy4cdx3333YbDZsNhsHDx406/2f//kfQkJCaNSoETNnzjTPe3l5MWbMGEJDQ2nUqBHjxo0zy44ePUr37t3NtkaNGgVAcXEx/fv3p3nz5vj7+xMYGEheXh4ZGRm0a9cOPz8/rFYrvXv3LqdeExEREZGKpP9r/Sb07rvvsnnzZoYMGcIjjzyCxWLh3LlzpKamAtC3b18iIiL48MMPMQyDF154gWnTpvHss88yadIkfvvtN2699VZycnJwcfm/3Nfd3Z21a9eyY8cOgoOD6dWrlzl7c+bMGVatWkVGRgZNmjShd+/e1K9fn9jYWEaMGEFERASFhYU88sgjLFq0CC8vL77//nu2b9+Oi4sLmZmZVK5cmTlz5uDl5cW3334LwKlTp8q9/0RERESk/ClxEQD69Oljfl68eDFr1qzh7bffBiA3N5fKlStTo0YNmjZtSs+ePWnXrh2dOnXC09PTvC8mJgaA++67Dzc3N44ePWqWny+rVasWjRs35sCBA9x+++0sW7aMY8eOmXVkZ2ezc+dOWrduTUFBAX369OGhhx6iU6dOuLi40KJFC/79738zePBgWrVqRfv27S/6PJMnT2by5MnmcXFBbin1lIiIiIhUBCUuAlBiU71hGCxevJjGjRtfcN2aNWtYtWoVqamptGjRgk8//ZTw8HDgjxmX81xdXSksLDSPL1ZWXFyMxWJh/fr1VKpU6YK2tm3bxg8//MDy5ct59dVXWbFiBS1btsRut5OSksKCBQsYOXIkGzduxNXVtcS98fHxxMfHm8du1T3+Rq+IiIiIiLPQHhe5QHR0NOPHjzcTj9OnT7N3716ysrI4duwY4eHhvP7664SFhbFx48a/3U716tUJDw9n/Pjx5rkjR46Qnp5ORkYGZ8+epV27drzxxht4eXmxfft2Dhw4QLVq1ejevTtTp05l9+7dZGdnX/Mzi4iIiIhzU+IiF3jnnXdwc3PDZrNhtVqJiooiLS2NzMxMunbtam6MLygoIDY29praSkpKYseOHfj5+eHn58fjjz/OyZMnOXz4MG3btsVqteLn54evry8dOnQgNTWVwMBAbDYbDz74IG+99Ra33XZbKT25iIiIiDgri2EYRkUHIVLW3Kp74PnyJxUdhojTSxvfqaJDEBGRm5Snpyfp6emXLNeMi4iIiIiIOD0lLiIiIiIi4vSUuIiIiIiIiNNT4iIiIiIiIk5PiYuIiIiIiDg9JS4iIiIiIuL03Co6AJHyUPc2d73mVUREROQ6phkXERERERFxekpcRERERETE6SlxERERERERp6fERUREREREnJ4SFxERERERcXp6q5jcFI5m5uE1fElFhyEiIiJyXXDGt7FqxkVERERERJyeEhcREREREXF6SlxERERERMTpKXERERERERGnp8RFREREREScnhIXERERERFxekpcRERERETE6SlxERERERERp6fE5RpYLBays7MrOoyrlpaWxgcffFBm9dvtdj7//PMyq19EREREbj5KXG5CFZ24FBYWllnbIiIiInJjUuJyFRYuXEjz5s1p2bIlY8eONc8vXbqUgIAArFYrrVq1Yvv27WbZzJkzsdls+Pv7ExQURFpaGmlpaXh4eJjXZGdnY7FYzGOLxcKbb75JcHAwjRs3JiUlhVdffZX7778fHx8ftm3bZl47e/ZsQkJCCAgIoFWrVmzduhWAjz/+mPbt2/PUU0/h5+dHUFAQ+/fvByAuLo7t27djs9mIjo6+7DNPmjSJ4OBgAgIC6NixI4cPHwZg9OjRPP3003Tu3Blvb29at27NqVOnOH78OKNGjSIlJQWbzUZcXJz5TG+//TaRkZG8+uqrHDt2jMceeww/Pz98fX1LJFJeXl68+uqrRERE0KRJEyZPngzA/Pnzad++vXldUVERDRs2LNHf502ePBlPT0/zp7gg97LPKSIiIiLOTYmLg44fP84LL7xAcnIyq1ev5pZbbgHg1KlT9OzZk08++YTNmzfTt29funfvDkBqair/+te/+O9//8umTZtYsWIFtWvXdqi9GjVqsG7dOiZMmECXLl0ICwtj48aNxMbG8q9//QuAn376iXnz5rFixQp++eUXxo0bR0xMjFnH2rVrGT9+PFu2bCEqKooJEyYAkJiYiLe3N3a7nS+//PKSMcydO5fdu3ezevVqfvnlF5566in69+9fov5PPvmE7du3U7t2bd5//31q165NQkICUVFR2O12EhMTzevPnTtHamoqb731FgMGDKB58+Zs2bKFZcuWMXbsWNatW2dee+zYMVasWMGaNWuYMmUKa9eupWvXruzatYs9e/YAsHjxYpo0aYK3t/cFscfHx5Oenm7+uFS61aF+FxERERHnpMTFQWvWrCEgIIBmzZoB0LdvXwA2bdqEzWbDz88PgJiYGNLT0/ntt99YsmQJzzzzDPXq1QOgSpUqVKlSxaH2nnzySQACAgJwcXGhU6dOAAQGBpozJ8nJyWzatImQkBBsNhv/+Mc/yMjIID8/H4CwsDAaNmwIQMuWLdm3b99VPfPixYtJSUkhMDAQm83GxIkTOXjwoFneoUMHatas6XD9ffr0MT+npKTw8ssvA1C7dm26du3K999/b5Y/99xzAHh4ePDYY4/x/fff4+rqSr9+/Zg+fToA06ZNK5FIiYiIiMiNy62iA7heGIZxyfN/XuZ13sXOnefm5kZRUZF5nJeXd8E17u7uALi6upqzO+ePz+8RMQyDPn36kJCQcNF2ztfx1/scZRgGI0eOLJFwXEv91apVK3H81z66XJ+dL3vhhRfw9fXlqaeeYv/+/Vdc6iYiIiIiNwbNuDioZcuWbNy4kd27dwMwY8YMAO6//37sdjs7duwAYN68eXh6elK3bl06d+7MrFmzOHr0KAA5OTnk5ORQt25dCgsL2bVrFwCzZs36WzGdr//8vpPi4mI2bNhwxftq1KhBZmbmFa+Ljo5m+vTpnDp1CoCCggI2btxYKvVHRUWZ+1oyMjJYtGgRrVu3NstnzpwJ/LEUb/HixbRp0waAO+64g86dO/P4448TFxeHq6vrFeMRERERkeufEhcH1a5dmw8++IDOnTsTGhqKi8sfXXfHHXcwe/ZsYmJi8Pf357333jPfqBUREcHIkSNp164d/v7+tGrVioyMDNzc3Hj33Xfp0KEDERERnDt37m/FFBERwRtvvEGXLl3w9/fH19eXzz777Ir3Wa1WmjVrhq+v72VnLHr16kXPnj2JjIzE398fm83G8uXLr1h/mzZtOHv2LP7+/ubm/L9699132bx5M1arlYceeogRI0YQHBxsljds2JDw8HCCg4MZMGBAibIXXniBjIwMnn/++SvGIiIiIiI3BotxqTVQIhXEy8uLr7/+Gl9f34uWT5w4kV27dvHRRx85XKdbdQ88X/6ktEIUERERuaGlje9U7m16enqSnp5+yXLtcZHrio+PDxaLhaVLl1Z0KCIiIiJSjpS4CEFBQRdsrPfx8SEpKalC4klLS7tk2Z+/w0ZEREREbh5KXMShDf0iIiIiIhVJm/NFRERERMTpKXERERERERGnp6ViclOoe5t7hbwdQ0RERERKh2ZcRERERETE6SlxERERERERp6fERUREREREnJ4SFxERERERcXpKXERERERExOnprWJyUziamYfX8CUV1r7eaCYiIiJybTTjIiIiIiIiTk+Ji4iIiIiIOD0lLiIiIiIi4vSUuIiIiIiIiNNT4iIiIiIiIk5PiYuIiIiIiDg9JS4iIiIiIuL0lLiIiIiIiIjTU+JSxkaPHk1+fj4Ao0aN4rPPPgMgNTWVb7/91rwuLS0NDw+PConRER9//DHdunWr6DBMH3/8Mbt3767oMERERESknChxKWNjxowxE5eEhASefPJJ4MLExZkUFhZWdAhXpMRFRERE5OaixKUMxcXFARAaGorNZqNjx45MmzYNu91OYmIis2bNwmazkZCQcMG969evp3Xr1gQFBREQEMCCBQsu2U52djY1a9akoKAAgICAAGJiYgDYv38/99xzj3ldnz598PX1xdfXlzFjxph1REZGMmLECNq0aUP79u3Jz8/nxRdf5N577+Whhx5i7dq1V3zeHTt20L59e6xWK1arlcTERAD27t1LVFQUVqsVm83G4sWLzXssFgvZ2dnmsYeHB2lpaQB4eXkxZswYQkNDadSoEePGjQNgxowZbNiwgQEDBmCz2fjmm28uiGXy5Ml4enqaP8UFuVeMX0REREScl1tFB3AjS0xM5P3332fVqlVUq1aNZ599FgCbzUZcXBzZ2dlMmjQJwPxjHeDMmTO8+OKLLFmyhHr16nHixAkCAwN58MEHqVu37gXtVKtWDR8fH1avXo2Pjw9FRUWsX78egO+++46oqCgAxo4dS35+Pps3byY3N5ewsDC8vb154oknALDb7SxdupRKlSoxdepUDhw4wLZt2ygoKCAiIgIvL69LPmthYSFdunRh3LhxdO/eHYATJ04AEBMTw3PPPUffvn3Zs2cPLVq0IDAwkLvvvvuKfXjmzBlWrVpFRkYGTZo0oXfv3jz//PPMmTOHIUOG8Mgjj1z0vvj4eOLj481jt+rOuwxPRERERK5MMy5OaNWqVezfv58OHTpgs9mIiorCMAx27dp1yXuioqJISUlh2bJltG/fnmbNmrF161ZSUlLMxCUlJYW4uDhcXFyoWrUqzzzzDCkpKWYdvXr1olKlSgAsX76c2NhYKlWqRJUqVejZs+dlY961axeFhYVm0gJ/zJ5kZWVht9t57rnnAGjatClhYWH8+OOPDvXF+ZmjWrVq0bhxYw4cOODQfSIiIiJyY9GMixMyDAOr1cqKFSscvicqKoqhQ4dy/PhxunbtSv369fnuu+/44YcfzCVbhmFgsVhK3Pfn42rVqpWIoTScr+dS7bq6ulJUVGSez8vLK3Gdu7u7+dnV1fW62H8jIiIiIqVPMy5lrHr16mRmZl5wvkaNGhc9D3/sidmzZw/Lli0zz9ntdnOT/8WEhISwc+dOUlJSCA8PJyoqiilTpnD33Xdz5513AtC2bVs+/PBDDMPg7NmzzJkzx5yN+as2bdowe/ZsCgsLyc3NZe7cuZd9zmbNmlG5cmXmz59vnjtx4gQ1atTAZrPxySefALBv3z5++uknHnzwQQDuuecec//MwoULOXv27GXbOe9y/SciIiIiNx4lLmVs8ODBtG7dGpvNxvHjx83zjz32GBs2bLjo5vw77riDr776irFjx+Lv74+3tzfDhw+nuLj4ku24ubkRFhZGgwYNuPXWW/Hx8aGgoKBEYvL6669jsVjw8/MjJCSE6OjoS77iuG/fvjRo0ABvb286depEeHj4ZZ/Tzc2N5ORkPvjgA/z8/LBareYLBZKSkpgzZw7+/v48/vjjzJgxw9zf8s477/Dyyy/z4IMP8ssvv5hJ1pX07duXhISES27OFxEREZEbi8UorTVBIk7MrboHni9/UmHtp43vVGFti4iIiFwPPD09SU9Pv2S5ZlxERERERMTpaXP+dcRut5uvVP6z2NhYBg0aVC4xzJgxg2nTpl1wfurUqVdcTiYiIiIi8ndpqZjcFLRUTERERMS5aamYiIiIiIhc95S4iIiIiIiI09MeF7kp1L3NXcu1RERERK5jmnERERERERGnp8RFREREREScnhIXERERERFxekpcRERERETE6SlxERERERERp6fERUREREREnJ4SFxERERERcXpKXERERERExOkpcREREREREaenxEVERERERJyexTAMo6KDEClrbm5u1K1bt6LDuC5kZ2dTrVq1ig7juqH+cpz6ynHqK8epr66O+stx6ivHlVZfZWRkcO7cuUuWu11zCyLXgbp165Kenl7RYVwXPD091VdXQf3lOPWV49RXjlNfXR31l+PUV44rr77SUjEREREREXF6SlxERERERMTpKXGRm0J8fHxFh3DdUF9dHfWX49RXjlNfOU59dXXUX45TXzmuvPpKm/NFRERERMTpacZFREREREScnhIXERERERFxekpc5Lq2Z88eQkNDuffeewkODmb79u0Xve6jjz6iadOm3HPPPfTt25fCwkKz7Ouvv6Z58+Y0adKExx9/nOzs7PIKv1xda1+lpaXh5uaGzWYzf/bt21eej1BuHOmrtLQ0IiMjue222wgKCrqgXOPq/1yurzSuSlq2bBkhISF4e3vj6+vLiBEj+POK7ptlXMG195fGVkmrV682+8HHx4cXX3yxxPdl3Cxj61r7SuPq4vLy8vD29r7g3/hSH1eGyHXsoYceMmbOnGkYhmHMnz/faNGixQXX7N+/36hXr55x9OhRo7i42OjcubORmJhoGIZhZGVlGbVr1zZ27NhhGIZhvPzyy8bw4cPLLf7ydK19deDAAePOO+8sz5ArjCN9dfLkSWPlypXG119/bQQGBpYo07gq6XJ9pXFV0i+//GLs27fPMAzDyM3NNR588EEjKSnJMIyba1wZxrX3l8ZWSWfPnjXy8/MNwzCMoqIi47HHHjOmTJliGMbNNbauta80ri4uPj7e6NOnT4l/48tiXClxkevWsWPHjNtuu80oKCgwDMMwiouLjTp16hgHDhwocd3EiRONfv36mcdLliwxWrVqZRiGYXz++edGx44dzbJt27YZDRs2LOvQy11p9NXN8o+1o3113vLlyy/4Y1zj6sBFr79YX2lcHbjsfS+//LIxduxYwzBunnFlGKXTXxpbBy55T25urvHwww8bU6dONQzj5hlbpdFXGlcHLrh2xYoVRufOnS/4N74sxpWWisl16/Dhw9x11124ubkBYLFYaNCgAYcOHSpx3aFDh2jYsKF57OXlZV5zsbJff/2V4uLicniC8lMafQXw+++/88ADDxAQEEBCQgJFRUXl8wDlyNG+uhyNK8f7CjSuLuXo0aN88cUXdOzYEbh5xhWUTn+BxtZfpaWlYbPZ8PDwoEaNGvTt2xe4ecZWafQVaFz92dmzZxk4cCDvvffeBXWUxbhS4iLXNYvFUuLYuMTbvf983V+v+WsdN6pr7at69eqRnp7O+vXrSUlJYeXKlbz99ttlE2wFc7SvrqaOG9W19pXG1cX9/vvvdO7cmVdeeYWAgIBL1nEju9b+0ti6kJeXF3a7naNHj3Lu3DkWLlx4yTpuVNfaVxpXJQ0dOpSXX36Z+vXrO1THtVLiItetu+++m/T0dHPzuGEYHD58mAYNGpS4rkGDBqSlpZnHBw8eNK/5a1laWhr169fHxeXG+k+jNPrqlltuoXbt2gDUrFmTPn36sHLlyvJ5gHLkaF9djsaV432lcXVhX2VlZfHwww8THR1d4kvdbpZxBaXTXxpbl/7vsFq1avTo0YOkpCTg5hlbpdFXGlcl++rHH38kISEBLy8vevTowZYtW/Dx8QHKZlzdWCNSbiq1a9fm/vvvZ86cOQAsWLAALy8vvLy8Slz3+OOPs2jRIo4dO4ZhGCQmJtKjRw8AHn74YdavX8/OnTsBmD59ull2IymNvjp+/DgFBQUA5v/7dP/995frc5QHR/vqcjSuvByuQ+PKq8R12dnZPPzww7Rv357XX3+9RNnNMq6gdPpLY8urxHX79u0z+yM/P5+FCxditVqBm2dslUZfaVx5lbhu8+bNpKWlkZaWxrx58/Dz82Pbtm1AGY2ra9ohI1LBdu7cabRo0cJo2rSpERgYaGzdutUwDMN47rnnjOTkZPO6Dz74wLjnnnuMRo0aGc8995z5thDDMIzk5GSjWbNmxj333GM8+uijRmZmZrk/R3m41r5asGCB4ePjY1itVsPb29vo37+/kZeXVyHPUtYc6au8vDyjfv36hoeHh1GpUiWjfv36Jd6WonHlWF9pXJXsq3Hjxhlubm6Gv7+/+TNu3DizjptlXBnGtfeXxlbJvpoxY8YF/ZGbm2vWcbOMrWvtK42rC/9uOO9iL2Ap7XFlMYy/sXhbRERERESkHGmpmIiIiIiIOD0lLiIiIiIi4vSUuIiIiIiIiNNT4iIiIiIiIk5PiYuIiIiIiDg9JS4iIiIiIuL0lLiIiMgNz8vLi+bNm2Oz2bDZbMTFxf3tuhYvXsy6detKMbqL+/LLLxk6dGiZt3MpZ86cYeLEiRXWvojIX+l7XERE5Ibn5eXF119/ja+v7zXX9eyzzxIUFET//v2v+t7CwkLc3NyuOYayVlhYSHp6OkFBQZw4caKiwxERATTjIiIiN7HZs2cTEhJCQEAArVq1YuvWrQBs2bKF8PBwAgIC8Pb25s033wTgm2++4csvv2T8+PHYbDZmzJhBamoqQUFBZp1bt27Fy8sLgLS0NDw8PEhISCA8PJypU6dy9OhRunfvTnBwMFarlVGjRl00to8//phu3boBkJqaas4U+fn5ERAQwNatW3nyySfx9vambdu2ZGdnAzB69Gi6d+9Ox44d8fX1JTo6mtOnTwOQnZ1Nnz598PX1xdfXlzFjxpjtRUZGMmLECNq0aUP79u2Ji4vjzJkz2Gw28/kmT57MAw88wP33309wcDBr164177dYLEyYMIGQkBAaNWrEzJkzzbIdO3bQvn17rFYrVquVxMREAIf7QkQEAENEROQG17BhQ6NZs2aGv7+/4e/vbyxcuND48ccfjY4dOxp5eXmGYRjGihUrDKvVahiGYfz+++/m+ZycHMNmsxnr1683DMMwYmNjjalTp5p1L1++3AgMDDSPt2zZYjRs2NAwDMM4cOCAARhJSUlmebt27YwffvjBMAzDKCgoMNq3b28sXLjwgphnzpxpPP7442Ybbm5uxsaNGw3DMIx+/foZ9evXNw4fPmwYhmF06NDBeP/99w3DMIx//vOfRt26dY2jR48ahmEYL730kvHSSy8ZhmEYr7zyihETE2MUFRUZ2dnZhs1mMz7//HPDMAyjVatWRseOHY38/Hwz9jvvvLNETMePHzc/r1692vDx8TGPAeOdd94xDMMwtm/fblSrVs0oKCgwCgoKjKZNmxqfffaZeW1GRsZV9YWIiGEYhvPPV4uIiJSCL774osRSsVdeeYVNmzYREhJinsvIyCA/P5/c3Fz69euH3W7HxcWFw4cPY7fbS8ysOMrd3Z2nnnoKgLNnz7Js2TKOHTtmlmdnZ7Nz584r1tOsWTNsNhsAAQEBHDx4EE9PTwACAwPZv3+/ee0jjzxCnTp1AOjbty/du3cHICUlhSlTpuDi4kLVqlV55plnSElJ4YknngCgV69eVKpU6ZIxbNy4kX/961+cPHkSNzc3tm/fTn5+PpUrVwYgJiYGgPvuuw83NzeOHj1KZmYmhYWFZgwAHh4e19QXInJzUuIiIiI3JcMw6NOnDwkJCReUvfbaa9SpU4eNGzfi5uZG165dycvLu2g9bm5uFBUVmcd/va5q1apYLBYAiouLsVgsrF+//rIJwsW4u7ubn11dXS84zs3NveS959s3DMP8/NcygGrVql2yjvz8fB5//HFSU1MJDAzk999/57bbbiuRuPw1psLCwkvWdy19ISI3J+1xERGRm1Lnzp2ZNWsWhw8fBv74Q3rDhg0AnD59Gk9PT9zc3Ni1axffffedeV+NGjXIzMw0jxs1asSBAwc4efIk8Me+mUupXr064eHhjB8/3jx35MgR0tPTS/XZlixZwvHjxwH46KOPiIqKAqBt27Z8+OGHGIbB2bNnmTNnjln2VzVq1CAnJ8dMPvLy8igoKODuu+8GYOrUqQ7F0qxZMypXrsz8+fPNcydOnCi3vhCRG4cSFxERuSlFRETwxhtv0KVLF/z9/fH19eWzzz4DYOTIkcyYMYMHHniAkSNH0rp1a/O+Xr16MXfuXHNzfv369RkyZAhBQUE89NBD3H777ZdtNykpiR07duDn54efnx+PP/64mfSUljZt2vDcc8/h6+vLwYMHGTduHACvv/46FosFPz8/QkJCiI6ONl8A8Fc1a9YkJiYGPz8/goKCqFGjBgkJCQQHBxMREcEtt9ziUCxubm4kJyfzwQcf4Ofnh9VqZcGCBUD59IWI3Dj0OmQREZEbyOjRo8nOzmbSpEkVHYqISKnSjIuIiIiIiDg9zbiIiIiIiIjT04yLiIiIiIg4PSUuIiIiIiLi9JS4iIiIiIiI01PiIiIiIiIiTk+Ji4iIiIiIOD0lLiIiIiIi4vT+P2c08HUGs3X2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 800x320 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Gradient Boosted Decision Trees\n",
    "gbdt = GradientBoostingClassifier()\n",
    "grid_values = {'learning_rate':[0.001,0.01,0.1], \"max_depth\":[2,3,4,5,100]}\n",
    "cv = 3\n",
    "gbdt_grid = GridSearchCV(gbdt, param_grid=grid_values, scoring='roc_auc', cv=cv)\n",
    "gbdt_grid.fit(X_train, y_train)\n",
    "y_val_scores_gbdt_grid = gbdt_grid.predict_proba(X_val)\n",
    "y_val_scores_gbdt_grid = y_val_scores_gbdt_grid[:,1]\n",
    "gbdt_grid_fpr, gbdt_grid_tpr, _ = roc_curve(y_val, y_val_scores_gbdt_grid)\n",
    "gbdt_grid_roc_auc = auc(gbdt_grid_fpr, gbdt_grid_tpr)\n",
    "print('Random Forest Grid best parameter (max. AUC): ', gbdt_grid.best_params_)\n",
    "# print('Random Forest Grid best score (AUC): ', gbdt_grid.best_score_)\n",
    "print('Gradient-Boosted Decision Trees Validation set AUC: ', roc_auc_score(y_val, y_val_scores_gbdt_grid))\n",
    "gbdt = GradientBoostingClassifier(max_depth=gbdt_grid.best_params_['max_depth'], learning_rate=gbdt_grid.best_params_['learning_rate'])\n",
    "gbdt.fit(X_train, y_train)\n",
    "gbdt_fig = plt.figure(figsize=(10,4), dpi=80)\n",
    "plot_feature_importances(gbdt, features)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a8d179ff",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "aef89dbca058b3768c5e369581c14bbb",
     "grade": false,
     "grade_id": "cell-f8da4477c345bf17",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SVC Validation set AUC:  0.8492305471098806\n",
      "Tree Validation set AUC:  0.8519227829395478\n",
      "Naive Bayes (Gaussian) Validation set AUC:  0.8202253155096564\n",
      "Random Forest Grid best parameter (max. AUC):  {'max_features': 3, 'n_estimators': 100}\n",
      "Random Forest Validation set AUC:  0.8714584784196491\n",
      "Random Forest Grid best parameter (max. AUC):  {'learning_rate': 0.1, 'max_depth': 4}\n",
      "Gradient-Boosted Decision Trees Validation set AUC:  0.8883698809301377\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "9240     0.008184\n",
       "9241     0.035667\n",
       "9242     0.089048\n",
       "9243     0.977667\n",
       "9244     0.009268\n",
       "           ...   \n",
       "11544    0.020335\n",
       "11545    0.011086\n",
       "11546    0.010711\n",
       "11547    0.941957\n",
       "11548    0.014962\n",
       "Name: engagement, Length: 2309, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def engagement_model():\n",
    "    rec = None\n",
    "    # YOUR CODE HERE\n",
    "    # Input data, both training/validation and test sets\n",
    "    trainval_df = pd.read_csv('assets/train.csv')\n",
    "    test_df = pd.read_csv('assets/test.csv')\n",
    "    # Break training/validation sets into targer and features\n",
    "    trainval_y = trainval_df['engagement']\n",
    "    trainval_X = trainval_df.loc[:, trainval_df.columns != 'engagement'].set_index('id')\n",
    "    features = list(trainval_X.columns)\n",
    "    X_test = test_df.set_index('id')\n",
    "    X_test_ids = X_test.index.tolist()\n",
    "    # Binarize the target values for the training and validation sets\n",
    "    trainval_y = trainval_y.astype(str)\n",
    "    trainval_y.replace(to_replace={'False':0,'True':1}, inplace=True)\n",
    "\n",
    "    # Create training and validation sets for both features and targets\n",
    "    X_train, X_val, y_train, y_val = train_test_split(trainval_X, trainval_y)\n",
    "\n",
    "    # Define the scaler, then fit to X_train and transform X_train, X_val, and X_test\n",
    "    scaler = MinMaxScaler()\n",
    "    X_train = scaler.fit_transform(X_train)\n",
    "    X_val = scaler.transform(X_val)\n",
    "    X_test = scaler.transform(X_test)\n",
    "    \n",
    "    model_scores = {}\n",
    "    \n",
    "    ## Dummy Models\n",
    "    # dummies = {}\n",
    "    # for strategy in DummyClass_Strategies:\n",
    "    #     dummies[strategy] = DummyClassifier(strategy=strategy).fit(X_train, y_train)\n",
    "    # for key, dummy in dummies.items():\n",
    "    #     print(\"Dummy Training Accuracy Score (Strategy = {})\".format(key),dummy.score(X_train, y_train))\n",
    "    #     print(\"Dummy Validation Accuracy Score (Strategy = {})\".format(key),dummy.score(X_val, y_val))\n",
    "\n",
    "    ## K-Nearest Neighbors with varying n\n",
    "    best_mean = [0,0]\n",
    "    n = 1\n",
    "    for n in range(1,len(features)*2):\n",
    "        knn = KNeighborsClassifier(n_neighbors = n)\n",
    "        knn.fit(X_train, y_train)\n",
    "        cv = 4\n",
    "        knn_aucs = cross_val_score(knn, X_train, y_train, cv=cv, scoring=\"roc_auc\")\n",
    "        mean = np.mean(knn_aucs)\n",
    "        if mean > best_mean[1]:\n",
    "            best_mean[1] = mean\n",
    "            best_mean[0] = n\n",
    "    # print('KNN (n={}) Best mean cross-validation ROC AUC score ({}-fold) on Training Data: {:.3f}'.format(best_mean[0], cv, best_mean[1]))\n",
    "    n = best_mean[0]\n",
    "    knn = KNeighborsClassifier(n_neighbors = n)\n",
    "    knn.fit(X_train, y_train)\n",
    "    # print(\"Accuracy Score for KNN on Training Data (n={}):\".format(n),knn.score(X_train, y_train))\n",
    "    # print(\"Accuracy Score for KNN on Validation Data (n={}):\".format(n),knn.score(X_val, y_val))\n",
    "    y_val_scores_knn = knn.predict_proba(X_val)\n",
    "    y_val_scores_knn = y_val_scores_knn[:,1]\n",
    "    knn_fpr, knn_tpr, _ = roc_curve(y_val, y_val_scores_knn)\n",
    "    knn_roc_auc = auc(knn_fpr, knn_tpr)\n",
    "    print('KNN Validation set AUC: ', knn_roc_auc)\n",
    "    model_scores['knn'] = (knn, knn_roc_auc)\n",
    "    \n",
    "    ## Linear Regression\n",
    "    ## Polynomial Option\n",
    "    # poly = PolynomialFeatures(degree=2)\n",
    "    # X_train = poly.fit_transform(X_train)\n",
    "    # X_val = poly.transform(X_val)\n",
    "    # linreg = LinearRegression().fit(X_train, y_train)\n",
    "    # linreg.fit(X_train, y_train)\n",
    "    # cv = 3\n",
    "    # linreg_aucs = cross_val_score(linreg, X_train, y_train, cv=cv, scoring=\"roc_auc\")\n",
    "    # print('LinReg Cross-validation scores ({}-fold):'.format(cv), linreg_aucs)\n",
    "    # print('LinReg Mean cross-validation score ({}-fold): {:.3f}'.format(cv,np.mean(linreg_aucs)))\n",
    "    # print(\"Accuracy Score for LinReg on Training Data:\",linreg.score(X_train, y_train))\n",
    "    # print(\"Accuracy Score for LinReg on Validation Data:\",linreg.score(X_val, y_val))\n",
    "       \n",
    "    ## Linear Regression Ridge\n",
    "    # linregridge = Ridge(alpha=0.1).fit(X_train, y_train)\n",
    "    # print(\"Accuracy Score for RidgeReg on Training Data:\",linregridge.score(X_train, y_train))\n",
    "    # print(\"Accuracy Score for RidgeReg on Validation Data:\",linregridge.score(X_val, y_val))\n",
    "    ## Linear Ridge with GridSearchCV\n",
    "    # linregridge = Ridge().fit(X_train, y_train)\n",
    "    # grid_values = {'alpha': [0.01, 0.1, 1, 10, 100]}\n",
    "    # cv = 2\n",
    "    # linregridge_grid = GridSearchCV(linregridge, param_grid=grid_values, scoring='roc_auc', cv=cv)\n",
    "    # linregridge_grid.fit(X_train, y_train)\n",
    "    # print('LinRegRidge Grid best parameter (max. AUC): ', linregridge_grid.best_params_)\n",
    "    # print('LinRegRidge Grid best score (AUC): ', linregridge_grid.best_score_)\n",
    "    # linregridge = Ridge(alpha=linregridge_grid.best_params_['alpha'])\n",
    "    \n",
    "    ## Linear Regression Lasso\n",
    "    # linreglasso = Lasso(alpha=0.1).fit(X_train, y_train)\n",
    "    # print(\"Accuracy Score for LassoReg on Training Data:\",linreglasso.score(X_train, y_train))\n",
    "    # print(\"Accuracy Score for LassoReg on Validation Data:\",linreglasso.score(X_val, y_val))\n",
    "    ## Linear Lasso with GridSearchCV\n",
    "    # linreglasso = Lasso().fit(X_train, y_train)\n",
    "    # grid_values = {'alpha': [0.01, 0.1, 1, 10, 100]}\n",
    "    # cv = 2\n",
    "    # linreglasso_grid = GridSearchCV(linreglasso, param_grid=grid_values, scoring='roc_auc', cv=cv)\n",
    "    # linreglasso_grid.fit(X_train, y_train)\n",
    "    # print('LinRegLasso Grid best parameter (max. AUC): ', linreglasso_grid.best_params_)\n",
    "    # print('LinRegLasso Grid best score (AUC): ', linreglasso_grid.best_score_)\n",
    "    # linreglasso = Lasso(alpha=linreglasso_grid.best_params_['alpha'])\n",
    "    \n",
    "    ## Logisitc Regression, C=10\n",
    "    # logreg = LogisticRegression(C=10).fit(X_train, y_train)\n",
    "    # print(\"Accuracy Score on Training Set:\",logreg.score(X_train, y_train))\n",
    "    # print(\"Accuracy Score on Validation Set:\",logreg.score(X_val, y_val))\n",
    "    # y_val_scores = logreg.fit(X_train, y_train).decision_function(X_val)\n",
    "    # logreg_fpr, logreg_tpr, _ = roc_curve(y_val, y_val_scores)\n",
    "    # logreg_roc_auc = auc(logreg_fpr, logreg_tpr)\n",
    "    # print(\"Logistic Regression ROC Curve AUC (C = {}):\".format(C),logreg_roc_auc)\n",
    "    ## Logisitc Regression & GridSearchCV and scoring on ROC AUC\n",
    "    # logreg = LogisticRegression()\n",
    "    # grid_values = {'C': [0.01, 0.1, 1, 10, 100]}\n",
    "    # cv = 2\n",
    "    # logreg_grid = GridSearchCV(logreg, param_grid=grid_values, scoring='roc_auc', cv=cv)\n",
    "    # logreg_grid.fit(X_train, y_train)\n",
    "    # y_val_scores_logreg_grid = logreg_grid.decision_function(X_val)\n",
    "    # logreg_grid_fpr, logreg_grid_tpr, _ = roc_curve(y_val, y_val_scores_logreg_grid)\n",
    "    # logreg_grid_roc_auc = auc(logreg_grid_fpr, logreg_grid_tpr)\n",
    "    # print('LogReg Grid best parameter (max. AUC): ', logreg_grid.best_params_)\n",
    "    # print('LogReg Grid best score (AUC): ', logreg_grid.best_score_)\n",
    "    # print('LogReg Validation set AUC: ', roc_auc_score(y_val, y_val_scores_logreg_grid))\n",
    "    # logreg = LogisticRegression(C=logreg_grid.best_params_['C'])\n",
    "    \n",
    "    ## Support Vector Machine with GridSearchCV\n",
    "    svc = SVC(kernel='rbf')\n",
    "    # grid_values = {'gamma': [0.01, 0.1, 1, 10, 100], 'C': [0.01, 0.1, 1, 10, 100]}\n",
    "    grid_values = {'gamma': [0.1, 1, 10, 100], 'C': [0.1, 1, 10, 100]}\n",
    "    cv = 3\n",
    "    svc_grid = GridSearchCV(svc, param_grid=grid_values, scoring='roc_auc', cv=cv)\n",
    "    svc_grid.fit(X_train, y_train)\n",
    "    y_val_scores_svc_grid = svc_grid.decision_function(X_val)\n",
    "    svc_grid_fpr, svc_grid_tpr, _ = roc_curve(y_val, y_val_scores_svc_grid)\n",
    "    svc_grid_roc_auc = auc(svc_grid_fpr, svc_grid_tpr)\n",
    "    # print('SVC Grid best parameter (max. AUC): ', svc_grid.best_params_)\n",
    "    # print('SVC Grid best score (AUC): ', svc_grid.best_score_)\n",
    "    print('SVC Validation set AUC: ', roc_auc_score(y_val, y_val_scores_svc_grid))\n",
    "    svc = SVC(kernel='rbf', C=svc_grid.best_params_['C'], gamma=svc_grid.best_params_['gamma'])\n",
    "    svc.fit(X_train, y_train)\n",
    "    model_scores['svc'] = (svc, svc_grid_roc_auc)\n",
    "    \n",
    "    ## Decision Tree Classifier\n",
    "    # tree = DecisionTreeClassifier().fit(X_train, y_train)\n",
    "    # print(\"Accuracy Score for Decision Tree on Training Data:\",tree.score(X_train, y_train))\n",
    "    # print(\"Accuracy Score for Decision Tree on Validation Data:\",tree.score(X_val, y_val))\n",
    "    ## Decision Tree Classifier with GridSearchCV\n",
    "    tree = DecisionTreeClassifier()\n",
    "    grid_values = {'max_depth': [2, 3, 4, 5, 6]}\n",
    "    cv = 3\n",
    "    tree_grid = GridSearchCV(tree, param_grid=grid_values, scoring='roc_auc', cv=cv)\n",
    "    tree_grid.fit(X_train, y_train)\n",
    "    y_val_scores_tree_grid = tree_grid.predict_proba(X_val)\n",
    "    y_val_scores_tree_grid = y_val_scores_tree_grid[:,1]\n",
    "    tree_grid_fpr, tree_grid_tpr, _ = roc_curve(y_val, y_val_scores_tree_grid)\n",
    "    tree_grid_roc_auc = auc(tree_grid_fpr, tree_grid_tpr)\n",
    "    # print('Tree Grid best parameter (max. AUC): ', tree_grid.best_params_)\n",
    "    # print('Tree Grid best score (AUC): ', tree_grid.best_score_)\n",
    "    print('Tree Validation set AUC: ', roc_auc_score(y_val, y_val_scores_tree_grid))\n",
    "    tree = DecisionTreeClassifier(max_depth=tree_grid.best_params_['max_depth'])\n",
    "    tree.fit(X_train, y_train)\n",
    "    model_scores['tree'] = (tree, tree_grid_roc_auc)\n",
    "    \n",
    "    ## Naive Bayes (Gaussian)\n",
    "    nbg = GaussianNB().fit(X_train, y_train)\n",
    "    y_val_scores_nbg = nbg.predict_proba(X_val)\n",
    "    y_val_scores_nbg = y_val_scores_nbg[:,1]\n",
    "    nbg_fpr, nbg_tpr, _ = roc_curve(y_val, y_val_scores_nbg)\n",
    "    nbg_roc_auc = auc(nbg_fpr, nbg_tpr)\n",
    "    print('Naive Bayes (Gaussian) Validation set AUC: ', nbg_roc_auc)\n",
    "    model_scores['nbg'] = (nbg, nbg_roc_auc)\n",
    "    \n",
    "    ## Random Forest\n",
    "    randfor = RandomForestClassifier()\n",
    "    sqrt_features = int(sqrt(len(features)))\n",
    "    num_features_list = list(range(sqrt_features-2,sqrt_features+2,1))\n",
    "    grid_values = {'max_features':num_features_list, \"n_estimators\":[10,15,25,50,100,200]}\n",
    "    cv = 3\n",
    "    randfor_grid = GridSearchCV(randfor, param_grid=grid_values, scoring='roc_auc', cv=cv)\n",
    "    randfor_grid.fit(X_train, y_train)\n",
    "    y_val_scores_randfor_grid = randfor_grid.predict_proba(X_val)\n",
    "    y_val_scores_randfor_grid = y_val_scores_randfor_grid[:,1]\n",
    "    randfor_grid_fpr, randfor_grid_tpr, _ = roc_curve(y_val, y_val_scores_randfor_grid)\n",
    "    randfor_grid_roc_auc = auc(randfor_grid_fpr, randfor_grid_tpr)\n",
    "    print('Random Forest Grid best parameter (max. AUC): ', randfor_grid.best_params_)\n",
    "    # print('Random Forest Grid best score (AUC): ', randfor_grid.best_score_)\n",
    "    print('Random Forest Validation set AUC: ', roc_auc_score(y_val, y_val_scores_randfor_grid))\n",
    "    randfor = RandomForestClassifier(max_features=randfor_grid.best_params_['max_features'], n_estimators=randfor_grid.best_params_['n_estimators'])\n",
    "    randfor.fit(X_train, y_train)\n",
    "    model_scores['randfor'] = (randfor, randfor_grid_roc_auc)\n",
    "    \n",
    "    ## Gradient Boosted Decision Trees\n",
    "    gbdt = GradientBoostingClassifier()\n",
    "    grid_values = {'learning_rate':[0.001,0.01,0.1], \"max_depth\":[2,3,4,5,100]}\n",
    "    cv = 3\n",
    "    gbdt_grid = GridSearchCV(gbdt, param_grid=grid_values, scoring='roc_auc', cv=cv)\n",
    "    gbdt_grid.fit(X_train, y_train)\n",
    "    y_val_scores_gbdt_grid = gbdt_grid.predict_proba(X_val)\n",
    "    y_val_scores_gbdt_grid = y_val_scores_gbdt_grid[:,1]\n",
    "    gbdt_grid_fpr, gbdt_grid_tpr, _ = roc_curve(y_val, y_val_scores_gbdt_grid)\n",
    "    gbdt_grid_roc_auc = auc(gbdt_grid_fpr, gbdt_grid_tpr)\n",
    "    print('Random Forest Grid best parameter (max. AUC): ', gbdt_grid.best_params_)\n",
    "    # print('Random Forest Grid best score (AUC): ', gbdt_grid.best_score_)\n",
    "    print('Gradient-Boosted Decision Trees Validation set AUC: ', roc_auc_score(y_val, y_val_scores_gbdt_grid))\n",
    "    gbdt = GradientBoostingClassifier(max_depth=gbdt_grid.best_params_['max_depth'], learning_rate=gbdt_grid.best_params_['learning_rate'])\n",
    "    gbdt.fit(X_train, y_train)\n",
    "    model_scores['gbdt'] = (gbdt, gbdt_grid_roc_auc)\n",
    "    \n",
    "    ## MLP Classifier\n",
    "    MLP = MLPClassifier()\n",
    "    # nn = MLPClassifier(solver='lbfgs', activation = 'tanh', alpha = 0.1,\n",
    "    #                    hidden_layer_sizes = [len(features)**2, len(features)]).fit(X_train, y_train)\n",
    "    \n",
    "    model_scores = sorted(model_scores.values(), key=lambda x:x[1], reverse=True)\n",
    "    best_model = model_scores[0][0]\n",
    "    predictions = best_model.predict_proba(X_test)\n",
    "    predictions = predictions[:,1]\n",
    "    return pd.Series(predictions,index=X_test_ids,name='engagement')\n",
    "    \n",
    "    raise NotImplementedError()\n",
    "    \n",
    "answer = engagement_model()\n",
    "answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "41e24667-3d1e-446b-87b6-bf6634fc2871",
   "metadata": {},
   "outputs": [],
   "source": [
    "stu_ans = answer\n",
    "assert isinstance(stu_ans, pd.Series), \"Your function should return a pd.Series. \"\n",
    "assert len(stu_ans) == 2309, \"Your series is of incorrect length: expected 2309 \"\n",
    "assert np.issubdtype(stu_ans.index.dtype, np.integer), \"Your answer pd.Series should have an index of integer type representing video id.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c78d565a",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e3b639d19639542702a374e84b2efb8e",
     "grade": false,
     "grade_id": "cell-ee9c764493852fe4",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "stu_ans = engagement_model()\n",
    "assert isinstance(stu_ans, pd.Series), \"Your function should return a pd.Series. \"\n",
    "assert len(stu_ans) == 2309, \"Your series is of incorrect length: expected 2309 \"\n",
    "assert np.issubdtype(stu_ans.index.dtype, np.integer), \"Your answer pd.Series should have an index of integer type representing video id.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24b6c9cb",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "bdbc39782ccccaa72a7a808b9025bba9",
     "grade": true,
     "grade_id": "cell-0372cfb70ab9b4de",
     "locked": true,
     "points": 4,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42118464",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "26a7c364781ea24c8e46c623e73c1d4a",
     "grade": true,
     "grade_id": "cell-df6ac3eec5bb54f7",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}