H2O_AutoML_IPYNB/h2oai_client_demo_bnp_paribas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Driverless AI: Using the Python API\n",
    "This notebook provides an H2OAI Client workflow, of model building and scoring, that parallels the Driverless AI workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Workflow Steps\n",
    "\n",
    "**Build an Experiment with Python API:**\n",
    "1. Sign in\n",
    "2. Import train & test set/new data\n",
    "3. Specify columns to drop & select target column\n",
    "4. Set problem type, configuration, and accuracy settings\n",
    "5. Launch Experiement\n",
    "6. View Results\n",
    "    \n",
    "**Build an Experiment in Web UI and Access Through Python:**\n",
    "1. Get pointer to experiment\n",
    "    \n",
    "**Scoring:**\n",
    "1. Score using Driverless\n",
    "2. Score using the Scoring Package\n",
    "\n",
    "**Train H2O Model:**\n",
    "1. Train H2O Model on raw data\n",
    "2. Train H2O Model on transformed data\n",
    "3. Compare model performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build an Experiment with Python API\n",
    "\n",
    "### Import Datasets\n",
    "\n",
    "#### 1. Sign In\n",
    "To upload a dataset to the Driverless AI Server, pass in your credentials through the Client class which creates an authentication token to send to the Driverless AI Server. In plain english: to sign into the Driverless AI webage (which then sends requests to the Driverless Server) instantiate the Client class with your Driverless AI address and login credentials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h2oai_client\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "# import h2o\n",
    "import requests\n",
    "import math\n",
    "from h2oai_client import Client, ModelParameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "address = 'http://ip_where_driverless_is_running:12345'\n",
    "username = 'username'\n",
    "password = 'password'\n",
    "h2oai = Client(address = address, username = username, password = password)\n",
    "# make sure to use the same user name and password when signing in through the GUI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Equivalent Steps in Driverless: Signing In\n",
    "![Equivalent Steps in Driverless: Signing In](h2oai_client_images/sign_in_home_page_0.png)\n",
    "![Equivalent Steps in Driverless: Signing In](h2oai_client_images/skip_sign_in_home_page_1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Upload Datasets\n",
    "\n",
    "In our example, we will use the BNP Paribas data.  We will split our data into training and validation so we can compare the performance of our Driverless AI model to an external model later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Data\n",
    "import pandas as pd\n",
    "data_path = '/data/Kaggle/BNPParibas/BNPParibas-train.csv'\n",
    "data = pd.read_csv(data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split Data\n",
    "import numpy as np\n",
    "train_indices = np.random.rand(len(data)) < 0.8\n",
    "train = data[train_indices]\n",
    "test = data[~train_indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export Data\n",
    "train_path = '/data/Kaggle/BNPParibas/train-80.csv'\n",
    "test_path = '/data/Kaggle/BNPParibas/valid-20.csv'\n",
    "train.to_csv(path_or_buf=train_path, index=False)\n",
    "test.to_csv(path_or_buf=test_path, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add Dataset to Driverless \n",
    "train = h2oai.create_dataset_sync(train_path)\n",
    "test = h2oai.create_dataset_sync(test_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Equivalent Steps in Driverless: Uploading Train & Test CSV Files\n",
    "![Equivalent Steps in Driverless: Uploading Train & Test CSV Files](h2oai_client_images/import_datasets_bnp.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### 3. Set Columns: Target, Ignore, Weight, and Fold\n",
    "\n",
    "* Target Column: the column we are trying to predict\n",
    "* Ignore Columns: the columns we do not want to use as predictors such as ID columns, columns with data leakage, etc\n",
    "* Weight Column: the column that indicates the per row observation weights - if none each row will have an observation weight of 1 \n",
    "* Fold Column: the column that indicates the fold - if none the folds will be determined by Driverless AI\n",
    "\n",
    "For this example, we will be predicting **`target`** and we will ignore the column: **`ID`**.  We do not have a `weight_col` or `fold_col`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set the parameters you want to pass to the UI\n",
    "target = \"target\"\n",
    "drop_cols = ['ID']\n",
    "weight_col = None\n",
    "fold_col = None\n",
    "time_col = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Equivalent Steps in Driverless: Set Target & Ignored Columns\n",
    "![Equivalent Steps in Driverless: Set Target & Ignored Columns](h2oai_client_images/add_drop_test_bnp.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Set Problem Type, Configuration & Accuracy Settings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pre-set parameters to pass model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "is_classification = True\n",
    "enable_gpus = True\n",
    "seed=True\n",
    "scorer_str = 'logloss'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pre-set accuracy knobs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy_value = 7\n",
    "time_value = 5\n",
    "interpretability = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Launch Experiment: Feature Engineering + Final Model Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment = h2oai.start_experiment_sync(ModelParameters(\n",
    "    dataset_key=train.key,\n",
    "    target_col=target,\n",
    "    is_classification=is_classification,\n",
    "    cols_to_drop=drop_cols,\n",
    "    testset_key=test.key, \n",
    "    enable_gpus=enable_gpus,\n",
    "    seed=seed,\n",
    "    accuracy=accuracy_value, \n",
    "    time= time_value,\n",
    "    interpretability=interpretability, \n",
    "    scorer=scorer_str,\n",
    "    weight_col = weight_col,\n",
    "    fold_col = fold_col,\n",
    "    time_col = time_col\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Equivalent Steps in Driverless: Set the Knobs, Configuration & Launch\n",
    "![Equivalent Steps in Driverless: Set the Knobs](h2oai_client_images/set_experiment_settings_bnp.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Equivalent Steps in Driverless: Launch Your Experiement](h2oai_client_images/exp_running_bnp.png)\n",
    "![Equivalent Steps in Driverless: Launch Your Experiement](h2oai_client_images/launch_experiment_bnp.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. View Results\n",
    "\n",
    "You can use the experiment object to get the final model score on the training and testing data. This final model may be an ensemble model depending on the accuracy setting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final Model Score on Train Data: 0.4585\n",
      "Final Model Score on Test Data: 0.46\n"
     ]
    }
   ],
   "source": [
    "print(\"Final Model Score on Train Data: \" + str(round(experiment.train_score, 4)))\n",
    "print(\"Final Model Score on Test Data: \" + str(round(experiment.test_score, 4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Equivalent Steps in Driverless: View Results\n",
    "![Equivalent Steps in Driverless: View Results](h2oai_client_images/experiment_done_bnp.png)\n",
    "\n",
    "#### All These Yellow Download Buttons Can Be Done through the Python API:\n",
    "* the munged test csv\n",
    "* the munged train csv\n",
    "* the predictions on the (holdout) train csv\n",
    "* the predictions on the test csv\n",
    "\n",
    "We will show an example of downloading the test predictions below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./test_preds.csv'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h2oai.download(src_path = experiment.test_predictions_path, dest_dir = \".\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>target.1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.646166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.932822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.756273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.917090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.659382</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   target.1\n",
       "0  0.646166\n",
       "1  0.932822\n",
       "2  0.756273\n",
       "3  0.917090\n",
       "4  0.659382"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_preds = pd.read_csv(\"./test_preds.csv\")\n",
    "test_preds.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build an Experiment in Web UI and Access Through Python\n",
    "\n",
    "It is also possible to use the Python API to examine an experiment that was started through the Web UI using the experiment key.\n",
    "\n",
    "![Experiments List](h2oai_client_images/launch_experiment_bnp.png)\n",
    "\n",
    "#### 1. Get pointer to experiment\n",
    "\n",
    "You can get a pointer to the experiment by referencing the experiment key in the web UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['797d7d']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment_list = list(map(lambda x: x.key, h2oai.list_models(offset = 0, limit = 100)))\n",
    "experiment_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment = h2oai.get_model_job(experiment_list[0]).entity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scoring\n",
    "\n",
    "There are two ways to score: through Driverless AI or by downloading the Scoring Package.  The Scoring Package contains a python module and is able to score independently of Driverless AI. \n",
    "\n",
    "#### 1. Score using Driverless AI\n",
    "\n",
    "We will start out by scoring through Driverless AI.  This is equivalent to the `SCORE ON ANOTHER DATASET` button in the Web UI.  In the example below, we will score on the test data and download the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>target.1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.646166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.932822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.756273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.917090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.659382</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   target.1\n",
       "0  0.646166\n",
       "1  0.932822\n",
       "2  0.756273\n",
       "3  0.917090\n",
       "4  0.659382"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction = h2oai.make_prediction_sync(experiment.key, test_path)\n",
    "pred_path = h2oai.download(prediction.predictions_csv_path, '.')\n",
    "pred_table = pd.read_csv(pred_path)\n",
    "pred_table.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### 2. Score using the Scoring Package\n",
    "\n",
    "We will download the scoring package and import the python scoring module to again score on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./scorer.zip'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Download the scoring package\n",
    "h2oai.download(experiment.scoring_package_path, '.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  scorer.zip\n",
      "   creating: scoring-package/\n",
      "  inflating: scoring-package/h2o4gpu-0.1.0+master.c200632-py36-none-any.whl  \n",
      "  inflating: scoring-package/example_client.py  \n",
      "  inflating: scoring-package/requirements.txt  \n",
      "  inflating: scoring-package/server.py  \n",
      "  inflating: scoring-package/example.py  \n",
      "  inflating: scoring-package/run_tcp_server.sh  \n",
      "  inflating: scoring-package/run_tcp_client.sh  \n",
      "  inflating: scoring-package/README.txt  \n",
      "  inflating: scoring-package/run_http_server.sh  \n",
      "  inflating: scoring-package/features.txt  \n",
      " extracting: scoring-package/server_requirements.txt  \n",
      "  inflating: scoring-package/run_http_client.sh  \n",
      "  inflating: scoring-package/run_example.sh  \n",
      "  inflating: scoring-package/scoring.thrift  \n",
      "  inflating: scoring-package/h2oaicore-1.0.10-cp36-cp36m-linux_x86_64.whl  \n",
      "  inflating: scoring-package/datatable-0.2.2+master.372.noomp-cp36-cp36m-linux_x86_64.whl  \n",
      "  inflating: scoring-package/scoring_797d7d_20171203000345_39780-1.0.0-py3-none-any.whl  \n",
      " extracting: scoring-package/client_requirements.txt  \n",
      "README.txt\n",
      "client_requirements.txt\n",
      "datatable-0.2.2+master.372.noomp-cp36-cp36m-linux_x86_64.whl\n",
      "example.py\n",
      "example_client.py\n",
      "features.txt\n",
      "h2o4gpu-0.1.0+master.c200632-py36-none-any.whl\n",
      "h2oaicore-1.0.10-cp36-cp36m-linux_x86_64.whl\n",
      "requirements.txt\n",
      "run_example.sh\n",
      "run_http_client.sh\n",
      "run_http_server.sh\n",
      "run_tcp_client.sh\n",
      "run_tcp_server.sh\n",
      "scoring.thrift\n",
      "scoring_797d7d_20171203000345_39780-1.0.0-py3-none-any.whl\n",
      "server.py\n",
      "server_requirements.txt\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Unzip scoring package and install the scoring python library\n",
    "unzip scorer\n",
    "\n",
    "cd scoring-package\n",
    "ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing ./scoring-package/scoring_797d7d_20171203000345_39780-1.0.0-py3-none-any.whl\n",
      "Installing collected packages: scoring-797d7d-20171203000345-39780\n",
      "Successfully installed scoring-797d7d-20171203000345-39780-1.0.0\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "# Import scoring module\n",
    "pip install scoring-package/scoring_797d7d_20171203000345_39780-1.0.0-py3-none-any.whl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from numpy import nan\n",
    "from scoring_797d7d_20171203000345_39780 import Scorer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/h2oai_env/lib/python3.6/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.\n",
      "  from pandas.core import datetools\n"
     ]
    }
   ],
   "source": [
    "#\n",
    "# Create a singleton Scorer instance.\n",
    "# For optimal performance, create a Scorer instance once, and call score() or score_batch() multiple times.\n",
    "#\n",
    "scorer = Scorer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import test data\n",
    "test_data = pd.read_csv(test_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Score the train data\n",
    "from IPython.utils import io\n",
    "with io.capture_output() as captured:\n",
    "    scorer.score_batch(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0_CV_TE_v107_0</th>\n",
       "      <th>1_CV_TE_v110_0</th>\n",
       "      <th>2_CV_TE_v112_0</th>\n",
       "      <th>3_CV_TE_v113_0</th>\n",
       "      <th>4_CV_TE_v125_0</th>\n",
       "      <th>5_CV_TE_v129_0</th>\n",
       "      <th>6_CV_TE_v22_0</th>\n",
       "      <th>7_CV_TE_v24_0</th>\n",
       "      <th>8_CV_TE_v3_0</th>\n",
       "      <th>9_CV_TE_v30_0</th>\n",
       "      <th>...</th>\n",
       "      <th>147_CV_CatNumEnc_v113_v56_v66__v12_std</th>\n",
       "      <th>147_CV_CatNumEnc_v113_v56_v66__v5_std</th>\n",
       "      <th>147_CV_CatNumEnc_v113_v56_v66__v50_std</th>\n",
       "      <th>147_CV_CatNumEnc_v113_v56_v66__v6_std</th>\n",
       "      <th>148_NumCatTE_v1_v11_v113_v4_v47_v5_v50_v56_v6_v66_v69_0</th>\n",
       "      <th>149_NumCatTE_v1_v11_v113_v12_v2_v24_v3_v31_v5_v50_v56_0</th>\n",
       "      <th>150_WoE_v110_v66_0</th>\n",
       "      <th>151_NumToCatTE_v1_v11_0</th>\n",
       "      <th>152_CV_TE_v56_0</th>\n",
       "      <th>153_NumCatTE_v1_v11_v110_v113_v12_v3_v31_v4_v5_v50_v56_v6_v66_v7_v79_0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.764832</td>\n",
       "      <td>0.6903</td>\n",
       "      <td>0.789034</td>\n",
       "      <td>0.683810</td>\n",
       "      <td>0.795302</td>\n",
       "      <td>0.726638</td>\n",
       "      <td>0.803030</td>\n",
       "      <td>0.800503</td>\n",
       "      <td>0.760087</td>\n",
       "      <td>0.774120</td>\n",
       "      <td>...</td>\n",
       "      <td>0.316153</td>\n",
       "      <td>1.740665</td>\n",
       "      <td>0.586085</td>\n",
       "      <td>0.596651</td>\n",
       "      <td>0.761961</td>\n",
       "      <td>0.761961</td>\n",
       "      <td>-0.504404</td>\n",
       "      <td>0.770112</td>\n",
       "      <td>0.636078</td>\n",
       "      <td>0.761961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.758349</td>\n",
       "      <td>0.6903</td>\n",
       "      <td>0.773467</td>\n",
       "      <td>0.683810</td>\n",
       "      <td>0.774168</td>\n",
       "      <td>0.726638</td>\n",
       "      <td>0.775436</td>\n",
       "      <td>0.755980</td>\n",
       "      <td>0.760087</td>\n",
       "      <td>0.761961</td>\n",
       "      <td>...</td>\n",
       "      <td>0.926541</td>\n",
       "      <td>2.039920</td>\n",
       "      <td>1.166030</td>\n",
       "      <td>0.602259</td>\n",
       "      <td>0.761961</td>\n",
       "      <td>0.761961</td>\n",
       "      <td>0.479076</td>\n",
       "      <td>0.770112</td>\n",
       "      <td>0.761961</td>\n",
       "      <td>0.761961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.758349</td>\n",
       "      <td>0.6903</td>\n",
       "      <td>0.778271</td>\n",
       "      <td>0.676176</td>\n",
       "      <td>0.784792</td>\n",
       "      <td>0.726638</td>\n",
       "      <td>0.761931</td>\n",
       "      <td>0.733728</td>\n",
       "      <td>0.760087</td>\n",
       "      <td>0.761961</td>\n",
       "      <td>...</td>\n",
       "      <td>0.639567</td>\n",
       "      <td>1.700511</td>\n",
       "      <td>0.843003</td>\n",
       "      <td>0.376991</td>\n",
       "      <td>0.761961</td>\n",
       "      <td>0.761961</td>\n",
       "      <td>-0.504404</td>\n",
       "      <td>0.770112</td>\n",
       "      <td>0.704082</td>\n",
       "      <td>0.761961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.743438</td>\n",
       "      <td>0.6903</td>\n",
       "      <td>0.747622</td>\n",
       "      <td>0.676176</td>\n",
       "      <td>0.750423</td>\n",
       "      <td>0.726638</td>\n",
       "      <td>0.692308</td>\n",
       "      <td>0.755980</td>\n",
       "      <td>0.760087</td>\n",
       "      <td>0.727316</td>\n",
       "      <td>...</td>\n",
       "      <td>0.430758</td>\n",
       "      <td>2.247023</td>\n",
       "      <td>1.217886</td>\n",
       "      <td>0.540155</td>\n",
       "      <td>0.840945</td>\n",
       "      <td>0.761961</td>\n",
       "      <td>-0.504404</td>\n",
       "      <td>0.770112</td>\n",
       "      <td>0.871929</td>\n",
       "      <td>0.795727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.758349</td>\n",
       "      <td>0.6903</td>\n",
       "      <td>0.773467</td>\n",
       "      <td>0.660176</td>\n",
       "      <td>0.774168</td>\n",
       "      <td>0.726638</td>\n",
       "      <td>0.681034</td>\n",
       "      <td>0.755980</td>\n",
       "      <td>0.760087</td>\n",
       "      <td>0.761961</td>\n",
       "      <td>...</td>\n",
       "      <td>0.364064</td>\n",
       "      <td>2.725092</td>\n",
       "      <td>0.622478</td>\n",
       "      <td>0.624809</td>\n",
       "      <td>0.525734</td>\n",
       "      <td>0.761961</td>\n",
       "      <td>0.479076</td>\n",
       "      <td>0.770112</td>\n",
       "      <td>0.771385</td>\n",
       "      <td>0.653876</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 168 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   0_CV_TE_v107_0  1_CV_TE_v110_0  2_CV_TE_v112_0  3_CV_TE_v113_0  \\\n",
       "0        0.764832          0.6903        0.789034        0.683810   \n",
       "1        0.758349          0.6903        0.773467        0.683810   \n",
       "2        0.758349          0.6903        0.778271        0.676176   \n",
       "3        0.743438          0.6903        0.747622        0.676176   \n",
       "4        0.758349          0.6903        0.773467        0.660176   \n",
       "\n",
       "   4_CV_TE_v125_0  5_CV_TE_v129_0  6_CV_TE_v22_0  7_CV_TE_v24_0  8_CV_TE_v3_0  \\\n",
       "0        0.795302        0.726638       0.803030       0.800503      0.760087   \n",
       "1        0.774168        0.726638       0.775436       0.755980      0.760087   \n",
       "2        0.784792        0.726638       0.761931       0.733728      0.760087   \n",
       "3        0.750423        0.726638       0.692308       0.755980      0.760087   \n",
       "4        0.774168        0.726638       0.681034       0.755980      0.760087   \n",
       "\n",
       "   9_CV_TE_v30_0  \\\n",
       "0       0.774120   \n",
       "1       0.761961   \n",
       "2       0.761961   \n",
       "3       0.727316   \n",
       "4       0.761961   \n",
       "\n",
       "                                    ...                                    \\\n",
       "0                                   ...                                     \n",
       "1                                   ...                                     \n",
       "2                                   ...                                     \n",
       "3                                   ...                                     \n",
       "4                                   ...                                     \n",
       "\n",
       "   147_CV_CatNumEnc_v113_v56_v66__v12_std  \\\n",
       "0                                0.316153   \n",
       "1                                0.926541   \n",
       "2                                0.639567   \n",
       "3                                0.430758   \n",
       "4                                0.364064   \n",
       "\n",
       "   147_CV_CatNumEnc_v113_v56_v66__v5_std  \\\n",
       "0                               1.740665   \n",
       "1                               2.039920   \n",
       "2                               1.700511   \n",
       "3                               2.247023   \n",
       "4                               2.725092   \n",
       "\n",
       "   147_CV_CatNumEnc_v113_v56_v66__v50_std  \\\n",
       "0                                0.586085   \n",
       "1                                1.166030   \n",
       "2                                0.843003   \n",
       "3                                1.217886   \n",
       "4                                0.622478   \n",
       "\n",
       "   147_CV_CatNumEnc_v113_v56_v66__v6_std  \\\n",
       "0                               0.596651   \n",
       "1                               0.602259   \n",
       "2                               0.376991   \n",
       "3                               0.540155   \n",
       "4                               0.624809   \n",
       "\n",
       "   148_NumCatTE_v1_v11_v113_v4_v47_v5_v50_v56_v6_v66_v69_0  \\\n",
       "0                                           0.761961         \n",
       "1                                           0.761961         \n",
       "2                                           0.761961         \n",
       "3                                           0.840945         \n",
       "4                                           0.525734         \n",
       "\n",
       "   149_NumCatTE_v1_v11_v113_v12_v2_v24_v3_v31_v5_v50_v56_0  \\\n",
       "0                                           0.761961         \n",
       "1                                           0.761961         \n",
       "2                                           0.761961         \n",
       "3                                           0.761961         \n",
       "4                                           0.761961         \n",
       "\n",
       "   150_WoE_v110_v66_0  151_NumToCatTE_v1_v11_0  152_CV_TE_v56_0  \\\n",
       "0           -0.504404                 0.770112         0.636078   \n",
       "1            0.479076                 0.770112         0.761961   \n",
       "2           -0.504404                 0.770112         0.704082   \n",
       "3           -0.504404                 0.770112         0.871929   \n",
       "4            0.479076                 0.770112         0.771385   \n",
       "\n",
       "   153_NumCatTE_v1_v11_v110_v113_v12_v3_v31_v4_v5_v50_v56_v6_v66_v7_v79_0  \n",
       "0                                           0.761961                       \n",
       "1                                           0.761961                       \n",
       "2                                           0.761961                       \n",
       "3                                           0.795727                       \n",
       "4                                           0.653876                       \n",
       "\n",
       "[5 rows x 168 columns]"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Transform test data\n",
    "transformed_test = scorer.transform_batch(test_data)\n",
    "transformed_test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train H2O Model \n",
    "\n",
    "We are not limited to using the scores from Driverless AI.  We can also use the transformed data to train our own model.  In this section, we will train a GBM model on the raw data and transformed data and compare the results to Driverless AI. \n",
    "\n",
    "#### 1. Train H2O Model on Raw Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting h2o==3.16.0.1 from http://h2o-release.s3.amazonaws.com/h2o/rel-wheeler/1/Python/h2o-3.16.0.1-py2.py3-none-any.whl\n",
      "  Downloading http://h2o-release.s3.amazonaws.com/h2o/rel-wheeler/1/Python/h2o-3.16.0.1-py2.py3-none-any.whl (119.6MB)\n",
      "\u001b[K    100% |################################| 119.6MB 109.4MB/s ta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: colorama>=0.3.8 in /h2oai_env/lib/python3.6/site-packages (from h2o==3.16.0.1)\n",
      "Requirement already satisfied: tabulate in /h2oai_env/lib/python3.6/site-packages (from h2o==3.16.0.1)\n",
      "Requirement already satisfied: future in /h2oai_env/lib/python3.6/site-packages (from h2o==3.16.0.1)\n",
      "Requirement already satisfied: requests in /h2oai_env/lib/python3.6/site-packages (from h2o==3.16.0.1)\n",
      "Installing collected packages: h2o\n",
      "Successfully installed h2o-3.16.0.1\n"
     ]
    }
   ],
   "source": [
    "# Install H2O\n",
    "\n",
    "! pip install http://h2o-release.s3.amazonaws.com/h2o/rel-wheeler/1/Python/h2o-3.16.0.1-py2.py3-none-any.whl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54323..... not found.\n",
      "Attempting to start a local H2O server...\n",
      "  Java Version: java version \"1.8.0_151\"; Java(TM) SE Runtime Environment (build 1.8.0_151-b12); Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)\n",
      "  Starting server from /h2oai_env/lib/python3.6/site-packages/h2o/backend/bin/h2o.jar\n",
      "  Ice root: /tmp/tmpzaf2vqc6\n",
      "  JVM stdout: /tmp/tmpzaf2vqc6/h2o_unknownUser_started_from_python.out\n",
      "  JVM stderr: /tmp/tmpzaf2vqc6/h2o_unknownUser_started_from_python.err\n",
      "  Server is running at http://127.0.0.1:54323\n",
      "Connecting to H2O server at http://127.0.0.1:54323... successful.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td>H2O cluster uptime:</td>\n",
       "<td>01 secs</td></tr>\n",
       "<tr><td>H2O cluster version:</td>\n",
       "<td>3.16.0.1</td></tr>\n",
       "<tr><td>H2O cluster version age:</td>\n",
       "<td>8 days </td></tr>\n",
       "<tr><td>H2O cluster name:</td>\n",
       "<td>H2O_from_python_unknownUser_2k4ws0</td></tr>\n",
       "<tr><td>H2O cluster total nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O cluster free memory:</td>\n",
       "<td>13.33 Gb</td></tr>\n",
       "<tr><td>H2O cluster total cores:</td>\n",
       "<td>4</td></tr>\n",
       "<tr><td>H2O cluster allowed cores:</td>\n",
       "<td>4</td></tr>\n",
       "<tr><td>H2O cluster status:</td>\n",
       "<td>accepting new members, healthy</td></tr>\n",
       "<tr><td>H2O connection url:</td>\n",
       "<td>http://127.0.0.1:54323</td></tr>\n",
       "<tr><td>H2O connection proxy:</td>\n",
       "<td>None</td></tr>\n",
       "<tr><td>H2O internal security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>H2O API Extensions:</td>\n",
       "<td>XGBoost, Algos, AutoML, Core V3, Core V4</td></tr>\n",
       "<tr><td>Python version:</td>\n",
       "<td>3.6.2 final</td></tr></table></div>"
      ],
      "text/plain": [
       "--------------------------  ----------------------------------------\n",
       "H2O cluster uptime:         01 secs\n",
       "H2O cluster version:        3.16.0.1\n",
       "H2O cluster version age:    8 days\n",
       "H2O cluster name:           H2O_from_python_unknownUser_2k4ws0\n",
       "H2O cluster total nodes:    1\n",
       "H2O cluster free memory:    13.33 Gb\n",
       "H2O cluster total cores:    4\n",
       "H2O cluster allowed cores:  4\n",
       "H2O cluster status:         accepting new members, healthy\n",
       "H2O connection url:         http://127.0.0.1:54323\n",
       "H2O connection proxy:\n",
       "H2O internal security:      False\n",
       "H2O API Extensions:         XGBoost, Algos, AutoML, Core V3, Core V4\n",
       "Python version:             3.6.2 final\n",
       "--------------------------  ----------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import H2O and connect\n",
    "import h2o\n",
    "h2o.init(port = 54323)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "train_h2o = h2o.import_file(train_path)\n",
    "test_h2o = h2o.import_file(test_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "if is_classification:\n",
    "    train_h2o[target] = train_h2o[target].asfactor()\n",
    "    test_h2o[target] = test_h2o[target].asfactor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gbm Model Build progress: |███████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "# Train GBM on Raw Data\n",
    "from h2o.estimators import H2OGradientBoostingEstimator\n",
    "gbm_raw = H2OGradientBoostingEstimator(model_id = \"raw_data\", \n",
    "                                       nfolds = 3, # 3-fold cross validation\n",
    "                                       ntrees = 1000,   # more than enough trees\n",
    "                                       ## early stopping once the validation logloss doesn't improve \n",
    "                                       ## by at least 0.1% for 3 consecutive scoring events\n",
    "                                       stopping_rounds = 3, stopping_tolerance = 1e-3, stopping_metric = \"logloss\", \n",
    "                                       score_tree_interval = 10,\n",
    "                                       seed = 1234)\n",
    "gbm_raw.train(y = target, training_frame = train_h2o)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Train H2O Model on Transformed Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "# Convert Transform Test Data to H2O Frame\n",
    "transformed_test = pd.concat([transformed_test, test_data[target]], axis = 1)\n",
    "transformed_test_h2o = h2o.H2OFrame(transformed_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "# Import Transformed Train Data as H2O Frame\n",
    "transformed_train_path = h2oai.download(src_path = experiment.train_transformed_csv_path, dest_dir = \".\")\n",
    "transformed_train_h2o = h2o.import_file(transformed_train_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "if is_classification:\n",
    "    transformed_test_h2o[target] = transformed_test_h2o[target].asfactor()\n",
    "    transformed_train_h2o[target] = transformed_train_h2o[target].asfactor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gbm Model Build progress: |███████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "# Train GBM on Transformed Data\n",
    "gbm_transformed = H2OGradientBoostingEstimator(model_id = \"transformed_data\", \n",
    "                                               nfolds = 3, # 3-fold cross validation\n",
    "                                               ntrees = 1000,   # more than enough trees \n",
    "                                               ## early stopping once the validation logloss doesn't improve \n",
    "                                               ## by at least 0.1% for 3 consecutive scoring events\n",
    "                                               stopping_rounds = 3, stopping_tolerance = 1e-3, stopping_metric = \"logloss\",\n",
    "                                               score_tree_interval = 10,\n",
    "                                               seed = 1234)\n",
    "gbm_transformed.train(y = target, training_frame = transformed_train_h2o)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Compare Model Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_data_logloss = gbm_raw.model_performance(test_data = test_h2o).logloss()\n",
    "transformed_data_logloss = gbm_transformed.model_performance(test_data = transformed_test_h2o).logloss()\n",
    "driverless_logloss = experiment.test_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw Data Logloss: 0.4683\n",
      "Transformed Data Logloss: 0.4685\n",
      "Driverless AI Logloss: 0.46\n"
     ]
    }
   ],
   "source": [
    "print(\"Raw Data Logloss: \" + str(round(raw_data_logloss, 4)))\n",
    "print(\"Transformed Data Logloss: \" + str(round(transformed_data_logloss, 4)))\n",
    "print(\"Driverless AI Logloss: \" + str(round(driverless_logloss, 4)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}