H2O_AutoML_IPYNB/H2OAutoML_as_sklearn_estimator.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using H2OAutoML with Scikit-Learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Since release `3.28.0.1`, it is finally possible to use `H2OAutoML` as a `Scikit-Learn` estimator and in combination with other `sklearn` components.\n",
    "\n",
    "Like for other `H2O-3` estimators (see `H2O_estimators_as_sklearn_estimators` notebook in the current folder for more details), the new `h2o.sklearn` module exposes 2 wrappers for `H2OAutoML`:\n",
    "- `H2OAutoMLClassifier`\n",
    "- `H2OAutoMLRegressor`\n",
    "\n",
    "Those wrappers expose the standard API familiar to `sklearn` users (`fit`, `predict`, `fit_predict`, `score`, `get_params`, `set_params`) and accept various format as input data (`H2OFrame`, `numpy` array, `pandas` Dataframe) which allow them to be combined with pure `sklearn` components in pipelines.\n",
    "\n",
    "Finally, user can still access the original `H2OAutoML` estimator from the wrapper thanks to its `estimator` property: this gives access to all the methods and properties declared in `H2OAutoML`, like `leaderboard`, `leader`, `event_log`, `training_info`, ...\n",
    "\n",
    "Again, for more details, it is recommended to have a look at the `H2O_estimators_as_sklearn_estimators` notebook in this folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `H2OAutoMLClassifier` in practice "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requirements"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# requirements for this tutorial (enable and run this cell to install)\n",
    "!pip install numpy pandas\n",
    "!pip install http://h2o-release.s3.amazonaws.com/h2o/rel-yu/2/Python/h2o-3.28.0.2-py2.py3-none-any.whl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(action='ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "from sklearn.feature_selection import f_classif, SelectKBest\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "from h2o.sklearn import H2OAutoMLClassifier\n",
    "\n",
    "seed = 2020\n",
    "\n",
    "X, y = datasets.load_breast_cancer(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y)\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('polyfeat', PolynomialFeatures(degree=2)),\n",
    "    ('featselect', SelectKBest(f_classif, k=5)),\n",
    "    ('classifier', H2OAutoMLClassifier(max_models=10, seed=seed, sort_metric='aucpr'))\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54321 . connected.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td>H2O cluster uptime:</td>\n",
       "<td>32 mins 25 secs</td></tr>\n",
       "<tr><td>H2O cluster timezone:</td>\n",
       "<td>Europe/Prague</td></tr>\n",
       "<tr><td>H2O data parsing timezone:</td>\n",
       "<td>UTC</td></tr>\n",
       "<tr><td>H2O cluster version:</td>\n",
       "<td>3.28.0.2</td></tr>\n",
       "<tr><td>H2O cluster version age:</td>\n",
       "<td>8 days </td></tr>\n",
       "<tr><td>H2O cluster name:</td>\n",
       "<td>H2O_from_python_seb_tdhiu9</td></tr>\n",
       "<tr><td>H2O cluster total nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O cluster free memory:</td>\n",
       "<td>3.218 Gb</td></tr>\n",
       "<tr><td>H2O cluster total cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O cluster allowed cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O cluster status:</td>\n",
       "<td>locked, healthy</td></tr>\n",
       "<tr><td>H2O connection url:</td>\n",
       "<td>http://localhost:54321</td></tr>\n",
       "<tr><td>H2O connection proxy:</td>\n",
       "<td>{'http': None, 'https': None}</td></tr>\n",
       "<tr><td>H2O internal security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>H2O API Extensions:</td>\n",
       "<td>Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4</td></tr>\n",
       "<tr><td>Python version:</td>\n",
       "<td>3.7.5 final</td></tr></table></div>"
      ],
      "text/plain": [
       "--------------------------  ------------------------------------------------------------------\n",
       "H2O cluster uptime:         32 mins 25 secs\n",
       "H2O cluster timezone:       Europe/Prague\n",
       "H2O data parsing timezone:  UTC\n",
       "H2O cluster version:        3.28.0.2\n",
       "H2O cluster version age:    8 days\n",
       "H2O cluster name:           H2O_from_python_seb_tdhiu9\n",
       "H2O cluster total nodes:    1\n",
       "H2O cluster free memory:    3.218 Gb\n",
       "H2O cluster total cores:    8\n",
       "H2O cluster allowed cores:  8\n",
       "H2O cluster status:         locked, healthy\n",
       "H2O connection url:         http://localhost:54321\n",
       "H2O connection proxy:       {'http': None, 'https': None}\n",
       "H2O internal security:      False\n",
       "H2O API Extensions:         Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4\n",
       "Python version:             3.7.5 final\n",
       "--------------------------  ------------------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "AutoML progress: |████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "glm prediction progress: |████████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "pipeline.fit(X_train, y_train)\n",
    "preds = pipeline.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.95      0.95        58\n",
      "           1       0.96      0.96      0.96        85\n",
      "\n",
      "    accuracy                           0.96       143\n",
      "   macro avg       0.96      0.96      0.96       143\n",
      "weighted avg       0.96      0.96      0.96       143\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, preds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also access the `H2OAutoML` instance with the `estimator` property:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>model_id                                           </th><th style=\"text-align: right;\">   aucpr</th><th style=\"text-align: right;\">     auc</th><th style=\"text-align: right;\">  logloss</th><th style=\"text-align: right;\">  mean_per_class_error</th><th style=\"text-align: right;\">    rmse</th><th style=\"text-align: right;\">      mse</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>GLM_1_AutoML_20200128_201119                       </td><td style=\"text-align: right;\">0.98864 </td><td style=\"text-align: right;\">0.989257</td><td style=\"text-align: right;\"> 0.11541 </td><td style=\"text-align: right;\">             0.0351652</td><td style=\"text-align: right;\">0.176139</td><td style=\"text-align: right;\">0.0310248</td></tr>\n",
       "<tr><td>StackedEnsemble_BestOfFamily_AutoML_20200128_201119</td><td style=\"text-align: right;\">0.984122</td><td style=\"text-align: right;\">0.984172</td><td style=\"text-align: right;\"> 0.125697</td><td style=\"text-align: right;\">             0.0370034</td><td style=\"text-align: right;\">0.176479</td><td style=\"text-align: right;\">0.0311449</td></tr>\n",
       "<tr><td>StackedEnsemble_AllModels_AutoML_20200128_201119   </td><td style=\"text-align: right;\">0.984022</td><td style=\"text-align: right;\">0.9841  </td><td style=\"text-align: right;\"> 0.125147</td><td style=\"text-align: right;\">             0.0416587</td><td style=\"text-align: right;\">0.176794</td><td style=\"text-align: right;\">0.0312561</td></tr>\n",
       "<tr><td>XGBoost_3_AutoML_20200128_201119                   </td><td style=\"text-align: right;\">0.981715</td><td style=\"text-align: right;\">0.983802</td><td style=\"text-align: right;\"> 0.135276</td><td style=\"text-align: right;\">             0.0522584</td><td style=\"text-align: right;\">0.191452</td><td style=\"text-align: right;\">0.0366538</td></tr>\n",
       "<tr><td>GBM_1_AutoML_20200128_201119                       </td><td style=\"text-align: right;\">0.974896</td><td style=\"text-align: right;\">0.973871</td><td style=\"text-align: right;\"> 0.146192</td><td style=\"text-align: right;\">             0.0430672</td><td style=\"text-align: right;\">0.193089</td><td style=\"text-align: right;\">0.0372835</td></tr>\n",
       "<tr><td>GBM_3_AutoML_20200128_201119                       </td><td style=\"text-align: right;\">0.968214</td><td style=\"text-align: right;\">0.972641</td><td style=\"text-align: right;\"> 0.150999</td><td style=\"text-align: right;\">             0.0560542</td><td style=\"text-align: right;\">0.196219</td><td style=\"text-align: right;\">0.0385017</td></tr>\n",
       "<tr><td>GBM_4_AutoML_20200128_201119                       </td><td style=\"text-align: right;\">0.962027</td><td style=\"text-align: right;\">0.973883</td><td style=\"text-align: right;\"> 0.147488</td><td style=\"text-align: right;\">             0.0476031</td><td style=\"text-align: right;\">0.193489</td><td style=\"text-align: right;\">0.0374382</td></tr>\n",
       "<tr><td>GBM_2_AutoML_20200128_201119                       </td><td style=\"text-align: right;\">0.960663</td><td style=\"text-align: right;\">0.972044</td><td style=\"text-align: right;\"> 0.156461</td><td style=\"text-align: right;\">             0.0494414</td><td style=\"text-align: right;\">0.201593</td><td style=\"text-align: right;\">0.0406398</td></tr>\n",
       "<tr><td>XGBoost_2_AutoML_20200128_201119                   </td><td style=\"text-align: right;\">0.925624</td><td style=\"text-align: right;\">0.983707</td><td style=\"text-align: right;\"> 0.189011</td><td style=\"text-align: right;\">             0.0634072</td><td style=\"text-align: right;\">0.213545</td><td style=\"text-align: right;\">0.0456016</td></tr>\n",
       "<tr><td>XGBoost_1_AutoML_20200128_201119                   </td><td style=\"text-align: right;\">0.861227</td><td style=\"text-align: right;\">0.983635</td><td style=\"text-align: right;\"> 0.145491</td><td style=\"text-align: right;\">             0.0574628</td><td style=\"text-align: right;\">0.195643</td><td style=\"text-align: right;\">0.0382763</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "automl = pipeline.named_steps.classifier.estimator\n",
    "\n",
    "automl.leaderboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `H2OAutoMLRegressor` example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `H2OAutoMLRegressor` can be used in exactly the same way as its classifier counterpart.\n",
    "\n",
    "Here, we will try to run multiple automl at once, using a small grid search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "AutoML progress: |████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "gbm prediction progress: |████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "AutoML progress: |████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "stackedensemble prediction progress: |████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "AutoML progress: |████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "stackedensemble prediction progress: |████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "AutoML progress: |████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "stackedensemble prediction progress: |████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "AutoML progress: |████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "gbm prediction progress: |████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "AutoML progress: |████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "stackedensemble prediction progress: |████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "AutoML progress: |████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "gbm prediction progress: |████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "AutoML progress: |████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "stackedensemble prediction progress: |████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n",
      "AutoML progress: |████████████████████████████████████████████████████████| 100%\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=2, error_score=nan,\n",
       "             estimator=H2OAutoMLRegressor(algo_parameters=None,\n",
       "                                          balance_classes=False,\n",
       "                                          class_sampling_factors=None,\n",
       "                                          data_conversion='auto',\n",
       "                                          exclude_algos=None,\n",
       "                                          export_checkpoints_dir=None,\n",
       "                                          include_algos=None,\n",
       "                                          keep_cross_validation_fold_assignment=False,\n",
       "                                          keep_cross_validation_models=False,\n",
       "                                          keep_cross_validation_predictions=False,\n",
       "                                          max_afte...\n",
       "                                          monotone_constraints=None, nfolds=5,\n",
       "                                          project_name=None, seed=2020,\n",
       "                                          sort_metric='AUTO',\n",
       "                                          stopping_metric='AUTO',\n",
       "                                          stopping_rounds=3,\n",
       "                                          stopping_tolerance=None,\n",
       "                                          verbosity='warn'),\n",
       "             iid='deprecated', n_jobs=None,\n",
       "             param_grid={'monotone_constraints': [None, {'AGE': 1},\n",
       "                                                  {'PTRATIO': 1},\n",
       "                                                  {'CRIM': -1}]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=0)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from sklearn import datasets\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "\n",
    "from h2o.sklearn import H2OAutoMLRegressor\n",
    "\n",
    "ds = datasets.load_boston()\n",
    "\n",
    "seed = 2020\n",
    "regressor = H2OAutoMLRegressor(max_models=10, max_runtime_secs_per_model=30, seed=seed)\n",
    "\n",
    "grid = GridSearchCV(regressor, cv=2, param_grid=dict(\n",
    "    monotone_constraints=[None, dict(AGE=1), dict(PTRATIO=1), dict(CRIM=-1)],\n",
    "))\n",
    "\n",
    "# converting the data into a pandas DataFrame as we need the column names for the monotone_constraints parameter\n",
    "X = pd.DataFrame(ds.data, columns=ds.feature_names)\n",
    "y = ds.target\n",
    "\n",
    "grid.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'monotone_constraints': {'AGE': 1}}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best = grid.best_estimator_\n",
    "grid.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([13.52753139, 12.98746252, 13.68160176, 13.30000401]),\n",
       " 'std_fit_time': array([0.29193354, 0.03823948, 0.21549892, 0.07853317]),\n",
       " 'mean_score_time': array([0.38817024, 0.48703146, 0.40478611, 0.39410508]),\n",
       " 'std_score_time': array([0.0828371 , 0.01597548, 0.15079379, 0.1262399 ]),\n",
       " 'param_monotone_constraints': masked_array(data=[None, {'AGE': 1}, {'PTRATIO': 1}, {'CRIM': -1}],\n",
       "              mask=[False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'monotone_constraints': None},\n",
       "  {'monotone_constraints': {'AGE': 1}},\n",
       "  {'monotone_constraints': {'PTRATIO': 1}},\n",
       "  {'monotone_constraints': {'CRIM': -1}}],\n",
       " 'split0_test_score': array([0.78161713, 0.82788077, 0.76359332, 0.77081437]),\n",
       " 'split1_test_score': array([0.62617193, 0.61747351, 0.56696796, 0.61541766]),\n",
       " 'mean_test_score': array([0.70389453, 0.72267714, 0.66528064, 0.69311602]),\n",
       " 'std_test_score': array([0.0777226 , 0.10520363, 0.09831268, 0.07769836]),\n",
       " 'rank_test_score': array([2, 1, 4, 3], dtype=int32)}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>model_id                                           </th><th style=\"text-align: right;\">  mean_residual_deviance</th><th style=\"text-align: right;\">   rmse</th><th style=\"text-align: right;\">     mse</th><th style=\"text-align: right;\">    mae</th><th style=\"text-align: right;\">   rmsle</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>XGBoost_3_AutoML_20200128_201323                   </td><td style=\"text-align: right;\">                 8.68469</td><td style=\"text-align: right;\">2.94698</td><td style=\"text-align: right;\"> 8.68469</td><td style=\"text-align: right;\">2.09068</td><td style=\"text-align: right;\">0.134579</td></tr>\n",
       "<tr><td>StackedEnsemble_AllModels_AutoML_20200128_201323   </td><td style=\"text-align: right;\">                 8.93475</td><td style=\"text-align: right;\">2.98911</td><td style=\"text-align: right;\"> 8.93475</td><td style=\"text-align: right;\">2.07317</td><td style=\"text-align: right;\">0.134023</td></tr>\n",
       "<tr><td>StackedEnsemble_BestOfFamily_AutoML_20200128_201323</td><td style=\"text-align: right;\">                 8.93475</td><td style=\"text-align: right;\">2.98911</td><td style=\"text-align: right;\"> 8.93475</td><td style=\"text-align: right;\">2.07317</td><td style=\"text-align: right;\">0.134023</td></tr>\n",
       "<tr><td>GBM_1_AutoML_20200128_201323                       </td><td style=\"text-align: right;\">                 9.60047</td><td style=\"text-align: right;\">3.09846</td><td style=\"text-align: right;\"> 9.60047</td><td style=\"text-align: right;\">2.20709</td><td style=\"text-align: right;\">0.137438</td></tr>\n",
       "<tr><td>DRF_1_AutoML_20200128_201323                       </td><td style=\"text-align: right;\">                10.1783 </td><td style=\"text-align: right;\">3.19034</td><td style=\"text-align: right;\">10.1783 </td><td style=\"text-align: right;\">2.14753</td><td style=\"text-align: right;\">0.137721</td></tr>\n",
       "<tr><td>GBM_2_AutoML_20200128_201323                       </td><td style=\"text-align: right;\">                10.2626 </td><td style=\"text-align: right;\">3.20353</td><td style=\"text-align: right;\">10.2626 </td><td style=\"text-align: right;\">2.20055</td><td style=\"text-align: right;\">0.140183</td></tr>\n",
       "<tr><td>XGBoost_1_AutoML_20200128_201323                   </td><td style=\"text-align: right;\">                10.281  </td><td style=\"text-align: right;\">3.2064 </td><td style=\"text-align: right;\">10.281  </td><td style=\"text-align: right;\">2.16717</td><td style=\"text-align: right;\">0.14362 </td></tr>\n",
       "<tr><td>GBM_4_AutoML_20200128_201323                       </td><td style=\"text-align: right;\">                10.537  </td><td style=\"text-align: right;\">3.24608</td><td style=\"text-align: right;\">10.537  </td><td style=\"text-align: right;\">2.22226</td><td style=\"text-align: right;\">0.143879</td></tr>\n",
       "<tr><td>XGBoost_2_AutoML_20200128_201323                   </td><td style=\"text-align: right;\">                10.7641 </td><td style=\"text-align: right;\">3.28087</td><td style=\"text-align: right;\">10.7641 </td><td style=\"text-align: right;\">2.21865</td><td style=\"text-align: right;\">0.145428</td></tr>\n",
       "<tr><td>GBM_3_AutoML_20200128_201323                       </td><td style=\"text-align: right;\">                11.2836 </td><td style=\"text-align: right;\">3.35911</td><td style=\"text-align: right;\">11.2836 </td><td style=\"text-align: right;\">2.26692</td><td style=\"text-align: right;\">0.149402</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "automl = best.estimator\n",
    "\n",
    "automl.leaderboard"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ve37-h2o",
   "language": "python",
   "name": "ve37-h2o"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}