cpd4.6/notebooks/python_sdk/experiments/autoai/Use AutoAI and Lale to predict credit risk.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Use AutoAI and Lale to predict credit risk with `ibm-watson-machine-learning`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This notebook contains the steps and code to demonstrate support of AutoAI experiments in Watson Machine Learning service. It introduces commands for data retrieval, training experiments, persisting pipelines, testing pipelines, refining pipelines, and scoring.\n",
        "\n",
        "Some familiarity with Python is helpful. This notebook uses Python 3.9.\n",
        "\n",
        "\n",
        "## Learning goals\n",
        "\n",
        "The learning goals of this notebook are:\n",
        "\n",
        "-  Work with Watson Machine Learning experiments to train AutoAI models.\n",
        "-  Compare trained models quality and select the best one for further refinement.\n",
        "-  Refine the best model and test new variations.\n",
        "-  Perform online deployment and score the trained model.\n",
        "\n",
        "\n",
        "## Contents\n",
        "\n",
        "This notebook contains the following parts:\n",
        "\n",
        "1.\t[Setup](#setup)\n",
        "2.\t[Optimizer definition](#definition)\n",
        "3.\t[Experiment Run](#run)\n",
        "4.\t[Pipelines comparison and testing](#comparison)\n",
        "5.\t[Historical runs](#runs)\n",
        "6.\t[Pipeline refinement and testing](#refinery)\n",
        "7.\t[Deploy and Score](#scoring)\n",
        "8.  [Clean up](#cleanup)\n",
        "8.\t[Summary and next steps](#summary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"setup\"></a>\n",
        "## 1. Set up the environment\n",
        "\n",
        "Before you use the sample code in this notebook, contact with your Cloud Pack for Data administrator and ask for your account credentials."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Connection to WML\n",
        "\n",
        "Authenticate the Watson Machine Learning service on IBM Cloud Pack for Data. You need to provide platform `url`, your `username` and `api_key`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "username = 'PASTE YOUR USERNAME HERE'\n",
        "api_key = 'PASTE YOUR API_KEY HERE'\n",
        "url = 'PASTE THE PLATFORM URL HERE'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {},
      "outputs": [],
      "source": [
        "wml_credentials = {\n",
        "    \"username\": username,\n",
        "    \"apikey\": api_key,\n",
        "    \"url\": url,\n",
        "    \"instance_id\": 'openshift',\n",
        "    \"version\": '4.6'\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Alternatively you can use `username` and `password` to authenticate WML services.\n",
        "\n",
        "```\n",
        "wml_credentials = {\n",
        "    \"username\": ***,\n",
        "    \"password\": ***,\n",
        "    \"url\": ***,\n",
        "    \"instance_id\": 'openshift',\n",
        "    \"version\": '4.6'\n",
        "}\n",
        "\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Install and import the `ibm-watson-machine-learning` package\n",
        "**Note:** `ibm-watson-machine-learning` documentation can be found <a href=\"http://ibm-wml-api-pyclient.mybluemix.net/\" target=\"_blank\" rel=\"noopener no referrer\">here</a>."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "!pip install ibm-watson-machine-learning | tail -n 1\n",
        "!pip install -U autoai-libs | tail -n 1\n",
        "!pip install -U lale | tail -n 1\n",
        "!pip install -U scikit-learn==1.0.2 | tail -n 1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "from ibm_watson_machine_learning import APIClient\n",
        "\n",
        "client = APIClient(wml_credentials)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Working with spaces\n",
        "\n",
        "First of all, you need to create a space that will be used for your work. If you do not have space already created, you can use `{PLATFORM_URL}/ml-runtime/spaces?context=icp4data` to create one.\n",
        "\n",
        "- Click New Deployment Space\n",
        "- Create an empty space\n",
        "- Go to space `Settings` tab\n",
        "- Copy `space_id` and paste it below\n",
        "\n",
        "**Tip**: You can also use SDK to prepare the space for your work. More information can be found [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd4.6/notebooks/python_sdk/instance-management/Space%20management.ipynb).\n",
        "\n",
        "**Action**: Assign space ID below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {},
      "outputs": [],
      "source": [
        "space_id = 'PASTE YOUR SPACE ID HERE'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can use the `list` method to print all existing spaces."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "client.spaces.list(limit=10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To be able to interact with all resources available in Watson Machine Learning, you need to set the **space** which you will be using."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'SUCCESS'"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "client.set.default_space(space_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"definition\"></a>\n",
        "## 2. Optimizer definition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Training data connection\n",
        "\n",
        "Define connection information to an external database.\n",
        "\n",
        "This example uses the German Credit Risk dataset. \n",
        "The dataset can be downloaded from [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/data/credit_risk/german_credit_data_biased_training.csv)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Connection configuration\n",
        "\n",
        "Credentials for database should be passed as a python dictionary.  \n",
        "For the vast number of supported datasets the credentials should follow the bellow pattern.  \n",
        "**Warning**: Database name should be slected from the list of all the supported databases, in order to look it up use `client.connections.list_datasource_types()`  \n",
        "**Warning**: Input table should be uploaded in database under the location `/schema_name/table_name`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "db_name = 'PUT YOUR DATABASE NAME HERE'\n",
        "schema_name = 'PUT YOUR SCHEMA NAME HERE'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "table_name = 'CREDIT_RISK'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "db_credentials = {\n",
        "    \"database\": \"***\",\n",
        "    \"password\": \"***\",\n",
        "    \"port\": \"***\",\n",
        "    \"host\": \"***\",\n",
        "    \"ssl\": \"***\",\n",
        "    \"username\": \"***\"\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Create connection"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "conn_meta_props= {\n",
        "    client.connections.ConfigurationMetaNames.NAME: f\"Connection to Database - {db_name} \",\n",
        "    client.connections.ConfigurationMetaNames.DATASOURCE_TYPE: client.connections.get_datasource_type_uid_by_name(db_name),\n",
        "    client.connections.ConfigurationMetaNames.DESCRIPTION: \"Connection to external Database\",\n",
        "    client.connections.ConfigurationMetaNames.PROPERTIES: db_credentials\n",
        "}\n",
        "\n",
        "conn_details = client.connections.create(meta_props=conn_meta_props)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "connection_id = client.connections.get_uid(conn_details)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Download training data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "!wget -q 'https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/cpd4.6/data/credit_risk/german_credit_data_biased_training.csv' -O \\\n",
        "            'credit_risk.csv'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Hint**: To install `pandas` execute `!pip install pandas`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "from ibm_watson_machine_learning.helpers import DataConnection, DatabaseLocation\n",
        "\n",
        "\n",
        "credit_risk_conn = DataConnection(\n",
        "    connection_asset_id=connection_id,\n",
        "    location=DatabaseLocation(\n",
        "        schema_name=schema_name,\n",
        "        table_name=table_name\n",
        "    )\n",
        ")\n",
        "\n",
        "credit_risk_conn._wml_client = client\n",
        "credit_risk_conn.write(pd.read_csv(\"credit_risk.csv\"))\n",
        "training_data_reference=[credit_risk_conn]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Optimizer configuration\n",
        "\n",
        "Provide the input information for AutoAI optimizer:\n",
        "- `name` - experiment name\n",
        "- `prediction_type` - type of the problem\n",
        "- `prediction_column` - target column name\n",
        "- `scoring` - optimization metric"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [],
      "source": [
        "from ibm_watson_machine_learning.experiment import AutoAI\n",
        "\n",
        "experiment = AutoAI(wml_credentials, space_id=space_id)\n",
        "\n",
        "pipeline_optimizer = experiment.optimizer(\n",
        "    name='Credit Risk Prediction - AutoAI',\n",
        "    desc='Sample notebook',\n",
        "    prediction_type=AutoAI.PredictionType.BINARY,\n",
        "    prediction_column='Risk',\n",
        "    scoring=AutoAI.Metrics.ROC_AUC_SCORE,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Configuration parameters can be retrieved via `get_params()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'name': 'Credit Risk Prediction - AutoAI',\n",
              " 'desc': 'Sample notebook',\n",
              " 'prediction_type': 'binary',\n",
              " 'prediction_column': 'Risk',\n",
              " 'scoring': 'roc_auc',\n",
              " 'test_size': 0.1,\n",
              " 'max_num_daub_ensembles': 2,\n",
              " 't_shirt_size': 'm',\n",
              " 'train_sample_rows_test_size': None,\n",
              " 'daub_include_only_estimators': ['ExtraTreesClassifierEstimator',\n",
              "  'GradientBoostingClassifierEstimator',\n",
              "  'LGBMClassifierEstimator',\n",
              "  'LogisticRegressionEstimator',\n",
              "  'RandomForestClassifierEstimator',\n",
              "  'XGBClassifierEstimator',\n",
              "  'DecisionTreeClassifierEstimator'],\n",
              " 'cognito_transform_names': None,\n",
              " 'data_join_graph': False,\n",
              " 'csv_separator': ',',\n",
              " 'excel_sheet': 0,\n",
              " 'encoding': 'utf-8',\n",
              " 'positive_label': None,\n",
              " 'run_id': None}"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipeline_optimizer.get_params()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"run\"></a>\n",
        "## 3. Experiment run\n",
        "\n",
        "Call the `fit()` method to trigger the AutoAI experiment. You can either use interactive mode (synchronous job) or background mode (asychronous job) by specifying `background_model=True`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training job 91e1541a-fee1-4bb9-8211-68eed6abfdc4 completed: 100%|████████| [10:01<00:00,  6.01s/it]\n"
          ]
        }
      ],
      "source": [
        "run_details = pipeline_optimizer.fit(\n",
        "            training_data_reference=training_data_reference,\n",
        "            background_mode=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can use the `get_run_status()` method to monitor AutoAI jobs in background mode."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'completed'"
            ]
          },
          "execution_count": 24,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipeline_optimizer.get_run_status()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"comparison\"></a>\n",
        "## 4. Pipelines comparison and testing\n",
        "\n",
        "You can list trained pipelines and evaluation metrics information in\n",
        "the form of a Pandas DataFrame by calling the `summary()` method. You can\n",
        "use the DataFrame to compare all discovered pipelines and select the one\n",
        "you like for further testing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Number of enhancements</th>\n",
              "      <th>Estimator</th>\n",
              "      <th>training_roc_auc_(optimized)</th>\n",
              "      <th>holdout_precision</th>\n",
              "      <th>training_average_precision</th>\n",
              "      <th>holdout_average_precision</th>\n",
              "      <th>training_neg_log_loss</th>\n",
              "      <th>holdout_recall</th>\n",
              "      <th>training_precision</th>\n",
              "      <th>holdout_accuracy</th>\n",
              "      <th>holdout_balanced_accuracy</th>\n",
              "      <th>training_recall</th>\n",
              "      <th>holdout_f1</th>\n",
              "      <th>holdout_neg_log_loss</th>\n",
              "      <th>training_accuracy</th>\n",
              "      <th>holdout_roc_auc</th>\n",
              "      <th>training_balanced_accuracy</th>\n",
              "      <th>training_f1</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline Name</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Pipeline_4</th>\n",
              "      <td>4</td>\n",
              "      <td>XGBClassifierEstimator</td>\n",
              "      <td>0.853971</td>\n",
              "      <td>0.702532</td>\n",
              "      <td>0.786750</td>\n",
              "      <td>0.762646</td>\n",
              "      <td>-0.432601</td>\n",
              "      <td>0.664671</td>\n",
              "      <td>0.686865</td>\n",
              "      <td>0.794</td>\n",
              "      <td>0.761765</td>\n",
              "      <td>0.649368</td>\n",
              "      <td>0.683077</td>\n",
              "      <td>-0.434525</td>\n",
              "      <td>0.783778</td>\n",
              "      <td>0.858985</td>\n",
              "      <td>0.750276</td>\n",
              "      <td>0.667448</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_7</th>\n",
              "      <td>3</td>\n",
              "      <td>GradientBoostingClassifierEstimator</td>\n",
              "      <td>0.853637</td>\n",
              "      <td>0.744186</td>\n",
              "      <td>0.781843</td>\n",
              "      <td>0.752330</td>\n",
              "      <td>-0.430732</td>\n",
              "      <td>0.574850</td>\n",
              "      <td>0.754036</td>\n",
              "      <td>0.792</td>\n",
              "      <td>0.737876</td>\n",
              "      <td>0.581504</td>\n",
              "      <td>0.648649</td>\n",
              "      <td>-0.435807</td>\n",
              "      <td>0.796667</td>\n",
              "      <td>0.859524</td>\n",
              "      <td>0.743037</td>\n",
              "      <td>0.656293</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_8</th>\n",
              "      <td>4</td>\n",
              "      <td>GradientBoostingClassifierEstimator</td>\n",
              "      <td>0.853637</td>\n",
              "      <td>0.744186</td>\n",
              "      <td>0.781843</td>\n",
              "      <td>0.752330</td>\n",
              "      <td>-0.430732</td>\n",
              "      <td>0.574850</td>\n",
              "      <td>0.754036</td>\n",
              "      <td>0.792</td>\n",
              "      <td>0.737876</td>\n",
              "      <td>0.581504</td>\n",
              "      <td>0.648649</td>\n",
              "      <td>-0.435807</td>\n",
              "      <td>0.796667</td>\n",
              "      <td>0.859524</td>\n",
              "      <td>0.743037</td>\n",
              "      <td>0.656293</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_2</th>\n",
              "      <td>2</td>\n",
              "      <td>XGBClassifierEstimator</td>\n",
              "      <td>0.853223</td>\n",
              "      <td>0.674847</td>\n",
              "      <td>0.783110</td>\n",
              "      <td>0.759399</td>\n",
              "      <td>-0.437251</td>\n",
              "      <td>0.658683</td>\n",
              "      <td>0.674910</td>\n",
              "      <td>0.780</td>\n",
              "      <td>0.749762</td>\n",
              "      <td>0.657352</td>\n",
              "      <td>0.666667</td>\n",
              "      <td>-0.439138</td>\n",
              "      <td>0.779556</td>\n",
              "      <td>0.856000</td>\n",
              "      <td>0.749096</td>\n",
              "      <td>0.665831</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_3</th>\n",
              "      <td>3</td>\n",
              "      <td>XGBClassifierEstimator</td>\n",
              "      <td>0.853223</td>\n",
              "      <td>0.674847</td>\n",
              "      <td>0.783110</td>\n",
              "      <td>0.759399</td>\n",
              "      <td>-0.437251</td>\n",
              "      <td>0.658683</td>\n",
              "      <td>0.674910</td>\n",
              "      <td>0.780</td>\n",
              "      <td>0.749762</td>\n",
              "      <td>0.657352</td>\n",
              "      <td>0.666667</td>\n",
              "      <td>-0.439138</td>\n",
              "      <td>0.779556</td>\n",
              "      <td>0.856000</td>\n",
              "      <td>0.749096</td>\n",
              "      <td>0.665831</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_6</th>\n",
              "      <td>2</td>\n",
              "      <td>GradientBoostingClassifierEstimator</td>\n",
              "      <td>0.850494</td>\n",
              "      <td>0.755906</td>\n",
              "      <td>0.778339</td>\n",
              "      <td>0.765539</td>\n",
              "      <td>-0.432066</td>\n",
              "      <td>0.574850</td>\n",
              "      <td>0.759112</td>\n",
              "      <td>0.796</td>\n",
              "      <td>0.740879</td>\n",
              "      <td>0.592149</td>\n",
              "      <td>0.653061</td>\n",
              "      <td>-0.430092</td>\n",
              "      <td>0.800667</td>\n",
              "      <td>0.856503</td>\n",
              "      <td>0.748694</td>\n",
              "      <td>0.665064</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_1</th>\n",
              "      <td>1</td>\n",
              "      <td>XGBClassifierEstimator</td>\n",
              "      <td>0.849883</td>\n",
              "      <td>0.692810</td>\n",
              "      <td>0.782921</td>\n",
              "      <td>0.760481</td>\n",
              "      <td>-0.437680</td>\n",
              "      <td>0.634731</td>\n",
              "      <td>0.692303</td>\n",
              "      <td>0.784</td>\n",
              "      <td>0.746795</td>\n",
              "      <td>0.638057</td>\n",
              "      <td>0.662500</td>\n",
              "      <td>-0.438008</td>\n",
              "      <td>0.783778</td>\n",
              "      <td>0.854903</td>\n",
              "      <td>0.747457</td>\n",
              "      <td>0.663639</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_5</th>\n",
              "      <td>1</td>\n",
              "      <td>GradientBoostingClassifierEstimator</td>\n",
              "      <td>0.848791</td>\n",
              "      <td>0.766129</td>\n",
              "      <td>0.780104</td>\n",
              "      <td>0.751335</td>\n",
              "      <td>-0.432413</td>\n",
              "      <td>0.568862</td>\n",
              "      <td>0.768005</td>\n",
              "      <td>0.798</td>\n",
              "      <td>0.740888</td>\n",
              "      <td>0.573520</td>\n",
              "      <td>0.652921</td>\n",
              "      <td>-0.442652</td>\n",
              "      <td>0.799333</td>\n",
              "      <td>0.848789</td>\n",
              "      <td>0.743049</td>\n",
              "      <td>0.656371</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               Number of enhancements                            Estimator  \\\n",
              "Pipeline Name                                                                \n",
              "Pipeline_4                          4               XGBClassifierEstimator   \n",
              "Pipeline_7                          3  GradientBoostingClassifierEstimator   \n",
              "Pipeline_8                          4  GradientBoostingClassifierEstimator   \n",
              "Pipeline_2                          2               XGBClassifierEstimator   \n",
              "Pipeline_3                          3               XGBClassifierEstimator   \n",
              "Pipeline_6                          2  GradientBoostingClassifierEstimator   \n",
              "Pipeline_1                          1               XGBClassifierEstimator   \n",
              "Pipeline_5                          1  GradientBoostingClassifierEstimator   \n",
              "\n",
              "               training_roc_auc_(optimized)  holdout_precision  \\\n",
              "Pipeline Name                                                    \n",
              "Pipeline_4                         0.853971           0.702532   \n",
              "Pipeline_7                         0.853637           0.744186   \n",
              "Pipeline_8                         0.853637           0.744186   \n",
              "Pipeline_2                         0.853223           0.674847   \n",
              "Pipeline_3                         0.853223           0.674847   \n",
              "Pipeline_6                         0.850494           0.755906   \n",
              "Pipeline_1                         0.849883           0.692810   \n",
              "Pipeline_5                         0.848791           0.766129   \n",
              "\n",
              "               training_average_precision  holdout_average_precision  \\\n",
              "Pipeline Name                                                          \n",
              "Pipeline_4                       0.786750                   0.762646   \n",
              "Pipeline_7                       0.781843                   0.752330   \n",
              "Pipeline_8                       0.781843                   0.752330   \n",
              "Pipeline_2                       0.783110                   0.759399   \n",
              "Pipeline_3                       0.783110                   0.759399   \n",
              "Pipeline_6                       0.778339                   0.765539   \n",
              "Pipeline_1                       0.782921                   0.760481   \n",
              "Pipeline_5                       0.780104                   0.751335   \n",
              "\n",
              "               training_neg_log_loss  holdout_recall  training_precision  \\\n",
              "Pipeline Name                                                              \n",
              "Pipeline_4                 -0.432601        0.664671            0.686865   \n",
              "Pipeline_7                 -0.430732        0.574850            0.754036   \n",
              "Pipeline_8                 -0.430732        0.574850            0.754036   \n",
              "Pipeline_2                 -0.437251        0.658683            0.674910   \n",
              "Pipeline_3                 -0.437251        0.658683            0.674910   \n",
              "Pipeline_6                 -0.432066        0.574850            0.759112   \n",
              "Pipeline_1                 -0.437680        0.634731            0.692303   \n",
              "Pipeline_5                 -0.432413        0.568862            0.768005   \n",
              "\n",
              "               holdout_accuracy  holdout_balanced_accuracy  training_recall  \\\n",
              "Pipeline Name                                                                 \n",
              "Pipeline_4                0.794                   0.761765         0.649368   \n",
              "Pipeline_7                0.792                   0.737876         0.581504   \n",
              "Pipeline_8                0.792                   0.737876         0.581504   \n",
              "Pipeline_2                0.780                   0.749762         0.657352   \n",
              "Pipeline_3                0.780                   0.749762         0.657352   \n",
              "Pipeline_6                0.796                   0.740879         0.592149   \n",
              "Pipeline_1                0.784                   0.746795         0.638057   \n",
              "Pipeline_5                0.798                   0.740888         0.573520   \n",
              "\n",
              "               holdout_f1  holdout_neg_log_loss  training_accuracy  \\\n",
              "Pipeline Name                                                        \n",
              "Pipeline_4       0.683077             -0.434525           0.783778   \n",
              "Pipeline_7       0.648649             -0.435807           0.796667   \n",
              "Pipeline_8       0.648649             -0.435807           0.796667   \n",
              "Pipeline_2       0.666667             -0.439138           0.779556   \n",
              "Pipeline_3       0.666667             -0.439138           0.779556   \n",
              "Pipeline_6       0.653061             -0.430092           0.800667   \n",
              "Pipeline_1       0.662500             -0.438008           0.783778   \n",
              "Pipeline_5       0.652921             -0.442652           0.799333   \n",
              "\n",
              "               holdout_roc_auc  training_balanced_accuracy  training_f1  \n",
              "Pipeline Name                                                            \n",
              "Pipeline_4            0.858985                    0.750276     0.667448  \n",
              "Pipeline_7            0.859524                    0.743037     0.656293  \n",
              "Pipeline_8            0.859524                    0.743037     0.656293  \n",
              "Pipeline_2            0.856000                    0.749096     0.665831  \n",
              "Pipeline_3            0.856000                    0.749096     0.665831  \n",
              "Pipeline_6            0.856503                    0.748694     0.665064  \n",
              "Pipeline_1            0.854903                    0.747457     0.663639  \n",
              "Pipeline_5            0.848789                    0.743049     0.656371  "
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "summary = pipeline_optimizer.summary()\n",
        "summary"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can visualize the scoring metric calculated on a holdout data set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxV9Z3/8dcnCWFNwhYCgcsmyCoJcFHrVlfcqtQqoG1nrNOW0QHHOtpqa3+ddpxp7eJ0bLVau4ydTquCiruotTq4VhMhhkVkh7CGLexLks/vj3uwMQ3cG0hyzk3ez8fDR+4959xz3/dK8r7ne+79XnN3REREjiYj7AAiIhJ9KgsREUlKZSEiIkmpLEREJCmVhYiIJKWyEBGRpFIqCzO7yMyWmNkyM7u9gfV5ZvaMmZWZ2UIzu67Ouq5m9piZfWhmi83sU8Hy7mb2spktDX52a7qHJSIiTSlpWZhZJnAfcDEwErjGzEbW22w6sMjdi4CzgbvNLDtYdw8wx92HA0XA4mD57cAr7j4UeCW4LiIiEZSVwjYnA8vcfQWAmT0CTAIW1dnGgRwzM6ALsA2oNrNc4CzgSwDufhA4GNxmEoliAfgd8Bpw29GC9OzZ0wcOHJhCZBEROay0tHSLu+cfzz5SKYu+wNo61yuAU+ptcy/wNLAeyAGmunutmQ0GKoH/NrMioBS4yd33AAXuvgHA3TeYWa9kQQYOHEhJSUkKkUVE5DAzW328+0jlnIU1sKz+HCEXAvOBQqAYuDc4qsgCxgH3u/tYYA+NHG4ys2lmVmJmJZWVlY25qYiINJFUyqICiNW53o/EEURd1wFPeMIyYCUwPLhthbv/JdjuMRLlAbDJzPoABD83N3Tn7v6gu8fdPZ6ff1xHUSIicoxSKYv3gKFmNig4aX01iSGnutYA5wGYWQEwDFjh7huBtWY2LNjuPP56ruNp4Nrg8rXAU8f8KEREpFklPWfh7tVmNgN4EcgEfuvuC83s+mD9A8CdwENmVk5i2Oo2d98S7OJG4A9B0awgcRQCcBcw08y+TKJsJjfh4xIRkSZk6TRFeTwed53gFhFpHDMrdff48exDn+AWEZGkVBYiIpJUKp+zkBZSW+ss3bybsoodHDhUE3aclJ3UryvFsa5hxxCRZqSyCNGeA9XMX7uD0tXbKVm9nXlrtrNrf3XYsRotOzODZ248g2G9c8KOIiLNRGXRgtbt2EfJqm28H5TD4g07qXUwgxN75XBZUSHj+3djbP+u5HZsF3bclOzeX82V97/FrbPKeOKfTqNdpkY2RVojlUUzOVRTy+INOylZtZ3SNdt5f/V2NlTtB6BTdibFsa7MOGcI4wZ0Y2z/buSlSTnU17NLe/79s6O54Q/v88v/W86Mc4eGHUlEmoHKoolU7T3E+2u2U7J6G6Wrt1O2top9wXmHvl07Eh/YnfiAbowf0I3hvXPIakWvwC8+qQ+fGdOHe15ZyvkjCxjeOzfsSCLSxFQWx8DdWbllD6Wrt3/839LNuwHIzDBGFeYydUKM+MBEOfTJ6xhy4ub3b5NG886Krdwys4wnp5+u4SiRVkZlkYL9h2ooX1eVOBG9ajvvr9nOtj2JmdZzO2QxfkA3JhUXMn5Ad4pieXTKbntPa/fO2fz7Z0/i+v8t5RevLuem8zUcJdKatL2/aimo3HWA0mA4qWT1dhasq+JQTeKT7oN6dubc4b0YP6Ab8QHdOCG/CxkZDU3M2/ZcNLo3k4oL+fmfl3L+yF6MKswLO5KINJE2XxY1tc7SzbsSRwxBOazZtheA7KwMivrl8Q9nDGJ8/8SQUo8u7UNOHG3fvWwUby7byq2zPuCp6aeTnaXhKJHWoM2Vxe4D1ZSt3fHxu5Tmrd7OrgOJzzb07NKe+IBu/N2pAxg/sBujCnNpn5UZcuL00q1zNt+/YjTTfl/Kfa8u4+YLTgw7kog0gTZRFm8s3cJLizZSWu+zDcMKcri8uDAYUupOrHtHEt8MK8dj4qjeXDG2L/e9uowLRhYwuq+Go0TSXZsoi9eWbOax0grG9u/KjHOHMn5A8MG3Dun52YZ08K+XjeTNZVu4dVYZT884Q8NRImmuTUxRvmv/ITq2y2xVn21IB68s3sSXf1fCjecO4ZaJw5LfQESahaYoT1FOh3YqihCcN6KAK8f14xevLae8oirsOCJyHPQXVJrVdy4bSc8u2dwyaz4HqtNnJl0R+SSVhTSrvI7tuOtzY/ho025+9srSsOOIyDFSWUizO2d4LyaP78f9ry2nbO2OsOOIyDFQWUiL+PZnRlKQ24FbZ5WxP42+2ElEElQW0iLyOrbjrivHsHTzbv7rTxqOEkk3KgtpMZ8+MZ+rJ8R4cO5y5q3ZHnYcEWkElYW0qDsuHUFvDUeJpB2VhbSonA7t+OFVY1heuYefvvxR2HFEJEUqC2lxZw7N5/On9OfB11dQulrDUSLpQGUhofjWJSMozOvI1zUcJZIWVBYSii7ts/jRVWNYsWUPP3lxSdhxRCQJlYWE5vQhPfniqf35zZsrKVm1Lew4InIUKZWFmV1kZkvMbJmZ3d7A+jwze8bMysxsoZldV2fdKjMrN7P5ZlZSZ3mxmb1zeLmZndw0D0nSyTcvHkHfrh25dVYZ+w5qOEokqpKWhZllAvcBFwMjgWvMbGS9zaYDi9y9CDgbuNvMsuusP8fdi+tNkfsj4HvuXgx8J7gubUzn9ln8+KoiVm3dy49e/DDsOCJyBKkcWZwMLHP3Fe5+EHgEmFRvGwdyLPE1c12AbUB1kv06kBtczgPWp5xaWpVPndCDaz81gIfeWsVfVmwNO46INCCVsugLrK1zvSJYVte9wAgSf/DLgZvcvTZY58BLZlZqZtPq3OZrwI/NbC3wE+CbDd25mU0LhqlKKisrU4gr6ei2i4cT69aJrz/2AXsPJnudISItLZWyaOhLqet/vd6FwHygECgG7jWzw0cNp7v7OBLDWNPN7Kxg+Q3Aze4eA24GftPQnbv7g+4ed/d4fn5+CnElHXXKzuLHV41hzba9/GiO3h0lEjWplEUFEKtzvR9/O2R0HfCEJywDVgLDAdx9ffBzMzCbxLAWwLXAE8HlWXWWSxt1yuAefOm0gTz01ireXq7hKJEoSaUs3gOGmtmg4KT11cDT9bZZA5wHYGYFwDBghZl1NrOcYHlnYCKwILjNeuDTweVzAU1FKnzjomEM7NGJbzxexp4DGo4SiYqkZeHu1cAM4EVgMTDT3Rea2fVmdn2w2Z3AaWZWDrwC3ObuW4AC4A0zKwPeBZ5z9znBbb5K4l1TZcD3gbrnM6SN6pSdxY8nF1GxfR93vaB3R4lEhbnXP/0QXfF43EtKSpJvKGnvzmcX8Zs3VvLHr5zCaUN6hh1HJK2ZWWm9jy40mj7BLZF068RhDOrZma8/9gG7NRwlEjqVhURSx+xMfjJ5DOur9vGD5xeHHUekzVNZSGSNH9Cdr5wxiD/8ZQ1vLN0SdhyRNk1lIZF2y8RhDM7vzG2Pf8Cu/YfCjiPSZqksJNI6tMvkJ5OL2FC1j+9rOEokNCoLibxx/bvx1bMG8/C7a5n7kaZ8EQmDykLSws3nn8iQXl247fEP2KnhKJEWp7KQtHB4OGrTzv38x7MajhJpaSoLSRvFsa7846dP4NGStby6ZHPYcUTaFJWFpJWvnT+UEwu68M3Hy6nap+EokZaispC00j4rMRxVufsAdz67KOw4Im2GykLSzph+Xbnh0yfwWGkFf/5wU9hxRNoElYWkpRvPG8Kwghxuf7ycqr0ajhJpbioLSUvtszK5e0oRW/cc5HvPLAw7jkirp7KQtDW6bx7Tzz6BJ+at4+VFGo4SaU4qC0lrM84dyvDeOXxrdjk79h4MO45Iq6WykLSWnZXB3VOK2L7nIN99WsNRIs1FZSFpb1RhHjPOHcKT89fz4sKNYccRaZVUFtIqTD9nCCP75HLH7HK27dFwlEhTU1lIq9AuM4OfTC6iat8h/lXDUSJNTmUhrcbIwlxuPHcoz5St54XyDWHHEWlVssIOINKUbjj7BF5atJFvP7mAkwd1p0eX9mFHarVqa52VW/dQtnYH89fuYNXWvXzhlP5cOKp32NGkGZi7h50hZfF43EtKSsKOIRG3ZOMuPvPz15k4qjf3fX5c2HFajS27D3xcDPPX7qBs7Q527q8GoHN2Jnkd27G+aj9XjO3Ldy8bRV6ndiEnlsPMrNTd48ezDx1ZSKszrHcOXzv/RH784hIuGb2BS8f0CTtS2tl3sIYF66soW7uDeUExVGzfB0CGwbDeuVw6ppDiWB7FsW4M6dWFWnfu/fMy7n11GW8t38IPrxzD2cN6hfxIpKnoyEJapeqaWj53/1tUbN/HSzefRU8NRx1RTa2zvHL3J44YPty4i5raxN+Gvl07UhzrSlFQDKP75tIp+8ivM8srqrhl1nw+2rSbqyfEuOPSEeR00FFGmJriyEJlIa3W0k27uPRnb3DeiF784gvjMLOwI0XCpp37Py6G+Wt2UL6uit0HEsNJOe2zKKpTDEWxPHrldGj0few/VMNP//QRD85dQWFeR348eQynndCzqR+KpEhlIZLE/a8t54dzPuTn14zlsqLCsOO0uD0HqilfV/VxMZRV7GBD1X4AsjKMEX1yg6OGrhTHujK4Z2cyMpquVEtXb+OWmWWs2rqXL502kNsuGk7H7Mwm27+kRmUhkkR1TS1XPvA2a7bu4aWbP01+TusdjqquqeWjTbspq/hrMXy0aRfBaBL9u3f6RDGMKsylQ7vm/8O972ANP5zzIQ+9tYpBPTvzk8ljGD+ge7Pfr/xVi5WFmV0E3ANkAr9297vqrc8D/hfoT+Kk+U/c/b+DdauAXUANUF03sJndCMwAqoHn3P0bR8uhspBjsWzzLi752RucMyyfB744vlUMR7k766v2f+LdSeUVVew7VANA107tKOqXKIaxQUF075wdaua3lm/h67M+YEPVPr561mBuPv/EFikraaGyMLNM4CPgAqACeA+4xt0X1dnmW0Ceu99mZvnAEqC3ux8MyiLu7lvq7fcc4A7gUnc/YGa93H3z0bKoLORY/fL/lvODFz7knquLmVTcN+w4jbZz/yHKK6r+eq5h7Q4qdx0AIDszg5GFieGkw/8N6NEpkqW4+0A1//HcIh5+dy1De3XhP6cUc1K/vLBjtXot9dbZk4Fl7r4iuNNHgElA3S9AdiDHEv86uwDbSBwtHM0NwF3ufgAgWVGIHI+vnDmYOQs38p2nFvLRpl0Y0ftD2pCNwcno5ZW7Ofy6bnDPzpw5pOfHw0nD++TQPis9XqF3aZ/FDz43homjenP74x/w2V+8yfRzhjDjnCFkZ2lCiShL5cjiKuAid/9KcP3vgFPcfUadbXKAp4HhQA4w1d2fC9atBLaTKJRfuvuDwfL5wFPARcB+4FZ3f6+B+58GTAPo37//+NWrVx/XA5a2a3nlbr7467+wOXhFng661RlOKo51pahf11bzYbeqvYf43jMLeWLeOkYV5nL3lCKG984NO1ar1FJHFg29BKvfMBcC84FzgROAl83sdXffCZzu7uvNrFew/EN3nxvcdzfgVGACMNPMBnu99grK5UFIDEM14rGJfMIJ+V14+5vnhR1DAnmd2vGfU4u5cHRv7phdzmU/f4ObLziRaWcOJitTRxlRk8r/kQogVud6P2B9vW2uA57whGXAShJHGbj7+uDnZmA2iWGtw/s9fJt3gVpAb8QWaWMuHNWbF792FuePKOBHc5Zw1QNvs7xyd9ixpJ5UyuI9YKiZDTKzbOBqEkNOda0BzgMwswJgGLDCzDoHQ1SYWWdgIrAguM2TJI5EMLMTgWxgCyLS5vTo0p5ffGEc91xdzMote7jkntf5zRsrqa3VYEJUJC0Ld68m8fbWF4HFwEx3X2hm15vZ9cFmdwKnmVk58ApwW/DupwLgDTMrA94l8fbYOcFtfgsMNrMFwCPAtfWHoESk7TAzJhX35eWbz+KMIT2589lFXPOrd1i7bW/Y0QR9KE9EIsjdmVVawZ3PLKLGnTsuHcHnT+4fybcDp4OmOMGts0giEjlmxpR4jDk3n8W4/t24Y/YC/v6377J+x76wo7VZKgsRiay+XTvy+y+fzJ2fHU3Jqu1c+F9zeay0gnQaEWktVBYiEmlmxt+dOoA5XzuTEb1zuXVWGV/9n1I279ofdrQ2RWUhImlhQI/OPDztVL596QjmLq3kwp/O5dkP6r+LX5qLykJE0kZmhvGVMwfz/D+fQf/unZjxx3lM/+P7bNtzMOxorZ7KQkTSzpBeOTx+w2l8/cJhvLRwIxN/OpeXF20KO1arprIQkbSUlZnB9HOG8NT0M8jPac9X/6eEW2aWUbXvUNjRWiWVhYiktZGFuTw1/XRuPHcIT85fx0X/NZe5H1WGHavVUVmISNrLzsrglonDeOKG0+jcPou//+27fGt2+cffLS7HT2UhIq1GUawrz954BtPOGszD767h4nvm8s6KrWHHahVUFiLSqnRol8m3LhnBzH/8FBlmXPOrd/i3ZxaxP/jKWTk2KgsRaZUmDOzOCzedyd+fOoDfvrmSS+55nffXbA87VtpSWYhIq9UpO4vvTRrNH75yCgeqa7nq/rf44ZwPOVCto4zGUlmISKt3+pCezPnamUweH+P+15Zz+c/fZPGGnWHHSisqCxFpE3I6tOOHV43ht1+Ks23vQab9vkRfrtQIKgsRaVPOHV7A//vMSNZu28dby/VOqVSpLESkzZk4soC8ju14tGRt2FHShspCRNqcDu0yuWJsX15cuJEdezUJYSpUFiLSJk2JxzhYXcuT89aFHSUtqCxEpE0aWZjLSX3zeLRE37yXCpWFiLRZUybEWLxhJwvX6220yagsRKTNuryokPZZGTz6nk50J6OyEJE2K69jOy4e3Zsn56/T3FFJqCxEpE2bMiHGrv3VzFmwMewokaayEJE27dRBPejfvZOGopJQWYhIm5aRYUyJ9+PtFVtZvXVP2HEiS2UhIm3eVeNjZBjMKqkIO0pkqSxEpM3rndeBT5+Yz2OlFdRocsEGpVQWZnaRmS0xs2VmdnsD6/PM7BkzKzOzhWZ2XZ11q8ys3Mzmm1lJA7e91czczHoe30MRETl2UyfE2LhzP3OXVoYdJZKSloWZZQL3ARcDI4FrzGxkvc2mA4vcvQg4G7jbzLLrrD/H3YvdPV5v3zHgAmDNsT8EEZHjd+7wAnp0zmamTnQ3KJUji5OBZe6+wt0PAo8Ak+pt40COmRnQBdgGVKew758C3whuLyISmuysDK4Y25c/Ld7E1t0Hwo4TOamURV+gbtVWBMvquhcYAawHyoGb3L02WOfAS2ZWambTDt/AzC4H1rl72dHu3MymmVmJmZVUVurwUESaz9QJMQ7VOLM1ueDfSKUsrIFl9Y8ELgTmA4VAMXCvmeUG605393EkhrGmm9lZZtYJuAP4TrI7d/cH3T3u7vH8/PwU4oqIHJuhBTmM7d+VR99bq8kF60mlLCqAWJ3r/UgcQdR1HfCEJywDVgLDAdx9ffBzMzCbxLDWCcAgoMzMVgX7fN/Meh/7QxEROX5T4zGWbt7NvLU7wo4SKamUxXvAUDMbFJy0vhp4ut42a4DzAMysABgGrDCzzmaWEyzvDEwEFrh7ubv3cveB7j6QRCGNc3d93l5EQvWZokI6ZWfqRHc9ScvC3auBGcCLwGJgprsvNLPrzez6YLM7gdPMrBx4BbjN3bcABcAbZlYGvAs85+5zmuOBiIg0hS7ts7j0pD48U7aevQdTeZ9O25CVykbu/jzwfL1lD9S5vJ7EUUP9260AilLY/8BUcoiItIQpE2LMKq3guQ82MDkeS36DNkCf4BYRqSc+oBuD8zszs0RDUYepLERE6jEzpsRjvLdqO8srd4cdJxJUFiIiDfjcuL5kZpiOLgIqCxGRBvTK6cC5w3vxeOk6DtXUJr9BK6eyEBE5gqnxGFt2H+DVDzeHHSV0KgsRkSM4e1g+vXLaaygKlYWIyBFlZWZw5fh+vLqkks0794cdJ1QqCxGRo5g8vh81tc7j77ftyQVVFiIiRzE4vwsnD+zOrJK2PbmgykJEJIkpE2Ks2LKH91ZtDztKaFQWIiJJXHJSb7q0z+LRNjy5oMpCRCSJTtlZXFZUyPPlG9i1/1DYcUKhshARScHUCTH2HarhmbINYUcJhcpCRCQFRf3yGFaQw6Nt9DMXKgsRkRSYGVMmxChbu4MlG3eFHafFqSxERFJ0xdi+tMtsm5MLqixERFLUvXM2F4wsYPa8dRysbluTC6osREQaYUo8xrY9B/nT4k1hR2lRKgsRkUY4c2g+hXkd2txnLlQWIiKNkJlhXDW+H3OXVrJ+x76w47QYlYWISCNNjsdwh8dKK8KO0mJUFiIijRTr3onTh/RgZslaamvbxuSCKgsRkWMwJR6jYvs+3lmxNewoLUJlISJyDC4c1ZvcDllt5hPdKgsRkWPQoV0mnx3blxcWbKRqb+ufXFBlISJyjKbEYxysruWpstb/LXoqCxGRYzS6bx6jCnPbxGcuVBYiIsdh6oQYC9fvZMG6qrCjNKuUysLMLjKzJWa2zMxub2B9npk9Y2ZlZrbQzK6rs26VmZWb2XwzK6mz/Mdm9qGZfWBms82sa9M8JBGRljOpqC/ZWRmtfnLBpGVhZpnAfcDFwEjgGjMbWW+z6cAidy8CzgbuNrPsOuvPcfdid4/XWfYyMNrdxwAfAd889ochIhKOvE7tuHh0b56ct479h2rCjtNsUjmyOBlY5u4r3P0g8Agwqd42DuSYmQFdgG1A9dF26u4vufvhbd4B+jUquYhIREyJx9i5v5oXF24MO0qzSaUs+gJ1j68qgmV13QuMANYD5cBN7n54/l4HXjKzUjObdoT7+AfghYZWmNk0Mysxs5LKysoU4oqItKxPDe5BrHvHVj0UlUpZWAPL6n++/UJgPlAIFAP3mllusO50dx9HYhhrupmd9Ymdm91B4ijkDw3dubs/6O5xd4/n5+enEFdEpGVlZBiTx8d4c9lW1m7bG3acZpFKWVQAsTrX+5E4gqjrOuAJT1gGrASGA7j7+uDnZmA2iWEtAMzsWuAzwBfcvW1MsCIirdJV4/thBrNa6dFFKmXxHjDUzAYFJ62vBp6ut80a4DwAMysAhgErzKyzmeUEyzsDE4EFwfWLgNuAy929dVaxiLQZhV07ctbQfGaVVlDTCicXTFoWwUnoGcCLwGJgprsvNLPrzez6YLM7gdPMrBx4BbjN3bcABcAbZlYGvAs85+5zgtvcC+QALwdvq32gSR+ZiEgLmzohxoaq/by+tPWdX81KZSN3fx54vt6yB+pcXk/iqKH+7VYARUfY55BGJRURibjzRxTQvXM2M0vWcvawXmHHaVL6BLeISBPJzsrgirF9eXnRJrbtORh2nCalshARaUJT4jEO1Tiz57WuyQVVFiIiTWhY7xyKYl2Z+d5aWtObPFUWIiJNbGo8xpJNuyiraD2TC6osRESa2GVFfejYLrNVTV2ushARaWI5HdpxyUl9eKZsPXsPHnWavLShshARaQZTJ8TYfaCa58tbx+SCKgsRkWYwYWA3BvXszMxWMhSlshARaQZmxuR4P95dtY0VlbvDjnPcVBYiIs3kqnH9yMwwZpVWhB3luKksRESaSa/cDpwzLJ/HSyuorqlNfoMIU1mIiDSjKfEYm3cd4LUl6T25oMpCRKQZnTO8Fz27tOfRNP+eC5WFiEgzapeZwZXj+/LnDzezedf+sOMcM5WFiEgzmxKPUVPrPPF++k4uqLIQEWlmJ+R3YcLAbmk9uaDKQkSkBUyOx1ixZQ+lq7eHHeWYqCxERFrApSf1oXN2+k4uqLIQEWkBndtncVlRIc+Vb2D3gfSbXFBlISLSQqZMiLH3YA3Plq0PO0qjqSxERFrI2FhXhvbqkpafuVBZiIi0EDNj6oQY89bsYOmmXWHHaRSVhYhIC7pibF+yMiztTnSrLEREWlCPLu05f0QBT8xbx8Hq9JlcUGUhItLCpk6IsW3PQf784aawo6RMZSEi0sLOOjGf3rkd0mooSmUhItLCMjOMq8b34/8+qmRjVXpMLphSWZjZRWa2xMyWmdntDazPM7NnzKzMzBaa2XV11q0ys3Izm29mJXWWdzezl81safCzW9M8JBGR6JsSj1Hr8FhpehxdJC0LM8sE7gMuBkYC15jZyHqbTQcWuXsRcDZwt5ll11l/jrsXu3u8zrLbgVfcfSjwSnBdRKRN6N+jE58a3IOZJRXU1kZ/csFUjixOBpa5+wp3Pwg8Akyqt40DOWZmQBdgG5Ds8+yTgN8Fl38HfDbl1CIircDUCTHWbNvLOyu3hh0lqVTKoi9Q9zipIlhW173ACGA9UA7c5O6H3xPmwEtmVmpm0+rcpsDdNwAEP3sdQ34RkbR10eje5HTIYmYanOhOpSysgWX1j5kuBOYDhUAxcK+Z5QbrTnf3cSSGsaab2VmNCWhm08ysxMxKKivT+ztsRUTq6tAuk0nFhbywYCNV+w6FHeeoUimLCiBW53o/EkcQdV0HPOEJy4CVwHAAd18f/NwMzCYxrAWwycz6AAQ/Nzd05+7+oLvH3T2en5+f2qMSEUkTU+P9OVBdy9MRn1wwlbJ4DxhqZoOCk9ZXA0/X22YNcB6AmRUAw4AVZtbZzHKC5Z2BicCC4DZPA9cGl68FnjqeByIiko5G981lRJ/cyA9FJS0Ld68GZgAvAouBme6+0MyuN7Prg83uBE4zs3IS72y6zd23AAXAG2ZWBrwLPOfuc4Lb3AVcYGZLgQuC6yIibYqZMTXej/J1VSxavzPsOEdk6fR9sPF43EtKSpJvKCKSRnbsPcjJ33+Fz5/cn+9ePqrJ929mpfU+utBo+gS3iEjIunbK5sJRvZk9bx37D9WEHadBKgsRkQiYGo9Rte8QLy2K5uSCKgsRkQg47YQe9O3aMbInulUWIiIRkJFhTI73441lW1i7bW/Ycf6GykJEJCImx2OYwWOlFWFH+RsqCxGRiOjbtSNnDOnJY6UV1ERsckGVhYhIhEydEGPdjn28uWxL2FE+QWUhIhIhF4wsoFundjxaEq0T3SoLEZEIaZ+VyWfH9uXlhZvYvudg2JY7T7wAAAh3SURBVHE+prIQEYmYqRNiHKypZfa8dWFH+ZjKQkQkYob3zmVMvzxmlqwlKlMyqSxERCJoSjzGhxt3Ub6uKuwogMpCRCSSLi8upEO7DB6NyCe6VRYiIhGU26Edl4zuw9Pz17PvYPiTC6osREQiasqEGLsOVPPCgg1hR1FZiIhE1SmDujOwR6dIDEWpLEREIsrMmByP8ZeV21i1ZU+oWVQWIiIRdtX4fozok0vl7gOh5sgK9d5FROSoCnI78MJNZ4YdQ0cWIiKSnMpCRESSUlmIiEhSKgsREUlKZSEiIkmpLEREJCmVhYiIJKWyEBGRpCwqX6yRCjOrBFYf4817AtH6BvSjS6e86ZQV0itvOmWF9MqbTlnh+PIOcPf847nztCqL42FmJe4eDztHqtIpbzplhfTKm05ZIb3yplNWCD+vhqFERCQplYWIiCTVlsriwbADNFI65U2nrJBeedMpK6RX3nTKCiHnbTPnLERE5Ni1pSMLERE5RioLERFJKjJlYWY1ZjbfzBaY2Swz6xQsf+s49vmamcWDy8+bWddmyPqWmbmZxaOYNdhfrZntC/7bZWY7guWRyxs8twvNbLeZbTezcjO7JIpZg/3Vmtn+4L9NZjYsWB65vMFzuzbIuiP4tzsyilnr5F0V5K0ys0eD5ZHLG2RdGvyO1ZrZ56OaNdjfl8ysMvg7Nt/MvpLsNlH6prx97l4MYGZ/AK4H/tPdT2uKnbv7JU2xn8A+dy82sxxgGcEHBSOaFWCvu3cBMLP3gNrgfqKYdx/wJnAvcAZQAfzC3Qc2xc6b4bk9APRw971m9i7wCDA2ws/tKHffGfyO7Se6v2OQyLcdGEvi38OS4H6imHcfcAGQCzwJTAT+GNGshz3q7jNS3TgyRxb1vA4MATCz3cHPs81srpnNNrNFZvaAmWUE6yaa2dtm9n5wVNKl/g6DVyg9zWygmS02s18Fr2BfMrOOwTYnmNkcMys1s9fNbHiSnHcCjwHt0iDrYd2BTRHP6yR+6Q7/O1gf4aw17r43uPwy0DvKz6277wwuvg7EAI9qVhK/V/e5+/Ygb9Sf21Xu/gGJ369IZz0m7h6J/4Ddwc8s4CnghnrLzybxSmMwkEniF/MqEh+Bnwt0Dra7DfhOcPk1IB5cXhVsOxCoBoqD5TOBLwaXXwGGBpdPAf58pKwkXu08EWT9CIhHMWu953BwkOufIv7c9gHKSbxa2wOMj2LWBv7drgCejvJzG/y8MXhetwFDo5g1WF8N/IjEkeY24Odp8NxmAWuAB6OaNVj/JWAD8AGJF7yxZH+jozQM1dHM5geXXwd+08A277r7CgAze5jEMMV+YCTwppkBZANvJ7mvle5++L5KgYFBm58GzAr2A9D+SFmDjGtI/A9qaCwxKlnhr89tLxKH8r+OcN6OQElw+dckfhl+H9Gs8Nfntluw3ecb2CYqeev+jv0WeBf4dkSzQuKP6jQSfyifAa4ws/8X0bx1n9s9wJ8b2CYqWSHxfD7s7gfM7Hrgd8C5R7vTKJXFx+csjqL+h0IcMOBld7+mEfd1oM7lGhJ/oDKAHSlkgMQr3v1AJ+AyEoecT/PJYb2oZIW/nmOZB0x394MNbBOVvPuAHcBF7r4WwMw6BFmilvVw3luBnwMT3H13A9tEJe8nfseCYZB7I5oVEq+gv+buDwV5XyFxJBTFvHXPuT4UZK8vKllx9611rv4K+GGy20T1nMWRnGxmg4J/5FOBN4B3gNPN7PA5jk5mdmJjd+yJsdyVZjY52I+ZWdFRtu/p7gM9ceL1HeByghPHUcsabDOMxKvfI71qiVLeNcB5wbYjgA588hctSlkzgF8Cl7v75iNsE5m8Zlb3j+2lwNKoZiXxB/ecYNuewIkkhvqimjeZyGQ1sz51rl4OLE52H+lWFm8DdwELgJXAbHevJDH+9rCZfUDiyT/WEztfAL5sZmXAQmBSK8p6DfCIBwOWEc97C/DVYNuHgwxRzZoNdCFx+D/fzJ5uYJso5Z1hiROj84F/Aa6NcNYaYKuZLQJeBb5e7xVxpPKa2QQzqwAmA780s4VRzQr8c/DvoAz4Z/72d+xvpM10H2Z2NnCru38m7CzJpFNWSK+86ZQV0itvOmWF9MqbTlmPJN2OLEREJARpc2QRFjO7g8RhZV2z3P0/wshzNOmUFdIrbzplhfTKm05ZIb3yNmVWlYWIiCSlYSgREUlKZSEiIkmpLCRtWZrMVGyJGT5rzWxMnWULzGzg8e5bpKWoLCSd7XP3YncfDRwkMVNxk85K6u47mmJfJGbPvaOJ9iXS4lQW0lpEfabiZ4FRFnzfRb37ud/MSoJ9f6/e/X8/yFliZuPM7EUzW26J+XwOb/d1M3vPzD6oe3uRpqSykLRnZlnAxSRmqq3vZBKfCD8JOAH4XDB1xLeB8919HImJC/8lyd0MJTFd9igSc1ddGSx/ELjR3ceTmCPqF0e4fS2JGVS/1cC6O9w9DowBPl13uApY6+6fIlGGD5GYqfRU4N+Cxz4xyHYyUAyMN7OzkjwWkUaL0kSCIo2VTjMVA/wRuMPMBtVbPsXMppH4fewTZPsgWHd4+pByoIu77wJ2WeLb47qS+JKdicC8YLsuJMpjbpLHI9IoKgtJZ+k0UzHuXm1md5P4zgIAguK4lcSMtdstMWNphwbut7ZehloSv78G/MDdf9mIxyLSaBqGktYuMjN9Bh4Czgfyg+u5JL7/oMrMCkgMpzXGi8A/HD7nYmZ9zaxXI/chkpTKQlq7KM30SfBdIj8j8UVUuHsZiSGkhSS+kOjNxty5u79EYnjrbTMrJ/GtZzmNfAwiSWm6D2m1WsNMnyJRoSMLERFJSkcWIiKSlI4sREQkKZWFiIgkpbIQEZGkVBYiIpKUykJERJJSWYiISFL/H6T5lnLflYNrAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "pd.options.plotting.backend = \"plotly\"\n",
        "\n",
        "summary.holdout_roc_auc.plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Get selected pipeline model\n",
        "\n",
        "Download and reconstruct a scikit-learn pipeline model object from the\n",
        "AutoAI training job."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "pycharm": {
          "is_executing": false
        },
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "best_pipeline = pipeline_optimizer.get_pipeline()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Check confusion matrix for selected pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>fn</th>\n",
              "      <th>fp</th>\n",
              "      <th>tn</th>\n",
              "      <th>tp</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>true_class</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>No Risk</th>\n",
              "      <td>47</td>\n",
              "      <td>56</td>\n",
              "      <td>111</td>\n",
              "      <td>286</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Risk</th>\n",
              "      <td>56</td>\n",
              "      <td>47</td>\n",
              "      <td>286</td>\n",
              "      <td>111</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            fn  fp   tn   tp\n",
              "true_class                  \n",
              "No Risk     47  56  111  286\n",
              "Risk        56  47  286  111"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipeline_optimizer.get_pipeline_details()['confusion_matrix']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Check features importance for selected pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>features_importance</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>NewFeature_14_round(sum(LoanDuration__Age))</th>\n",
              "      <td>1.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_1_sum(LoanDuration__Age)</th>\n",
              "      <td>0.89</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_6_pca_15</th>\n",
              "      <td>0.38</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ExistingCreditsCount</th>\n",
              "      <td>0.32</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>OwnsProperty</th>\n",
              "      <td>0.31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_3_pca_0</th>\n",
              "      <td>0.27</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>EmploymentDuration</th>\n",
              "      <td>0.25</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_11_round(LoanAmount)</th>\n",
              "      <td>0.23</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CurrentResidenceDuration</th>\n",
              "      <td>0.18</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>OthersOnLoan</th>\n",
              "      <td>0.18</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_5_pca_13</th>\n",
              "      <td>0.17</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Telephone</th>\n",
              "      <td>0.11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_8_pca_21</th>\n",
              "      <td>0.08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CheckingStatus</th>\n",
              "      <td>0.06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Housing</th>\n",
              "      <td>0.06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_19_round(pca_15)</th>\n",
              "      <td>0.05</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_7_pca_20</th>\n",
              "      <td>0.05</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CreditHistory</th>\n",
              "      <td>0.04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_0_sum(LoanDuration__LoanAmount)</th>\n",
              "      <td>0.04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>InstallmentPlans</th>\n",
              "      <td>0.04</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>LoanAmount</th>\n",
              "      <td>0.03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_2_sum(LoanAmount__Age)</th>\n",
              "      <td>0.03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Age</th>\n",
              "      <td>0.03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_15_round(sum(LoanAmount__Age))</th>\n",
              "      <td>0.03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_12_round(Age)</th>\n",
              "      <td>0.03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_9_pca_22</th>\n",
              "      <td>0.03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_18_round(pca_9)</th>\n",
              "      <td>0.02</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Job</th>\n",
              "      <td>0.02</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_4_pca_1</th>\n",
              "      <td>0.02</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Sex</th>\n",
              "      <td>0.02</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_10_round(LoanDuration)</th>\n",
              "      <td>0.02</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_13_round(sum(LoanDuration__LoanAmount))</th>\n",
              "      <td>0.02</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_17_round(pca_1)</th>\n",
              "      <td>0.01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Dependents</th>\n",
              "      <td>0.01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_16_round(pca_0)</th>\n",
              "      <td>0.01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>LoanDuration</th>\n",
              "      <td>0.01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ExistingSavings</th>\n",
              "      <td>0.01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>LoanPurpose</th>\n",
              "      <td>0.01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>InstallmentPercent</th>\n",
              "      <td>0.01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ForeignWorker</th>\n",
              "      <td>0.00</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                    features_importance\n",
              "NewFeature_14_round(sum(LoanDuration__Age))                        1.00\n",
              "NewFeature_1_sum(LoanDuration__Age)                                0.89\n",
              "NewFeature_6_pca_15                                                0.38\n",
              "ExistingCreditsCount                                               0.32\n",
              "OwnsProperty                                                       0.31\n",
              "NewFeature_3_pca_0                                                 0.27\n",
              "EmploymentDuration                                                 0.25\n",
              "NewFeature_11_round(LoanAmount)                                    0.23\n",
              "CurrentResidenceDuration                                           0.18\n",
              "OthersOnLoan                                                       0.18\n",
              "NewFeature_5_pca_13                                                0.17\n",
              "Telephone                                                          0.11\n",
              "NewFeature_8_pca_21                                                0.08\n",
              "CheckingStatus                                                     0.06\n",
              "Housing                                                            0.06\n",
              "NewFeature_19_round(pca_15)                                        0.05\n",
              "NewFeature_7_pca_20                                                0.05\n",
              "CreditHistory                                                      0.04\n",
              "NewFeature_0_sum(LoanDuration__LoanAmount)                         0.04\n",
              "InstallmentPlans                                                   0.04\n",
              "LoanAmount                                                         0.03\n",
              "NewFeature_2_sum(LoanAmount__Age)                                  0.03\n",
              "Age                                                                0.03\n",
              "NewFeature_15_round(sum(LoanAmount__Age))                          0.03\n",
              "NewFeature_12_round(Age)                                           0.03\n",
              "NewFeature_9_pca_22                                                0.03\n",
              "NewFeature_18_round(pca_9)                                         0.02\n",
              "Job                                                                0.02\n",
              "NewFeature_4_pca_1                                                 0.02\n",
              "Sex                                                                0.02\n",
              "NewFeature_10_round(LoanDuration)                                  0.02\n",
              "NewFeature_13_round(sum(LoanDuration__LoanAmount))                 0.02\n",
              "NewFeature_17_round(pca_1)                                         0.01\n",
              "Dependents                                                         0.01\n",
              "NewFeature_16_round(pca_0)                                         0.01\n",
              "LoanDuration                                                       0.01\n",
              "ExistingSavings                                                    0.01\n",
              "LoanPurpose                                                        0.01\n",
              "InstallmentPercent                                                 0.01\n",
              "ForeignWorker                                                      0.00"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipeline_optimizer.get_pipeline_details()['features_importance']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Convert the pipeline model to a Python script and download it"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<a download=\"pipeline_script.py\" href=\"data:binary;base64,ZnJvbSBhdXRvYWlfbGlicy50cmFuc2Zvcm1lcnMuZXhwb3J0YWJsZSBpbXBvcnQgTnVtcHlDb2x1bW5TZWxlY3Rvcgpmcm9tIGF1dG9haV9saWJzLnRyYW5zZm9ybWVycy5leHBvcnRhYmxlIGltcG9ydCBDb21wcmVzc1N0cmluZ3MKZnJvbSBhdXRvYWlfbGlicy50cmFuc2Zvcm1lcnMuZXhwb3J0YWJsZSBpbXBvcnQgTnVtcHlSZXBsYWNlTWlzc2luZ1ZhbHVlcwpmcm9tIGF1dG9haV9saWJzLnRyYW5zZm9ybWVycy5leHBvcnRhYmxlIGltcG9ydCBOdW1weVJlcGxhY2VVbmtub3duVmFsdWVzCmZyb20gYXV0b2FpX2xpYnMudHJhbnNmb3JtZXJzLmV4cG9ydGFibGUgaW1wb3J0IGJvb2xlYW4yZmxvYXQKZnJvbSBhdXRvYWlfbGlicy50cmFuc2Zvcm1lcnMuZXhwb3J0YWJsZSBpbXBvcnQgQ2F0SW1wdXRlcgpmcm9tIGF1dG9haV9saWJzLnRyYW5zZm9ybWVycy5leHBvcnRhYmxlIGltcG9ydCBDYXRFbmNvZGVyCmltcG9ydCBudW1weSBhcyBucApmcm9tIGF1dG9haV9saWJzLnRyYW5zZm9ybWVycy5leHBvcnRhYmxlIGltcG9ydCBmbG9hdDMyX3RyYW5zZm9ybQpmcm9tIGF1dG9haV9saWJzLnRyYW5zZm9ybWVycy5leHBvcnRhYmxlIGltcG9ydCBGbG9hdFN0cjJGbG9hdApmcm9tIGF1dG9haV9saWJzLnRyYW5zZm9ybWVycy5leHBvcnRhYmxlIGltcG9ydCBOdW1JbXB1dGVyCmZyb20gYXV0b2FpX2xpYnMudHJhbnNmb3JtZXJzLmV4cG9ydGFibGUgaW1wb3J0IE9wdFN0YW5kYXJkU2NhbGVyCmZyb20gbGFsZS5saWIubGFsZSBpbXBvcnQgQ29uY2F0RmVhdHVyZXMKZnJvbSBhdXRvYWlfbGlicy50cmFuc2Zvcm1lcnMuZXhwb3J0YWJsZSBpbXBvcnQgTnVtcHlQZXJtdXRlQXJyYXkKZnJvbSBhdXRvYWlfbGlicy5jb2duaXRvLnRyYW5zZm9ybXMudHJhbnNmb3JtX3V0aWxzIGltcG9ydCBUTm9PcApmcm9tIHhnYm9vc3QgaW1wb3J0IFhHQkNsYXNzaWZpZXIKaW1wb3J0IGxhbGUKCmxhbGUud3JhcF9pbXBvcnRlZF9vcGVyYXRvcnMoKQpudW1weV9jb2x1bW5fc2VsZWN0b3JfMCA9IE51bXB5Q29sdW1uU2VsZWN0b3IoCiAgICBjb2x1bW5zPVsKICAgICAgICAwLCAxLCAyLCAzLCA1LCA2LCA3LCA4LCA5LCAxMCwgMTEsIDEyLCAxMywgMTQsIDE1LCAxNiwgMTcsIDE4LCAxOSwKICAgIF0KKQpjb21wcmVzc19zdHJpbmdzID0gQ29tcHJlc3NTdHJpbmdzKAogICAgY29tcHJlc3NfdHlwZT0iaGFzaCIsCiAgICBkdHlwZXNfbGlzdD1bCiAgICAgICAgImNoYXJfc3RyIiwgImludF9udW0iLCAiY2hhcl9zdHIiLCAiY2hhcl9zdHIiLCAiY2hhcl9zdHIiLCAiY2hhcl9zdHIiLAogICAgICAgICJpbnRfbnVtIiwgImNoYXJfc3RyIiwgImNoYXJfc3RyIiwgImludF9udW0iLCAiY2hhcl9zdHIiLCAiaW50X251bSIsCiAgICAgICAgImNoYXJfc3RyIiwgImNoYXJfc3RyIiwgImludF9udW0iLCAiY2hhcl9zdHIiLCAiaW50X251bSIsICJjaGFyX3N0ciIsCiAgICAgICAgImNoYXJfc3RyIiwKICAgIF0sCiAgICBtaXNzaW5nX3ZhbHVlc19yZWZlcmVuY2VfbGlzdD1bIiIsICItIiwgIj8iLCBmbG9hdCgibmFuIildLAogICAgbWlzc2xpc3RfbGlzdD1bCiAgICAgICAgW10sIFtdLCBbXSwgW10sIFtdLCBbXSwgW10sIFtdLCBbXSwgW10sIFtdLCBbXSwgW10sIFtdLCBbXSwgW10sIFtdLAogICAgICAgIFtdLCBbXSwKICAgIF0sCikKbnVtcHlfcmVwbGFjZV9taXNzaW5nX3ZhbHVlc18wID0gTnVtcHlSZXBsYWNlTWlzc2luZ1ZhbHVlcygKICAgIG1pc3NpbmdfdmFsdWVzPVtdLCBmaWxsaW5nX3ZhbHVlcz1mbG9hdCgibmFuIikKKQpudW1weV9yZXBsYWNlX3Vua25vd25fdmFsdWVzID0gTnVtcHlSZXBsYWNlVW5rbm93blZhbHVlcygKICAgIGZpbGxpbmdfdmFsdWVzPWZsb2F0KCJuYW4iKSwKICAgIGZpbGxpbmdfdmFsdWVzX2xpc3Q9WwogICAgICAgIGZsb2F0KCJuYW4iKSwgZmxvYXQoIm5hbiIpLCBmbG9hdCgibmFuIiksIGZsb2F0KCJuYW4iKSwgZmxvYXQoIm5hbiIpLAogICAgICAgIGZsb2F0KCJuYW4iKSwgZmxvYXQoIm5hbiIpLCBmbG9hdCgibmFuIiksIGZsb2F0KCJuYW4iKSwgZmxvYXQoIm5hbiIpLAogICAgICAgIGZsb2F0KCJuYW4iKSwgZmxvYXQoIm5hbiIpLCBmbG9hdCgibmFuIiksIGZsb2F0KCJuYW4iKSwgZmxvYXQoIm5hbiIpLAogICAgICAgIGZsb2F0KCJuYW4iKSwgZmxvYXQoIm5hbiIpLCBmbG9hdCgibmFuIiksIGZsb2F0KCJuYW4iKSwKICAgIF0sCiAgICBrbm93bl92YWx1ZXNfbGlzdD1bCiAgICAgICAgWwogICAgICAgICAgICAyMjcyNTkyNjQ2ODg3NTM2NDY4MTAwNzczNzU3OTA5MDgyODY1MDgsCiAgICAgICAgICAgIDI1MzczMjIxNDkxMDgxNTIzODEzNDUwOTI4ODExMTQwMjQ4NjcyMiwKICAgICAgICAgICAgMzAzODE5MTQ0MzQ1MDk4NjI2NTU0NDU2MDExNDk2MjE3MjIzNTc1LAogICAgICAgICAgICAyODAzNTM2MDY4NzI5MzkzODg2MTQzMTU5MDExODYwOTQzMjY5NDksCiAgICAgICAgXSwKICAgICAgICBbCiAgICAgICAgICAgIDQsIDUsIDYsIDcsIDgsIDksIDEwLCAxMSwgMTIsIDEzLCAxNCwgMTUsIDE2LCAxNywgMTgsIDE5LCAyMCwgMjEsCiAgICAgICAgICAgIDIyLCAyMywgMjQsIDI1LCAyNiwgMjcsIDI4LCAyOSwgMzAsIDMxLCAzMiwgMzMsIDM0LCAzNSwgMzYsIDM3LAogICAgICAgICAgICAzOCwgMzksIDQwLCA0MSwgNDIsIDQzLCA0NCwgNDUsIDQ2LCA0NywgNDgsIDQ5LCA1MCwgNTEsIDUyLCA1MywKICAgICAgICAgICAgNTQsIDU2LCA1NywgNTksIDYwLCA2MSwgNjIsIDYzLCA2NCwKICAgICAgICBdLAogICAgICAgIFsKICAgICAgICAgICAgMzEwODYxNDM0MjkyNzI0MjY2ODI4NTEyNzQyMTcwMTA2ODc5MjM1LAogICAgICAgICAgICAyNTkwNzg1NDY3MjgzMjMwMDY4MjM2MjExNDkzODIxMjMyNzQ4NTEsCiAgICAgICAgICAgIDI0MTMwNDE5MTUxNTM1NTYwMDE0MTM2OTY3MjkwNTQ4MTQ2MjIyOCwKICAgICAgICAgICAgMjEwNDIxMzYzOTYzOTM3MjY0MTM0NTk2OTY0NTc2NTI1OTIyOTU4LAogICAgICAgICAgICAzMjIxMjkzOTM0MTA3NDUzMjA2Njg0NTczMTgxNjk3MDAwMzY5MywKICAgICAgICBdLAogICAgICAgIFsKICAgICAgICAgICAgMzM4NDczNjI5ODQzNTgxMTY1NzIwNTQ0NDMyMjgxMTI1MjM5NjA5LAogICAgICAgICAgICAzMjY3ODE3OTMyMDYyNTg0NDI2NzM2OTI5Njk2MjM4NDM0MzUxMTMsCiAgICAgICAgICAgIDI3MjAwMjUzMDM5MTg5MzA4NDY4NDQyOTMyNDYxNjM1MDAwNzk3NSwKICAgICAgICAgICAgMjQ5NjYxNTc4MDk4ODUyNTY5MDMwODYzMjY4MzM2NDc1MTA0MjU5LAogICAgICAgICAgICAyNzc0NTI5OTIzMTEwNjEyMjM0OTg1NDgwMDQwNjE3NDA5NTQ5MjMsCiAgICAgICAgICAgIDMzNzAwMDYyNDEzMzIwNjc4OTgyNTExNTE4MjgwOTk5MTg0NDIwNSwKICAgICAgICAgICAgMTYxMzMwODcwMDI1NDAxNzUzMzQxMjM1NzM4MTM3OTg0ODAyNDc5LAogICAgICAgICAgICAxMzM0MDE3MjczNjc3NDIwNDI2ODEzMTgwODM1Nzc1NTE4MjEyMDMsCiAgICAgICAgICAgIDI2MDQ5MTExMjg3MzIwMjM5NDUxOTk0NTUyMzYzNjQ5NDg3NDY0NiwKICAgICAgICAgICAgMjA4MTMxNTkzNzQ3MTc0NDQ3MjIzMTU0NjI2OTQ5MDE1Njk1ODM0LAogICAgICAgICAgICAzMTcwOTMxOTA3NzIxNDExNzk0MTI1Mzk2MTY2NzkyOTY4NTg1OTcsCiAgICAgICAgXSwKICAgICAgICBbCiAgICAgICAgICAgIDE1NTQ2NjExNDUzOTQxMzk5MTg1MTU4MjQxNzEyNjM2NDg5NTMzOSwKICAgICAgICAgICAgMjQ2Nzg5MzUyNDAzMTA5MzI5OTMwNjQ4MTc2MzI5MTI1OTQ2OTgxLAogICAgICAgICAgICAyOTU5MzY2MjE0NTExNjk2ODk2OTkyMTg0NjkyNDgyNTQyNzUzNjEsCiAgICAgICAgICAgIDIyMzY4MTQ3NjM2MTY1MjQ1NTgwODE1MDMxNzQwODE0NTY2NjEyMiwKICAgICAgICAgICAgMjMwNzE1MTE0NDMwMzIxNzI0ODUwOTM0NzEzNzgzMTk3OTMyMTA2LAogICAgICAgIF0sCiAgICAgICAgWwogICAgICAgICAgICAxODUzNDA1NTg5MjgxMzg5MDk2MjY2MDM4MTI3NjM0ODgxOTA1ODgsCiAgICAgICAgICAgIDMxOTMyODY2MTM3ODA0NjU4MzcxNTE3NjQ3NjYyNzQ5MjM4NTc3OCwKICAgICAgICAgICAgMjYzNzk5NzQ5MDg2NjkyNTc2MDgzMTQ4ODU1NzA1MzU0MDk5ODcyLAogICAgICAgICAgICAxNzkyMjI1MDI3NzAzODMxMzg4NzYzMDI3NTU4MzY0MjQ4OTM1ODYsCiAgICAgICAgICAgIDI4MzM2NDMxMjI3MTY2MDk5NjQwMDg4Mzc2MzQ5MTk0OTQxOTg2MSwKICAgICAgICBdLAogICAgICAgIFsxLCAyLCAzLCA0LCA1LCA2XSwKICAgICAgICBbCiAgICAgICAgICAgIDUyMTQ5Mzc5MDAxMjY0OTMyMDY4NzU3NDg3MDU5MTc3MzUxNDA1LAogICAgICAgICAgICAxMDM4MTAxNTA4OTE0Nzc1MzAzMzU4MzM4NjU3MDk4NTkzOTYyOSwKICAgICAgICBdLAogICAgICAgIFsKICAgICAgICAgICAgOTAzODA1MTMxNTk4Mzk0MjQyMDU2NTcxNDE0NjY2MDMzMDk3ODAsCiAgICAgICAgICAgIDcyMDIwMTQ0MzYwMzE4OTA2Nzg4MjcwOTc0NDI1Mzc1NjI4NDk0LAogICAgICAgICAgICA2ODE4Njc0OTI4NjY2MzExMzcwNDQ3MjIxMDI0Njg0NDU0MDY2NCwKICAgICAgICBdLAogICAgICAgIFsxLCAyLCAzLCA0LCA1LCA2XSwKICAgICAgICBbCiAgICAgICAgICAgIDE4NDIxMDE4Mzc5NzU4MDczNTE5NzQ0MjU0MTc2MjUwODA4ODcyMywKICAgICAgICAgICAgMTEyNzg0NjQ1MTU5NzUyNDg2MzIzMDg2MDk1ODk3NTg2NTc5MTAyLAogICAgICAgICAgICAxOTQwNjQ2MzEwMjQ2MTY0ODUwMDUwMjIyMTM3NTAyMjQ2NzkxNzEsCiAgICAgICAgICAgIDIzMDcxNTExNDQzMDMyMTcyNDg1MDkzNDcxMzc4MzE5NzkzMjEwNiwKICAgICAgICBdLAogICAgICAgIFsKICAgICAgICAgICAgMTksIDIwLCAyMSwgMjIsIDIzLCAyNCwgMjUsIDI2LCAyNywgMjgsIDI5LCAzMCwgMzEsIDMyLCAzMywgMzQsCiAgICAgICAgICAgIDM1LCAzNiwgMzcsIDM4LCAzOSwgNDAsIDQxLCA0MiwgNDMsIDQ0LCA0NSwgNDYsIDQ3LCA0OCwgNDksIDUwLAogICAgICAgICAgICA1MSwgNTIsIDUzLCA1NCwgNTUsIDU2LCA1NywgNTgsIDU5LCA2MCwgNjEsIDYyLCA2MywgNjQsIDY1LCA2NiwKICAgICAgICAgICAgNjcsIDcwLCA3MSwgNzMsIDc0LAogICAgICAgIF0sCiAgICAgICAgWwogICAgICAgICAgICAyNTE2OTYzMDU0NzY2MDYyNjEyMTk5OTg4NDkyOTk0ODQ4MzE2MjYsCiAgICAgICAgICAgIDY4MTg2NzQ5Mjg2NjYzMTEzNzA0NDcyMjEwMjQ2ODQ0NTQwNjY0LAogICAgICAgICAgICAxMjk4NDM5NjkzNTg5NTM4NTcwNDk5OTc3Mzc5MjAzMDc4NjQ1NzMsCiAgICAgICAgXSwKICAgICAgICBbCiAgICAgICAgICAgIDIyNjIwNDY0OTk5MzI0ODcwNDIzNTc0NzE5NjczMTYzODE2MTgxMiwKICAgICAgICAgICAgMjQwNzAzODc5OTk3NDk2ODQ0Njk5NDU1MDA5ODgwNDk4MjA1NjY1LAogICAgICAgICAgICAyNDMxODQ4ODg2MDA2NjUyMjE4MzY4OTU2OTc5NTQ3MDEzNTcxNzcsCiAgICAgICAgXSwKICAgICAgICBbMSwgMiwgMywgNF0sCiAgICAgICAgWwogICAgICAgICAgICAzMjgyODY1MjcyOTU2NjM1ODI2NjMzNjU1MDMzMTk5MDI2MzI2NzYsCiAgICAgICAgICAgIDExOTY0MTcwNzYwNzkzOTAzODkxNDQ2NTAwMDg2NDI5MDI4ODg4MCwKICAgICAgICAgICAgMjgzMzY0MzEyMjcxNjYwOTk2NDAwODgzNzYzNDkxOTQ5NDE5ODYxLAogICAgICAgICAgICAyNzc0MTAxOTUwODk3NzA1NTgwNzQyMzk5MTc1MzQ2ODgxOTUyOCwKICAgICAgICBdLAogICAgICAgIFsxLCAyXSwKICAgICAgICBbCiAgICAgICAgICAgIDY4MTg2NzQ5Mjg2NjYzMTEzNzA0NDcyMjEwMjQ2ODQ0NTQwNjY0LAogICAgICAgICAgICAyMjA3MzY3OTA4NTQwNTA3NTA0MDA5Njg1NjE5MjIwNzYwNTk1NTAsCiAgICAgICAgXSwKICAgICAgICBbCiAgICAgICAgICAgIDE2OTY2MjAxOTc1NDg1OTY3NDkwNzM3MDMwNzMyNDQ3NjYwNjkxOSwKICAgICAgICAgICAgMjIwNzM2NzkwODU0MDUwNzUwNDAwOTY4NTYxOTIyMDc2MDU5NTUwLAogICAgICAgIF0sCiAgICBdLAogICAgbWlzc2luZ192YWx1ZXNfcmVmZXJlbmNlX2xpc3Q9WyIiLCAiLSIsICI/IiwgZmxvYXQoIm5hbiIpXSwKKQpjYXRfaW1wdXRlciA9IENhdEltcHV0ZXIoCiAgICBzdHJhdGVneT0ibW9zdF9mcmVxdWVudCIsCiAgICBtaXNzaW5nX3ZhbHVlcz1mbG9hdCgibmFuIiksCiAgICBza2xlYXJuX3ZlcnNpb25fZmFtaWx5PSIyMyIsCikKY2F0X2VuY29kZXIgPSBDYXRFbmNvZGVyKAogICAgZW5jb2Rpbmc9Im9yZGluYWwiLAogICAgY2F0ZWdvcmllcz0iYXV0byIsCiAgICBkdHlwZT1ucC5mbG9hdDY0LAogICAgaGFuZGxlX3Vua25vd249ImVycm9yIiwKICAgIHNrbGVhcm5fdmVyc2lvbl9mYW1pbHk9IjIzIiwKKQpudW1weV9jb2x1bW5fc2VsZWN0b3JfMSA9IE51bXB5Q29sdW1uU2VsZWN0b3IoY29sdW1ucz1bNF0pCmZsb2F0X3N0cjJfZmxvYXQgPSBGbG9hdFN0cjJGbG9hdCgKICAgIGR0eXBlc19saXN0PVsiaW50X251bSJdLCBtaXNzaW5nX3ZhbHVlc19yZWZlcmVuY2VfbGlzdD1bXQopCm51bXB5X3JlcGxhY2VfbWlzc2luZ192YWx1ZXNfMSA9IE51bXB5UmVwbGFjZU1pc3NpbmdWYWx1ZXMoCiAgICBtaXNzaW5nX3ZhbHVlcz1bXSwgZmlsbGluZ192YWx1ZXM9ZmxvYXQoIm5hbiIpCikKbnVtX2ltcHV0ZXIgPSBOdW1JbXB1dGVyKHN0cmF0ZWd5PSJtZWRpYW4iLCBtaXNzaW5nX3ZhbHVlcz1mbG9hdCgibmFuIikpCm9wdF9zdGFuZGFyZF9zY2FsZXIgPSBPcHRTdGFuZGFyZFNjYWxlcigKICAgIG51bV9zY2FsZXJfY29weT1Ob25lLAogICAgbnVtX3NjYWxlcl93aXRoX21lYW49Tm9uZSwKICAgIG51bV9zY2FsZXJfd2l0aF9zdGQ9Tm9uZSwKICAgIHVzZV9zY2FsZXJfZmxhZz1GYWxzZSwKKQpudW1weV9wZXJtdXRlX2FycmF5ID0gTnVtcHlQZXJtdXRlQXJyYXkoCiAgICBheGlzPTAsCiAgICBwZXJtdXRhdGlvbl9pbmRpY2VzPVsKICAgICAgICAwLCAxLCAyLCAzLCA1LCA2LCA3LCA4LCA5LCAxMCwgMTEsIDEyLCAxMywgMTQsIDE1LCAxNiwgMTcsIDE4LCAxOSwgNCwKICAgIF0sCikKdF9ub19vcCA9IFROb09wKAogICAgZnVuPU5vbmUsIG5hbWU9Im5vX2FjdGlvbiIsIGRhdGF0eXBlcz0ieCIsIGZlYXRfY29uc3RyYWludHM9W10KKQp4Z2JfY2xhc3NpZmllciA9IFhHQkNsYXNzaWZpZXIoCiAgICBnYW1tYT0wLjQ0NTc5ODAwMDkxMzgxOTY2LAogICAgbGVhcm5pbmdfcmF0ZT0wLjAyMTMzMDgwNjU1ODM3OTk4MiwKICAgIG1pbl9jaGlsZF93ZWlnaHQ9NSwKICAgIG1pc3Npbmc9ZmxvYXQoIm5hbiIpLAogICAgbl9lc3RpbWF0b3JzPTU3NCwKICAgIG5fam9icz0yLAogICAgcmFuZG9tX3N0YXRlPTMzLAogICAgcmVnX2FscGhhPTAuNzI0ODIyNTM4OTc4NjQ3LAogICAgcmVnX2xhbWJkYT0wLjI1MDY3NDIxNDEwMTQ5NTEsCiAgICBzaWxlbnQ9VHJ1ZSwKICAgIHN1YnNhbXBsZT0wLjQxMTQ3NTI3NzE0ODcxMTk2LAogICAgdmVyYm9zaXR5PTAsCiAgICB0cmVlX21ldGhvZD0iaGlzdCIsCikKcGlwZWxpbmUgPSAoCiAgICAoCiAgICAgICAgKAogICAgICAgICAgICBudW1weV9jb2x1bW5fc2VsZWN0b3JfMAogICAgICAgICAgICA+PiBjb21wcmVzc19zdHJpbmdzCiAgICAgICAgICAgID4+IG51bXB5X3JlcGxhY2VfbWlzc2luZ192YWx1ZXNfMAogICAgICAgICAgICA+PiBudW1weV9yZXBsYWNlX3Vua25vd25fdmFsdWVzCiAgICAgICAgICAgID4+IGJvb2xlYW4yZmxvYXQoKQogICAgICAgICAgICA+PiBjYXRfaW1wdXRlcgogICAgICAgICAgICA+PiBjYXRfZW5jb2RlcgogICAgICAgICAgICA+PiBmbG9hdDMyX3RyYW5zZm9ybSgpCiAgICAgICAgKQogICAgICAgICYgKAogICAgICAgICAgICBudW1weV9jb2x1bW5fc2VsZWN0b3JfMQogICAgICAgICAgICA+PiBmbG9hdF9zdHIyX2Zsb2F0CiAgICAgICAgICAgID4+IG51bXB5X3JlcGxhY2VfbWlzc2luZ192YWx1ZXNfMQogICAgICAgICAgICA+PiBudW1faW1wdXRlcgogICAgICAgICAgICA+PiBvcHRfc3RhbmRhcmRfc2NhbGVyCiAgICAgICAgICAgID4+IGZsb2F0MzJfdHJhbnNmb3JtKCkKICAgICAgICApCiAgICApCiAgICA+PiBDb25jYXRGZWF0dXJlcygpCiAgICA+PiBudW1weV9wZXJtdXRlX2FycmF5CiAgICA+PiB0X25vX29wCiAgICA+PiB4Z2JfY2xhc3NpZmllcgop\" target=\"_blank\">Download file.</a>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from ibm_watson_machine_learning.helpers import pipeline_to_script\n",
        "pipeline_to_script(best_pipeline)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Visualize pipeline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": [
              "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
              "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
              " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
              "<!-- Generated by graphviz version 2.44.1 (20200629.0846)\n",
              " -->\n",
              "<!-- Title: cluster:(root) Pages: 1 -->\n",
              "<svg width=\"1241pt\" height=\"149pt\"\n",
              " viewBox=\"0.00 0.00 1241.02 148.54\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
              "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 144.54)\">\n",
              "<title>cluster:(root)</title>\n",
              "<g id=\"a_graph0\"><a xlink:title=\"(root) = ...\">\n",
              "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-144.54 1237.02,-144.54 1237.02,4 -4,4\"/>\n",
              "</a>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0 -->\n",
              "<g id=\"node1\" class=\"node\">\n",
              "<title>numpy_column_selector_0</title>\n",
              "<g id=\"a_node1\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_0 = NumpyColumnSelector(columns=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"33.23\" cy=\"-103.77\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-112.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-100.97\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-88.97\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings -->\n",
              "<g id=\"node2\" class=\"node\">\n",
              "<title>compress_strings</title>\n",
              "<g id=\"a_node2\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.compress_strings.html\" xlink:title=\"compress_strings = CompressStrings(compress_type=&#39;hash&#39;, dtypes_list=[&#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_s...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"142.77\" cy=\"-103.77\" rx=\"40.11\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Compress&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Strings</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0&#45;&gt;compress_strings -->\n",
              "<g id=\"edge1\" class=\"edge\">\n",
              "<title>numpy_column_selector_0&#45;&gt;compress_strings</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M66.5,-103.77C74.59,-103.77 83.46,-103.77 92.19,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"92.3,-107.27 102.3,-103.77 92.3,-100.27 92.3,-107.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0 -->\n",
              "<g id=\"node3\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_0</title>\n",
              "<g id=\"a_node3\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_0 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-103.77\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-118.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-82.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings&#45;&gt;numpy_replace_missing_values_0 -->\n",
              "<g id=\"edge2\" class=\"edge\">\n",
              "<title>compress_strings&#45;&gt;numpy_replace_missing_values_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M183.11,-103.77C191.43,-103.77 200.26,-103.77 208.74,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"208.85,-107.27 218.85,-103.77 208.85,-100.27 208.85,-107.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values -->\n",
              "<g id=\"node4\" class=\"node\">\n",
              "<title>numpy_replace_unknown_values</title>\n",
              "<g id=\"a_node4\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_unknown_values.html\" xlink:title=\"numpy_replace_unknown_values = NumpyReplaceUnknownValues(filling_values=float(&#39;nan&#39;), filling_values_list=[float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-103.77\" rx=\"38.78\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-118.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Unknown&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-82.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values -->\n",
              "<g id=\"edge3\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M288.46,-103.77C296.6,-103.77 305.47,-103.77 314.13,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"314.16,-107.27 324.16,-103.77 314.16,-100.27 314.16,-107.27\"/>\n",
              "</g>\n",
              "<!-- boolean2float -->\n",
              "<g id=\"node5\" class=\"node\">\n",
              "<title>boolean2float</title>\n",
              "<g id=\"a_node5\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.boolean2float.html\" xlink:title=\"boolean2float = boolean2float()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-100.77\" rx=\"37.91\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-97.97\" font-family=\"Times,serif\" font-size=\"11.00\">boolean2float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values&#45;&gt;boolean2float -->\n",
              "<g id=\"edge4\" class=\"edge\">\n",
              "<title>numpy_replace_unknown_values&#45;&gt;boolean2float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M402.41,-102.74C410.6,-102.52 419.36,-102.28 427.85,-102.05\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"428.12,-105.54 438.02,-101.77 427.93,-98.55 428.12,-105.54\"/>\n",
              "</g>\n",
              "<!-- cat_imputer -->\n",
              "<g id=\"node6\" class=\"node\">\n",
              "<title>cat_imputer</title>\n",
              "<g id=\"a_node6\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_imputer.html\" xlink:title=\"cat_imputer = CatImputer(strategy=&#39;most_frequent&#39;, missing_values=float(&#39;nan&#39;), sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-98.77\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-101.97\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-89.97\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- boolean2float&#45;&gt;cat_imputer -->\n",
              "<g id=\"edge5\" class=\"edge\">\n",
              "<title>boolean2float&#45;&gt;cat_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M514.07,-100.05C522.45,-99.88 531.39,-99.71 539.88,-99.54\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"540.02,-103.04 549.95,-99.35 539.88,-96.04 540.02,-103.04\"/>\n",
              "</g>\n",
              "<!-- cat_encoder -->\n",
              "<g id=\"node7\" class=\"node\">\n",
              "<title>cat_encoder</title>\n",
              "<g id=\"a_node7\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_encoder.html\" xlink:title=\"cat_encoder = CatEncoder(encoding=&#39;ordinal&#39;, categories=&#39;auto&#39;, dtype=np.float64, handle_unknown=&#39;error&#39;, sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-97.77\" rx=\"31.64\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-100.97\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-88.97\" font-family=\"Times,serif\" font-size=\"11.00\">Encoder</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_imputer&#45;&gt;cat_encoder -->\n",
              "<g id=\"edge6\" class=\"edge\">\n",
              "<title>cat_imputer&#45;&gt;cat_encoder</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-98.47C620.35,-98.38 630.77,-98.28 640.7,-98.18\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"640.84,-101.68 650.8,-98.08 640.77,-94.68 640.84,-101.68\"/>\n",
              "</g>\n",
              "<!-- float32_transform_0 -->\n",
              "<g id=\"node8\" class=\"node\">\n",
              "<title>float32_transform_0</title>\n",
              "<g id=\"a_node8\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_0 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-94.77\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-97.97\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-85.97\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_encoder&#45;&gt;float32_transform_0 -->\n",
              "<g id=\"edge7\" class=\"edge\">\n",
              "<title>cat_encoder&#45;&gt;float32_transform_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M714.92,-96.89C724.18,-96.63 734.54,-96.34 744.53,-96.06\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.85,-99.55 754.75,-95.77 744.66,-92.55 744.85,-99.55\"/>\n",
              "</g>\n",
              "<!-- concat_features -->\n",
              "<g id=\"node15\" class=\"node\">\n",
              "<title>concat_features</title>\n",
              "<g id=\"a_node15\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.lale.concat_features.html\" xlink:title=\"concat_features = ConcatFeatures()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"896.36\" cy=\"-68.77\" rx=\"33.47\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-71.97\" font-family=\"Times,serif\" font-size=\"11.00\">Concat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-59.97\" font-family=\"Times,serif\" font-size=\"11.00\">Features</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float32_transform_0&#45;&gt;concat_features -->\n",
              "<g id=\"edge13\" class=\"edge\">\n",
              "<title>float32_transform_0&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.18,-86.69C834.15,-84.18 845.25,-81.38 855.66,-78.76\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"856.53,-82.15 865.38,-76.32 854.82,-75.36 856.53,-82.15\"/>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1 -->\n",
              "<g id=\"node9\" class=\"node\">\n",
              "<title>numpy_column_selector_1</title>\n",
              "<g id=\"a_node9\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_1 = NumpyColumnSelector(columns=[4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-31.77\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-40.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-28.97\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-16.97\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float -->\n",
              "<g id=\"node10\" class=\"node\">\n",
              "<title>float_str2_float</title>\n",
              "<g id=\"a_node10\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float_str2_float.html\" xlink:title=\"float_str2_float = FloatStr2Float(dtypes_list=[&#39;int_num&#39;], missing_values_reference_list=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-31.77\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-40.97\" font-family=\"Times,serif\" font-size=\"11.00\">Float&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-28.97\" font-family=\"Times,serif\" font-size=\"11.00\">Str2&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-16.97\" font-family=\"Times,serif\" font-size=\"11.00\">Float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1&#45;&gt;float_str2_float -->\n",
              "<g id=\"edge8\" class=\"edge\">\n",
              "<title>numpy_column_selector_1&#45;&gt;float_str2_float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M286.99,-31.77C299.27,-31.77 313.34,-31.77 325.96,-31.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"326.16,-35.27 336.16,-31.77 326.16,-28.27 326.16,-35.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1 -->\n",
              "<g id=\"node11\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_1</title>\n",
              "<g id=\"a_node11\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_1 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-36.77\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-51.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-39.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-27.97\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-15.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float&#45;&gt;numpy_replace_missing_values_1 -->\n",
              "<g id=\"edge9\" class=\"edge\">\n",
              "<title>float_str2_float&#45;&gt;numpy_replace_missing_values_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M390.25,-32.94C402.39,-33.49 417.18,-34.16 430.99,-34.78\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"431.15,-38.29 441.3,-35.24 431.47,-31.3 431.15,-38.29\"/>\n",
              "</g>\n",
              "<!-- num_imputer -->\n",
              "<g id=\"node12\" class=\"node\">\n",
              "<title>num_imputer</title>\n",
              "<g id=\"a_node12\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.num_imputer.html\" xlink:title=\"num_imputer = NumImputer(strategy=&#39;median&#39;, missing_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-39.77\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-42.97\" font-family=\"Times,serif\" font-size=\"11.00\">Num&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-30.97\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1&#45;&gt;num_imputer -->\n",
              "<g id=\"edge10\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_1&#45;&gt;num_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M510.91,-37.76C520.15,-38.03 530.24,-38.33 539.78,-38.61\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"539.74,-42.11 549.83,-38.9 539.94,-35.11 539.74,-42.11\"/>\n",
              "</g>\n",
              "<!-- opt_standard_scaler -->\n",
              "<g id=\"node13\" class=\"node\">\n",
              "<title>opt_standard_scaler</title>\n",
              "<g id=\"a_node13\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.opt_standard_scaler.html\" xlink:title=\"opt_standard_scaler = OptStandardScaler(num_scaler_copy=None, num_scaler_with_mean=None, num_scaler_with_std=None, use_scaler_flag=False)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-41.77\" rx=\"36.12\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-50.97\" font-family=\"Times,serif\" font-size=\"11.00\">Opt&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-38.97\" font-family=\"Times,serif\" font-size=\"11.00\">Standard&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-26.97\" font-family=\"Times,serif\" font-size=\"11.00\">Scaler</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- num_imputer&#45;&gt;opt_standard_scaler -->\n",
              "<g id=\"edge11\" class=\"edge\">\n",
              "<title>num_imputer&#45;&gt;opt_standard_scaler</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-40.36C619.14,-40.52 628.07,-40.7 636.8,-40.87\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"636.81,-44.37 646.88,-41.07 636.95,-37.37 636.81,-44.37\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1 -->\n",
              "<g id=\"node14\" class=\"node\">\n",
              "<title>float32_transform_1</title>\n",
              "<g id=\"a_node14\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_1 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-45.77\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-48.97\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-36.97\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- opt_standard_scaler&#45;&gt;float32_transform_1 -->\n",
              "<g id=\"edge12\" class=\"edge\">\n",
              "<title>opt_standard_scaler&#45;&gt;float32_transform_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M718.98,-43.09C727.31,-43.4 736.32,-43.74 745.03,-44.07\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.93,-47.57 755.06,-44.45 745.2,-40.58 744.93,-47.57\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1&#45;&gt;concat_features -->\n",
              "<g id=\"edge14\" class=\"edge\">\n",
              "<title>float32_transform_1&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.75,-53.05C834.41,-55.2 845.1,-57.58 855.18,-59.82\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"854.47,-63.25 865,-62.01 856,-56.42 854.47,-63.25\"/>\n",
              "</g>\n",
              "<!-- numpy_permute_array -->\n",
              "<g id=\"node16\" class=\"node\">\n",
              "<title>numpy_permute_array</title>\n",
              "<g id=\"a_node16\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_permute_array.html\" xlink:title=\"numpy_permute_array = NumpyPermuteArray(axis=0, permutation_indices=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1000.24\" cy=\"-68.77\" rx=\"34.8\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-77.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-65.97\" font-family=\"Times,serif\" font-size=\"11.00\">Permute&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-53.97\" font-family=\"Times,serif\" font-size=\"11.00\">Array</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- concat_features&#45;&gt;numpy_permute_array -->\n",
              "<g id=\"edge15\" class=\"edge\">\n",
              "<title>concat_features&#45;&gt;numpy_permute_array</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M929.6,-68.77C937.81,-68.77 946.78,-68.77 955.46,-68.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"955.46,-72.27 965.46,-68.77 955.46,-65.27 955.46,-72.27\"/>\n",
              "</g>\n",
              "<!-- t_no_op -->\n",
              "<g id=\"node17\" class=\"node\">\n",
              "<title>t_no_op</title>\n",
              "<g id=\"a_node17\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.t_no_op.html\" xlink:title=\"t_no_op = TNoOp(fun=None, name=&#39;no_action&#39;, datatypes=&#39;x&#39;, feat_constraints=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1097.89\" cy=\"-68.77\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-77.97\" font-family=\"Times,serif\" font-size=\"11.00\">T&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-65.97\" font-family=\"Times,serif\" font-size=\"11.00\">No&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-53.97\" font-family=\"Times,serif\" font-size=\"11.00\">Op</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_permute_array&#45;&gt;t_no_op -->\n",
              "<g id=\"edge16\" class=\"edge\">\n",
              "<title>numpy_permute_array&#45;&gt;t_no_op</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1034.96,-68.77C1043.21,-68.77 1052.07,-68.77 1060.43,-68.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1060.68,-72.27 1070.68,-68.77 1060.68,-65.27 1060.68,-72.27\"/>\n",
              "</g>\n",
              "<!-- xgb_classifier -->\n",
              "<g id=\"node18\" class=\"node\">\n",
              "<title>xgb_classifier</title>\n",
              "<g id=\"a_node18\"><a xlink:title=\"xgb_classifier = XGBClassifier(gamma=0.44579800091381966, learning_rate=0.021330806558379982, min_child_weight=5, missing=float(&#39;nan&#39;), n_estimators=574, n_jobs=2, random_state=33, reg_alpha=0.724822538978647, reg_lambda=0.2506742141014951, silent=True, subsample=0.41147527714871196...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1196.95\" cy=\"-68.77\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"1196.95\" y=\"-71.97\" font-family=\"Times,serif\" font-size=\"11.00\">XGB&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1196.95\" y=\"-59.97\" font-family=\"Times,serif\" font-size=\"11.00\">Classifier</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- t_no_op&#45;&gt;xgb_classifier -->\n",
              "<g id=\"edge17\" class=\"edge\">\n",
              "<title>t_no_op&#45;&gt;xgb_classifier</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1124.9,-68.77C1132.81,-68.77 1141.75,-68.77 1150.55,-68.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1150.74,-72.27 1160.74,-68.77 1150.74,-65.27 1150.74,-72.27\"/>\n",
              "</g>\n",
              "</g>\n",
              "</svg>\n"
            ],
            "text/plain": [
              "<graphviz.dot.Digraph at 0x7fd8d2f82790>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "best_pipeline.visualize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Each node in the visualization is a machine-learning operator\n",
        "(transformer or estimator). Each edge indicates data flow (transformed\n",
        "output from one operator becomes input to the next).  The input to the\n",
        "root nodes is the initial dataset and the output from the sink node\n",
        "is the final prediction.  When you hover the mouse pointer over a\n",
        "node, a tooltip shows you the configuration arguments of the\n",
        "corresponding operator (tuned hyperparameters). When you click on the\n",
        "hyperlink of a node, it brings you to a documentation page for the\n",
        "operator."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Pipeline source code"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "best_pipeline.pretty_print(ipython_display=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In the pretty-printed code, `>>` is the pipe combinator (dataflow\n",
        "edge) and `&` is the and combinator (combining multiple subpipelines).\n",
        "They correspond to the `make_pipeline` and `make_union` functions from\n",
        "scikit-learn, respectively. If you prefer the functions, you can\n",
        "instead pretty-print your pipeline with\n",
        "`best_pipeline.pretty_print(ipython_display=True, combinators=False)`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Reading training data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [],
      "source": [
        "train_df = pipeline_optimizer.get_data_connections()[0].read()\n",
        "\n",
        "train_X = train_df.drop(['Risk'], axis=1).values\n",
        "train_y = train_df.Risk.values"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Test pipeline model locally"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array(['No Risk', 'No Risk', 'No Risk', 'No Risk', 'Risk'], dtype=object)"
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "predicted_y = best_pipeline.predict(train_X)\n",
        "predicted_y[:5]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"runs\"></a>\n",
        "## 5. Historical runs\n",
        "\n",
        "In this section you learn to work with historical AutoPipelines fit jobs (runs)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To list historical runs use method `list()`. You can filter runs by providing experiment name."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>timestamp</th>\n",
              "      <th>run_id</th>\n",
              "      <th>state</th>\n",
              "      <th>auto_pipeline_optimizer name</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2020-12-08T12:50:22.481Z</td>\n",
              "      <td>91e1541a-fee1-4bb9-8211-68eed6abfdc4</td>\n",
              "      <td>completed</td>\n",
              "      <td>Credit Risk Prediction - AutoAI</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  timestamp                                run_id      state  \\\n",
              "0  2020-12-08T12:50:22.481Z  91e1541a-fee1-4bb9-8211-68eed6abfdc4  completed   \n",
              "\n",
              "      auto_pipeline_optimizer name  \n",
              "0  Credit Risk Prediction - AutoAI  "
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "experiment.runs(filter='Credit Risk Prediction - AutoAI').list()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To work with historical pipelines found during a particular optimizer\n",
        "run, you need to first provide the `run_id` to select the fitted\n",
        "optimizer.\n",
        "\n",
        "**Note**: you can assign selected `run_id` to the run_id variable."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [],
      "source": [
        "run_id = run_details['metadata']['id']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Get executed optimizer's configuration parameters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'name': 'Credit Risk Prediction - AutoAI',\n",
              " 'desc': 'Sample notebook',\n",
              " 'prediction_type': 'binary',\n",
              " 'prediction_column': 'Risk',\n",
              " 'scoring': 'roc_auc',\n",
              " 'test_size': 0.1,\n",
              " 'max_num_daub_ensembles': 2.0,\n",
              " 't_shirt_size': 'c076e82c-b2a7-4d20-9c0f-1f0c2fdf5a24',\n",
              " 'daub_include_only_estimators': ['ExtraTreesClassifierEstimator',\n",
              "  'GradientBoostingClassifierEstimator',\n",
              "  'LGBMClassifierEstimator',\n",
              "  'LogisticRegressionEstimator',\n",
              "  'RandomForestClassifierEstimator',\n",
              "  'XGBClassifierEstimator',\n",
              "  'DecisionTreeClassifierEstimator'],\n",
              " 'cognito_transform_names': None,\n",
              " 'train_sample_rows_test_size': None,\n",
              " 'csv_separator': ',',\n",
              " 'excel_sheet': 0,\n",
              " 'encoding': 'utf-8'}"
            ]
          },
          "execution_count": 26,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "experiment.runs(filter='Credit Risk Prediction - AutoAI').get_params(run_id=run_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "### Get historical optimizer instance and training details"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [],
      "source": [
        "historical_opt = experiment.runs.get_optimizer(run_id)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [],
      "source": [
        "run_details = historical_opt.get_run_details()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### List trained pipelines for selected optimizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Number of enhancements</th>\n",
              "      <th>Estimator</th>\n",
              "      <th>training_roc_auc_(optimized)</th>\n",
              "      <th>holdout_precision</th>\n",
              "      <th>training_average_precision</th>\n",
              "      <th>holdout_average_precision</th>\n",
              "      <th>training_neg_log_loss</th>\n",
              "      <th>holdout_recall</th>\n",
              "      <th>training_precision</th>\n",
              "      <th>holdout_accuracy</th>\n",
              "      <th>holdout_balanced_accuracy</th>\n",
              "      <th>training_recall</th>\n",
              "      <th>holdout_f1</th>\n",
              "      <th>holdout_neg_log_loss</th>\n",
              "      <th>training_accuracy</th>\n",
              "      <th>holdout_roc_auc</th>\n",
              "      <th>training_balanced_accuracy</th>\n",
              "      <th>training_f1</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline Name</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Pipeline_4</th>\n",
              "      <td>4</td>\n",
              "      <td>XGBClassifierEstimator</td>\n",
              "      <td>0.853971</td>\n",
              "      <td>0.702532</td>\n",
              "      <td>0.786750</td>\n",
              "      <td>0.762646</td>\n",
              "      <td>-0.432601</td>\n",
              "      <td>0.664671</td>\n",
              "      <td>0.686865</td>\n",
              "      <td>0.794</td>\n",
              "      <td>0.761765</td>\n",
              "      <td>0.649368</td>\n",
              "      <td>0.683077</td>\n",
              "      <td>-0.434525</td>\n",
              "      <td>0.783778</td>\n",
              "      <td>0.858985</td>\n",
              "      <td>0.750276</td>\n",
              "      <td>0.667448</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_7</th>\n",
              "      <td>3</td>\n",
              "      <td>GradientBoostingClassifierEstimator</td>\n",
              "      <td>0.853637</td>\n",
              "      <td>0.744186</td>\n",
              "      <td>0.781843</td>\n",
              "      <td>0.752330</td>\n",
              "      <td>-0.430732</td>\n",
              "      <td>0.574850</td>\n",
              "      <td>0.754036</td>\n",
              "      <td>0.792</td>\n",
              "      <td>0.737876</td>\n",
              "      <td>0.581504</td>\n",
              "      <td>0.648649</td>\n",
              "      <td>-0.435807</td>\n",
              "      <td>0.796667</td>\n",
              "      <td>0.859524</td>\n",
              "      <td>0.743037</td>\n",
              "      <td>0.656293</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_8</th>\n",
              "      <td>4</td>\n",
              "      <td>GradientBoostingClassifierEstimator</td>\n",
              "      <td>0.853637</td>\n",
              "      <td>0.744186</td>\n",
              "      <td>0.781843</td>\n",
              "      <td>0.752330</td>\n",
              "      <td>-0.430732</td>\n",
              "      <td>0.574850</td>\n",
              "      <td>0.754036</td>\n",
              "      <td>0.792</td>\n",
              "      <td>0.737876</td>\n",
              "      <td>0.581504</td>\n",
              "      <td>0.648649</td>\n",
              "      <td>-0.435807</td>\n",
              "      <td>0.796667</td>\n",
              "      <td>0.859524</td>\n",
              "      <td>0.743037</td>\n",
              "      <td>0.656293</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_2</th>\n",
              "      <td>2</td>\n",
              "      <td>XGBClassifierEstimator</td>\n",
              "      <td>0.853223</td>\n",
              "      <td>0.674847</td>\n",
              "      <td>0.783110</td>\n",
              "      <td>0.759399</td>\n",
              "      <td>-0.437251</td>\n",
              "      <td>0.658683</td>\n",
              "      <td>0.674910</td>\n",
              "      <td>0.780</td>\n",
              "      <td>0.749762</td>\n",
              "      <td>0.657352</td>\n",
              "      <td>0.666667</td>\n",
              "      <td>-0.439138</td>\n",
              "      <td>0.779556</td>\n",
              "      <td>0.856000</td>\n",
              "      <td>0.749096</td>\n",
              "      <td>0.665831</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_3</th>\n",
              "      <td>3</td>\n",
              "      <td>XGBClassifierEstimator</td>\n",
              "      <td>0.853223</td>\n",
              "      <td>0.674847</td>\n",
              "      <td>0.783110</td>\n",
              "      <td>0.759399</td>\n",
              "      <td>-0.437251</td>\n",
              "      <td>0.658683</td>\n",
              "      <td>0.674910</td>\n",
              "      <td>0.780</td>\n",
              "      <td>0.749762</td>\n",
              "      <td>0.657352</td>\n",
              "      <td>0.666667</td>\n",
              "      <td>-0.439138</td>\n",
              "      <td>0.779556</td>\n",
              "      <td>0.856000</td>\n",
              "      <td>0.749096</td>\n",
              "      <td>0.665831</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_6</th>\n",
              "      <td>2</td>\n",
              "      <td>GradientBoostingClassifierEstimator</td>\n",
              "      <td>0.850494</td>\n",
              "      <td>0.755906</td>\n",
              "      <td>0.778339</td>\n",
              "      <td>0.765539</td>\n",
              "      <td>-0.432066</td>\n",
              "      <td>0.574850</td>\n",
              "      <td>0.759112</td>\n",
              "      <td>0.796</td>\n",
              "      <td>0.740879</td>\n",
              "      <td>0.592149</td>\n",
              "      <td>0.653061</td>\n",
              "      <td>-0.430092</td>\n",
              "      <td>0.800667</td>\n",
              "      <td>0.856503</td>\n",
              "      <td>0.748694</td>\n",
              "      <td>0.665064</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_1</th>\n",
              "      <td>1</td>\n",
              "      <td>XGBClassifierEstimator</td>\n",
              "      <td>0.849883</td>\n",
              "      <td>0.692810</td>\n",
              "      <td>0.782921</td>\n",
              "      <td>0.760481</td>\n",
              "      <td>-0.437680</td>\n",
              "      <td>0.634731</td>\n",
              "      <td>0.692303</td>\n",
              "      <td>0.784</td>\n",
              "      <td>0.746795</td>\n",
              "      <td>0.638057</td>\n",
              "      <td>0.662500</td>\n",
              "      <td>-0.438008</td>\n",
              "      <td>0.783778</td>\n",
              "      <td>0.854903</td>\n",
              "      <td>0.747457</td>\n",
              "      <td>0.663639</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_5</th>\n",
              "      <td>1</td>\n",
              "      <td>GradientBoostingClassifierEstimator</td>\n",
              "      <td>0.848791</td>\n",
              "      <td>0.766129</td>\n",
              "      <td>0.780104</td>\n",
              "      <td>0.751335</td>\n",
              "      <td>-0.432413</td>\n",
              "      <td>0.568862</td>\n",
              "      <td>0.768005</td>\n",
              "      <td>0.798</td>\n",
              "      <td>0.740888</td>\n",
              "      <td>0.573520</td>\n",
              "      <td>0.652921</td>\n",
              "      <td>-0.442652</td>\n",
              "      <td>0.799333</td>\n",
              "      <td>0.848789</td>\n",
              "      <td>0.743049</td>\n",
              "      <td>0.656371</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               Number of enhancements                            Estimator  \\\n",
              "Pipeline Name                                                                \n",
              "Pipeline_4                          4               XGBClassifierEstimator   \n",
              "Pipeline_7                          3  GradientBoostingClassifierEstimator   \n",
              "Pipeline_8                          4  GradientBoostingClassifierEstimator   \n",
              "Pipeline_2                          2               XGBClassifierEstimator   \n",
              "Pipeline_3                          3               XGBClassifierEstimator   \n",
              "Pipeline_6                          2  GradientBoostingClassifierEstimator   \n",
              "Pipeline_1                          1               XGBClassifierEstimator   \n",
              "Pipeline_5                          1  GradientBoostingClassifierEstimator   \n",
              "\n",
              "               training_roc_auc_(optimized)  holdout_precision  \\\n",
              "Pipeline Name                                                    \n",
              "Pipeline_4                         0.853971           0.702532   \n",
              "Pipeline_7                         0.853637           0.744186   \n",
              "Pipeline_8                         0.853637           0.744186   \n",
              "Pipeline_2                         0.853223           0.674847   \n",
              "Pipeline_3                         0.853223           0.674847   \n",
              "Pipeline_6                         0.850494           0.755906   \n",
              "Pipeline_1                         0.849883           0.692810   \n",
              "Pipeline_5                         0.848791           0.766129   \n",
              "\n",
              "               training_average_precision  holdout_average_precision  \\\n",
              "Pipeline Name                                                          \n",
              "Pipeline_4                       0.786750                   0.762646   \n",
              "Pipeline_7                       0.781843                   0.752330   \n",
              "Pipeline_8                       0.781843                   0.752330   \n",
              "Pipeline_2                       0.783110                   0.759399   \n",
              "Pipeline_3                       0.783110                   0.759399   \n",
              "Pipeline_6                       0.778339                   0.765539   \n",
              "Pipeline_1                       0.782921                   0.760481   \n",
              "Pipeline_5                       0.780104                   0.751335   \n",
              "\n",
              "               training_neg_log_loss  holdout_recall  training_precision  \\\n",
              "Pipeline Name                                                              \n",
              "Pipeline_4                 -0.432601        0.664671            0.686865   \n",
              "Pipeline_7                 -0.430732        0.574850            0.754036   \n",
              "Pipeline_8                 -0.430732        0.574850            0.754036   \n",
              "Pipeline_2                 -0.437251        0.658683            0.674910   \n",
              "Pipeline_3                 -0.437251        0.658683            0.674910   \n",
              "Pipeline_6                 -0.432066        0.574850            0.759112   \n",
              "Pipeline_1                 -0.437680        0.634731            0.692303   \n",
              "Pipeline_5                 -0.432413        0.568862            0.768005   \n",
              "\n",
              "               holdout_accuracy  holdout_balanced_accuracy  training_recall  \\\n",
              "Pipeline Name                                                                 \n",
              "Pipeline_4                0.794                   0.761765         0.649368   \n",
              "Pipeline_7                0.792                   0.737876         0.581504   \n",
              "Pipeline_8                0.792                   0.737876         0.581504   \n",
              "Pipeline_2                0.780                   0.749762         0.657352   \n",
              "Pipeline_3                0.780                   0.749762         0.657352   \n",
              "Pipeline_6                0.796                   0.740879         0.592149   \n",
              "Pipeline_1                0.784                   0.746795         0.638057   \n",
              "Pipeline_5                0.798                   0.740888         0.573520   \n",
              "\n",
              "               holdout_f1  holdout_neg_log_loss  training_accuracy  \\\n",
              "Pipeline Name                                                        \n",
              "Pipeline_4       0.683077             -0.434525           0.783778   \n",
              "Pipeline_7       0.648649             -0.435807           0.796667   \n",
              "Pipeline_8       0.648649             -0.435807           0.796667   \n",
              "Pipeline_2       0.666667             -0.439138           0.779556   \n",
              "Pipeline_3       0.666667             -0.439138           0.779556   \n",
              "Pipeline_6       0.653061             -0.430092           0.800667   \n",
              "Pipeline_1       0.662500             -0.438008           0.783778   \n",
              "Pipeline_5       0.652921             -0.442652           0.799333   \n",
              "\n",
              "               holdout_roc_auc  training_balanced_accuracy  training_f1  \n",
              "Pipeline Name                                                            \n",
              "Pipeline_4            0.858985                    0.750276     0.667448  \n",
              "Pipeline_7            0.859524                    0.743037     0.656293  \n",
              "Pipeline_8            0.859524                    0.743037     0.656293  \n",
              "Pipeline_2            0.856000                    0.749096     0.665831  \n",
              "Pipeline_3            0.856000                    0.749096     0.665831  \n",
              "Pipeline_6            0.856503                    0.748694     0.665064  \n",
              "Pipeline_1            0.854903                    0.747457     0.663639  \n",
              "Pipeline_5            0.848789                    0.743049     0.656371  "
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "historical_opt.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Get selected pipeline and test locally"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [],
      "source": [
        "hist_pipeline = historical_opt.get_pipeline(pipeline_name='Pipeline_3')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "pycharm": {
          "is_executing": false
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array(['No Risk', 'No Risk', 'No Risk', 'No Risk', 'Risk'], dtype=object)"
            ]
          },
          "execution_count": 31,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "predicted_y = hist_pipeline.predict(train_X)\n",
        "predicted_y[:5]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"refinery\"></a>\n",
        "## 6. Pipeline refinement with [Lale](https://github.com/IBM/lale) and testing\n",
        "\n",
        "In this section you learn how to refine and retrain the best\n",
        "pipeline returned by AutoAI. There are many ways to refine a pipeline.\n",
        "For illustration, simply replace the final estimator in the\n",
        "pipeline by an interpretable model. The call to\n",
        "`wrap_imported_operators()` augments scikit-learn operators with\n",
        "schemas for hyperparameter tuning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.linear_model import LogisticRegression as LR\n",
        "from sklearn.tree import DecisionTreeClassifier as Tree\n",
        "from sklearn.neighbors import KNeighborsClassifier as KNN\n",
        "from lale.lib.lale import Hyperopt\n",
        "from lale import wrap_imported_operators\n",
        "\n",
        "wrap_imported_operators()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Pipeline decomposition and new definition\n",
        "\n",
        "Start by removing the last step of the pipeline, i.e., the final estimator."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "image/svg+xml": [
              "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
              "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
              " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
              "<!-- Generated by graphviz version 2.44.1 (20200629.0846)\n",
              " -->\n",
              "<!-- Title: cluster:(root) Pages: 1 -->\n",
              "<svg width=\"1133pt\" height=\"149pt\"\n",
              " viewBox=\"0.00 0.00 1132.89 148.54\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
              "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 144.54)\">\n",
              "<title>cluster:(root)</title>\n",
              "<g id=\"a_graph0\"><a xlink:title=\"(root) = ...\">\n",
              "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-144.54 1128.89,-144.54 1128.89,4 -4,4\"/>\n",
              "</a>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0 -->\n",
              "<g id=\"node1\" class=\"node\">\n",
              "<title>numpy_column_selector_0</title>\n",
              "<g id=\"a_node1\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_0 = NumpyColumnSelector(columns=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"33.23\" cy=\"-103.77\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-112.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-100.97\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-88.97\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings -->\n",
              "<g id=\"node2\" class=\"node\">\n",
              "<title>compress_strings</title>\n",
              "<g id=\"a_node2\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.compress_strings.html\" xlink:title=\"compress_strings = CompressStrings(compress_type=&#39;hash&#39;, dtypes_list=[&#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_s...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"142.77\" cy=\"-103.77\" rx=\"40.11\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Compress&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Strings</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0&#45;&gt;compress_strings -->\n",
              "<g id=\"edge1\" class=\"edge\">\n",
              "<title>numpy_column_selector_0&#45;&gt;compress_strings</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M66.5,-103.77C74.59,-103.77 83.46,-103.77 92.19,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"92.3,-107.27 102.3,-103.77 92.3,-100.27 92.3,-107.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0 -->\n",
              "<g id=\"node3\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_0</title>\n",
              "<g id=\"a_node3\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_0 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-103.77\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-118.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-82.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings&#45;&gt;numpy_replace_missing_values_0 -->\n",
              "<g id=\"edge2\" class=\"edge\">\n",
              "<title>compress_strings&#45;&gt;numpy_replace_missing_values_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M183.11,-103.77C191.43,-103.77 200.26,-103.77 208.74,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"208.85,-107.27 218.85,-103.77 208.85,-100.27 208.85,-107.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values -->\n",
              "<g id=\"node4\" class=\"node\">\n",
              "<title>numpy_replace_unknown_values</title>\n",
              "<g id=\"a_node4\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_unknown_values.html\" xlink:title=\"numpy_replace_unknown_values = NumpyReplaceUnknownValues(filling_values=float(&#39;nan&#39;), filling_values_list=[float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-103.77\" rx=\"38.78\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-118.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Unknown&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-82.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values -->\n",
              "<g id=\"edge3\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M288.46,-103.77C296.6,-103.77 305.47,-103.77 314.13,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"314.16,-107.27 324.16,-103.77 314.16,-100.27 314.16,-107.27\"/>\n",
              "</g>\n",
              "<!-- boolean2float -->\n",
              "<g id=\"node5\" class=\"node\">\n",
              "<title>boolean2float</title>\n",
              "<g id=\"a_node5\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.boolean2float.html\" xlink:title=\"boolean2float = boolean2float()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-100.77\" rx=\"37.91\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-97.97\" font-family=\"Times,serif\" font-size=\"11.00\">boolean2float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values&#45;&gt;boolean2float -->\n",
              "<g id=\"edge4\" class=\"edge\">\n",
              "<title>numpy_replace_unknown_values&#45;&gt;boolean2float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M402.41,-102.74C410.6,-102.52 419.36,-102.28 427.85,-102.05\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"428.12,-105.54 438.02,-101.77 427.93,-98.55 428.12,-105.54\"/>\n",
              "</g>\n",
              "<!-- cat_imputer -->\n",
              "<g id=\"node6\" class=\"node\">\n",
              "<title>cat_imputer</title>\n",
              "<g id=\"a_node6\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_imputer.html\" xlink:title=\"cat_imputer = CatImputer(strategy=&#39;most_frequent&#39;, missing_values=float(&#39;nan&#39;), sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-98.77\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-101.97\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-89.97\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- boolean2float&#45;&gt;cat_imputer -->\n",
              "<g id=\"edge5\" class=\"edge\">\n",
              "<title>boolean2float&#45;&gt;cat_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M514.07,-100.05C522.45,-99.88 531.39,-99.71 539.88,-99.54\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"540.02,-103.04 549.95,-99.35 539.88,-96.04 540.02,-103.04\"/>\n",
              "</g>\n",
              "<!-- cat_encoder -->\n",
              "<g id=\"node7\" class=\"node\">\n",
              "<title>cat_encoder</title>\n",
              "<g id=\"a_node7\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_encoder.html\" xlink:title=\"cat_encoder = CatEncoder(encoding=&#39;ordinal&#39;, categories=&#39;auto&#39;, dtype=np.float64, handle_unknown=&#39;error&#39;, sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-97.77\" rx=\"31.64\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-100.97\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-88.97\" font-family=\"Times,serif\" font-size=\"11.00\">Encoder</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_imputer&#45;&gt;cat_encoder -->\n",
              "<g id=\"edge6\" class=\"edge\">\n",
              "<title>cat_imputer&#45;&gt;cat_encoder</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-98.47C620.35,-98.38 630.77,-98.28 640.7,-98.18\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"640.84,-101.68 650.8,-98.08 640.77,-94.68 640.84,-101.68\"/>\n",
              "</g>\n",
              "<!-- float32_transform_0 -->\n",
              "<g id=\"node8\" class=\"node\">\n",
              "<title>float32_transform_0</title>\n",
              "<g id=\"a_node8\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_0 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-94.77\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-97.97\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-85.97\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_encoder&#45;&gt;float32_transform_0 -->\n",
              "<g id=\"edge7\" class=\"edge\">\n",
              "<title>cat_encoder&#45;&gt;float32_transform_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M714.92,-96.89C724.18,-96.63 734.54,-96.34 744.53,-96.06\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.85,-99.55 754.75,-95.77 744.66,-92.55 744.85,-99.55\"/>\n",
              "</g>\n",
              "<!-- concat_features -->\n",
              "<g id=\"node15\" class=\"node\">\n",
              "<title>concat_features</title>\n",
              "<g id=\"a_node15\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.lale.concat_features.html\" xlink:title=\"concat_features = ConcatFeatures()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"896.36\" cy=\"-68.77\" rx=\"33.47\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-71.97\" font-family=\"Times,serif\" font-size=\"11.00\">Concat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-59.97\" font-family=\"Times,serif\" font-size=\"11.00\">Features</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float32_transform_0&#45;&gt;concat_features -->\n",
              "<g id=\"edge13\" class=\"edge\">\n",
              "<title>float32_transform_0&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.18,-86.69C834.15,-84.18 845.25,-81.38 855.66,-78.76\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"856.53,-82.15 865.38,-76.32 854.82,-75.36 856.53,-82.15\"/>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1 -->\n",
              "<g id=\"node9\" class=\"node\">\n",
              "<title>numpy_column_selector_1</title>\n",
              "<g id=\"a_node9\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_1 = NumpyColumnSelector(columns=[4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-31.77\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-40.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-28.97\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-16.97\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float -->\n",
              "<g id=\"node10\" class=\"node\">\n",
              "<title>float_str2_float</title>\n",
              "<g id=\"a_node10\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float_str2_float.html\" xlink:title=\"float_str2_float = FloatStr2Float(dtypes_list=[&#39;int_num&#39;], missing_values_reference_list=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-31.77\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-40.97\" font-family=\"Times,serif\" font-size=\"11.00\">Float&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-28.97\" font-family=\"Times,serif\" font-size=\"11.00\">Str2&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-16.97\" font-family=\"Times,serif\" font-size=\"11.00\">Float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1&#45;&gt;float_str2_float -->\n",
              "<g id=\"edge8\" class=\"edge\">\n",
              "<title>numpy_column_selector_1&#45;&gt;float_str2_float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M286.99,-31.77C299.27,-31.77 313.34,-31.77 325.96,-31.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"326.16,-35.27 336.16,-31.77 326.16,-28.27 326.16,-35.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1 -->\n",
              "<g id=\"node11\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_1</title>\n",
              "<g id=\"a_node11\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_1 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-36.77\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-51.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-39.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-27.97\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-15.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float&#45;&gt;numpy_replace_missing_values_1 -->\n",
              "<g id=\"edge9\" class=\"edge\">\n",
              "<title>float_str2_float&#45;&gt;numpy_replace_missing_values_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M390.25,-32.94C402.39,-33.49 417.18,-34.16 430.99,-34.78\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"431.15,-38.29 441.3,-35.24 431.47,-31.3 431.15,-38.29\"/>\n",
              "</g>\n",
              "<!-- num_imputer -->\n",
              "<g id=\"node12\" class=\"node\">\n",
              "<title>num_imputer</title>\n",
              "<g id=\"a_node12\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.num_imputer.html\" xlink:title=\"num_imputer = NumImputer(strategy=&#39;median&#39;, missing_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-39.77\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-42.97\" font-family=\"Times,serif\" font-size=\"11.00\">Num&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-30.97\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1&#45;&gt;num_imputer -->\n",
              "<g id=\"edge10\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_1&#45;&gt;num_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M510.91,-37.76C520.15,-38.03 530.24,-38.33 539.78,-38.61\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"539.74,-42.11 549.83,-38.9 539.94,-35.11 539.74,-42.11\"/>\n",
              "</g>\n",
              "<!-- opt_standard_scaler -->\n",
              "<g id=\"node13\" class=\"node\">\n",
              "<title>opt_standard_scaler</title>\n",
              "<g id=\"a_node13\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.opt_standard_scaler.html\" xlink:title=\"opt_standard_scaler = OptStandardScaler(num_scaler_copy=None, num_scaler_with_mean=None, num_scaler_with_std=None, use_scaler_flag=False)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-41.77\" rx=\"36.12\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-50.97\" font-family=\"Times,serif\" font-size=\"11.00\">Opt&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-38.97\" font-family=\"Times,serif\" font-size=\"11.00\">Standard&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-26.97\" font-family=\"Times,serif\" font-size=\"11.00\">Scaler</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- num_imputer&#45;&gt;opt_standard_scaler -->\n",
              "<g id=\"edge11\" class=\"edge\">\n",
              "<title>num_imputer&#45;&gt;opt_standard_scaler</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-40.36C619.14,-40.52 628.07,-40.7 636.8,-40.87\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"636.81,-44.37 646.88,-41.07 636.95,-37.37 636.81,-44.37\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1 -->\n",
              "<g id=\"node14\" class=\"node\">\n",
              "<title>float32_transform_1</title>\n",
              "<g id=\"a_node14\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_1 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-45.77\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-48.97\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-36.97\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- opt_standard_scaler&#45;&gt;float32_transform_1 -->\n",
              "<g id=\"edge12\" class=\"edge\">\n",
              "<title>opt_standard_scaler&#45;&gt;float32_transform_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M718.98,-43.09C727.31,-43.4 736.32,-43.74 745.03,-44.07\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.93,-47.57 755.06,-44.45 745.2,-40.58 744.93,-47.57\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1&#45;&gt;concat_features -->\n",
              "<g id=\"edge14\" class=\"edge\">\n",
              "<title>float32_transform_1&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.75,-53.05C834.41,-55.2 845.1,-57.58 855.18,-59.82\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"854.47,-63.25 865,-62.01 856,-56.42 854.47,-63.25\"/>\n",
              "</g>\n",
              "<!-- numpy_permute_array -->\n",
              "<g id=\"node16\" class=\"node\">\n",
              "<title>numpy_permute_array</title>\n",
              "<g id=\"a_node16\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_permute_array.html\" xlink:title=\"numpy_permute_array = NumpyPermuteArray(axis=0, permutation_indices=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1000.24\" cy=\"-68.77\" rx=\"34.8\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-77.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-65.97\" font-family=\"Times,serif\" font-size=\"11.00\">Permute&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-53.97\" font-family=\"Times,serif\" font-size=\"11.00\">Array</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- concat_features&#45;&gt;numpy_permute_array -->\n",
              "<g id=\"edge15\" class=\"edge\">\n",
              "<title>concat_features&#45;&gt;numpy_permute_array</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M929.6,-68.77C937.81,-68.77 946.78,-68.77 955.46,-68.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"955.46,-72.27 965.46,-68.77 955.46,-65.27 955.46,-72.27\"/>\n",
              "</g>\n",
              "<!-- t_no_op -->\n",
              "<g id=\"node17\" class=\"node\">\n",
              "<title>t_no_op</title>\n",
              "<g id=\"a_node17\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.t_no_op.html\" xlink:title=\"t_no_op = TNoOp(fun=None, name=&#39;no_action&#39;, datatypes=&#39;x&#39;, feat_constraints=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1097.89\" cy=\"-68.77\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-77.97\" font-family=\"Times,serif\" font-size=\"11.00\">T&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-65.97\" font-family=\"Times,serif\" font-size=\"11.00\">No&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-53.97\" font-family=\"Times,serif\" font-size=\"11.00\">Op</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_permute_array&#45;&gt;t_no_op -->\n",
              "<g id=\"edge16\" class=\"edge\">\n",
              "<title>numpy_permute_array&#45;&gt;t_no_op</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1034.96,-68.77C1043.21,-68.77 1052.07,-68.77 1060.43,-68.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1060.68,-72.27 1070.68,-68.77 1060.68,-65.27 1060.68,-72.27\"/>\n",
              "</g>\n",
              "</g>\n",
              "</svg>\n"
            ],
            "text/plain": [
              "<graphviz.dot.Digraph at 0x7fd8cfc8a990>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prefix = hist_pipeline.remove_last().freeze_trainable()\n",
        "prefix.visualize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Next, add a new final step, which consists of a choice of three\n",
        "estimators. In this code, `|` is the or combinator (algorithmic\n",
        "choice). It defines a search space for another optimizer run."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "image/svg+xml": [
              "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
              "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
              " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
              "<!-- Generated by graphviz version 2.44.1 (20200629.0846)\n",
              " -->\n",
              "<!-- Title: cluster:(root) Pages: 1 -->\n",
              "<svg width=\"1239pt\" height=\"200pt\"\n",
              " viewBox=\"0.00 0.00 1238.89 199.77\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
              "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 195.77)\">\n",
              "<title>cluster:(root)</title>\n",
              "<g id=\"a_graph0\"><a xlink:title=\"(root) = ...\">\n",
              "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-195.77 1234.89,-195.77 1234.89,4 -4,4\"/>\n",
              "</a>\n",
              "</g>\n",
              "<g id=\"clust1\" class=\"cluster\">\n",
              "<title>cluster:choice</title>\n",
              "<g id=\"a_clust1\"><a xlink:title=\"choice = lr | tree | knn\">\n",
              "<polygon fill=\"#7ec0ee\" stroke=\"black\" points=\"1152.89,-8 1152.89,-169 1222.89,-169 1222.89,-8 1152.89,-8\"/>\n",
              "<text text-anchor=\"middle\" x=\"1187.89\" y=\"-153.8\" font-family=\"Times,serif\" font-size=\"14.00\">Choice</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0 -->\n",
              "<g id=\"node1\" class=\"node\">\n",
              "<title>numpy_column_selector_0</title>\n",
              "<g id=\"a_node1\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_0 = NumpyColumnSelector(columns=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"33.23\" cy=\"-155\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-164.2\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-152.2\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-140.2\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings -->\n",
              "<g id=\"node2\" class=\"node\">\n",
              "<title>compress_strings</title>\n",
              "<g id=\"a_node2\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.compress_strings.html\" xlink:title=\"compress_strings = CompressStrings(compress_type=&#39;hash&#39;, dtypes_list=[&#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_s...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"142.77\" cy=\"-155\" rx=\"40.11\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-158.2\" font-family=\"Times,serif\" font-size=\"11.00\">Compress&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-146.2\" font-family=\"Times,serif\" font-size=\"11.00\">Strings</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0&#45;&gt;compress_strings -->\n",
              "<g id=\"edge1\" class=\"edge\">\n",
              "<title>numpy_column_selector_0&#45;&gt;compress_strings</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M66.5,-155C74.59,-155 83.46,-155 92.19,-155\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"92.3,-158.5 102.3,-155 92.3,-151.5 92.3,-158.5\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0 -->\n",
              "<g id=\"node3\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_0</title>\n",
              "<g id=\"a_node3\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_0 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-155\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-170.2\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-158.2\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-146.2\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-134.2\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings&#45;&gt;numpy_replace_missing_values_0 -->\n",
              "<g id=\"edge2\" class=\"edge\">\n",
              "<title>compress_strings&#45;&gt;numpy_replace_missing_values_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M183.11,-155C191.43,-155 200.26,-155 208.74,-155\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"208.85,-158.5 218.85,-155 208.85,-151.5 208.85,-158.5\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values -->\n",
              "<g id=\"node4\" class=\"node\">\n",
              "<title>numpy_replace_unknown_values</title>\n",
              "<g id=\"a_node4\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_unknown_values.html\" xlink:title=\"numpy_replace_unknown_values = NumpyReplaceUnknownValues(filling_values=float(&#39;nan&#39;), filling_values_list=[float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-155\" rx=\"38.78\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-170.2\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-158.2\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-146.2\" font-family=\"Times,serif\" font-size=\"11.00\">Unknown&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-134.2\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values -->\n",
              "<g id=\"edge3\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M288.46,-155C296.6,-155 305.47,-155 314.13,-155\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"314.16,-158.5 324.16,-155 314.16,-151.5 314.16,-158.5\"/>\n",
              "</g>\n",
              "<!-- boolean2float -->\n",
              "<g id=\"node5\" class=\"node\">\n",
              "<title>boolean2float</title>\n",
              "<g id=\"a_node5\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.boolean2float.html\" xlink:title=\"boolean2float = boolean2float()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-152\" rx=\"37.91\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-149.2\" font-family=\"Times,serif\" font-size=\"11.00\">boolean2float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values&#45;&gt;boolean2float -->\n",
              "<g id=\"edge4\" class=\"edge\">\n",
              "<title>numpy_replace_unknown_values&#45;&gt;boolean2float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M402.41,-153.97C410.6,-153.75 419.36,-153.51 427.85,-153.28\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"428.12,-156.77 438.02,-153 427.93,-149.78 428.12,-156.77\"/>\n",
              "</g>\n",
              "<!-- cat_imputer -->\n",
              "<g id=\"node6\" class=\"node\">\n",
              "<title>cat_imputer</title>\n",
              "<g id=\"a_node6\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_imputer.html\" xlink:title=\"cat_imputer = CatImputer(strategy=&#39;most_frequent&#39;, missing_values=float(&#39;nan&#39;), sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-150\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-153.2\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-141.2\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- boolean2float&#45;&gt;cat_imputer -->\n",
              "<g id=\"edge5\" class=\"edge\">\n",
              "<title>boolean2float&#45;&gt;cat_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M514.07,-151.28C522.45,-151.11 531.39,-150.94 539.88,-150.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"540.02,-154.27 549.95,-150.58 539.88,-147.27 540.02,-154.27\"/>\n",
              "</g>\n",
              "<!-- cat_encoder -->\n",
              "<g id=\"node7\" class=\"node\">\n",
              "<title>cat_encoder</title>\n",
              "<g id=\"a_node7\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_encoder.html\" xlink:title=\"cat_encoder = CatEncoder(encoding=&#39;ordinal&#39;, categories=&#39;auto&#39;, dtype=np.float64, handle_unknown=&#39;error&#39;, sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-149\" rx=\"31.64\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-152.2\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-140.2\" font-family=\"Times,serif\" font-size=\"11.00\">Encoder</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_imputer&#45;&gt;cat_encoder -->\n",
              "<g id=\"edge6\" class=\"edge\">\n",
              "<title>cat_imputer&#45;&gt;cat_encoder</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-149.71C620.35,-149.61 630.77,-149.51 640.7,-149.41\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"640.84,-152.91 650.8,-149.31 640.77,-145.91 640.84,-152.91\"/>\n",
              "</g>\n",
              "<!-- float32_transform_0 -->\n",
              "<g id=\"node8\" class=\"node\">\n",
              "<title>float32_transform_0</title>\n",
              "<g id=\"a_node8\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_0 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-146\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-149.2\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-137.2\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_encoder&#45;&gt;float32_transform_0 -->\n",
              "<g id=\"edge7\" class=\"edge\">\n",
              "<title>cat_encoder&#45;&gt;float32_transform_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M714.92,-148.12C724.18,-147.86 734.54,-147.57 744.53,-147.29\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.85,-150.78 754.75,-147 744.66,-143.78 744.85,-150.78\"/>\n",
              "</g>\n",
              "<!-- concat_features -->\n",
              "<g id=\"node15\" class=\"node\">\n",
              "<title>concat_features</title>\n",
              "<g id=\"a_node15\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.lale.concat_features.html\" xlink:title=\"concat_features = ConcatFeatures()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"896.36\" cy=\"-120\" rx=\"33.47\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-123.2\" font-family=\"Times,serif\" font-size=\"11.00\">Concat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-111.2\" font-family=\"Times,serif\" font-size=\"11.00\">Features</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float32_transform_0&#45;&gt;concat_features -->\n",
              "<g id=\"edge13\" class=\"edge\">\n",
              "<title>float32_transform_0&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.18,-137.92C834.15,-135.41 845.25,-132.61 855.66,-129.99\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"856.53,-133.38 865.38,-127.55 854.82,-126.59 856.53,-133.38\"/>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1 -->\n",
              "<g id=\"node9\" class=\"node\">\n",
              "<title>numpy_column_selector_1</title>\n",
              "<g id=\"a_node9\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_1 = NumpyColumnSelector(columns=[4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-83\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-92.2\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-80.2\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-68.2\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float -->\n",
              "<g id=\"node10\" class=\"node\">\n",
              "<title>float_str2_float</title>\n",
              "<g id=\"a_node10\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float_str2_float.html\" xlink:title=\"float_str2_float = FloatStr2Float(dtypes_list=[&#39;int_num&#39;], missing_values_reference_list=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-83\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-92.2\" font-family=\"Times,serif\" font-size=\"11.00\">Float&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-80.2\" font-family=\"Times,serif\" font-size=\"11.00\">Str2&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-68.2\" font-family=\"Times,serif\" font-size=\"11.00\">Float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1&#45;&gt;float_str2_float -->\n",
              "<g id=\"edge8\" class=\"edge\">\n",
              "<title>numpy_column_selector_1&#45;&gt;float_str2_float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M286.99,-83C299.27,-83 313.34,-83 325.96,-83\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"326.16,-86.5 336.16,-83 326.16,-79.5 326.16,-86.5\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1 -->\n",
              "<g id=\"node11\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_1</title>\n",
              "<g id=\"a_node11\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_1 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-88\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-103.2\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-91.2\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-79.2\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-67.2\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float&#45;&gt;numpy_replace_missing_values_1 -->\n",
              "<g id=\"edge9\" class=\"edge\">\n",
              "<title>float_str2_float&#45;&gt;numpy_replace_missing_values_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M390.25,-84.17C402.39,-84.72 417.18,-85.39 430.99,-86.01\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"431.15,-89.52 441.3,-86.47 431.47,-82.53 431.15,-89.52\"/>\n",
              "</g>\n",
              "<!-- num_imputer -->\n",
              "<g id=\"node12\" class=\"node\">\n",
              "<title>num_imputer</title>\n",
              "<g id=\"a_node12\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.num_imputer.html\" xlink:title=\"num_imputer = NumImputer(strategy=&#39;median&#39;, missing_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-91\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-94.2\" font-family=\"Times,serif\" font-size=\"11.00\">Num&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-82.2\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1&#45;&gt;num_imputer -->\n",
              "<g id=\"edge10\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_1&#45;&gt;num_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M510.91,-88.99C520.15,-89.26 530.24,-89.56 539.78,-89.84\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"539.74,-93.34 549.83,-90.13 539.94,-86.34 539.74,-93.34\"/>\n",
              "</g>\n",
              "<!-- opt_standard_scaler -->\n",
              "<g id=\"node13\" class=\"node\">\n",
              "<title>opt_standard_scaler</title>\n",
              "<g id=\"a_node13\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.opt_standard_scaler.html\" xlink:title=\"opt_standard_scaler = OptStandardScaler(num_scaler_copy=None, num_scaler_with_mean=None, num_scaler_with_std=None, use_scaler_flag=False)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-93\" rx=\"36.12\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-102.2\" font-family=\"Times,serif\" font-size=\"11.00\">Opt&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-90.2\" font-family=\"Times,serif\" font-size=\"11.00\">Standard&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-78.2\" font-family=\"Times,serif\" font-size=\"11.00\">Scaler</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- num_imputer&#45;&gt;opt_standard_scaler -->\n",
              "<g id=\"edge11\" class=\"edge\">\n",
              "<title>num_imputer&#45;&gt;opt_standard_scaler</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-91.59C619.14,-91.75 628.07,-91.93 636.8,-92.1\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"636.81,-95.6 646.88,-92.3 636.95,-88.6 636.81,-95.6\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1 -->\n",
              "<g id=\"node14\" class=\"node\">\n",
              "<title>float32_transform_1</title>\n",
              "<g id=\"a_node14\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_1 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-97\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-100.2\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-88.2\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- opt_standard_scaler&#45;&gt;float32_transform_1 -->\n",
              "<g id=\"edge12\" class=\"edge\">\n",
              "<title>opt_standard_scaler&#45;&gt;float32_transform_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M718.98,-94.32C727.31,-94.63 736.32,-94.97 745.03,-95.3\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.93,-98.8 755.06,-95.68 745.2,-91.81 744.93,-98.8\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1&#45;&gt;concat_features -->\n",
              "<g id=\"edge14\" class=\"edge\">\n",
              "<title>float32_transform_1&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.75,-104.28C834.41,-106.43 845.1,-108.81 855.18,-111.05\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"854.47,-114.48 865,-113.24 856,-107.65 854.47,-114.48\"/>\n",
              "</g>\n",
              "<!-- numpy_permute_array -->\n",
              "<g id=\"node16\" class=\"node\">\n",
              "<title>numpy_permute_array</title>\n",
              "<g id=\"a_node16\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_permute_array.html\" xlink:title=\"numpy_permute_array = NumpyPermuteArray(axis=0, permutation_indices=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1000.24\" cy=\"-120\" rx=\"34.8\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-129.2\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-117.2\" font-family=\"Times,serif\" font-size=\"11.00\">Permute&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-105.2\" font-family=\"Times,serif\" font-size=\"11.00\">Array</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- concat_features&#45;&gt;numpy_permute_array -->\n",
              "<g id=\"edge15\" class=\"edge\">\n",
              "<title>concat_features&#45;&gt;numpy_permute_array</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M929.6,-120C937.81,-120 946.78,-120 955.46,-120\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"955.46,-123.5 965.46,-120 955.46,-116.5 955.46,-123.5\"/>\n",
              "</g>\n",
              "<!-- t_no_op -->\n",
              "<g id=\"node17\" class=\"node\">\n",
              "<title>t_no_op</title>\n",
              "<g id=\"a_node17\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.t_no_op.html\" xlink:title=\"t_no_op = TNoOp(fun=None, name=&#39;no_action&#39;, datatypes=&#39;x&#39;, feat_constraints=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1097.89\" cy=\"-120\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-129.2\" font-family=\"Times,serif\" font-size=\"11.00\">T&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-117.2\" font-family=\"Times,serif\" font-size=\"11.00\">No&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-105.2\" font-family=\"Times,serif\" font-size=\"11.00\">Op</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_permute_array&#45;&gt;t_no_op -->\n",
              "<g id=\"edge16\" class=\"edge\">\n",
              "<title>numpy_permute_array&#45;&gt;t_no_op</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1034.96,-120C1043.21,-120 1052.07,-120 1060.43,-120\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1060.68,-123.5 1070.68,-120 1060.68,-116.5 1060.68,-123.5\"/>\n",
              "</g>\n",
              "<!-- lr -->\n",
              "<g id=\"node18\" class=\"node\">\n",
              "<title>lr</title>\n",
              "<g id=\"a_node18\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.sklearn.logistic_regression.html\" xlink:title=\"lr = LR\">\n",
              "<ellipse fill=\"#7ec0ee\" stroke=\"black\" cx=\"1187.89\" cy=\"-120\" rx=\"27\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"1187.89\" y=\"-117.2\" font-family=\"Times,serif\" font-size=\"11.00\">LR</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- t_no_op&#45;&gt;lr -->\n",
              "<g id=\"edge17\" class=\"edge\">\n",
              "<title>t_no_op&#45;&gt;lr</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1125.29,-120C1133.28,-120 1142.2,-120 1150.72,-120\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1142.89,-123.5 1152.89,-120 1142.89,-116.5 1142.89,-123.5\"/>\n",
              "</g>\n",
              "<!-- tree -->\n",
              "<g id=\"node19\" class=\"node\">\n",
              "<title>tree</title>\n",
              "<g id=\"a_node19\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.sklearn.decision_tree_classifier.html\" xlink:title=\"tree = Tree\">\n",
              "<ellipse fill=\"#7ec0ee\" stroke=\"black\" cx=\"1187.89\" cy=\"-77\" rx=\"27\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"1187.89\" y=\"-74.2\" font-family=\"Times,serif\" font-size=\"11.00\">Tree</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- knn -->\n",
              "<g id=\"node20\" class=\"node\">\n",
              "<title>knn</title>\n",
              "<g id=\"a_node20\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.sklearn.k_neighbors_classifier.html\" xlink:title=\"knn = KNN\">\n",
              "<ellipse fill=\"#7ec0ee\" stroke=\"black\" cx=\"1187.89\" cy=\"-34\" rx=\"27\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"1187.89\" y=\"-31.2\" font-family=\"Times,serif\" font-size=\"11.00\">KNN</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "</g>\n",
              "</svg>\n"
            ],
            "text/plain": [
              "<graphviz.dot.Digraph at 0x7fd8d30cb890>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "new_pipeline = prefix >> (LR | Tree | KNN)\n",
        "new_pipeline.visualize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### New optimizer `Hyperopt` configuration and training\n",
        "\n",
        "To automatically select the algorithm and tune its hyperparameters, we\n",
        "create an instance of the `Hyperopt` optimizer and `fit` it to the\n",
        "data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(train_X, train_y, test_size=0.15, random_state=33)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100%|██████████| 20/20 [02:12<00:00,  6.63s/trial, best loss: -0.8336535460343567]\n"
          ]
        }
      ],
      "source": [
        "hyperopt = Hyperopt(estimator=new_pipeline, cv=3, max_evals=20, scoring='roc_auc')\n",
        "hyperopt_pipelines = hyperopt.fit(X_train, y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {},
      "outputs": [],
      "source": [
        "pipeline_model = hyperopt_pipelines.get_pipeline()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Pipeline model tests and visualization"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "roc_auc_score 75.1%\n"
          ]
        }
      ],
      "source": [
        "from sklearn.metrics import roc_auc_score\n",
        "\n",
        "predicted_y = pipeline_model.predict(X_test)\n",
        "score = roc_auc_score(predicted_y=='Risk', y_test=='Risk')\n",
        "print(f'roc_auc_score {score:.1%}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "image/svg+xml": [
              "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
              "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
              " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
              "<!-- Generated by graphviz version 2.44.1 (20200629.0846)\n",
              " -->\n",
              "<!-- Title: cluster:(root) Pages: 1 -->\n",
              "<svg width=\"1223pt\" height=\"149pt\"\n",
              " viewBox=\"0.00 0.00 1222.89 148.54\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
              "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 144.54)\">\n",
              "<title>cluster:(root)</title>\n",
              "<g id=\"a_graph0\"><a xlink:title=\"(root) = ...\">\n",
              "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-144.54 1218.89,-144.54 1218.89,4 -4,4\"/>\n",
              "</a>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0 -->\n",
              "<g id=\"node1\" class=\"node\">\n",
              "<title>numpy_column_selector_0</title>\n",
              "<g id=\"a_node1\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_0 = NumpyColumnSelector(columns=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"33.23\" cy=\"-103.77\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-112.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-100.97\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-88.97\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings -->\n",
              "<g id=\"node2\" class=\"node\">\n",
              "<title>compress_strings</title>\n",
              "<g id=\"a_node2\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.compress_strings.html\" xlink:title=\"compress_strings = CompressStrings(compress_type=&#39;hash&#39;, dtypes_list=[&#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_s...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"142.77\" cy=\"-103.77\" rx=\"40.11\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Compress&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Strings</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0&#45;&gt;compress_strings -->\n",
              "<g id=\"edge1\" class=\"edge\">\n",
              "<title>numpy_column_selector_0&#45;&gt;compress_strings</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M66.5,-103.77C74.59,-103.77 83.46,-103.77 92.19,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"92.3,-107.27 102.3,-103.77 92.3,-100.27 92.3,-107.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0 -->\n",
              "<g id=\"node3\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_0</title>\n",
              "<g id=\"a_node3\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_0 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-103.77\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-118.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-82.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings&#45;&gt;numpy_replace_missing_values_0 -->\n",
              "<g id=\"edge2\" class=\"edge\">\n",
              "<title>compress_strings&#45;&gt;numpy_replace_missing_values_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M183.11,-103.77C191.43,-103.77 200.26,-103.77 208.74,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"208.85,-107.27 218.85,-103.77 208.85,-100.27 208.85,-107.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values -->\n",
              "<g id=\"node4\" class=\"node\">\n",
              "<title>numpy_replace_unknown_values</title>\n",
              "<g id=\"a_node4\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_unknown_values.html\" xlink:title=\"numpy_replace_unknown_values = NumpyReplaceUnknownValues(filling_values=float(&#39;nan&#39;), filling_values_list=[float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-103.77\" rx=\"38.78\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-118.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Unknown&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-82.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values -->\n",
              "<g id=\"edge3\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M288.46,-103.77C296.6,-103.77 305.47,-103.77 314.13,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"314.16,-107.27 324.16,-103.77 314.16,-100.27 314.16,-107.27\"/>\n",
              "</g>\n",
              "<!-- boolean2float -->\n",
              "<g id=\"node5\" class=\"node\">\n",
              "<title>boolean2float</title>\n",
              "<g id=\"a_node5\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.boolean2float.html\" xlink:title=\"boolean2float = boolean2float()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-100.77\" rx=\"37.91\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-97.97\" font-family=\"Times,serif\" font-size=\"11.00\">boolean2float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values&#45;&gt;boolean2float -->\n",
              "<g id=\"edge4\" class=\"edge\">\n",
              "<title>numpy_replace_unknown_values&#45;&gt;boolean2float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M402.41,-102.74C410.6,-102.52 419.36,-102.28 427.85,-102.05\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"428.12,-105.54 438.02,-101.77 427.93,-98.55 428.12,-105.54\"/>\n",
              "</g>\n",
              "<!-- cat_imputer -->\n",
              "<g id=\"node6\" class=\"node\">\n",
              "<title>cat_imputer</title>\n",
              "<g id=\"a_node6\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_imputer.html\" xlink:title=\"cat_imputer = CatImputer(strategy=&#39;most_frequent&#39;, missing_values=float(&#39;nan&#39;), sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-98.77\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-101.97\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-89.97\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- boolean2float&#45;&gt;cat_imputer -->\n",
              "<g id=\"edge5\" class=\"edge\">\n",
              "<title>boolean2float&#45;&gt;cat_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M514.07,-100.05C522.45,-99.88 531.39,-99.71 539.88,-99.54\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"540.02,-103.04 549.95,-99.35 539.88,-96.04 540.02,-103.04\"/>\n",
              "</g>\n",
              "<!-- cat_encoder -->\n",
              "<g id=\"node7\" class=\"node\">\n",
              "<title>cat_encoder</title>\n",
              "<g id=\"a_node7\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_encoder.html\" xlink:title=\"cat_encoder = CatEncoder(encoding=&#39;ordinal&#39;, categories=&#39;auto&#39;, dtype=np.float64, handle_unknown=&#39;error&#39;, sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-97.77\" rx=\"31.64\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-100.97\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-88.97\" font-family=\"Times,serif\" font-size=\"11.00\">Encoder</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_imputer&#45;&gt;cat_encoder -->\n",
              "<g id=\"edge6\" class=\"edge\">\n",
              "<title>cat_imputer&#45;&gt;cat_encoder</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-98.47C620.35,-98.38 630.77,-98.28 640.7,-98.18\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"640.84,-101.68 650.8,-98.08 640.77,-94.68 640.84,-101.68\"/>\n",
              "</g>\n",
              "<!-- float32_transform_0 -->\n",
              "<g id=\"node8\" class=\"node\">\n",
              "<title>float32_transform_0</title>\n",
              "<g id=\"a_node8\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_0 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-94.77\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-97.97\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-85.97\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_encoder&#45;&gt;float32_transform_0 -->\n",
              "<g id=\"edge7\" class=\"edge\">\n",
              "<title>cat_encoder&#45;&gt;float32_transform_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M714.92,-96.89C724.18,-96.63 734.54,-96.34 744.53,-96.06\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.85,-99.55 754.75,-95.77 744.66,-92.55 744.85,-99.55\"/>\n",
              "</g>\n",
              "<!-- concat_features -->\n",
              "<g id=\"node15\" class=\"node\">\n",
              "<title>concat_features</title>\n",
              "<g id=\"a_node15\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.lale.concat_features.html\" xlink:title=\"concat_features = ConcatFeatures()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"896.36\" cy=\"-68.77\" rx=\"33.47\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-71.97\" font-family=\"Times,serif\" font-size=\"11.00\">Concat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-59.97\" font-family=\"Times,serif\" font-size=\"11.00\">Features</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float32_transform_0&#45;&gt;concat_features -->\n",
              "<g id=\"edge13\" class=\"edge\">\n",
              "<title>float32_transform_0&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.18,-86.69C834.15,-84.18 845.25,-81.38 855.66,-78.76\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"856.53,-82.15 865.38,-76.32 854.82,-75.36 856.53,-82.15\"/>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1 -->\n",
              "<g id=\"node9\" class=\"node\">\n",
              "<title>numpy_column_selector_1</title>\n",
              "<g id=\"a_node9\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_1 = NumpyColumnSelector(columns=[4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-31.77\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-40.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-28.97\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-16.97\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float -->\n",
              "<g id=\"node10\" class=\"node\">\n",
              "<title>float_str2_float</title>\n",
              "<g id=\"a_node10\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float_str2_float.html\" xlink:title=\"float_str2_float = FloatStr2Float(dtypes_list=[&#39;int_num&#39;], missing_values_reference_list=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-31.77\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-40.97\" font-family=\"Times,serif\" font-size=\"11.00\">Float&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-28.97\" font-family=\"Times,serif\" font-size=\"11.00\">Str2&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-16.97\" font-family=\"Times,serif\" font-size=\"11.00\">Float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1&#45;&gt;float_str2_float -->\n",
              "<g id=\"edge8\" class=\"edge\">\n",
              "<title>numpy_column_selector_1&#45;&gt;float_str2_float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M286.99,-31.77C299.27,-31.77 313.34,-31.77 325.96,-31.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"326.16,-35.27 336.16,-31.77 326.16,-28.27 326.16,-35.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1 -->\n",
              "<g id=\"node11\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_1</title>\n",
              "<g id=\"a_node11\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_1 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-36.77\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-51.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-39.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-27.97\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-15.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float&#45;&gt;numpy_replace_missing_values_1 -->\n",
              "<g id=\"edge9\" class=\"edge\">\n",
              "<title>float_str2_float&#45;&gt;numpy_replace_missing_values_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M390.25,-32.94C402.39,-33.49 417.18,-34.16 430.99,-34.78\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"431.15,-38.29 441.3,-35.24 431.47,-31.3 431.15,-38.29\"/>\n",
              "</g>\n",
              "<!-- num_imputer -->\n",
              "<g id=\"node12\" class=\"node\">\n",
              "<title>num_imputer</title>\n",
              "<g id=\"a_node12\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.num_imputer.html\" xlink:title=\"num_imputer = NumImputer(strategy=&#39;median&#39;, missing_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-39.77\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-42.97\" font-family=\"Times,serif\" font-size=\"11.00\">Num&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-30.97\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1&#45;&gt;num_imputer -->\n",
              "<g id=\"edge10\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_1&#45;&gt;num_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M510.91,-37.76C520.15,-38.03 530.24,-38.33 539.78,-38.61\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"539.74,-42.11 549.83,-38.9 539.94,-35.11 539.74,-42.11\"/>\n",
              "</g>\n",
              "<!-- opt_standard_scaler -->\n",
              "<g id=\"node13\" class=\"node\">\n",
              "<title>opt_standard_scaler</title>\n",
              "<g id=\"a_node13\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.opt_standard_scaler.html\" xlink:title=\"opt_standard_scaler = OptStandardScaler(num_scaler_copy=None, num_scaler_with_mean=None, num_scaler_with_std=None, use_scaler_flag=False)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-41.77\" rx=\"36.12\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-50.97\" font-family=\"Times,serif\" font-size=\"11.00\">Opt&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-38.97\" font-family=\"Times,serif\" font-size=\"11.00\">Standard&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-26.97\" font-family=\"Times,serif\" font-size=\"11.00\">Scaler</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- num_imputer&#45;&gt;opt_standard_scaler -->\n",
              "<g id=\"edge11\" class=\"edge\">\n",
              "<title>num_imputer&#45;&gt;opt_standard_scaler</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-40.36C619.14,-40.52 628.07,-40.7 636.8,-40.87\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"636.81,-44.37 646.88,-41.07 636.95,-37.37 636.81,-44.37\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1 -->\n",
              "<g id=\"node14\" class=\"node\">\n",
              "<title>float32_transform_1</title>\n",
              "<g id=\"a_node14\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_1 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-45.77\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-48.97\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-36.97\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- opt_standard_scaler&#45;&gt;float32_transform_1 -->\n",
              "<g id=\"edge12\" class=\"edge\">\n",
              "<title>opt_standard_scaler&#45;&gt;float32_transform_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M718.98,-43.09C727.31,-43.4 736.32,-43.74 745.03,-44.07\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.93,-47.57 755.06,-44.45 745.2,-40.58 744.93,-47.57\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1&#45;&gt;concat_features -->\n",
              "<g id=\"edge14\" class=\"edge\">\n",
              "<title>float32_transform_1&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.75,-53.05C834.41,-55.2 845.1,-57.58 855.18,-59.82\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"854.47,-63.25 865,-62.01 856,-56.42 854.47,-63.25\"/>\n",
              "</g>\n",
              "<!-- numpy_permute_array -->\n",
              "<g id=\"node16\" class=\"node\">\n",
              "<title>numpy_permute_array</title>\n",
              "<g id=\"a_node16\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_permute_array.html\" xlink:title=\"numpy_permute_array = NumpyPermuteArray(axis=0, permutation_indices=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1000.24\" cy=\"-68.77\" rx=\"34.8\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-77.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-65.97\" font-family=\"Times,serif\" font-size=\"11.00\">Permute&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-53.97\" font-family=\"Times,serif\" font-size=\"11.00\">Array</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- concat_features&#45;&gt;numpy_permute_array -->\n",
              "<g id=\"edge15\" class=\"edge\">\n",
              "<title>concat_features&#45;&gt;numpy_permute_array</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M929.6,-68.77C937.81,-68.77 946.78,-68.77 955.46,-68.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"955.46,-72.27 965.46,-68.77 955.46,-65.27 955.46,-72.27\"/>\n",
              "</g>\n",
              "<!-- t_no_op -->\n",
              "<g id=\"node17\" class=\"node\">\n",
              "<title>t_no_op</title>\n",
              "<g id=\"a_node17\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.t_no_op.html\" xlink:title=\"t_no_op = TNoOp(fun=None, name=&#39;no_action&#39;, datatypes=&#39;x&#39;, feat_constraints=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1097.89\" cy=\"-68.77\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-77.97\" font-family=\"Times,serif\" font-size=\"11.00\">T&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-65.97\" font-family=\"Times,serif\" font-size=\"11.00\">No&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-53.97\" font-family=\"Times,serif\" font-size=\"11.00\">Op</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_permute_array&#45;&gt;t_no_op -->\n",
              "<g id=\"edge16\" class=\"edge\">\n",
              "<title>numpy_permute_array&#45;&gt;t_no_op</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1034.96,-68.77C1043.21,-68.77 1052.07,-68.77 1060.43,-68.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1060.68,-72.27 1070.68,-68.77 1060.68,-65.27 1060.68,-72.27\"/>\n",
              "</g>\n",
              "<!-- lr -->\n",
              "<g id=\"node18\" class=\"node\">\n",
              "<title>lr</title>\n",
              "<g id=\"a_node18\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.sklearn.logistic_regression.html\" xlink:title=\"lr = LR(intercept_scaling=0.02888452505133443, max_iter=737, multi_class=&#39;multinomial&#39;, tol=0.006400103615640232)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1187.89\" cy=\"-68.77\" rx=\"27\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"1187.89\" y=\"-65.97\" font-family=\"Times,serif\" font-size=\"11.00\">LR</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- t_no_op&#45;&gt;lr -->\n",
              "<g id=\"edge17\" class=\"edge\">\n",
              "<title>t_no_op&#45;&gt;lr</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1125.29,-68.77C1133.28,-68.77 1142.2,-68.77 1150.72,-68.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1150.81,-72.27 1160.81,-68.77 1150.81,-65.27 1150.81,-72.27\"/>\n",
              "</g>\n",
              "</g>\n",
              "</svg>\n"
            ],
            "text/plain": [
              "<graphviz.dot.Digraph at 0x7fd8d2fb7c90>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "pipeline_model.visualize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"scoring\"></a>\n",
        "## 7. Deploy and Score\n",
        "\n",
        "In this section you will learn how to deploy and score pipeline model as webservice and batch using WML instance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Webservice deployment creation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Preparing an AutoAI Deployment...\n",
            "Published model uid: 014ffcfa-dcf1-4c26-add5-369243b140dd\n",
            "Deploying model 014ffcfa-dcf1-4c26-add5-369243b140dd using V4 client.\n",
            "\n",
            "\n",
            "#######################################################################################\n",
            "\n",
            "Synchronous deployment creation for uid: '014ffcfa-dcf1-4c26-add5-369243b140dd' started\n",
            "\n",
            "#######################################################################################\n",
            "\n",
            "\n",
            "initializing..\n",
            "ready\n",
            "\n",
            "\n",
            "------------------------------------------------------------------------------------------------\n",
            "Successfully finished deployment creation, deployment_uid='e0f705fd-2fcb-437f-bc83-b17d5296b01d'\n",
            "------------------------------------------------------------------------------------------------\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "from ibm_watson_machine_learning.deployment import WebService\n",
        "\n",
        "service = WebService(wml_credentials, source_space_id=space_id)\n",
        "\n",
        "service.create(\n",
        "    experiment_run_id=run_id,\n",
        "    model='Pipeline_1', \n",
        "    deployment_name=\"Credit Risk Deployment AutoAI\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Deployment object could be printed to show basic information:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "name: Credit Risk Deployment AutoAI, id: e0f705fd-2fcb-437f-bc83-b17d5296b01d, scoring_url: https://cpd-wmlautoai-may12.apps.ocp46wmlautoaai.cp.fyre.ibm.com/ml/v4/deployments/e0f705fd-2fcb-437f-bc83-b17d5296b01d/predictions, asset_id: 014ffcfa-dcf1-4c26-add5-369243b140dd\n"
          ]
        }
      ],
      "source": [
        "print(service)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To show all available information about the deployment use the `.get_params()` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'entity': {'asset': {'id': '014ffcfa-dcf1-4c26-add5-369243b140dd'},\n",
              "  'custom': {},\n",
              "  'deployed_asset_type': 'model',\n",
              "  'hardware_spec': {'id': 'c076e82c-b2a7-4d20-9c0f-1f0c2fdf5a24',\n",
              "   'name': 'M',\n",
              "   'num_nodes': 1},\n",
              "  'hybrid_pipeline_hardware_specs': [{'hardware_spec': {'name': 'S',\n",
              "     'num_nodes': 1},\n",
              "    'node_runtime_id': 'auto_ai.kb'}],\n",
              "  'name': 'Credit Risk Deployment AutoAI',\n",
              "  'online': {},\n",
              "  'space_id': 'fbd2dc3a-634a-46d5-9edd-8faa22d27b8d',\n",
              "  'status': {'online_url': {'url': 'https://cpd-wmlautoai-may12.apps.ocp46wmlautoaai.cp.fyre.ibm.com/ml/v4/deployments/e0f705fd-2fcb-437f-bc83-b17d5296b01d/predictions'},\n",
              "   'state': 'ready'}},\n",
              " 'metadata': {'created_at': '2021-05-20T08:23:53.001Z',\n",
              "  'id': 'e0f705fd-2fcb-437f-bc83-b17d5296b01d',\n",
              "  'modified_at': '2021-05-20T08:23:53.001Z',\n",
              "  'name': 'Credit Risk Deployment AutoAI',\n",
              "  'owner': '1000330999',\n",
              "  'space_id': 'fbd2dc3a-634a-46d5-9edd-8faa22d27b8d'}}"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "service.get_params()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Scoring of webservice\n",
        "You can make scoring request by calling `score()` on deployed pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'predictions': [{'fields': ['prediction', 'probability'],\n",
              "   'values': [['No Risk', [0.6743601502735059, 0.32563984972649407]],\n",
              "    ['No Risk', [0.7892372484049005, 0.21076275159509952]],\n",
              "    ['No Risk', [0.560817617610811, 0.43918238238918905]],\n",
              "    ['No Risk', [0.8068670237279646, 0.1931329762720355]],\n",
              "    ['Risk', [0.3344503240926395, 0.6655496759073605]],\n",
              "    ['Risk', [0.10565678467912125, 0.8943432153208788]],\n",
              "    ['No Risk', [0.7361802572321648, 0.2638197427678351]],\n",
              "    ['No Risk', [0.8315661007516614, 0.1684338992483386]],\n",
              "    ['No Risk', [0.8599408631598607, 0.1400591368401393]],\n",
              "    ['Risk', [0.12661051706745757, 0.8733894829325424]]]}]}"
            ]
          },
          "execution_count": 43,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "predictions = service.score(payload=train_df.drop(['Risk'], axis=1).iloc[:10])\n",
        "predictions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you want to work with the web service in an external Python application, you can retrieve the service object:\n",
        " - Initialize the service using `service = WebService(wml_credentials)`\n",
        " - Get deployment_id using the `service.list()` method\n",
        " - Get webservice object using the `service.get('deployment_id')` method\n",
        "\n",
        "After that you can call `service.score()` method."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Deleting deployment\n",
        "You can delete the existing deployment by calling the `service.delete()` command.\n",
        "To list the existing web services you can use `service.list()`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Batch deployment creation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "A batch deployment processes input data from a inline data and return predictions in scoring details or processes from data asset and writes the output to a file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CheckingStatus</th>\n",
              "      <th>LoanDuration</th>\n",
              "      <th>CreditHistory</th>\n",
              "      <th>LoanPurpose</th>\n",
              "      <th>LoanAmount</th>\n",
              "      <th>ExistingSavings</th>\n",
              "      <th>EmploymentDuration</th>\n",
              "      <th>InstallmentPercent</th>\n",
              "      <th>Sex</th>\n",
              "      <th>OthersOnLoan</th>\n",
              "      <th>CurrentResidenceDuration</th>\n",
              "      <th>OwnsProperty</th>\n",
              "      <th>Age</th>\n",
              "      <th>InstallmentPlans</th>\n",
              "      <th>Housing</th>\n",
              "      <th>ExistingCreditsCount</th>\n",
              "      <th>Job</th>\n",
              "      <th>Dependents</th>\n",
              "      <th>Telephone</th>\n",
              "      <th>ForeignWorker</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0_to_200</td>\n",
              "      <td>31</td>\n",
              "      <td>credits_paid_to_date</td>\n",
              "      <td>other</td>\n",
              "      <td>1889</td>\n",
              "      <td>100_to_500</td>\n",
              "      <td>less_1</td>\n",
              "      <td>3</td>\n",
              "      <td>female</td>\n",
              "      <td>none</td>\n",
              "      <td>3</td>\n",
              "      <td>savings_insurance</td>\n",
              "      <td>32</td>\n",
              "      <td>none</td>\n",
              "      <td>own</td>\n",
              "      <td>1</td>\n",
              "      <td>skilled</td>\n",
              "      <td>1</td>\n",
              "      <td>none</td>\n",
              "      <td>yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>less_0</td>\n",
              "      <td>18</td>\n",
              "      <td>credits_paid_to_date</td>\n",
              "      <td>car_new</td>\n",
              "      <td>462</td>\n",
              "      <td>less_100</td>\n",
              "      <td>1_to_4</td>\n",
              "      <td>2</td>\n",
              "      <td>female</td>\n",
              "      <td>none</td>\n",
              "      <td>2</td>\n",
              "      <td>savings_insurance</td>\n",
              "      <td>37</td>\n",
              "      <td>stores</td>\n",
              "      <td>own</td>\n",
              "      <td>2</td>\n",
              "      <td>skilled</td>\n",
              "      <td>1</td>\n",
              "      <td>none</td>\n",
              "      <td>yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>less_0</td>\n",
              "      <td>15</td>\n",
              "      <td>prior_payments_delayed</td>\n",
              "      <td>furniture</td>\n",
              "      <td>250</td>\n",
              "      <td>less_100</td>\n",
              "      <td>1_to_4</td>\n",
              "      <td>2</td>\n",
              "      <td>male</td>\n",
              "      <td>none</td>\n",
              "      <td>3</td>\n",
              "      <td>real_estate</td>\n",
              "      <td>28</td>\n",
              "      <td>none</td>\n",
              "      <td>own</td>\n",
              "      <td>2</td>\n",
              "      <td>skilled</td>\n",
              "      <td>1</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0_to_200</td>\n",
              "      <td>28</td>\n",
              "      <td>credits_paid_to_date</td>\n",
              "      <td>retraining</td>\n",
              "      <td>3693</td>\n",
              "      <td>less_100</td>\n",
              "      <td>greater_7</td>\n",
              "      <td>3</td>\n",
              "      <td>male</td>\n",
              "      <td>none</td>\n",
              "      <td>2</td>\n",
              "      <td>savings_insurance</td>\n",
              "      <td>32</td>\n",
              "      <td>none</td>\n",
              "      <td>own</td>\n",
              "      <td>1</td>\n",
              "      <td>skilled</td>\n",
              "      <td>1</td>\n",
              "      <td>none</td>\n",
              "      <td>yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>no_checking</td>\n",
              "      <td>28</td>\n",
              "      <td>prior_payments_delayed</td>\n",
              "      <td>education</td>\n",
              "      <td>6235</td>\n",
              "      <td>500_to_1000</td>\n",
              "      <td>greater_7</td>\n",
              "      <td>3</td>\n",
              "      <td>male</td>\n",
              "      <td>none</td>\n",
              "      <td>3</td>\n",
              "      <td>unknown</td>\n",
              "      <td>57</td>\n",
              "      <td>none</td>\n",
              "      <td>own</td>\n",
              "      <td>2</td>\n",
              "      <td>skilled</td>\n",
              "      <td>1</td>\n",
              "      <td>none</td>\n",
              "      <td>yes</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  CheckingStatus  LoanDuration           CreditHistory LoanPurpose  \\\n",
              "0       0_to_200            31    credits_paid_to_date       other   \n",
              "1         less_0            18    credits_paid_to_date     car_new   \n",
              "2         less_0            15  prior_payments_delayed   furniture   \n",
              "3       0_to_200            28    credits_paid_to_date  retraining   \n",
              "4    no_checking            28  prior_payments_delayed   education   \n",
              "\n",
              "   LoanAmount ExistingSavings EmploymentDuration  InstallmentPercent     Sex  \\\n",
              "0        1889      100_to_500             less_1                   3  female   \n",
              "1         462        less_100             1_to_4                   2  female   \n",
              "2         250        less_100             1_to_4                   2    male   \n",
              "3        3693        less_100          greater_7                   3    male   \n",
              "4        6235     500_to_1000          greater_7                   3    male   \n",
              "\n",
              "  OthersOnLoan  CurrentResidenceDuration       OwnsProperty  Age  \\\n",
              "0         none                         3  savings_insurance   32   \n",
              "1         none                         2  savings_insurance   37   \n",
              "2         none                         3        real_estate   28   \n",
              "3         none                         2  savings_insurance   32   \n",
              "4         none                         3            unknown   57   \n",
              "\n",
              "  InstallmentPlans Housing  ExistingCreditsCount      Job  Dependents  \\\n",
              "0             none     own                     1  skilled           1   \n",
              "1           stores     own                     2  skilled           1   \n",
              "2             none     own                     2  skilled           1   \n",
              "3             none     own                     1  skilled           1   \n",
              "4             none     own                     2  skilled           1   \n",
              "\n",
              "  Telephone ForeignWorker  \n",
              "0      none           yes  \n",
              "1      none           yes  \n",
              "2       yes            no  \n",
              "3      none           yes  \n",
              "4      none           yes  "
            ]
          },
          "execution_count": 44,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "batch_payload_df = train_df.drop(['Risk'], axis=1)[:5]\n",
        "batch_payload_df"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create batch deployment for `Pipeline_2` created in AutoAI experiment with the `run_id`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Preparing an AutoAI Deployment...\n",
            "Published model uid: 35a98cac-c3b7-448e-9a02-245400c93a82\n",
            "Deploying model 35a98cac-c3b7-448e-9a02-245400c93a82 using V4 client.\n",
            "\n",
            "\n",
            "#######################################################################################\n",
            "\n",
            "Synchronous deployment creation for uid: '35a98cac-c3b7-448e-9a02-245400c93a82' started\n",
            "\n",
            "#######################################################################################\n",
            "\n",
            "\n",
            "ready.\n",
            "\n",
            "\n",
            "------------------------------------------------------------------------------------------------\n",
            "Successfully finished deployment creation, deployment_uid='756558ad-78f1-4d74-9e6d-555aa9cfff21'\n",
            "------------------------------------------------------------------------------------------------\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "from ibm_watson_machine_learning.deployment import Batch\n",
        "\n",
        "service_batch = Batch(wml_credentials,source_space_id=space_id)\n",
        "service_batch.create(\n",
        "            experiment_run_id=run_id,\n",
        "            model=\"Pipeline_2\",\n",
        "            deployment_name=\"Credit Risk Batch Deployment AutoAI\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Score batch deployment with inline payload as pandas DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "##########################################################################\n",
            "\n",
            "Synchronous scoring for id: '478b6b5b-a80f-43f5-98f9-63e9da889720' started\n",
            "\n",
            "##########################################################################\n",
            "\n",
            "\n",
            "queued..\n",
            "running\n",
            "completed\n",
            "Scoring job  '478b6b5b-a80f-43f5-98f9-63e9da889720' finished successfully.\n"
          ]
        }
      ],
      "source": [
        "scoring_params = service_batch.run_job(\n",
        "            payload=batch_payload_df,\n",
        "            background_mode=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 47,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'fields': ['prediction', 'probability'],\n",
              "  'values': [['No Risk', [0.7743273377418518, 0.225672647356987]],\n",
              "   ['No Risk', [0.8309863209724426, 0.16901369392871857]],\n",
              "   ['No Risk', [0.8892747759819031, 0.11072523891925812]],\n",
              "   ['No Risk', [0.7456686496734619, 0.2543313801288605]],\n",
              "   ['Risk', [0.22297507524490356, 0.7770249247550964]]]}]"
            ]
          },
          "execution_count": 47,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "scoring_params['entity']['scoring'].get('predictions')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Score batch deployment with payload as connected asset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Simmilary to training use created connection in order to locate tabe in used database."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 48,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Creating data asset...\n",
            "SUCCESS\n"
          ]
        }
      ],
      "source": [
        "from ibm_watson_machine_learning.helpers.connections import AssetLocation, DeploymentOutputAssetLocation\n",
        "\n",
        "\n",
        "batch_payload_filename = \"credit_risk_batch_payload.csv\"\n",
        "batch_payload_df.to_csv(batch_payload_filename, index=False)\n",
        "\n",
        "asset_details = client.data_assets.create(\n",
        "            name=batch_payload_filename,\n",
        "            file_path=batch_payload_filename)\n",
        "asset_id = client.data_assets.get_id(asset_details)\n",
        "\n",
        "payload_reference = DataConnection(data_asset_id=asset_id)\n",
        "results_reference = DataConnection(\n",
        "            location=DeploymentOutputAssetLocation(name=\"batch_output_credit_risk.csv\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Run scoring job for batch deployment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "##########################################################################\n",
            "\n",
            "Synchronous scoring for id: '80664cc6-3bf8-4fae-9309-f8622972a119' started\n",
            "\n",
            "##########################################################################\n",
            "\n",
            "\n",
            "queued.........\n",
            "running.\n",
            "completed\n",
            "Scoring job  '80664cc6-3bf8-4fae-9309-f8622972a119' finished successfully.\n"
          ]
        }
      ],
      "source": [
        "scoring_params = service_batch.run_job(\n",
        "            payload=[payload_reference],\n",
        "            output_data_reference=results_reference,\n",
        "            background_mode=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Deleting deployment\n",
        "You can delete the existing deployment by calling the `service_batch.delete()` command.\n",
        "To list the existing: \n",
        "- batch services you can use `service_batch.list()`, \n",
        "- scoring jobs you can use `service_batch.list_jobs()`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"cleanup\"></a>\n",
        "## 8. Clean up "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you want to clean up all created assets:\n",
        "- experiments\n",
        "- trainings\n",
        "- pipelines\n",
        "- model definitions\n",
        "- models\n",
        "- functions\n",
        "- deployments\n",
        "\n",
        "please follow up this sample [notebook](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd4.6/notebooks/python_sdk/instance-management/Machine%20Learning%20artifacts%20management.ipynb)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a id=\"summary\"></a>\n",
        "## 9. Summary and next steps\n",
        "\n",
        " You successfully completed this notebook!.\n",
        " \n",
        " You learned how to use `ibm-watson-machine-learning` to run AutoAI experiments. \n",
        " \n",
        " Check out our _[Online Documentation](https://www.ibm.com/cloud/watson-studio/autoai)_ for more samples, tutorials, documentation, how-tos, and blog posts. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Authors\n",
        "\n",
        "**Lukasz Cmielowski**, PhD, is an Automation Architect and Data Scientist at IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.\n",
        "\n",
        "**Amadeusz Masny**, Python Software Developer in Watson Machine Learning at IBM\n",
        "\n",
        "**Kiran Kate**, Senior Software Engineer at IBM Research AI\n",
        "\n",
        "**Martin Hirzel**, Research Staff Member and Manager at IBM Research AI\n",
        "\n",
        "**Jan Sołtysik**, Intern in Watson Machine Learning "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Copyright © 2020-2025 IBM. This notebook and its source code are released under the terms of the MIT License."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}