cpd5.1/notebooks/python_sdk/experiments/autoai/Use AutoAI and Lale to predict credit risk.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "# Use AutoAI and Lale to predict credit risk with `ibm-watsonx-ai`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "This notebook contains the steps and code to demonstrate support of AutoAI experiments in Watson Machine Learning service. It introduces commands for data retrieval, training experiments, persisting pipelines, testing pipelines, refining pipelines, and scoring.\n",
        "\n",
        "Some familiarity with Python is helpful. This notebook uses Python 3.11.\n",
        "\n",
        "\n",
        "## Learning goals\n",
        "\n",
        "The learning goals of this notebook are:\n",
        "\n",
        "-  Work with Watson Machine Learning experiments to train AutoAI models.\n",
        "-  Compare trained models quality and select the best one for further refinement.\n",
        "-  Refine the best model and test new variations.\n",
        "-  Perform online deployment and score the trained model.\n",
        "\n",
        "\n",
        "## Contents\n",
        "\n",
        "This notebook contains the following parts:\n",
        "\n",
        "1. [Setup](#setup)\n",
        "2. [Optimizer definition](#definition)\n",
        "3. [Experiment Run](#run)\n",
        "4. [Pipelines comparison and testing](#comparison)\n",
        "5. [Historical runs](#runs)\n",
        "6. [Pipeline refinement and testing](#refinery)\n",
        "7. [Deploy and Score](#scoring)\n",
        "8. [Clean up](#cleanup)\n",
        "8. [Summary and next steps](#summary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"setup\"></a>\n",
        "## 1. Set up the environment\n",
        "\n",
        "Before you use the sample code in this notebook, contact with your Cloud Pack for Data administrator and ask for your account credentials."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Install and import the `ibm-watsonx-ai` and dependecies\n",
        "**Note:** `ibm-watsonx-ai` documentation can be found <a href=\"https://ibm.github.io/watsonx-ai-python-sdk/index.html\" target=\"_blank\" rel=\"noopener no referrer\">here</a>."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "!pip install -U ibm-watsonx-ai | tail -n 1\n",
        "!pip install -U autoai-libs | tail -n 1\n",
        "!pip install -U lale | tail -n 1\n",
        "!pip install \"scikit-learn==1.3.0\" | tail -n 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Connection to WML\n",
        "\n",
        "Authenticate the Watson Machine Learning service on IBM Cloud Pack for Data. You need to provide platform `url`, your `username` and `api_key`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "username = 'PASTE YOUR USERNAME HERE'\n",
        "api_key = 'PASTE YOUR API_KEY HERE'\n",
        "url = 'PASTE THE PLATFORM URL HERE'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from ibm_watsonx_ai import Credentials\n",
        "\n",
        "credentials = Credentials(\n",
        "    username=username,\n",
        "    api_key=api_key,\n",
        "    url=url,\n",
        "    instance_id=\"openshift\",\n",
        "    version=\"5.1\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Alternatively you can use `username` and `password` to authenticate WML services.\n",
        "\n",
        "```python\n",
        "credentials = Credentials(\n",
        "    username=***,\n",
        "    password=***,\n",
        "    url=***,\n",
        "    instance_id=\"openshift\",\n",
        "    version=\"5.1\"\n",
        ")\n",
        "\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from ibm_watsonx_ai import APIClient\n",
        "\n",
        "client = APIClient(credentials)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Working with spaces\n",
        "\n",
        "First of all, you need to create a space that will be used for your work. If you do not have space already created, you can use `{PLATFORM_URL}/ml-runtime/spaces?context=icp4data` to create one.\n",
        "\n",
        "- Click New Deployment Space\n",
        "- Create an empty space\n",
        "- Go to space `Settings` tab\n",
        "- Copy `space_id` and paste it below\n",
        "\n",
        "**Tip**: You can also use SDK to prepare the space for your work. More information can be found [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd5.1/notebooks/python_sdk/instance-management/Space%20management.ipynb).\n",
        "\n",
        "**Action**: Assign space ID below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "space_id = 'PASTE YOUR SPACE ID HERE'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "You can use the `list` method to print all existing spaces."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "client.spaces.list(limit=10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "To be able to interact with all resources available in Watson Machine Learning, you need to set the **space** which you will be using."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'SUCCESS'"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "client.set.default_space(space_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"definition\"></a>\n",
        "## 2. Optimizer definition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Training data connection\n",
        "\n",
        "Define connection information to an external database.\n",
        "\n",
        "This example uses the German Credit Risk dataset. \n",
        "The dataset can be downloaded from [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd5.1/data/credit_risk/german_credit_data_biased_training.csv)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Connection configuration\n",
        "\n",
        "Credentials for database should be passed as a python dictionary.  \n",
        "For the vast number of supported datasets the credentials should follow the bellow pattern.  \n",
        "**Warning**: Database name should be slected from the list of all the supported databases, in order to look it up use `client.connections.list_datasource_types()`  \n",
        "**Warning**: Input table should be uploaded in database under the location `/schema_name/table_name`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "db_name = 'PUT YOUR DATABASE NAME HERE'\n",
        "schema_name = 'PUT YOUR SCHEMA NAME HERE'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "table_name = 'CREDIT_RISK'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "db_credentials = {\n",
        "    \"database\": \"***\",\n",
        "    \"password\": \"***\",\n",
        "    \"port\": \"***\",\n",
        "    \"host\": \"***\",\n",
        "    \"ssl\": \"***\",\n",
        "    \"username\": \"***\"\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "#### Create connection"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Creating connections...\n",
            "SUCCESS\n"
          ]
        }
      ],
      "source": [
        "conn_meta_props= {\n",
        "    client.connections.ConfigurationMetaNames.NAME: f\"Connection to Database - {db_name} \",\n",
        "    client.connections.ConfigurationMetaNames.DATASOURCE_TYPE: client.connections.get_datasource_type_id_by_name(db_name),\n",
        "    client.connections.ConfigurationMetaNames.DESCRIPTION: \"Connection to external Database\",\n",
        "    client.connections.ConfigurationMetaNames.PROPERTIES: db_credentials\n",
        "}\n",
        "\n",
        "conn_details = client.connections.create(meta_props=conn_meta_props)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "connection_id = client.connections.get_id(conn_details)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "#### Download training data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "!wget -q 'https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/cpd5.1/data/credit_risk/german_credit_data_biased_training.csv' -O \\\n",
        "            'credit_risk.csv'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "**Hint**: To install `pandas` execute `!pip install pandas`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "from ibm_watsonx_ai.helpers import DataConnection, DatabaseLocation\n",
        "\n",
        "\n",
        "credit_risk_conn = DataConnection(\n",
        "    connection_asset_id=connection_id,\n",
        "    location=DatabaseLocation(\n",
        "        schema_name=schema_name,\n",
        "        table_name=table_name\n",
        "    )\n",
        ")\n",
        "\n",
        "credit_risk_conn._api_client = client\n",
        "credit_risk_conn.write(pd.read_csv(\"credit_risk.csv\"))\n",
        "training_data_reference=[credit_risk_conn]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Optimizer configuration\n",
        "\n",
        "Provide the input information for AutoAI optimizer:\n",
        "- `name` - experiment name\n",
        "- `prediction_type` - type of the problem\n",
        "- `prediction_column` - target column name\n",
        "- `scoring` - optimization metric"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from ibm_watsonx_ai.experiment import AutoAI\n",
        "\n",
        "experiment = AutoAI(credentials, space_id=space_id)\n",
        "\n",
        "pipeline_optimizer = experiment.optimizer(\n",
        "    name='Credit Risk Prediction - AutoAI',\n",
        "    desc='Sample notebook',\n",
        "    prediction_type=AutoAI.PredictionType.BINARY,\n",
        "    prediction_column='Risk',\n",
        "    scoring=AutoAI.Metrics.ROC_AUC_SCORE\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Configuration parameters can be retrieved via `get_params()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'name': 'Credit Risk Prediction - AutoAI',\n",
              " 'desc': 'Sample notebook',\n",
              " 'prediction_type': 'binary',\n",
              " 'prediction_column': 'Risk',\n",
              " 'prediction_columns': None,\n",
              " 'timestamp_column_name': None,\n",
              " 'scoring': 'roc_auc',\n",
              " 'holdout_size': None,\n",
              " 'max_num_daub_ensembles': None,\n",
              " 't_shirt_size': 'm',\n",
              " 'train_sample_rows_test_size': None,\n",
              " 'include_only_estimators': [<ClassificationAlgorithms.XGB: 'XGBClassifier'>],\n",
              " 'include_batched_ensemble_estimators': None,\n",
              " 'backtest_num': None,\n",
              " 'lookback_window': None,\n",
              " 'forecast_window': None,\n",
              " 'backtest_gap_length': None,\n",
              " 'cognito_transform_names': None,\n",
              " 'csv_separator': ',',\n",
              " 'excel_sheet': None,\n",
              " 'encoding': 'utf-8',\n",
              " 'positive_label': None,\n",
              " 'drop_duplicates': True,\n",
              " 'outliers_columns': None,\n",
              " 'text_processing': None,\n",
              " 'word2vec_feature_number': None,\n",
              " 'daub_give_priority_to_runtime': None,\n",
              " 'text_columns_names': None,\n",
              " 'sampling_type': None,\n",
              " 'sample_size_limit': None,\n",
              " 'sample_rows_limit': None,\n",
              " 'sample_percentage_limit': None,\n",
              " 'number_of_batch_rows': None,\n",
              " 'n_parallel_data_connections': None,\n",
              " 'test_data_csv_separator': ',',\n",
              " 'test_data_excel_sheet': None,\n",
              " 'test_data_encoding': 'utf-8',\n",
              " 'categorical_imputation_strategy': None,\n",
              " 'numerical_imputation_strategy': None,\n",
              " 'numerical_imputation_value': None,\n",
              " 'imputation_threshold': None,\n",
              " 'retrain_on_holdout': True,\n",
              " 'feature_columns': None,\n",
              " 'pipeline_types': None,\n",
              " 'supporting_features_at_forecast': None,\n",
              " 'numerical_columns': None,\n",
              " 'categorical_columns': None,\n",
              " 'confidence_level': None,\n",
              " 'incremental_learning': None,\n",
              " 'early_stop_enabled': None,\n",
              " 'early_stop_window_size': None,\n",
              " 'time_ordered_data': None,\n",
              " 'feature_selector_mode': None,\n",
              " 'run_id': None}"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipeline_optimizer.get_params()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"run\"></a>\n",
        "## 3. Experiment run\n",
        "\n",
        "Call the `fit()` method to trigger the AutoAI experiment. You can either use interactive mode (synchronous job) or background mode (asychronous job) by specifying `background_model=True`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training job fc5431ed-d8b7-4e84-af97-bd657e13ad85 completed: 100%|████████| [02:22<00:00,  1.43s/it]\n"
          ]
        }
      ],
      "source": [
        "run_details = pipeline_optimizer.fit(\n",
        "            training_data_reference=training_data_reference,\n",
        "            background_mode=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "You can use the `get_run_status()` method to monitor AutoAI jobs in background mode."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'completed'"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipeline_optimizer.get_run_status()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"comparison\"></a>\n",
        "## 4. Pipelines comparison and testing\n",
        "\n",
        "You can list trained pipelines and evaluation metrics information in\n",
        "the form of a Pandas DataFrame by calling the `summary()` method. You can\n",
        "use the DataFrame to compare all discovered pipelines and select the one\n",
        "you like for further testing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Enhancements</th>\n",
              "      <th>Estimator</th>\n",
              "      <th>training_roc_auc_(optimized)</th>\n",
              "      <th>holdout_average_precision</th>\n",
              "      <th>holdout_log_loss</th>\n",
              "      <th>training_accuracy</th>\n",
              "      <th>holdout_roc_auc</th>\n",
              "      <th>training_balanced_accuracy</th>\n",
              "      <th>training_f1</th>\n",
              "      <th>holdout_precision</th>\n",
              "      <th>training_average_precision</th>\n",
              "      <th>training_log_loss</th>\n",
              "      <th>holdout_recall</th>\n",
              "      <th>training_precision</th>\n",
              "      <th>holdout_accuracy</th>\n",
              "      <th>holdout_balanced_accuracy</th>\n",
              "      <th>training_recall</th>\n",
              "      <th>holdout_f1</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline Name</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Pipeline_3</th>\n",
              "      <td>HPO, FE</td>\n",
              "      <td>XGBClassifier</td>\n",
              "      <td>0.854398</td>\n",
              "      <td>0.470253</td>\n",
              "      <td>0.386171</td>\n",
              "      <td>0.801698</td>\n",
              "      <td>0.853329</td>\n",
              "      <td>0.751417</td>\n",
              "      <td>0.858331</td>\n",
              "      <td>0.801047</td>\n",
              "      <td>0.916451</td>\n",
              "      <td>0.430152</td>\n",
              "      <td>0.921687</td>\n",
              "      <td>0.817077</td>\n",
              "      <td>0.795591</td>\n",
              "      <td>0.733298</td>\n",
              "      <td>0.904031</td>\n",
              "      <td>0.857143</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_2</th>\n",
              "      <td>HPO</td>\n",
              "      <td>XGBClassifier</td>\n",
              "      <td>0.853384</td>\n",
              "      <td>0.473649</td>\n",
              "      <td>0.401209</td>\n",
              "      <td>0.801697</td>\n",
              "      <td>0.850642</td>\n",
              "      <td>0.750097</td>\n",
              "      <td>0.858698</td>\n",
              "      <td>0.802111</td>\n",
              "      <td>0.915805</td>\n",
              "      <td>0.429192</td>\n",
              "      <td>0.915663</td>\n",
              "      <td>0.815568</td>\n",
              "      <td>0.793587</td>\n",
              "      <td>0.733280</td>\n",
              "      <td>0.906715</td>\n",
              "      <td>0.855134</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_4</th>\n",
              "      <td>HPO, FE, HPO</td>\n",
              "      <td>XGBClassifier</td>\n",
              "      <td>0.856033</td>\n",
              "      <td>0.456397</td>\n",
              "      <td>0.301605</td>\n",
              "      <td>0.802367</td>\n",
              "      <td>0.850570</td>\n",
              "      <td>0.755547</td>\n",
              "      <td>0.857892</td>\n",
              "      <td>0.867403</td>\n",
              "      <td>0.916986</td>\n",
              "      <td>0.428606</td>\n",
              "      <td>0.945783</td>\n",
              "      <td>0.821532</td>\n",
              "      <td>0.867735</td>\n",
              "      <td>0.829179</td>\n",
              "      <td>0.897655</td>\n",
              "      <td>0.904899</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_5</th>\n",
              "      <td>HPO, FE, HPO, Ensemble</td>\n",
              "      <td>BatchedTreeEnsembleClassifier(XGBClassifier)</td>\n",
              "      <td>0.856033</td>\n",
              "      <td>0.456397</td>\n",
              "      <td>0.301605</td>\n",
              "      <td>0.802367</td>\n",
              "      <td>0.850570</td>\n",
              "      <td>0.755547</td>\n",
              "      <td>0.857892</td>\n",
              "      <td>0.867403</td>\n",
              "      <td>0.916986</td>\n",
              "      <td>0.428606</td>\n",
              "      <td>0.945783</td>\n",
              "      <td>0.821532</td>\n",
              "      <td>0.867735</td>\n",
              "      <td>0.829179</td>\n",
              "      <td>0.897655</td>\n",
              "      <td>0.904899</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_1</th>\n",
              "      <td></td>\n",
              "      <td>XGBClassifier</td>\n",
              "      <td>0.846630</td>\n",
              "      <td>0.461541</td>\n",
              "      <td>0.331796</td>\n",
              "      <td>0.799912</td>\n",
              "      <td>0.848351</td>\n",
              "      <td>0.752877</td>\n",
              "      <td>0.856075</td>\n",
              "      <td>0.842246</td>\n",
              "      <td>0.911442</td>\n",
              "      <td>0.440900</td>\n",
              "      <td>0.948795</td>\n",
              "      <td>0.819926</td>\n",
              "      <td>0.847695</td>\n",
              "      <td>0.797751</td>\n",
              "      <td>0.895640</td>\n",
              "      <td>0.892351</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                         Enhancements  \\\n",
              "Pipeline Name                           \n",
              "Pipeline_3                    HPO, FE   \n",
              "Pipeline_2                        HPO   \n",
              "Pipeline_4               HPO, FE, HPO   \n",
              "Pipeline_5     HPO, FE, HPO, Ensemble   \n",
              "Pipeline_1                              \n",
              "\n",
              "                                                  Estimator  \\\n",
              "Pipeline Name                                                 \n",
              "Pipeline_3                                    XGBClassifier   \n",
              "Pipeline_2                                    XGBClassifier   \n",
              "Pipeline_4                                    XGBClassifier   \n",
              "Pipeline_5     BatchedTreeEnsembleClassifier(XGBClassifier)   \n",
              "Pipeline_1                                    XGBClassifier   \n",
              "\n",
              "               training_roc_auc_(optimized)  holdout_average_precision  \\\n",
              "Pipeline Name                                                            \n",
              "Pipeline_3                         0.854398                   0.470253   \n",
              "Pipeline_2                         0.853384                   0.473649   \n",
              "Pipeline_4                         0.856033                   0.456397   \n",
              "Pipeline_5                         0.856033                   0.456397   \n",
              "Pipeline_1                         0.846630                   0.461541   \n",
              "\n",
              "               holdout_log_loss  training_accuracy  holdout_roc_auc  \\\n",
              "Pipeline Name                                                         \n",
              "Pipeline_3             0.386171           0.801698         0.853329   \n",
              "Pipeline_2             0.401209           0.801697         0.850642   \n",
              "Pipeline_4             0.301605           0.802367         0.850570   \n",
              "Pipeline_5             0.301605           0.802367         0.850570   \n",
              "Pipeline_1             0.331796           0.799912         0.848351   \n",
              "\n",
              "               training_balanced_accuracy  training_f1  holdout_precision  \\\n",
              "Pipeline Name                                                               \n",
              "Pipeline_3                       0.751417     0.858331           0.801047   \n",
              "Pipeline_2                       0.750097     0.858698           0.802111   \n",
              "Pipeline_4                       0.755547     0.857892           0.867403   \n",
              "Pipeline_5                       0.755547     0.857892           0.867403   \n",
              "Pipeline_1                       0.752877     0.856075           0.842246   \n",
              "\n",
              "               training_average_precision  training_log_loss  holdout_recall  \\\n",
              "Pipeline Name                                                                  \n",
              "Pipeline_3                       0.916451           0.430152        0.921687   \n",
              "Pipeline_2                       0.915805           0.429192        0.915663   \n",
              "Pipeline_4                       0.916986           0.428606        0.945783   \n",
              "Pipeline_5                       0.916986           0.428606        0.945783   \n",
              "Pipeline_1                       0.911442           0.440900        0.948795   \n",
              "\n",
              "               training_precision  holdout_accuracy  \\\n",
              "Pipeline Name                                         \n",
              "Pipeline_3               0.817077          0.795591   \n",
              "Pipeline_2               0.815568          0.793587   \n",
              "Pipeline_4               0.821532          0.867735   \n",
              "Pipeline_5               0.821532          0.867735   \n",
              "Pipeline_1               0.819926          0.847695   \n",
              "\n",
              "               holdout_balanced_accuracy  training_recall  holdout_f1  \n",
              "Pipeline Name                                                          \n",
              "Pipeline_3                      0.733298         0.904031    0.857143  \n",
              "Pipeline_2                      0.733280         0.906715    0.855134  \n",
              "Pipeline_4                      0.829179         0.897655    0.904899  \n",
              "Pipeline_5                      0.829179         0.897655    0.904899  \n",
              "Pipeline_1                      0.797751         0.895640    0.892351  "
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "summary = pipeline_optimizer.summary()\n",
        "summary"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "You can visualize the scoring metric calculated on a holdout data set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "application/vnd.plotly.v1+json": {
              "config": {
                "plotlyServerURL": "https://plot.ly"
              },
              "data": [
                {
                  "hovertemplate": "variable=holdout_roc_auc<br>Pipeline Name=%{x}<br>value=%{y}<extra></extra>",
                  "legendgroup": "holdout_roc_auc",
                  "line": {
                    "color": "#636efa",
                    "dash": "solid"
                  },
                  "marker": {
                    "symbol": "circle"
                  },
                  "mode": "lines",
                  "name": "holdout_roc_auc",
                  "orientation": "v",
                  "showlegend": true,
                  "type": "scatter",
                  "x": [
                    "Pipeline_3",
                    "Pipeline_2",
                    "Pipeline_4",
                    "Pipeline_5",
                    "Pipeline_1"
                  ],
                  "xaxis": "x",
                  "y": [
                    0.8533294856070992,
                    0.8506420893153452,
                    0.8505699444484526,
                    0.8505699444484526,
                    0.8483514897915013
                  ],
                  "yaxis": "y"
                }
              ],
              "layout": {
                "legend": {
                  "title": {
                    "text": "variable"
                  },
                  "tracegroupgap": 0
                },
                "margin": {
                  "t": 60
                },
                "template": {
                  "data": {
                    "bar": [
                      {
                        "error_x": {
                          "color": "#2a3f5f"
                        },
                        "error_y": {
                          "color": "#2a3f5f"
                        },
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "bar"
                      }
                    ],
                    "barpolar": [
                      {
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "barpolar"
                      }
                    ],
                    "carpet": [
                      {
                        "aaxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "baxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "type": "carpet"
                      }
                    ],
                    "choropleth": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "choropleth"
                      }
                    ],
                    "contour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "contour"
                      }
                    ],
                    "contourcarpet": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "contourcarpet"
                      }
                    ],
                    "heatmap": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmap"
                      }
                    ],
                    "heatmapgl": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmapgl"
                      }
                    ],
                    "histogram": [
                      {
                        "marker": {
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "histogram"
                      }
                    ],
                    "histogram2d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2d"
                      }
                    ],
                    "histogram2dcontour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2dcontour"
                      }
                    ],
                    "mesh3d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "mesh3d"
                      }
                    ],
                    "parcoords": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "parcoords"
                      }
                    ],
                    "pie": [
                      {
                        "automargin": true,
                        "type": "pie"
                      }
                    ],
                    "scatter": [
                      {
                        "fillpattern": {
                          "fillmode": "overlay",
                          "size": 10,
                          "solidity": 0.2
                        },
                        "type": "scatter"
                      }
                    ],
                    "scatter3d": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatter3d"
                      }
                    ],
                    "scattercarpet": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattercarpet"
                      }
                    ],
                    "scattergeo": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergeo"
                      }
                    ],
                    "scattergl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergl"
                      }
                    ],
                    "scattermapbox": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattermapbox"
                      }
                    ],
                    "scatterpolar": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolar"
                      }
                    ],
                    "scatterpolargl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolargl"
                      }
                    ],
                    "scatterternary": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterternary"
                      }
                    ],
                    "surface": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "surface"
                      }
                    ],
                    "table": [
                      {
                        "cells": {
                          "fill": {
                            "color": "#EBF0F8"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "header": {
                          "fill": {
                            "color": "#C8D4E3"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "type": "table"
                      }
                    ]
                  },
                  "layout": {
                    "annotationdefaults": {
                      "arrowcolor": "#2a3f5f",
                      "arrowhead": 0,
                      "arrowwidth": 1
                    },
                    "autotypenumbers": "strict",
                    "coloraxis": {
                      "colorbar": {
                        "outlinewidth": 0,
                        "ticks": ""
                      }
                    },
                    "colorscale": {
                      "diverging": [
                        [
                          0,
                          "#8e0152"
                        ],
                        [
                          0.1,
                          "#c51b7d"
                        ],
                        [
                          0.2,
                          "#de77ae"
                        ],
                        [
                          0.3,
                          "#f1b6da"
                        ],
                        [
                          0.4,
                          "#fde0ef"
                        ],
                        [
                          0.5,
                          "#f7f7f7"
                        ],
                        [
                          0.6,
                          "#e6f5d0"
                        ],
                        [
                          0.7,
                          "#b8e186"
                        ],
                        [
                          0.8,
                          "#7fbc41"
                        ],
                        [
                          0.9,
                          "#4d9221"
                        ],
                        [
                          1,
                          "#276419"
                        ]
                      ],
                      "sequential": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ],
                      "sequentialminus": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ]
                    },
                    "colorway": [
                      "#636efa",
                      "#EF553B",
                      "#00cc96",
                      "#ab63fa",
                      "#FFA15A",
                      "#19d3f3",
                      "#FF6692",
                      "#B6E880",
                      "#FF97FF",
                      "#FECB52"
                    ],
                    "font": {
                      "color": "#2a3f5f"
                    },
                    "geo": {
                      "bgcolor": "white",
                      "lakecolor": "white",
                      "landcolor": "#E5ECF6",
                      "showlakes": true,
                      "showland": true,
                      "subunitcolor": "white"
                    },
                    "hoverlabel": {
                      "align": "left"
                    },
                    "hovermode": "closest",
                    "mapbox": {
                      "style": "light"
                    },
                    "paper_bgcolor": "white",
                    "plot_bgcolor": "#E5ECF6",
                    "polar": {
                      "angularaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "radialaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "scene": {
                      "xaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "yaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "zaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      }
                    },
                    "shapedefaults": {
                      "line": {
                        "color": "#2a3f5f"
                      }
                    },
                    "ternary": {
                      "aaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "baxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "caxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "title": {
                      "x": 0.05
                    },
                    "xaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    },
                    "yaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    }
                  }
                },
                "xaxis": {
                  "anchor": "y",
                  "domain": [
                    0,
                    1
                  ],
                  "title": {
                    "text": "Pipeline Name"
                  }
                },
                "yaxis": {
                  "anchor": "x",
                  "domain": [
                    0,
                    1
                  ],
                  "title": {
                    "text": "value"
                  }
                }
              }
            }
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "pd.options.plotting.backend = \"plotly\"\n",
        "\n",
        "summary.holdout_roc_auc.plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Get selected pipeline model\n",
        "\n",
        "Download and reconstruct a scikit-learn pipeline model object from the\n",
        "AutoAI training job."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        },
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "best_pipeline = pipeline_optimizer.get_pipeline()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Check confusion matrix for selected pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>fn</th>\n",
              "      <th>fp</th>\n",
              "      <th>tn</th>\n",
              "      <th>tp</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>true_class</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Risk</th>\n",
              "      <td>76</td>\n",
              "      <td>26</td>\n",
              "      <td>306</td>\n",
              "      <td>91</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>No Risk</th>\n",
              "      <td>26</td>\n",
              "      <td>76</td>\n",
              "      <td>91</td>\n",
              "      <td>306</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            fn  fp   tn   tp\n",
              "true_class                  \n",
              "Risk        76  26  306   91\n",
              "No Risk     26  76   91  306"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipeline_optimizer.get_pipeline_details()['confusion_matrix']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Check features importance for selected pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>features_importance</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>NewFeature_1_sum(LoanDuration__Age)</th>\n",
              "      <td>0.1405</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ExistingCreditsCount</th>\n",
              "      <td>0.1051</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>OthersOnLoan</th>\n",
              "      <td>0.0618</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Telephone</th>\n",
              "      <td>0.0574</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CurrentResidenceDuration</th>\n",
              "      <td>0.0509</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_0_sum(LoanDuration__LoanAmount)</th>\n",
              "      <td>0.0368</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>OwnsProperty</th>\n",
              "      <td>0.0335</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Age</th>\n",
              "      <td>0.0321</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>InstallmentPlans</th>\n",
              "      <td>0.0312</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Job</th>\n",
              "      <td>0.0271</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>EmploymentDuration</th>\n",
              "      <td>0.0265</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_14_pca_15</th>\n",
              "      <td>0.0235</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Housing</th>\n",
              "      <td>0.0231</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CheckingStatus</th>\n",
              "      <td>0.0231</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_2_sum(LoanAmount__Age)</th>\n",
              "      <td>0.0224</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>LoanDuration</th>\n",
              "      <td>0.0199</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>LoanAmount</th>\n",
              "      <td>0.0195</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_5_pca_4</th>\n",
              "      <td>0.0174</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_12_pca_13</th>\n",
              "      <td>0.0158</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CreditHistory</th>\n",
              "      <td>0.0154</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_15_pca_16</th>\n",
              "      <td>0.0148</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>InstallmentPercent</th>\n",
              "      <td>0.0145</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_18_pca_21</th>\n",
              "      <td>0.0144</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_11_pca_12</th>\n",
              "      <td>0.0143</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_3_pca_0</th>\n",
              "      <td>0.0141</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_8_pca_9</th>\n",
              "      <td>0.0133</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_16_pca_18</th>\n",
              "      <td>0.0131</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_6_pca_6</th>\n",
              "      <td>0.0130</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_19_pca_22</th>\n",
              "      <td>0.0129</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_9_pca_10</th>\n",
              "      <td>0.0128</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_4_pca_1</th>\n",
              "      <td>0.0128</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_13_pca_14</th>\n",
              "      <td>0.0123</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_7_pca_8</th>\n",
              "      <td>0.0115</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_10_pca_11</th>\n",
              "      <td>0.0115</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NewFeature_17_pca_20</th>\n",
              "      <td>0.0115</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>LoanPurpose</th>\n",
              "      <td>0.0105</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ExistingSavings</th>\n",
              "      <td>0.0097</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ForeignWorker</th>\n",
              "      <td>0.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Sex</th>\n",
              "      <td>0.0000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Dependents</th>\n",
              "      <td>0.0000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                            features_importance\n",
              "NewFeature_1_sum(LoanDuration__Age)                      0.1405\n",
              "ExistingCreditsCount                                     0.1051\n",
              "OthersOnLoan                                             0.0618\n",
              "Telephone                                                0.0574\n",
              "CurrentResidenceDuration                                 0.0509\n",
              "NewFeature_0_sum(LoanDuration__LoanAmount)               0.0368\n",
              "OwnsProperty                                             0.0335\n",
              "Age                                                      0.0321\n",
              "InstallmentPlans                                         0.0312\n",
              "Job                                                      0.0271\n",
              "EmploymentDuration                                       0.0265\n",
              "NewFeature_14_pca_15                                     0.0235\n",
              "Housing                                                  0.0231\n",
              "CheckingStatus                                           0.0231\n",
              "NewFeature_2_sum(LoanAmount__Age)                        0.0224\n",
              "LoanDuration                                             0.0199\n",
              "LoanAmount                                               0.0195\n",
              "NewFeature_5_pca_4                                       0.0174\n",
              "NewFeature_12_pca_13                                     0.0158\n",
              "CreditHistory                                            0.0154\n",
              "NewFeature_15_pca_16                                     0.0148\n",
              "InstallmentPercent                                       0.0145\n",
              "NewFeature_18_pca_21                                     0.0144\n",
              "NewFeature_11_pca_12                                     0.0143\n",
              "NewFeature_3_pca_0                                       0.0141\n",
              "NewFeature_8_pca_9                                       0.0133\n",
              "NewFeature_16_pca_18                                     0.0131\n",
              "NewFeature_6_pca_6                                       0.0130\n",
              "NewFeature_19_pca_22                                     0.0129\n",
              "NewFeature_9_pca_10                                      0.0128\n",
              "NewFeature_4_pca_1                                       0.0128\n",
              "NewFeature_13_pca_14                                     0.0123\n",
              "NewFeature_7_pca_8                                       0.0115\n",
              "NewFeature_10_pca_11                                     0.0115\n",
              "NewFeature_17_pca_20                                     0.0115\n",
              "LoanPurpose                                              0.0105\n",
              "ExistingSavings                                          0.0097\n",
              "ForeignWorker                                            0.0000\n",
              "Sex                                                      0.0000\n",
              "Dependents                                               0.0000"
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipeline_optimizer.get_pipeline_details()['features_importance']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Convert the pipeline model to a Python script and download it"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from ibm_watsonx_ai.helpers import pipeline_to_script\n",
        "pipeline_to_script(best_pipeline)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Visualize pipeline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": [
              "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
              "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
              " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
              "<!-- Generated by graphviz version 10.0.1 (20240210.2158)\n",
              " -->\n",
              "<!-- Title: cluster:(root) Pages: 1 -->\n",
              "<svg width=\"1517pt\" height=\"163pt\"\n",
              " viewBox=\"0.00 0.00 1516.79 163.48\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
              "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 159.48)\">\n",
              "<title>cluster:(root)</title>\n",
              "<g id=\"a_graph0\"><a xlink:title=\"(root) = ...\">\n",
              "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-159.48 1512.79,-159.48 1512.79,4 -4,4\"/>\n",
              "</a>\n",
              "</g>\n",
              "<g id=\"clust1\" class=\"cluster\">\n",
              "<title>cluster:tam</title>\n",
              "<g id=\"a_clust1\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.tam.html\" xlink:title=\"tam = TAM(tans_class=pca, name=&#39;pca&#39;, col_names=[&#39;CheckingStatus&#39;, &#39;LoanDuration&#39;, &#39;CreditHistory&#39;, &#39;LoanPurpose&#39;, &#39;LoanAmount&#39;, &#39;ExistingSavings&#39;, &#39;EmploymentDuration&#39;, &#39;InstallmentPercent&#39;, &#39;Sex&#39;, &#39;OthersOnLoan&#39;, &#39;CurrentResidenceDuration&#39;, &#39;OwnsProperty&#39;, &#39;Ag...)\" target=\"_blank\">\n",
              "<polygon fill=\"white\" stroke=\"black\" points=\"1247.05,-48.24 1247.05,-125.24 1317.05,-125.24 1317.05,-48.24 1247.05,-48.24\"/>\n",
              "<text text-anchor=\"middle\" x=\"1282.05\" y=\"-107.94\" font-family=\"Times,serif\" font-size=\"14.00\">TAM</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0 -->\n",
              "<g id=\"node1\" class=\"node\">\n",
              "<title>numpy_column_selector_0</title>\n",
              "<g id=\"a_node1\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_0 = NumpyColumnSelector(columns=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"33.16\" cy=\"-114.24\" rx=\"33.16\" ry=\"31.69\"/>\n",
              "<text text-anchor=\"middle\" x=\"33.16\" y=\"-124.04\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.16\" y=\"-110.54\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.16\" y=\"-97.04\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings -->\n",
              "<g id=\"node2\" class=\"node\">\n",
              "<title>compress_strings</title>\n",
              "<g id=\"a_node2\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.compress_strings.html\" xlink:title=\"compress_strings = CompressStrings(compress_type=&#39;hash&#39;, dtypes_list=[&#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_s...)\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"142.36\" cy=\"-114.24\" rx=\"40.05\" ry=\"22.15\"/>\n",
              "<text text-anchor=\"middle\" x=\"142.36\" y=\"-117.29\" font-family=\"Times,serif\" font-size=\"11.00\">Compress&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"142.36\" y=\"-103.79\" font-family=\"Times,serif\" font-size=\"11.00\">Strings</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0&#45;&gt;compress_strings -->\n",
              "<g id=\"edge1\" class=\"edge\">\n",
              "<title>numpy_column_selector_0&#45;&gt;compress_strings</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M66.61,-114.24C74.15,-114.24 82.36,-114.24 90.48,-114.24\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"90.31,-117.74 100.31,-114.24 90.31,-110.74 90.31,-117.74\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0 -->\n",
              "<g id=\"node3\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_0</title>\n",
              "<g id=\"a_node3\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_0 = NumpyReplaceMissingValues(filling_values=float(&#39;nan&#39;), missing_values=[])\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.16\" cy=\"-114.24\" rx=\"34.75\" ry=\"41.24\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.16\" y=\"-130.79\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.16\" y=\"-117.29\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.16\" y=\"-103.79\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.16\" y=\"-90.29\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings&#45;&gt;numpy_replace_missing_values_0 -->\n",
              "<g id=\"edge2\" class=\"edge\">\n",
              "<title>compress_strings&#45;&gt;numpy_replace_missing_values_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M182.64,-114.24C190.41,-114.24 198.63,-114.24 206.59,-114.24\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"206.47,-117.74 216.47,-114.24 206.47,-110.74 206.47,-117.74\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values -->\n",
              "<g id=\"node4\" class=\"node\">\n",
              "<title>numpy_replace_unknown_values</title>\n",
              "<g id=\"a_node4\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_unknown_values.html\" xlink:title=\"numpy_replace_unknown_values = NumpyReplaceUnknownValues(filling_values=float(&#39;nan&#39;), filling_values_list=[float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(...)\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"362.9\" cy=\"-114.24\" rx=\"38.99\" ry=\"41.24\"/>\n",
              "<text text-anchor=\"middle\" x=\"362.9\" y=\"-130.79\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"362.9\" y=\"-117.29\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"362.9\" y=\"-103.79\" font-family=\"Times,serif\" font-size=\"11.00\">Unknown&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"362.9\" y=\"-90.29\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values -->\n",
              "<g id=\"edge3\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M288.25,-114.24C295.88,-114.24 304.12,-114.24 312.23,-114.24\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"312.02,-117.74 322.02,-114.24 312.02,-110.74 312.02,-117.74\"/>\n",
              "</g>\n",
              "<!-- boolean2float -->\n",
              "<g id=\"node5\" class=\"node\">\n",
              "<title>boolean2float</title>\n",
              "<g id=\"a_node5\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.boolean2float.html\" xlink:title=\"boolean2float = boolean2float()\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"477.39\" cy=\"-110.24\" rx=\"39.5\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"477.39\" y=\"-106.54\" font-family=\"Times,serif\" font-size=\"11.00\">boolean2float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values&#45;&gt;boolean2float -->\n",
              "<g id=\"edge4\" class=\"edge\">\n",
              "<title>numpy_replace_unknown_values&#45;&gt;boolean2float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M402.3,-112.87C410.08,-112.6 418.38,-112.3 426.5,-112.01\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"426.4,-115.52 436.27,-111.67 426.15,-108.52 426.4,-115.52\"/>\n",
              "</g>\n",
              "<!-- cat_imputer -->\n",
              "<g id=\"node6\" class=\"node\">\n",
              "<title>cat_imputer</title>\n",
              "<g id=\"a_node6\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_imputer.html\" xlink:title=\"cat_imputer = CatImputer(missing_values=float(&#39;nan&#39;), sklearn_version_family=&#39;1&#39;, strategy=&#39;most_frequent&#39;)\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"583.39\" cy=\"-108.24\" rx=\"30.5\" ry=\"22.15\"/>\n",
              "<text text-anchor=\"middle\" x=\"583.39\" y=\"-111.29\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"583.39\" y=\"-97.79\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- boolean2float&#45;&gt;cat_imputer -->\n",
              "<g id=\"edge5\" class=\"edge\">\n",
              "<title>boolean2float&#45;&gt;cat_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M517.11,-109.49C524.95,-109.34 533.22,-109.18 541.14,-109.03\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"540.96,-112.54 550.89,-108.84 540.83,-105.54 540.96,-112.54\"/>\n",
              "</g>\n",
              "<!-- cat_encoder -->\n",
              "<g id=\"node7\" class=\"node\">\n",
              "<title>cat_encoder</title>\n",
              "<g id=\"a_node7\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_encoder.html\" xlink:title=\"cat_encoder = CatEncoder(dtype=np.float64, handle_unknown=&#39;error&#39;, sklearn_version_family=&#39;1&#39;, encoding=&#39;ordinal&#39;, categories=&#39;auto&#39;)\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"686.23\" cy=\"-106.24\" rx=\"32.1\" ry=\"22.15\"/>\n",
              "<text text-anchor=\"middle\" x=\"686.23\" y=\"-109.29\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"686.23\" y=\"-95.79\" font-family=\"Times,serif\" font-size=\"11.00\">Encoder</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_imputer&#45;&gt;cat_encoder -->\n",
              "<g id=\"edge6\" class=\"edge\">\n",
              "<title>cat_imputer&#45;&gt;cat_encoder</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M614.37,-107.64C623.21,-107.47 633.05,-107.27 642.51,-107.09\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"642.35,-110.59 652.27,-106.89 642.21,-103.59 642.35,-110.59\"/>\n",
              "</g>\n",
              "<!-- float32_transform_0 -->\n",
              "<g id=\"node8\" class=\"node\">\n",
              "<title>float32_transform_0</title>\n",
              "<g id=\"a_node8\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_0 = float32_transform()\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"794.91\" cy=\"-102.24\" rx=\"36.34\" ry=\"22.15\"/>\n",
              "<text text-anchor=\"middle\" x=\"794.91\" y=\"-105.29\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"794.91\" y=\"-91.79\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_encoder&#45;&gt;float32_transform_0 -->\n",
              "<g id=\"edge7\" class=\"edge\">\n",
              "<title>cat_encoder&#45;&gt;float32_transform_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M718.66,-105.06C727.49,-104.73 737.28,-104.36 746.81,-104\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"746.84,-107.51 756.7,-103.63 746.57,-100.51 746.84,-107.51\"/>\n",
              "</g>\n",
              "<!-- concat_features -->\n",
              "<g id=\"node15\" class=\"node\">\n",
              "<title>concat_features</title>\n",
              "<g id=\"a_node15\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.rasl.concat_features.html\" xlink:title=\"concat_features = ConcatFeatures()\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"900.4\" cy=\"-74.24\" rx=\"33.16\" ry=\"22.15\"/>\n",
              "<text text-anchor=\"middle\" x=\"900.4\" y=\"-77.29\" font-family=\"Times,serif\" font-size=\"11.00\">Concat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"900.4\" y=\"-63.79\" font-family=\"Times,serif\" font-size=\"11.00\">Features</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float32_transform_0&#45;&gt;concat_features -->\n",
              "<g id=\"edge13\" class=\"edge\">\n",
              "<title>float32_transform_0&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M828.65,-93.38C838.12,-90.82 848.57,-87.99 858.47,-85.31\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"859.13,-88.76 867.87,-82.77 857.3,-82 859.13,-88.76\"/>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1 -->\n",
              "<g id=\"node9\" class=\"node\">\n",
              "<title>numpy_column_selector_1</title>\n",
              "<g id=\"a_node9\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_1 = NumpyColumnSelector(columns=[4])\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.16\" cy=\"-34.24\" rx=\"33.16\" ry=\"31.69\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.16\" y=\"-44.04\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.16\" y=\"-30.54\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.16\" y=\"-17.04\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float -->\n",
              "<g id=\"node10\" class=\"node\">\n",
              "<title>float_str2_float</title>\n",
              "<g id=\"a_node10\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float_str2_float.html\" xlink:title=\"float_str2_float = FloatStr2Float(dtypes_list=[&#39;int_num&#39;], missing_values_reference_list=[])\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"362.9\" cy=\"-34.24\" rx=\"27\" ry=\"31.69\"/>\n",
              "<text text-anchor=\"middle\" x=\"362.9\" y=\"-44.04\" font-family=\"Times,serif\" font-size=\"11.00\">Float&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"362.9\" y=\"-30.54\" font-family=\"Times,serif\" font-size=\"11.00\">Str2&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"362.9\" y=\"-17.04\" font-family=\"Times,serif\" font-size=\"11.00\">Float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1&#45;&gt;float_str2_float -->\n",
              "<g id=\"edge8\" class=\"edge\">\n",
              "<title>numpy_column_selector_1&#45;&gt;float_str2_float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M286.78,-34.24C298.61,-34.24 312.06,-34.24 324.28,-34.24\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"324.02,-37.74 334.02,-34.24 324.02,-30.74 324.02,-37.74\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1 -->\n",
              "<g id=\"node11\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_1</title>\n",
              "<g id=\"a_node11\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_1 = NumpyReplaceMissingValues(filling_values=float(&#39;nan&#39;), missing_values=[])\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"477.39\" cy=\"-41.24\" rx=\"34.75\" ry=\"41.24\"/>\n",
              "<text text-anchor=\"middle\" x=\"477.39\" y=\"-57.79\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"477.39\" y=\"-44.29\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"477.39\" y=\"-30.79\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"477.39\" y=\"-17.29\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float&#45;&gt;numpy_replace_missing_values_1 -->\n",
              "<g id=\"edge9\" class=\"edge\">\n",
              "<title>float_str2_float&#45;&gt;numpy_replace_missing_values_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M390.27,-35.88C402.45,-36.64 417.25,-37.56 431.13,-38.42\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"430.78,-41.91 440.98,-39.04 431.22,-34.92 430.78,-41.91\"/>\n",
              "</g>\n",
              "<!-- num_imputer -->\n",
              "<g id=\"node12\" class=\"node\">\n",
              "<title>num_imputer</title>\n",
              "<g id=\"a_node12\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.num_imputer.html\" xlink:title=\"num_imputer = NumImputer(missing_values=float(&#39;nan&#39;), strategy=&#39;median&#39;)\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"583.39\" cy=\"-43.24\" rx=\"30.5\" ry=\"22.15\"/>\n",
              "<text text-anchor=\"middle\" x=\"583.39\" y=\"-46.29\" font-family=\"Times,serif\" font-size=\"11.00\">Num&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"583.39\" y=\"-32.79\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1&#45;&gt;num_imputer -->\n",
              "<g id=\"edge10\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_1&#45;&gt;num_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M512.44,-41.89C521.62,-42.07 531.65,-42.26 541.18,-42.45\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"540.88,-45.94 550.95,-42.63 541.02,-38.94 540.88,-45.94\"/>\n",
              "</g>\n",
              "<!-- opt_standard_scaler -->\n",
              "<g id=\"node13\" class=\"node\">\n",
              "<title>opt_standard_scaler</title>\n",
              "<g id=\"a_node13\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.opt_standard_scaler.html\" xlink:title=\"opt_standard_scaler = OptStandardScaler(use_scaler_flag=False)\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"686.23\" cy=\"-44.24\" rx=\"36.34\" ry=\"31.69\"/>\n",
              "<text text-anchor=\"middle\" x=\"686.23\" y=\"-54.04\" font-family=\"Times,serif\" font-size=\"11.00\">Opt&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"686.23\" y=\"-40.54\" font-family=\"Times,serif\" font-size=\"11.00\">Standard&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"686.23\" y=\"-27.04\" font-family=\"Times,serif\" font-size=\"11.00\">Scaler</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- num_imputer&#45;&gt;opt_standard_scaler -->\n",
              "<g id=\"edge11\" class=\"edge\">\n",
              "<title>num_imputer&#45;&gt;opt_standard_scaler</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M614.37,-43.54C621.87,-43.61 630.09,-43.69 638.2,-43.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"637.95,-47.27 647.98,-43.87 638.02,-40.27 637.95,-47.27\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1 -->\n",
              "<g id=\"node14\" class=\"node\">\n",
              "<title>float32_transform_1</title>\n",
              "<g id=\"a_node14\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_1 = float32_transform()\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"794.91\" cy=\"-49.24\" rx=\"36.34\" ry=\"22.15\"/>\n",
              "<text text-anchor=\"middle\" x=\"794.91\" y=\"-52.29\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"794.91\" y=\"-38.79\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- opt_standard_scaler&#45;&gt;float32_transform_1 -->\n",
              "<g id=\"edge12\" class=\"edge\">\n",
              "<title>opt_standard_scaler&#45;&gt;float32_transform_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M722.75,-45.9C730.5,-46.27 738.81,-46.66 746.92,-47.04\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"746.51,-50.52 756.67,-47.49 746.84,-43.53 746.51,-50.52\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1&#45;&gt;concat_features -->\n",
              "<g id=\"edge14\" class=\"edge\">\n",
              "<title>float32_transform_1&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M829.22,-57.29C838.4,-59.5 848.46,-61.93 858.03,-64.24\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"856.94,-67.58 867.48,-66.53 858.58,-60.78 856.94,-67.58\"/>\n",
              "</g>\n",
              "<!-- numpy_permute_array -->\n",
              "<g id=\"node16\" class=\"node\">\n",
              "<title>numpy_permute_array</title>\n",
              "<g id=\"a_node16\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_permute_array.html\" xlink:title=\"numpy_permute_array = NumpyPermuteArray(axis=0, permutation_indices=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 4])\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1004.31\" cy=\"-74.24\" rx=\"34.75\" ry=\"31.69\"/>\n",
              "<text text-anchor=\"middle\" x=\"1004.31\" y=\"-84.04\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1004.31\" y=\"-70.54\" font-family=\"Times,serif\" font-size=\"11.00\">Permute&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1004.31\" y=\"-57.04\" font-family=\"Times,serif\" font-size=\"11.00\">Array</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- concat_features&#45;&gt;numpy_permute_array -->\n",
              "<g id=\"edge15\" class=\"edge\">\n",
              "<title>concat_features&#45;&gt;numpy_permute_array</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M933.93,-74.24C941.61,-74.24 949.93,-74.24 958.06,-74.24\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"957.82,-77.74 967.82,-74.24 957.82,-70.74 957.82,-77.74\"/>\n",
              "</g>\n",
              "<!-- ta2 -->\n",
              "<g id=\"node17\" class=\"node\">\n",
              "<title>ta2</title>\n",
              "<g id=\"a_node17\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.ta2.html\" xlink:title=\"ta2 = TA2(fun=np.add, name=&#39;sum&#39;, datatypes1=[&#39;intc&#39;, &#39;intp&#39;, &#39;int_&#39;, &#39;uint8&#39;, &#39;uint16&#39;, &#39;uint32&#39;, &#39;uint64&#39;, &#39;int8&#39;, &#39;int16&#39;, &#39;int32&#39;, &#39;int64&#39;, &#39;short&#39;, &#39;long&#39;, &#39;longlong&#39;, &#39;float16&#39;, &#39;float32&#39;, &#39;float64&#39;], feat_constraints1=[autoai_libs.utils.fc_methods.is_not_...)\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1102.05\" cy=\"-74.24\" rx=\"27\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"1102.05\" y=\"-70.54\" font-family=\"Times,serif\" font-size=\"11.00\">TA2</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_permute_array&#45;&gt;ta2 -->\n",
              "<g id=\"edge16\" class=\"edge\">\n",
              "<title>numpy_permute_array&#45;&gt;ta2</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1039.33,-74.24C1047.13,-74.24 1055.47,-74.24 1063.41,-74.24\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1063.15,-77.74 1073.15,-74.24 1063.15,-70.74 1063.15,-77.74\"/>\n",
              "</g>\n",
              "<!-- fs1_0 -->\n",
              "<g id=\"node18\" class=\"node\">\n",
              "<title>fs1_0</title>\n",
              "<g id=\"a_node18\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.fs1.html\" xlink:title=\"fs1_0 = FS1(cols_ids_must_keep=range(0, 20), additional_col_count_to_keep=20, ptype=&#39;classification&#39;)\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1192.05\" cy=\"-74.24\" rx=\"27\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"1192.05\" y=\"-70.54\" font-family=\"Times,serif\" font-size=\"11.00\">FS1</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- ta2&#45;&gt;fs1_0 -->\n",
              "<g id=\"edge17\" class=\"edge\">\n",
              "<title>ta2&#45;&gt;fs1_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1129.46,-74.24C1136.94,-74.24 1145.24,-74.24 1153.25,-74.24\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1153.16,-77.74 1163.16,-74.24 1153.16,-70.74 1153.16,-77.74\"/>\n",
              "</g>\n",
              "<!-- pca -->\n",
              "<g id=\"node19\" class=\"node\">\n",
              "<title>pca</title>\n",
              "<g id=\"a_node19\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.sklearn.pca.html\" xlink:title=\"pca = PCA()\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1282.05\" cy=\"-74.24\" rx=\"27\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"1282.05\" y=\"-70.54\" font-family=\"Times,serif\" font-size=\"11.00\">PCA</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- fs1_0&#45;&gt;pca -->\n",
              "<g id=\"edge18\" class=\"edge\">\n",
              "<title>fs1_0&#45;&gt;pca</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1219.46,-74.24C1226.94,-74.24 1235.24,-74.24 1243.25,-74.24\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1235.54,-77.74 1245.54,-74.24 1235.54,-70.74 1235.54,-77.74\"/>\n",
              "</g>\n",
              "<!-- fs1_1 -->\n",
              "<g id=\"node20\" class=\"node\">\n",
              "<title>fs1_1</title>\n",
              "<g id=\"a_node20\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.fs1.html\" xlink:title=\"fs1_1 = FS1(cols_ids_must_keep=range(0, 20), additional_col_count_to_keep=20, ptype=&#39;classification&#39;)\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1372.05\" cy=\"-74.24\" rx=\"27\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"1372.05\" y=\"-70.54\" font-family=\"Times,serif\" font-size=\"11.00\">FS1</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- pca&#45;&gt;fs1_1 -->\n",
              "<g id=\"edge19\" class=\"edge\">\n",
              "<title>pca&#45;&gt;fs1_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1317.05,-74.24C1322.36,-74.24 1327.87,-74.24 1333.25,-74.24\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1333.16,-77.74 1343.16,-74.24 1333.16,-70.74 1333.16,-77.74\"/>\n",
              "</g>\n",
              "<!-- xgb_classifier -->\n",
              "<g id=\"node21\" class=\"node\">\n",
              "<title>xgb_classifier</title>\n",
              "<g id=\"a_node21\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.xgboost.xgb_classifier.html\" xlink:title=\"xgb_classifier = XGBClassifier(gamma=0, learning_rate=0.030992390423914655, max_depth=4, min_child_weight=16, missing=float(&#39;nan&#39;), n_estimators=239, n_jobs=4, random_state=33, reg_alpha=1, reg_lambda=0.21117015412922918, subsample=0.43148270828427676, tree_method=&#39;hist&#39;, verbosity=...)\" target=\"_blank\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1471.92\" cy=\"-74.24\" rx=\"36.87\" ry=\"22.15\"/>\n",
              "<text text-anchor=\"middle\" x=\"1471.92\" y=\"-77.29\" font-family=\"Times,serif\" font-size=\"11.00\">XGB&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1471.92\" y=\"-63.79\" font-family=\"Times,serif\" font-size=\"11.00\">Classifier</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- fs1_1&#45;&gt;xgb_classifier -->\n",
              "<g id=\"edge20\" class=\"edge\">\n",
              "<title>fs1_1&#45;&gt;xgb_classifier</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1399.54,-74.24C1406.93,-74.24 1415.19,-74.24 1423.39,-74.24\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1423.35,-77.74 1433.35,-74.24 1423.35,-70.74 1423.35,-77.74\"/>\n",
              "</g>\n",
              "</g>\n",
              "</svg>\n"
            ],
            "text/plain": [
              "<graphviz.graphs.Digraph at 0x149ea95d0>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "best_pipeline.visualize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Each node in the visualization is a machine-learning operator\n",
        "(transformer or estimator). Each edge indicates data flow (transformed\n",
        "output from one operator becomes input to the next).  The input to the\n",
        "root nodes is the initial dataset and the output from the sink node\n",
        "is the final prediction.  When you hover the mouse pointer over a\n",
        "node, a tooltip shows you the configuration arguments of the\n",
        "corresponding operator (tuned hyperparameters). When you click on the\n",
        "hyperlink of a node, it brings you to a documentation page for the\n",
        "operator."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Pipeline source code"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "best_pipeline.pretty_print(ipython_display=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "In the pretty-printed code, `>>` is the pipe combinator (dataflow\n",
        "edge) and `&` is the and combinator (combining multiple subpipelines).\n",
        "They correspond to the `make_pipeline` and `make_union` functions from\n",
        "scikit-learn, respectively. If you prefer the functions, you can\n",
        "instead pretty-print your pipeline with\n",
        "`best_pipeline.pretty_print(ipython_display=True, combinators=False)`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Reading training data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "train_df = pipeline_optimizer.get_data_connections()[0].read()\n",
        "\n",
        "train_X = train_df.drop(['Risk'], axis=1).values\n",
        "train_y = train_df.Risk.values"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Test pipeline model locally"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "predicted_y = best_pipeline.predict(train_X)\n",
        "predicted_y[:5]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"runs\"></a>\n",
        "## 5. Historical runs\n",
        "\n",
        "In this section you learn to work with historical AutoPipelines fit jobs (runs)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "To list historical runs use method `list()`. You can filter runs by providing experiment name."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>timestamp</th>\n",
              "      <th>run_id</th>\n",
              "      <th>state</th>\n",
              "      <th>auto_pipeline_optimizer name</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2020-12-08T12:50:22.481Z</td>\n",
              "      <td>91e1541a-fee1-4bb9-8211-68eed6abfdc4</td>\n",
              "      <td>completed</td>\n",
              "      <td>Credit Risk Prediction - AutoAI</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  timestamp                                run_id      state  \\\n",
              "0  2020-12-08T12:50:22.481Z  91e1541a-fee1-4bb9-8211-68eed6abfdc4  completed   \n",
              "\n",
              "      auto_pipeline_optimizer name  \n",
              "0  Credit Risk Prediction - AutoAI  "
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "experiment.runs(filter='Credit Risk Prediction - AutoAI').list()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "To work with historical pipelines found during a particular optimizer\n",
        "run, you need to first provide the `run_id` to select the fitted\n",
        "optimizer.\n",
        "\n",
        "**Note**: you can assign selected `run_id` to the run_id variable."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "run_id = run_details['metadata']['id']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Get executed optimizer's configuration parameters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'name': 'Credit Risk Prediction - AutoAI',\n",
              " 'desc': 'Sample notebook',\n",
              " 'prediction_type': 'binary',\n",
              " 'prediction_column': 'Risk',\n",
              " 'scoring': 'roc_auc',\n",
              " 'test_size': 0.1,\n",
              " 'max_num_daub_ensembles': 2.0,\n",
              " 't_shirt_size': 'c076e82c-b2a7-4d20-9c0f-1f0c2fdf5a24',\n",
              " 'daub_include_only_estimators': ['ExtraTreesClassifierEstimator',\n",
              "  'GradientBoostingClassifierEstimator',\n",
              "  'LGBMClassifierEstimator',\n",
              "  'LogisticRegressionEstimator',\n",
              "  'RandomForestClassifierEstimator',\n",
              "  'XGBClassifierEstimator',\n",
              "  'DecisionTreeClassifierEstimator'],\n",
              " 'cognito_transform_names': None,\n",
              " 'train_sample_rows_test_size': None,\n",
              " 'csv_separator': ',',\n",
              " 'excel_sheet': 0,\n",
              " 'encoding': 'utf-8'}"
            ]
          },
          "execution_count": 26,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "experiment.runs(filter='Credit Risk Prediction - AutoAI').get_params(run_id=run_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "\n",
        "### Get historical optimizer instance and training details"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "historical_opt = experiment.runs.get_optimizer(run_id)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "run_details = historical_opt.get_run_details()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### List trained pipelines for selected optimizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Number of enhancements</th>\n",
              "      <th>Estimator</th>\n",
              "      <th>training_roc_auc_(optimized)</th>\n",
              "      <th>holdout_precision</th>\n",
              "      <th>training_average_precision</th>\n",
              "      <th>holdout_average_precision</th>\n",
              "      <th>training_neg_log_loss</th>\n",
              "      <th>holdout_recall</th>\n",
              "      <th>training_precision</th>\n",
              "      <th>holdout_accuracy</th>\n",
              "      <th>holdout_balanced_accuracy</th>\n",
              "      <th>training_recall</th>\n",
              "      <th>holdout_f1</th>\n",
              "      <th>holdout_neg_log_loss</th>\n",
              "      <th>training_accuracy</th>\n",
              "      <th>holdout_roc_auc</th>\n",
              "      <th>training_balanced_accuracy</th>\n",
              "      <th>training_f1</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline Name</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Pipeline_4</th>\n",
              "      <td>4</td>\n",
              "      <td>XGBClassifierEstimator</td>\n",
              "      <td>0.853971</td>\n",
              "      <td>0.702532</td>\n",
              "      <td>0.786750</td>\n",
              "      <td>0.762646</td>\n",
              "      <td>-0.432601</td>\n",
              "      <td>0.664671</td>\n",
              "      <td>0.686865</td>\n",
              "      <td>0.794</td>\n",
              "      <td>0.761765</td>\n",
              "      <td>0.649368</td>\n",
              "      <td>0.683077</td>\n",
              "      <td>-0.434525</td>\n",
              "      <td>0.783778</td>\n",
              "      <td>0.858985</td>\n",
              "      <td>0.750276</td>\n",
              "      <td>0.667448</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_7</th>\n",
              "      <td>3</td>\n",
              "      <td>GradientBoostingClassifierEstimator</td>\n",
              "      <td>0.853637</td>\n",
              "      <td>0.744186</td>\n",
              "      <td>0.781843</td>\n",
              "      <td>0.752330</td>\n",
              "      <td>-0.430732</td>\n",
              "      <td>0.574850</td>\n",
              "      <td>0.754036</td>\n",
              "      <td>0.792</td>\n",
              "      <td>0.737876</td>\n",
              "      <td>0.581504</td>\n",
              "      <td>0.648649</td>\n",
              "      <td>-0.435807</td>\n",
              "      <td>0.796667</td>\n",
              "      <td>0.859524</td>\n",
              "      <td>0.743037</td>\n",
              "      <td>0.656293</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_8</th>\n",
              "      <td>4</td>\n",
              "      <td>GradientBoostingClassifierEstimator</td>\n",
              "      <td>0.853637</td>\n",
              "      <td>0.744186</td>\n",
              "      <td>0.781843</td>\n",
              "      <td>0.752330</td>\n",
              "      <td>-0.430732</td>\n",
              "      <td>0.574850</td>\n",
              "      <td>0.754036</td>\n",
              "      <td>0.792</td>\n",
              "      <td>0.737876</td>\n",
              "      <td>0.581504</td>\n",
              "      <td>0.648649</td>\n",
              "      <td>-0.435807</td>\n",
              "      <td>0.796667</td>\n",
              "      <td>0.859524</td>\n",
              "      <td>0.743037</td>\n",
              "      <td>0.656293</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_2</th>\n",
              "      <td>2</td>\n",
              "      <td>XGBClassifierEstimator</td>\n",
              "      <td>0.853223</td>\n",
              "      <td>0.674847</td>\n",
              "      <td>0.783110</td>\n",
              "      <td>0.759399</td>\n",
              "      <td>-0.437251</td>\n",
              "      <td>0.658683</td>\n",
              "      <td>0.674910</td>\n",
              "      <td>0.780</td>\n",
              "      <td>0.749762</td>\n",
              "      <td>0.657352</td>\n",
              "      <td>0.666667</td>\n",
              "      <td>-0.439138</td>\n",
              "      <td>0.779556</td>\n",
              "      <td>0.856000</td>\n",
              "      <td>0.749096</td>\n",
              "      <td>0.665831</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_3</th>\n",
              "      <td>3</td>\n",
              "      <td>XGBClassifierEstimator</td>\n",
              "      <td>0.853223</td>\n",
              "      <td>0.674847</td>\n",
              "      <td>0.783110</td>\n",
              "      <td>0.759399</td>\n",
              "      <td>-0.437251</td>\n",
              "      <td>0.658683</td>\n",
              "      <td>0.674910</td>\n",
              "      <td>0.780</td>\n",
              "      <td>0.749762</td>\n",
              "      <td>0.657352</td>\n",
              "      <td>0.666667</td>\n",
              "      <td>-0.439138</td>\n",
              "      <td>0.779556</td>\n",
              "      <td>0.856000</td>\n",
              "      <td>0.749096</td>\n",
              "      <td>0.665831</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_6</th>\n",
              "      <td>2</td>\n",
              "      <td>GradientBoostingClassifierEstimator</td>\n",
              "      <td>0.850494</td>\n",
              "      <td>0.755906</td>\n",
              "      <td>0.778339</td>\n",
              "      <td>0.765539</td>\n",
              "      <td>-0.432066</td>\n",
              "      <td>0.574850</td>\n",
              "      <td>0.759112</td>\n",
              "      <td>0.796</td>\n",
              "      <td>0.740879</td>\n",
              "      <td>0.592149</td>\n",
              "      <td>0.653061</td>\n",
              "      <td>-0.430092</td>\n",
              "      <td>0.800667</td>\n",
              "      <td>0.856503</td>\n",
              "      <td>0.748694</td>\n",
              "      <td>0.665064</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_1</th>\n",
              "      <td>1</td>\n",
              "      <td>XGBClassifierEstimator</td>\n",
              "      <td>0.849883</td>\n",
              "      <td>0.692810</td>\n",
              "      <td>0.782921</td>\n",
              "      <td>0.760481</td>\n",
              "      <td>-0.437680</td>\n",
              "      <td>0.634731</td>\n",
              "      <td>0.692303</td>\n",
              "      <td>0.784</td>\n",
              "      <td>0.746795</td>\n",
              "      <td>0.638057</td>\n",
              "      <td>0.662500</td>\n",
              "      <td>-0.438008</td>\n",
              "      <td>0.783778</td>\n",
              "      <td>0.854903</td>\n",
              "      <td>0.747457</td>\n",
              "      <td>0.663639</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_5</th>\n",
              "      <td>1</td>\n",
              "      <td>GradientBoostingClassifierEstimator</td>\n",
              "      <td>0.848791</td>\n",
              "      <td>0.766129</td>\n",
              "      <td>0.780104</td>\n",
              "      <td>0.751335</td>\n",
              "      <td>-0.432413</td>\n",
              "      <td>0.568862</td>\n",
              "      <td>0.768005</td>\n",
              "      <td>0.798</td>\n",
              "      <td>0.740888</td>\n",
              "      <td>0.573520</td>\n",
              "      <td>0.652921</td>\n",
              "      <td>-0.442652</td>\n",
              "      <td>0.799333</td>\n",
              "      <td>0.848789</td>\n",
              "      <td>0.743049</td>\n",
              "      <td>0.656371</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               Number of enhancements                            Estimator  \\\n",
              "Pipeline Name                                                                \n",
              "Pipeline_4                          4               XGBClassifierEstimator   \n",
              "Pipeline_7                          3  GradientBoostingClassifierEstimator   \n",
              "Pipeline_8                          4  GradientBoostingClassifierEstimator   \n",
              "Pipeline_2                          2               XGBClassifierEstimator   \n",
              "Pipeline_3                          3               XGBClassifierEstimator   \n",
              "Pipeline_6                          2  GradientBoostingClassifierEstimator   \n",
              "Pipeline_1                          1               XGBClassifierEstimator   \n",
              "Pipeline_5                          1  GradientBoostingClassifierEstimator   \n",
              "\n",
              "               training_roc_auc_(optimized)  holdout_precision  \\\n",
              "Pipeline Name                                                    \n",
              "Pipeline_4                         0.853971           0.702532   \n",
              "Pipeline_7                         0.853637           0.744186   \n",
              "Pipeline_8                         0.853637           0.744186   \n",
              "Pipeline_2                         0.853223           0.674847   \n",
              "Pipeline_3                         0.853223           0.674847   \n",
              "Pipeline_6                         0.850494           0.755906   \n",
              "Pipeline_1                         0.849883           0.692810   \n",
              "Pipeline_5                         0.848791           0.766129   \n",
              "\n",
              "               training_average_precision  holdout_average_precision  \\\n",
              "Pipeline Name                                                          \n",
              "Pipeline_4                       0.786750                   0.762646   \n",
              "Pipeline_7                       0.781843                   0.752330   \n",
              "Pipeline_8                       0.781843                   0.752330   \n",
              "Pipeline_2                       0.783110                   0.759399   \n",
              "Pipeline_3                       0.783110                   0.759399   \n",
              "Pipeline_6                       0.778339                   0.765539   \n",
              "Pipeline_1                       0.782921                   0.760481   \n",
              "Pipeline_5                       0.780104                   0.751335   \n",
              "\n",
              "               training_neg_log_loss  holdout_recall  training_precision  \\\n",
              "Pipeline Name                                                              \n",
              "Pipeline_4                 -0.432601        0.664671            0.686865   \n",
              "Pipeline_7                 -0.430732        0.574850            0.754036   \n",
              "Pipeline_8                 -0.430732        0.574850            0.754036   \n",
              "Pipeline_2                 -0.437251        0.658683            0.674910   \n",
              "Pipeline_3                 -0.437251        0.658683            0.674910   \n",
              "Pipeline_6                 -0.432066        0.574850            0.759112   \n",
              "Pipeline_1                 -0.437680        0.634731            0.692303   \n",
              "Pipeline_5                 -0.432413        0.568862            0.768005   \n",
              "\n",
              "               holdout_accuracy  holdout_balanced_accuracy  training_recall  \\\n",
              "Pipeline Name                                                                 \n",
              "Pipeline_4                0.794                   0.761765         0.649368   \n",
              "Pipeline_7                0.792                   0.737876         0.581504   \n",
              "Pipeline_8                0.792                   0.737876         0.581504   \n",
              "Pipeline_2                0.780                   0.749762         0.657352   \n",
              "Pipeline_3                0.780                   0.749762         0.657352   \n",
              "Pipeline_6                0.796                   0.740879         0.592149   \n",
              "Pipeline_1                0.784                   0.746795         0.638057   \n",
              "Pipeline_5                0.798                   0.740888         0.573520   \n",
              "\n",
              "               holdout_f1  holdout_neg_log_loss  training_accuracy  \\\n",
              "Pipeline Name                                                        \n",
              "Pipeline_4       0.683077             -0.434525           0.783778   \n",
              "Pipeline_7       0.648649             -0.435807           0.796667   \n",
              "Pipeline_8       0.648649             -0.435807           0.796667   \n",
              "Pipeline_2       0.666667             -0.439138           0.779556   \n",
              "Pipeline_3       0.666667             -0.439138           0.779556   \n",
              "Pipeline_6       0.653061             -0.430092           0.800667   \n",
              "Pipeline_1       0.662500             -0.438008           0.783778   \n",
              "Pipeline_5       0.652921             -0.442652           0.799333   \n",
              "\n",
              "               holdout_roc_auc  training_balanced_accuracy  training_f1  \n",
              "Pipeline Name                                                            \n",
              "Pipeline_4            0.858985                    0.750276     0.667448  \n",
              "Pipeline_7            0.859524                    0.743037     0.656293  \n",
              "Pipeline_8            0.859524                    0.743037     0.656293  \n",
              "Pipeline_2            0.856000                    0.749096     0.665831  \n",
              "Pipeline_3            0.856000                    0.749096     0.665831  \n",
              "Pipeline_6            0.856503                    0.748694     0.665064  \n",
              "Pipeline_1            0.854903                    0.747457     0.663639  \n",
              "Pipeline_5            0.848789                    0.743049     0.656371  "
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "historical_opt.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Get selected pipeline and test locally"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "hist_pipeline = historical_opt.get_pipeline(pipeline_name='Pipeline_3')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "pycharm": {
          "is_executing": false,
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array(['No Risk', 'No Risk', 'No Risk', 'No Risk', 'Risk'], dtype=object)"
            ]
          },
          "execution_count": 31,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "predicted_y = hist_pipeline.predict(train_X)\n",
        "predicted_y[:5]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"refinery\"></a>\n",
        "## 6. Pipeline refinement with [Lale](https://github.com/IBM/lale) and testing\n",
        "\n",
        "In this section you learn how to refine and retrain the best\n",
        "pipeline returned by AutoAI. There are many ways to refine a pipeline.\n",
        "For illustration, simply replace the final estimator in the\n",
        "pipeline by an interpretable model. The call to\n",
        "`wrap_imported_operators()` augments scikit-learn operators with\n",
        "schemas for hyperparameter tuning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from sklearn.linear_model import LogisticRegression as LR\n",
        "from sklearn.tree import DecisionTreeClassifier as Tree\n",
        "from sklearn.neighbors import KNeighborsClassifier as KNN\n",
        "from lale.lib.lale import Hyperopt\n",
        "from lale import wrap_imported_operators\n",
        "\n",
        "wrap_imported_operators()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Pipeline decomposition and new definition\n",
        "\n",
        "Start by removing the last step of the pipeline, i.e., the final estimator."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": [
              "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
              "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
              " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
              "<!-- Generated by graphviz version 2.44.1 (20200629.0846)\n",
              " -->\n",
              "<!-- Title: cluster:(root) Pages: 1 -->\n",
              "<svg width=\"1133pt\" height=\"149pt\"\n",
              " viewBox=\"0.00 0.00 1132.89 148.54\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
              "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 144.54)\">\n",
              "<title>cluster:(root)</title>\n",
              "<g id=\"a_graph0\"><a xlink:title=\"(root) = ...\">\n",
              "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-144.54 1128.89,-144.54 1128.89,4 -4,4\"/>\n",
              "</a>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0 -->\n",
              "<g id=\"node1\" class=\"node\">\n",
              "<title>numpy_column_selector_0</title>\n",
              "<g id=\"a_node1\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_0 = NumpyColumnSelector(columns=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"33.23\" cy=\"-103.77\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-112.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-100.97\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-88.97\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings -->\n",
              "<g id=\"node2\" class=\"node\">\n",
              "<title>compress_strings</title>\n",
              "<g id=\"a_node2\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.compress_strings.html\" xlink:title=\"compress_strings = CompressStrings(compress_type=&#39;hash&#39;, dtypes_list=[&#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_s...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"142.77\" cy=\"-103.77\" rx=\"40.11\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Compress&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Strings</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0&#45;&gt;compress_strings -->\n",
              "<g id=\"edge1\" class=\"edge\">\n",
              "<title>numpy_column_selector_0&#45;&gt;compress_strings</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M66.5,-103.77C74.59,-103.77 83.46,-103.77 92.19,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"92.3,-107.27 102.3,-103.77 92.3,-100.27 92.3,-107.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0 -->\n",
              "<g id=\"node3\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_0</title>\n",
              "<g id=\"a_node3\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_0 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-103.77\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-118.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-82.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings&#45;&gt;numpy_replace_missing_values_0 -->\n",
              "<g id=\"edge2\" class=\"edge\">\n",
              "<title>compress_strings&#45;&gt;numpy_replace_missing_values_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M183.11,-103.77C191.43,-103.77 200.26,-103.77 208.74,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"208.85,-107.27 218.85,-103.77 208.85,-100.27 208.85,-107.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values -->\n",
              "<g id=\"node4\" class=\"node\">\n",
              "<title>numpy_replace_unknown_values</title>\n",
              "<g id=\"a_node4\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_unknown_values.html\" xlink:title=\"numpy_replace_unknown_values = NumpyReplaceUnknownValues(filling_values=float(&#39;nan&#39;), filling_values_list=[float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-103.77\" rx=\"38.78\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-118.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Unknown&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-82.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values -->\n",
              "<g id=\"edge3\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M288.46,-103.77C296.6,-103.77 305.47,-103.77 314.13,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"314.16,-107.27 324.16,-103.77 314.16,-100.27 314.16,-107.27\"/>\n",
              "</g>\n",
              "<!-- boolean2float -->\n",
              "<g id=\"node5\" class=\"node\">\n",
              "<title>boolean2float</title>\n",
              "<g id=\"a_node5\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.boolean2float.html\" xlink:title=\"boolean2float = boolean2float()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-100.77\" rx=\"37.91\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-97.97\" font-family=\"Times,serif\" font-size=\"11.00\">boolean2float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values&#45;&gt;boolean2float -->\n",
              "<g id=\"edge4\" class=\"edge\">\n",
              "<title>numpy_replace_unknown_values&#45;&gt;boolean2float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M402.41,-102.74C410.6,-102.52 419.36,-102.28 427.85,-102.05\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"428.12,-105.54 438.02,-101.77 427.93,-98.55 428.12,-105.54\"/>\n",
              "</g>\n",
              "<!-- cat_imputer -->\n",
              "<g id=\"node6\" class=\"node\">\n",
              "<title>cat_imputer</title>\n",
              "<g id=\"a_node6\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_imputer.html\" xlink:title=\"cat_imputer = CatImputer(strategy=&#39;most_frequent&#39;, missing_values=float(&#39;nan&#39;), sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-98.77\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-101.97\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-89.97\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- boolean2float&#45;&gt;cat_imputer -->\n",
              "<g id=\"edge5\" class=\"edge\">\n",
              "<title>boolean2float&#45;&gt;cat_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M514.07,-100.05C522.45,-99.88 531.39,-99.71 539.88,-99.54\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"540.02,-103.04 549.95,-99.35 539.88,-96.04 540.02,-103.04\"/>\n",
              "</g>\n",
              "<!-- cat_encoder -->\n",
              "<g id=\"node7\" class=\"node\">\n",
              "<title>cat_encoder</title>\n",
              "<g id=\"a_node7\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_encoder.html\" xlink:title=\"cat_encoder = CatEncoder(encoding=&#39;ordinal&#39;, categories=&#39;auto&#39;, dtype=np.float64, handle_unknown=&#39;error&#39;, sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-97.77\" rx=\"31.64\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-100.97\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-88.97\" font-family=\"Times,serif\" font-size=\"11.00\">Encoder</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_imputer&#45;&gt;cat_encoder -->\n",
              "<g id=\"edge6\" class=\"edge\">\n",
              "<title>cat_imputer&#45;&gt;cat_encoder</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-98.47C620.35,-98.38 630.77,-98.28 640.7,-98.18\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"640.84,-101.68 650.8,-98.08 640.77,-94.68 640.84,-101.68\"/>\n",
              "</g>\n",
              "<!-- float32_transform_0 -->\n",
              "<g id=\"node8\" class=\"node\">\n",
              "<title>float32_transform_0</title>\n",
              "<g id=\"a_node8\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_0 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-94.77\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-97.97\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-85.97\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_encoder&#45;&gt;float32_transform_0 -->\n",
              "<g id=\"edge7\" class=\"edge\">\n",
              "<title>cat_encoder&#45;&gt;float32_transform_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M714.92,-96.89C724.18,-96.63 734.54,-96.34 744.53,-96.06\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.85,-99.55 754.75,-95.77 744.66,-92.55 744.85,-99.55\"/>\n",
              "</g>\n",
              "<!-- concat_features -->\n",
              "<g id=\"node15\" class=\"node\">\n",
              "<title>concat_features</title>\n",
              "<g id=\"a_node15\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.lale.concat_features.html\" xlink:title=\"concat_features = ConcatFeatures()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"896.36\" cy=\"-68.77\" rx=\"33.47\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-71.97\" font-family=\"Times,serif\" font-size=\"11.00\">Concat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-59.97\" font-family=\"Times,serif\" font-size=\"11.00\">Features</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float32_transform_0&#45;&gt;concat_features -->\n",
              "<g id=\"edge13\" class=\"edge\">\n",
              "<title>float32_transform_0&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.18,-86.69C834.15,-84.18 845.25,-81.38 855.66,-78.76\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"856.53,-82.15 865.38,-76.32 854.82,-75.36 856.53,-82.15\"/>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1 -->\n",
              "<g id=\"node9\" class=\"node\">\n",
              "<title>numpy_column_selector_1</title>\n",
              "<g id=\"a_node9\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_1 = NumpyColumnSelector(columns=[4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-31.77\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-40.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-28.97\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-16.97\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float -->\n",
              "<g id=\"node10\" class=\"node\">\n",
              "<title>float_str2_float</title>\n",
              "<g id=\"a_node10\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float_str2_float.html\" xlink:title=\"float_str2_float = FloatStr2Float(dtypes_list=[&#39;int_num&#39;], missing_values_reference_list=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-31.77\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-40.97\" font-family=\"Times,serif\" font-size=\"11.00\">Float&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-28.97\" font-family=\"Times,serif\" font-size=\"11.00\">Str2&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-16.97\" font-family=\"Times,serif\" font-size=\"11.00\">Float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1&#45;&gt;float_str2_float -->\n",
              "<g id=\"edge8\" class=\"edge\">\n",
              "<title>numpy_column_selector_1&#45;&gt;float_str2_float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M286.99,-31.77C299.27,-31.77 313.34,-31.77 325.96,-31.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"326.16,-35.27 336.16,-31.77 326.16,-28.27 326.16,-35.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1 -->\n",
              "<g id=\"node11\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_1</title>\n",
              "<g id=\"a_node11\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_1 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-36.77\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-51.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-39.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-27.97\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-15.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float&#45;&gt;numpy_replace_missing_values_1 -->\n",
              "<g id=\"edge9\" class=\"edge\">\n",
              "<title>float_str2_float&#45;&gt;numpy_replace_missing_values_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M390.25,-32.94C402.39,-33.49 417.18,-34.16 430.99,-34.78\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"431.15,-38.29 441.3,-35.24 431.47,-31.3 431.15,-38.29\"/>\n",
              "</g>\n",
              "<!-- num_imputer -->\n",
              "<g id=\"node12\" class=\"node\">\n",
              "<title>num_imputer</title>\n",
              "<g id=\"a_node12\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.num_imputer.html\" xlink:title=\"num_imputer = NumImputer(strategy=&#39;median&#39;, missing_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-39.77\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-42.97\" font-family=\"Times,serif\" font-size=\"11.00\">Num&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-30.97\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1&#45;&gt;num_imputer -->\n",
              "<g id=\"edge10\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_1&#45;&gt;num_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M510.91,-37.76C520.15,-38.03 530.24,-38.33 539.78,-38.61\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"539.74,-42.11 549.83,-38.9 539.94,-35.11 539.74,-42.11\"/>\n",
              "</g>\n",
              "<!-- opt_standard_scaler -->\n",
              "<g id=\"node13\" class=\"node\">\n",
              "<title>opt_standard_scaler</title>\n",
              "<g id=\"a_node13\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.opt_standard_scaler.html\" xlink:title=\"opt_standard_scaler = OptStandardScaler(num_scaler_copy=None, num_scaler_with_mean=None, num_scaler_with_std=None, use_scaler_flag=False)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-41.77\" rx=\"36.12\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-50.97\" font-family=\"Times,serif\" font-size=\"11.00\">Opt&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-38.97\" font-family=\"Times,serif\" font-size=\"11.00\">Standard&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-26.97\" font-family=\"Times,serif\" font-size=\"11.00\">Scaler</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- num_imputer&#45;&gt;opt_standard_scaler -->\n",
              "<g id=\"edge11\" class=\"edge\">\n",
              "<title>num_imputer&#45;&gt;opt_standard_scaler</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-40.36C619.14,-40.52 628.07,-40.7 636.8,-40.87\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"636.81,-44.37 646.88,-41.07 636.95,-37.37 636.81,-44.37\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1 -->\n",
              "<g id=\"node14\" class=\"node\">\n",
              "<title>float32_transform_1</title>\n",
              "<g id=\"a_node14\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_1 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-45.77\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-48.97\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-36.97\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- opt_standard_scaler&#45;&gt;float32_transform_1 -->\n",
              "<g id=\"edge12\" class=\"edge\">\n",
              "<title>opt_standard_scaler&#45;&gt;float32_transform_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M718.98,-43.09C727.31,-43.4 736.32,-43.74 745.03,-44.07\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.93,-47.57 755.06,-44.45 745.2,-40.58 744.93,-47.57\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1&#45;&gt;concat_features -->\n",
              "<g id=\"edge14\" class=\"edge\">\n",
              "<title>float32_transform_1&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.75,-53.05C834.41,-55.2 845.1,-57.58 855.18,-59.82\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"854.47,-63.25 865,-62.01 856,-56.42 854.47,-63.25\"/>\n",
              "</g>\n",
              "<!-- numpy_permute_array -->\n",
              "<g id=\"node16\" class=\"node\">\n",
              "<title>numpy_permute_array</title>\n",
              "<g id=\"a_node16\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_permute_array.html\" xlink:title=\"numpy_permute_array = NumpyPermuteArray(axis=0, permutation_indices=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1000.24\" cy=\"-68.77\" rx=\"34.8\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-77.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-65.97\" font-family=\"Times,serif\" font-size=\"11.00\">Permute&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-53.97\" font-family=\"Times,serif\" font-size=\"11.00\">Array</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- concat_features&#45;&gt;numpy_permute_array -->\n",
              "<g id=\"edge15\" class=\"edge\">\n",
              "<title>concat_features&#45;&gt;numpy_permute_array</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M929.6,-68.77C937.81,-68.77 946.78,-68.77 955.46,-68.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"955.46,-72.27 965.46,-68.77 955.46,-65.27 955.46,-72.27\"/>\n",
              "</g>\n",
              "<!-- t_no_op -->\n",
              "<g id=\"node17\" class=\"node\">\n",
              "<title>t_no_op</title>\n",
              "<g id=\"a_node17\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.t_no_op.html\" xlink:title=\"t_no_op = TNoOp(fun=None, name=&#39;no_action&#39;, datatypes=&#39;x&#39;, feat_constraints=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1097.89\" cy=\"-68.77\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-77.97\" font-family=\"Times,serif\" font-size=\"11.00\">T&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-65.97\" font-family=\"Times,serif\" font-size=\"11.00\">No&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-53.97\" font-family=\"Times,serif\" font-size=\"11.00\">Op</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_permute_array&#45;&gt;t_no_op -->\n",
              "<g id=\"edge16\" class=\"edge\">\n",
              "<title>numpy_permute_array&#45;&gt;t_no_op</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1034.96,-68.77C1043.21,-68.77 1052.07,-68.77 1060.43,-68.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1060.68,-72.27 1070.68,-68.77 1060.68,-65.27 1060.68,-72.27\"/>\n",
              "</g>\n",
              "</g>\n",
              "</svg>\n"
            ],
            "text/plain": [
              "<graphviz.dot.Digraph at 0x7fd8cfc8a990>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "prefix = hist_pipeline.remove_last().freeze_trainable()\n",
        "prefix.visualize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Next, add a new final step, which consists of a choice of three\n",
        "estimators. In this code, `|` is the or combinator (algorithmic\n",
        "choice). It defines a search space for another optimizer run."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": [
              "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
              "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
              " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
              "<!-- Generated by graphviz version 2.44.1 (20200629.0846)\n",
              " -->\n",
              "<!-- Title: cluster:(root) Pages: 1 -->\n",
              "<svg width=\"1239pt\" height=\"200pt\"\n",
              " viewBox=\"0.00 0.00 1238.89 199.77\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
              "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 195.77)\">\n",
              "<title>cluster:(root)</title>\n",
              "<g id=\"a_graph0\"><a xlink:title=\"(root) = ...\">\n",
              "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-195.77 1234.89,-195.77 1234.89,4 -4,4\"/>\n",
              "</a>\n",
              "</g>\n",
              "<g id=\"clust1\" class=\"cluster\">\n",
              "<title>cluster:choice</title>\n",
              "<g id=\"a_clust1\"><a xlink:title=\"choice = lr | tree | knn\">\n",
              "<polygon fill=\"#7ec0ee\" stroke=\"black\" points=\"1152.89,-8 1152.89,-169 1222.89,-169 1222.89,-8 1152.89,-8\"/>\n",
              "<text text-anchor=\"middle\" x=\"1187.89\" y=\"-153.8\" font-family=\"Times,serif\" font-size=\"14.00\">Choice</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0 -->\n",
              "<g id=\"node1\" class=\"node\">\n",
              "<title>numpy_column_selector_0</title>\n",
              "<g id=\"a_node1\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_0 = NumpyColumnSelector(columns=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"33.23\" cy=\"-155\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-164.2\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-152.2\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-140.2\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings -->\n",
              "<g id=\"node2\" class=\"node\">\n",
              "<title>compress_strings</title>\n",
              "<g id=\"a_node2\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.compress_strings.html\" xlink:title=\"compress_strings = CompressStrings(compress_type=&#39;hash&#39;, dtypes_list=[&#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_s...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"142.77\" cy=\"-155\" rx=\"40.11\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-158.2\" font-family=\"Times,serif\" font-size=\"11.00\">Compress&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-146.2\" font-family=\"Times,serif\" font-size=\"11.00\">Strings</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0&#45;&gt;compress_strings -->\n",
              "<g id=\"edge1\" class=\"edge\">\n",
              "<title>numpy_column_selector_0&#45;&gt;compress_strings</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M66.5,-155C74.59,-155 83.46,-155 92.19,-155\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"92.3,-158.5 102.3,-155 92.3,-151.5 92.3,-158.5\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0 -->\n",
              "<g id=\"node3\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_0</title>\n",
              "<g id=\"a_node3\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_0 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-155\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-170.2\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-158.2\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-146.2\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-134.2\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings&#45;&gt;numpy_replace_missing_values_0 -->\n",
              "<g id=\"edge2\" class=\"edge\">\n",
              "<title>compress_strings&#45;&gt;numpy_replace_missing_values_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M183.11,-155C191.43,-155 200.26,-155 208.74,-155\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"208.85,-158.5 218.85,-155 208.85,-151.5 208.85,-158.5\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values -->\n",
              "<g id=\"node4\" class=\"node\">\n",
              "<title>numpy_replace_unknown_values</title>\n",
              "<g id=\"a_node4\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_unknown_values.html\" xlink:title=\"numpy_replace_unknown_values = NumpyReplaceUnknownValues(filling_values=float(&#39;nan&#39;), filling_values_list=[float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-155\" rx=\"38.78\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-170.2\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-158.2\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-146.2\" font-family=\"Times,serif\" font-size=\"11.00\">Unknown&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-134.2\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values -->\n",
              "<g id=\"edge3\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M288.46,-155C296.6,-155 305.47,-155 314.13,-155\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"314.16,-158.5 324.16,-155 314.16,-151.5 314.16,-158.5\"/>\n",
              "</g>\n",
              "<!-- boolean2float -->\n",
              "<g id=\"node5\" class=\"node\">\n",
              "<title>boolean2float</title>\n",
              "<g id=\"a_node5\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.boolean2float.html\" xlink:title=\"boolean2float = boolean2float()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-152\" rx=\"37.91\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-149.2\" font-family=\"Times,serif\" font-size=\"11.00\">boolean2float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values&#45;&gt;boolean2float -->\n",
              "<g id=\"edge4\" class=\"edge\">\n",
              "<title>numpy_replace_unknown_values&#45;&gt;boolean2float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M402.41,-153.97C410.6,-153.75 419.36,-153.51 427.85,-153.28\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"428.12,-156.77 438.02,-153 427.93,-149.78 428.12,-156.77\"/>\n",
              "</g>\n",
              "<!-- cat_imputer -->\n",
              "<g id=\"node6\" class=\"node\">\n",
              "<title>cat_imputer</title>\n",
              "<g id=\"a_node6\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_imputer.html\" xlink:title=\"cat_imputer = CatImputer(strategy=&#39;most_frequent&#39;, missing_values=float(&#39;nan&#39;), sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-150\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-153.2\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-141.2\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- boolean2float&#45;&gt;cat_imputer -->\n",
              "<g id=\"edge5\" class=\"edge\">\n",
              "<title>boolean2float&#45;&gt;cat_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M514.07,-151.28C522.45,-151.11 531.39,-150.94 539.88,-150.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"540.02,-154.27 549.95,-150.58 539.88,-147.27 540.02,-154.27\"/>\n",
              "</g>\n",
              "<!-- cat_encoder -->\n",
              "<g id=\"node7\" class=\"node\">\n",
              "<title>cat_encoder</title>\n",
              "<g id=\"a_node7\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_encoder.html\" xlink:title=\"cat_encoder = CatEncoder(encoding=&#39;ordinal&#39;, categories=&#39;auto&#39;, dtype=np.float64, handle_unknown=&#39;error&#39;, sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-149\" rx=\"31.64\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-152.2\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-140.2\" font-family=\"Times,serif\" font-size=\"11.00\">Encoder</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_imputer&#45;&gt;cat_encoder -->\n",
              "<g id=\"edge6\" class=\"edge\">\n",
              "<title>cat_imputer&#45;&gt;cat_encoder</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-149.71C620.35,-149.61 630.77,-149.51 640.7,-149.41\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"640.84,-152.91 650.8,-149.31 640.77,-145.91 640.84,-152.91\"/>\n",
              "</g>\n",
              "<!-- float32_transform_0 -->\n",
              "<g id=\"node8\" class=\"node\">\n",
              "<title>float32_transform_0</title>\n",
              "<g id=\"a_node8\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_0 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-146\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-149.2\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-137.2\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_encoder&#45;&gt;float32_transform_0 -->\n",
              "<g id=\"edge7\" class=\"edge\">\n",
              "<title>cat_encoder&#45;&gt;float32_transform_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M714.92,-148.12C724.18,-147.86 734.54,-147.57 744.53,-147.29\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.85,-150.78 754.75,-147 744.66,-143.78 744.85,-150.78\"/>\n",
              "</g>\n",
              "<!-- concat_features -->\n",
              "<g id=\"node15\" class=\"node\">\n",
              "<title>concat_features</title>\n",
              "<g id=\"a_node15\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.lale.concat_features.html\" xlink:title=\"concat_features = ConcatFeatures()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"896.36\" cy=\"-120\" rx=\"33.47\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-123.2\" font-family=\"Times,serif\" font-size=\"11.00\">Concat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-111.2\" font-family=\"Times,serif\" font-size=\"11.00\">Features</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float32_transform_0&#45;&gt;concat_features -->\n",
              "<g id=\"edge13\" class=\"edge\">\n",
              "<title>float32_transform_0&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.18,-137.92C834.15,-135.41 845.25,-132.61 855.66,-129.99\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"856.53,-133.38 865.38,-127.55 854.82,-126.59 856.53,-133.38\"/>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1 -->\n",
              "<g id=\"node9\" class=\"node\">\n",
              "<title>numpy_column_selector_1</title>\n",
              "<g id=\"a_node9\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_1 = NumpyColumnSelector(columns=[4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-83\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-92.2\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-80.2\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-68.2\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float -->\n",
              "<g id=\"node10\" class=\"node\">\n",
              "<title>float_str2_float</title>\n",
              "<g id=\"a_node10\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float_str2_float.html\" xlink:title=\"float_str2_float = FloatStr2Float(dtypes_list=[&#39;int_num&#39;], missing_values_reference_list=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-83\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-92.2\" font-family=\"Times,serif\" font-size=\"11.00\">Float&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-80.2\" font-family=\"Times,serif\" font-size=\"11.00\">Str2&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-68.2\" font-family=\"Times,serif\" font-size=\"11.00\">Float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1&#45;&gt;float_str2_float -->\n",
              "<g id=\"edge8\" class=\"edge\">\n",
              "<title>numpy_column_selector_1&#45;&gt;float_str2_float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M286.99,-83C299.27,-83 313.34,-83 325.96,-83\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"326.16,-86.5 336.16,-83 326.16,-79.5 326.16,-86.5\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1 -->\n",
              "<g id=\"node11\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_1</title>\n",
              "<g id=\"a_node11\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_1 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-88\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-103.2\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-91.2\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-79.2\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-67.2\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float&#45;&gt;numpy_replace_missing_values_1 -->\n",
              "<g id=\"edge9\" class=\"edge\">\n",
              "<title>float_str2_float&#45;&gt;numpy_replace_missing_values_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M390.25,-84.17C402.39,-84.72 417.18,-85.39 430.99,-86.01\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"431.15,-89.52 441.3,-86.47 431.47,-82.53 431.15,-89.52\"/>\n",
              "</g>\n",
              "<!-- num_imputer -->\n",
              "<g id=\"node12\" class=\"node\">\n",
              "<title>num_imputer</title>\n",
              "<g id=\"a_node12\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.num_imputer.html\" xlink:title=\"num_imputer = NumImputer(strategy=&#39;median&#39;, missing_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-91\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-94.2\" font-family=\"Times,serif\" font-size=\"11.00\">Num&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-82.2\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1&#45;&gt;num_imputer -->\n",
              "<g id=\"edge10\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_1&#45;&gt;num_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M510.91,-88.99C520.15,-89.26 530.24,-89.56 539.78,-89.84\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"539.74,-93.34 549.83,-90.13 539.94,-86.34 539.74,-93.34\"/>\n",
              "</g>\n",
              "<!-- opt_standard_scaler -->\n",
              "<g id=\"node13\" class=\"node\">\n",
              "<title>opt_standard_scaler</title>\n",
              "<g id=\"a_node13\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.opt_standard_scaler.html\" xlink:title=\"opt_standard_scaler = OptStandardScaler(num_scaler_copy=None, num_scaler_with_mean=None, num_scaler_with_std=None, use_scaler_flag=False)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-93\" rx=\"36.12\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-102.2\" font-family=\"Times,serif\" font-size=\"11.00\">Opt&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-90.2\" font-family=\"Times,serif\" font-size=\"11.00\">Standard&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-78.2\" font-family=\"Times,serif\" font-size=\"11.00\">Scaler</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- num_imputer&#45;&gt;opt_standard_scaler -->\n",
              "<g id=\"edge11\" class=\"edge\">\n",
              "<title>num_imputer&#45;&gt;opt_standard_scaler</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-91.59C619.14,-91.75 628.07,-91.93 636.8,-92.1\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"636.81,-95.6 646.88,-92.3 636.95,-88.6 636.81,-95.6\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1 -->\n",
              "<g id=\"node14\" class=\"node\">\n",
              "<title>float32_transform_1</title>\n",
              "<g id=\"a_node14\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_1 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-97\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-100.2\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-88.2\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- opt_standard_scaler&#45;&gt;float32_transform_1 -->\n",
              "<g id=\"edge12\" class=\"edge\">\n",
              "<title>opt_standard_scaler&#45;&gt;float32_transform_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M718.98,-94.32C727.31,-94.63 736.32,-94.97 745.03,-95.3\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.93,-98.8 755.06,-95.68 745.2,-91.81 744.93,-98.8\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1&#45;&gt;concat_features -->\n",
              "<g id=\"edge14\" class=\"edge\">\n",
              "<title>float32_transform_1&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.75,-104.28C834.41,-106.43 845.1,-108.81 855.18,-111.05\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"854.47,-114.48 865,-113.24 856,-107.65 854.47,-114.48\"/>\n",
              "</g>\n",
              "<!-- numpy_permute_array -->\n",
              "<g id=\"node16\" class=\"node\">\n",
              "<title>numpy_permute_array</title>\n",
              "<g id=\"a_node16\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_permute_array.html\" xlink:title=\"numpy_permute_array = NumpyPermuteArray(axis=0, permutation_indices=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1000.24\" cy=\"-120\" rx=\"34.8\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-129.2\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-117.2\" font-family=\"Times,serif\" font-size=\"11.00\">Permute&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-105.2\" font-family=\"Times,serif\" font-size=\"11.00\">Array</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- concat_features&#45;&gt;numpy_permute_array -->\n",
              "<g id=\"edge15\" class=\"edge\">\n",
              "<title>concat_features&#45;&gt;numpy_permute_array</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M929.6,-120C937.81,-120 946.78,-120 955.46,-120\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"955.46,-123.5 965.46,-120 955.46,-116.5 955.46,-123.5\"/>\n",
              "</g>\n",
              "<!-- t_no_op -->\n",
              "<g id=\"node17\" class=\"node\">\n",
              "<title>t_no_op</title>\n",
              "<g id=\"a_node17\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.t_no_op.html\" xlink:title=\"t_no_op = TNoOp(fun=None, name=&#39;no_action&#39;, datatypes=&#39;x&#39;, feat_constraints=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1097.89\" cy=\"-120\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-129.2\" font-family=\"Times,serif\" font-size=\"11.00\">T&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-117.2\" font-family=\"Times,serif\" font-size=\"11.00\">No&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-105.2\" font-family=\"Times,serif\" font-size=\"11.00\">Op</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_permute_array&#45;&gt;t_no_op -->\n",
              "<g id=\"edge16\" class=\"edge\">\n",
              "<title>numpy_permute_array&#45;&gt;t_no_op</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1034.96,-120C1043.21,-120 1052.07,-120 1060.43,-120\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1060.68,-123.5 1070.68,-120 1060.68,-116.5 1060.68,-123.5\"/>\n",
              "</g>\n",
              "<!-- lr -->\n",
              "<g id=\"node18\" class=\"node\">\n",
              "<title>lr</title>\n",
              "<g id=\"a_node18\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.sklearn.logistic_regression.html\" xlink:title=\"lr = LR\">\n",
              "<ellipse fill=\"#7ec0ee\" stroke=\"black\" cx=\"1187.89\" cy=\"-120\" rx=\"27\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"1187.89\" y=\"-117.2\" font-family=\"Times,serif\" font-size=\"11.00\">LR</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- t_no_op&#45;&gt;lr -->\n",
              "<g id=\"edge17\" class=\"edge\">\n",
              "<title>t_no_op&#45;&gt;lr</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1125.29,-120C1133.28,-120 1142.2,-120 1150.72,-120\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1142.89,-123.5 1152.89,-120 1142.89,-116.5 1142.89,-123.5\"/>\n",
              "</g>\n",
              "<!-- tree -->\n",
              "<g id=\"node19\" class=\"node\">\n",
              "<title>tree</title>\n",
              "<g id=\"a_node19\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.sklearn.decision_tree_classifier.html\" xlink:title=\"tree = Tree\">\n",
              "<ellipse fill=\"#7ec0ee\" stroke=\"black\" cx=\"1187.89\" cy=\"-77\" rx=\"27\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"1187.89\" y=\"-74.2\" font-family=\"Times,serif\" font-size=\"11.00\">Tree</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- knn -->\n",
              "<g id=\"node20\" class=\"node\">\n",
              "<title>knn</title>\n",
              "<g id=\"a_node20\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.sklearn.k_neighbors_classifier.html\" xlink:title=\"knn = KNN\">\n",
              "<ellipse fill=\"#7ec0ee\" stroke=\"black\" cx=\"1187.89\" cy=\"-34\" rx=\"27\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"1187.89\" y=\"-31.2\" font-family=\"Times,serif\" font-size=\"11.00\">KNN</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "</g>\n",
              "</svg>\n"
            ],
            "text/plain": [
              "<graphviz.dot.Digraph at 0x7fd8d30cb890>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "new_pipeline = prefix >> (LR | Tree | KNN)\n",
        "new_pipeline.visualize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### New optimizer `Hyperopt` configuration and training\n",
        "\n",
        "To automatically select the algorithm and tune its hyperparameters, we\n",
        "create an instance of the `Hyperopt` optimizer and `fit` it to the\n",
        "data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(train_X, train_y, test_size=0.15, random_state=33)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "100%|██████████| 20/20 [02:12<00:00,  6.63s/trial, best loss: -0.8336535460343567]\n"
          ]
        }
      ],
      "source": [
        "hyperopt = Hyperopt(estimator=new_pipeline, cv=3, max_evals=20, scoring='roc_auc')\n",
        "hyperopt_pipelines = hyperopt.fit(X_train, y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "pipeline_model = hyperopt_pipelines.get_pipeline()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Pipeline model tests and visualization"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "roc_auc_score 75.1%\n"
          ]
        }
      ],
      "source": [
        "from sklearn.metrics import roc_auc_score\n",
        "\n",
        "predicted_y = pipeline_model.predict(X_test)\n",
        "score = roc_auc_score(predicted_y=='Risk', y_test=='Risk')\n",
        "print(f'roc_auc_score {score:.1%}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "image/svg+xml": [
              "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
              "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
              " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
              "<!-- Generated by graphviz version 2.44.1 (20200629.0846)\n",
              " -->\n",
              "<!-- Title: cluster:(root) Pages: 1 -->\n",
              "<svg width=\"1223pt\" height=\"149pt\"\n",
              " viewBox=\"0.00 0.00 1222.89 148.54\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
              "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 144.54)\">\n",
              "<title>cluster:(root)</title>\n",
              "<g id=\"a_graph0\"><a xlink:title=\"(root) = ...\">\n",
              "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-144.54 1218.89,-144.54 1218.89,4 -4,4\"/>\n",
              "</a>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0 -->\n",
              "<g id=\"node1\" class=\"node\">\n",
              "<title>numpy_column_selector_0</title>\n",
              "<g id=\"a_node1\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_0 = NumpyColumnSelector(columns=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"33.23\" cy=\"-103.77\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-112.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-100.97\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"33.23\" y=\"-88.97\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings -->\n",
              "<g id=\"node2\" class=\"node\">\n",
              "<title>compress_strings</title>\n",
              "<g id=\"a_node2\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.compress_strings.html\" xlink:title=\"compress_strings = CompressStrings(compress_type=&#39;hash&#39;, dtypes_list=[&#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;int_num&#39;, &#39;char_str&#39;, &#39;char_s...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"142.77\" cy=\"-103.77\" rx=\"40.11\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Compress&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"142.77\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Strings</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_0&#45;&gt;compress_strings -->\n",
              "<g id=\"edge1\" class=\"edge\">\n",
              "<title>numpy_column_selector_0&#45;&gt;compress_strings</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M66.5,-103.77C74.59,-103.77 83.46,-103.77 92.19,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"92.3,-107.27 102.3,-103.77 92.3,-100.27 92.3,-107.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0 -->\n",
              "<g id=\"node3\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_0</title>\n",
              "<g id=\"a_node3\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_0 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-103.77\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-118.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-82.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- compress_strings&#45;&gt;numpy_replace_missing_values_0 -->\n",
              "<g id=\"edge2\" class=\"edge\">\n",
              "<title>compress_strings&#45;&gt;numpy_replace_missing_values_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M183.11,-103.77C191.43,-103.77 200.26,-103.77 208.74,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"208.85,-107.27 218.85,-103.77 208.85,-100.27 208.85,-107.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values -->\n",
              "<g id=\"node4\" class=\"node\">\n",
              "<title>numpy_replace_unknown_values</title>\n",
              "<g id=\"a_node4\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_unknown_values.html\" xlink:title=\"numpy_replace_unknown_values = NumpyReplaceUnknownValues(filling_values=float(&#39;nan&#39;), filling_values_list=[float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(&#39;nan&#39;), float(...)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-103.77\" rx=\"38.78\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-118.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-106.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-94.97\" font-family=\"Times,serif\" font-size=\"11.00\">Unknown&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-82.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values -->\n",
              "<g id=\"edge3\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_0&#45;&gt;numpy_replace_unknown_values</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M288.46,-103.77C296.6,-103.77 305.47,-103.77 314.13,-103.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"314.16,-107.27 324.16,-103.77 314.16,-100.27 314.16,-107.27\"/>\n",
              "</g>\n",
              "<!-- boolean2float -->\n",
              "<g id=\"node5\" class=\"node\">\n",
              "<title>boolean2float</title>\n",
              "<g id=\"a_node5\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.boolean2float.html\" xlink:title=\"boolean2float = boolean2float()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-100.77\" rx=\"37.91\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-97.97\" font-family=\"Times,serif\" font-size=\"11.00\">boolean2float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_unknown_values&#45;&gt;boolean2float -->\n",
              "<g id=\"edge4\" class=\"edge\">\n",
              "<title>numpy_replace_unknown_values&#45;&gt;boolean2float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M402.41,-102.74C410.6,-102.52 419.36,-102.28 427.85,-102.05\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"428.12,-105.54 438.02,-101.77 427.93,-98.55 428.12,-105.54\"/>\n",
              "</g>\n",
              "<!-- cat_imputer -->\n",
              "<g id=\"node6\" class=\"node\">\n",
              "<title>cat_imputer</title>\n",
              "<g id=\"a_node6\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_imputer.html\" xlink:title=\"cat_imputer = CatImputer(strategy=&#39;most_frequent&#39;, missing_values=float(&#39;nan&#39;), sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-98.77\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-101.97\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-89.97\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- boolean2float&#45;&gt;cat_imputer -->\n",
              "<g id=\"edge5\" class=\"edge\">\n",
              "<title>boolean2float&#45;&gt;cat_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M514.07,-100.05C522.45,-99.88 531.39,-99.71 539.88,-99.54\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"540.02,-103.04 549.95,-99.35 539.88,-96.04 540.02,-103.04\"/>\n",
              "</g>\n",
              "<!-- cat_encoder -->\n",
              "<g id=\"node7\" class=\"node\">\n",
              "<title>cat_encoder</title>\n",
              "<g id=\"a_node7\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.cat_encoder.html\" xlink:title=\"cat_encoder = CatEncoder(encoding=&#39;ordinal&#39;, categories=&#39;auto&#39;, dtype=np.float64, handle_unknown=&#39;error&#39;, sklearn_version_family=&#39;23&#39;)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-97.77\" rx=\"31.64\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-100.97\" font-family=\"Times,serif\" font-size=\"11.00\">Cat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-88.97\" font-family=\"Times,serif\" font-size=\"11.00\">Encoder</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_imputer&#45;&gt;cat_encoder -->\n",
              "<g id=\"edge6\" class=\"edge\">\n",
              "<title>cat_imputer&#45;&gt;cat_encoder</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-98.47C620.35,-98.38 630.77,-98.28 640.7,-98.18\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"640.84,-101.68 650.8,-98.08 640.77,-94.68 640.84,-101.68\"/>\n",
              "</g>\n",
              "<!-- float32_transform_0 -->\n",
              "<g id=\"node8\" class=\"node\">\n",
              "<title>float32_transform_0</title>\n",
              "<g id=\"a_node8\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_0 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-94.77\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-97.97\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-85.97\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- cat_encoder&#45;&gt;float32_transform_0 -->\n",
              "<g id=\"edge7\" class=\"edge\">\n",
              "<title>cat_encoder&#45;&gt;float32_transform_0</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M714.92,-96.89C724.18,-96.63 734.54,-96.34 744.53,-96.06\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.85,-99.55 754.75,-95.77 744.66,-92.55 744.85,-99.55\"/>\n",
              "</g>\n",
              "<!-- concat_features -->\n",
              "<g id=\"node15\" class=\"node\">\n",
              "<title>concat_features</title>\n",
              "<g id=\"a_node15\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.lale.concat_features.html\" xlink:title=\"concat_features = ConcatFeatures()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"896.36\" cy=\"-68.77\" rx=\"33.47\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-71.97\" font-family=\"Times,serif\" font-size=\"11.00\">Concat&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"896.36\" y=\"-59.97\" font-family=\"Times,serif\" font-size=\"11.00\">Features</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float32_transform_0&#45;&gt;concat_features -->\n",
              "<g id=\"edge13\" class=\"edge\">\n",
              "<title>float32_transform_0&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.18,-86.69C834.15,-84.18 845.25,-81.38 855.66,-78.76\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"856.53,-82.15 865.38,-76.32 854.82,-75.36 856.53,-82.15\"/>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1 -->\n",
              "<g id=\"node9\" class=\"node\">\n",
              "<title>numpy_column_selector_1</title>\n",
              "<g id=\"a_node9\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_column_selector.html\" xlink:title=\"numpy_column_selector_1 = NumpyColumnSelector(columns=[4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"253.73\" cy=\"-31.77\" rx=\"33.47\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-40.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-28.97\" font-family=\"Times,serif\" font-size=\"11.00\">Column&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"253.73\" y=\"-16.97\" font-family=\"Times,serif\" font-size=\"11.00\">Selector</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float -->\n",
              "<g id=\"node10\" class=\"node\">\n",
              "<title>float_str2_float</title>\n",
              "<g id=\"a_node10\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float_str2_float.html\" xlink:title=\"float_str2_float = FloatStr2Float(dtypes_list=[&#39;int_num&#39;], missing_values_reference_list=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"363.27\" cy=\"-31.77\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-40.97\" font-family=\"Times,serif\" font-size=\"11.00\">Float&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-28.97\" font-family=\"Times,serif\" font-size=\"11.00\">Str2&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"363.27\" y=\"-16.97\" font-family=\"Times,serif\" font-size=\"11.00\">Float</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_column_selector_1&#45;&gt;float_str2_float -->\n",
              "<g id=\"edge8\" class=\"edge\">\n",
              "<title>numpy_column_selector_1&#45;&gt;float_str2_float</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M286.99,-31.77C299.27,-31.77 313.34,-31.77 325.96,-31.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"326.16,-35.27 336.16,-31.77 326.16,-28.27 326.16,-35.27\"/>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1 -->\n",
              "<g id=\"node11\" class=\"node\">\n",
              "<title>numpy_replace_missing_values_1</title>\n",
              "<g id=\"a_node11\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_replace_missing_values.html\" xlink:title=\"numpy_replace_missing_values_1 = NumpyReplaceMissingValues(missing_values=[], filling_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"476.11\" cy=\"-36.77\" rx=\"34.8\" ry=\"36.54\"/>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-51.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-39.97\" font-family=\"Times,serif\" font-size=\"11.00\">Replace&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-27.97\" font-family=\"Times,serif\" font-size=\"11.00\">Missing&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"476.11\" y=\"-15.97\" font-family=\"Times,serif\" font-size=\"11.00\">Values</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- float_str2_float&#45;&gt;numpy_replace_missing_values_1 -->\n",
              "<g id=\"edge9\" class=\"edge\">\n",
              "<title>float_str2_float&#45;&gt;numpy_replace_missing_values_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M390.25,-32.94C402.39,-33.49 417.18,-34.16 430.99,-34.78\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"431.15,-38.29 441.3,-35.24 431.47,-31.3 431.15,-38.29\"/>\n",
              "</g>\n",
              "<!-- num_imputer -->\n",
              "<g id=\"node12\" class=\"node\">\n",
              "<title>num_imputer</title>\n",
              "<g id=\"a_node12\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.num_imputer.html\" xlink:title=\"num_imputer = NumImputer(strategy=&#39;median&#39;, missing_values=float(&#39;nan&#39;))\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"580.47\" cy=\"-39.77\" rx=\"30.31\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-42.97\" font-family=\"Times,serif\" font-size=\"11.00\">Num&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"580.47\" y=\"-30.97\" font-family=\"Times,serif\" font-size=\"11.00\">Imputer</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_replace_missing_values_1&#45;&gt;num_imputer -->\n",
              "<g id=\"edge10\" class=\"edge\">\n",
              "<title>numpy_replace_missing_values_1&#45;&gt;num_imputer</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M510.91,-37.76C520.15,-38.03 530.24,-38.33 539.78,-38.61\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"539.74,-42.11 549.83,-38.9 539.94,-35.11 539.74,-42.11\"/>\n",
              "</g>\n",
              "<!-- opt_standard_scaler -->\n",
              "<g id=\"node13\" class=\"node\">\n",
              "<title>opt_standard_scaler</title>\n",
              "<g id=\"a_node13\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.opt_standard_scaler.html\" xlink:title=\"opt_standard_scaler = OptStandardScaler(num_scaler_copy=None, num_scaler_with_mean=None, num_scaler_with_std=None, use_scaler_flag=False)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"682.94\" cy=\"-41.77\" rx=\"36.12\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-50.97\" font-family=\"Times,serif\" font-size=\"11.00\">Opt&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-38.97\" font-family=\"Times,serif\" font-size=\"11.00\">Standard&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"682.94\" y=\"-26.97\" font-family=\"Times,serif\" font-size=\"11.00\">Scaler</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- num_imputer&#45;&gt;opt_standard_scaler -->\n",
              "<g id=\"edge11\" class=\"edge\">\n",
              "<title>num_imputer&#45;&gt;opt_standard_scaler</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M611.07,-40.36C619.14,-40.52 628.07,-40.7 636.8,-40.87\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"636.81,-44.37 646.88,-41.07 636.95,-37.37 636.81,-44.37\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1 -->\n",
              "<g id=\"node14\" class=\"node\">\n",
              "<title>float32_transform_1</title>\n",
              "<g id=\"a_node14\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.float32_transform.html\" xlink:title=\"float32_transform_1 = float32_transform()\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"791.06\" cy=\"-45.77\" rx=\"36.12\" ry=\"19.6\"/>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-48.97\" font-family=\"Times,serif\" font-size=\"11.00\">float32_&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"791.06\" y=\"-36.97\" font-family=\"Times,serif\" font-size=\"11.00\">transform</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- opt_standard_scaler&#45;&gt;float32_transform_1 -->\n",
              "<g id=\"edge12\" class=\"edge\">\n",
              "<title>opt_standard_scaler&#45;&gt;float32_transform_1</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M718.98,-43.09C727.31,-43.4 736.32,-43.74 745.03,-44.07\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"744.93,-47.57 755.06,-44.45 745.2,-40.58 744.93,-47.57\"/>\n",
              "</g>\n",
              "<!-- float32_transform_1&#45;&gt;concat_features -->\n",
              "<g id=\"edge14\" class=\"edge\">\n",
              "<title>float32_transform_1&#45;&gt;concat_features</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M824.75,-53.05C834.41,-55.2 845.1,-57.58 855.18,-59.82\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"854.47,-63.25 865,-62.01 856,-56.42 854.47,-63.25\"/>\n",
              "</g>\n",
              "<!-- numpy_permute_array -->\n",
              "<g id=\"node16\" class=\"node\">\n",
              "<title>numpy_permute_array</title>\n",
              "<g id=\"a_node16\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.numpy_permute_array.html\" xlink:title=\"numpy_permute_array = NumpyPermuteArray(axis=0, permutation_indices=[0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 4])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1000.24\" cy=\"-68.77\" rx=\"34.8\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-77.97\" font-family=\"Times,serif\" font-size=\"11.00\">Numpy&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-65.97\" font-family=\"Times,serif\" font-size=\"11.00\">Permute&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1000.24\" y=\"-53.97\" font-family=\"Times,serif\" font-size=\"11.00\">Array</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- concat_features&#45;&gt;numpy_permute_array -->\n",
              "<g id=\"edge15\" class=\"edge\">\n",
              "<title>concat_features&#45;&gt;numpy_permute_array</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M929.6,-68.77C937.81,-68.77 946.78,-68.77 955.46,-68.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"955.46,-72.27 965.46,-68.77 955.46,-65.27 955.46,-72.27\"/>\n",
              "</g>\n",
              "<!-- t_no_op -->\n",
              "<g id=\"node17\" class=\"node\">\n",
              "<title>t_no_op</title>\n",
              "<g id=\"a_node17\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.autoai_libs.t_no_op.html\" xlink:title=\"t_no_op = TNoOp(fun=None, name=&#39;no_action&#39;, datatypes=&#39;x&#39;, feat_constraints=[])\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1097.89\" cy=\"-68.77\" rx=\"27\" ry=\"28.07\"/>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-77.97\" font-family=\"Times,serif\" font-size=\"11.00\">T&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-65.97\" font-family=\"Times,serif\" font-size=\"11.00\">No&#45;</text>\n",
              "<text text-anchor=\"middle\" x=\"1097.89\" y=\"-53.97\" font-family=\"Times,serif\" font-size=\"11.00\">Op</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- numpy_permute_array&#45;&gt;t_no_op -->\n",
              "<g id=\"edge16\" class=\"edge\">\n",
              "<title>numpy_permute_array&#45;&gt;t_no_op</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1034.96,-68.77C1043.21,-68.77 1052.07,-68.77 1060.43,-68.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1060.68,-72.27 1070.68,-68.77 1060.68,-65.27 1060.68,-72.27\"/>\n",
              "</g>\n",
              "<!-- lr -->\n",
              "<g id=\"node18\" class=\"node\">\n",
              "<title>lr</title>\n",
              "<g id=\"a_node18\"><a xlink:href=\"https://lale.readthedocs.io/en/latest/modules/lale.lib.sklearn.logistic_regression.html\" xlink:title=\"lr = LR(intercept_scaling=0.02888452505133443, max_iter=737, multi_class=&#39;multinomial&#39;, tol=0.006400103615640232)\">\n",
              "<ellipse fill=\"white\" stroke=\"black\" cx=\"1187.89\" cy=\"-68.77\" rx=\"27\" ry=\"18\"/>\n",
              "<text text-anchor=\"middle\" x=\"1187.89\" y=\"-65.97\" font-family=\"Times,serif\" font-size=\"11.00\">LR</text>\n",
              "</a>\n",
              "</g>\n",
              "</g>\n",
              "<!-- t_no_op&#45;&gt;lr -->\n",
              "<g id=\"edge17\" class=\"edge\">\n",
              "<title>t_no_op&#45;&gt;lr</title>\n",
              "<path fill=\"none\" stroke=\"black\" d=\"M1125.29,-68.77C1133.28,-68.77 1142.2,-68.77 1150.72,-68.77\"/>\n",
              "<polygon fill=\"black\" stroke=\"black\" points=\"1150.81,-72.27 1160.81,-68.77 1150.81,-65.27 1150.81,-72.27\"/>\n",
              "</g>\n",
              "</g>\n",
              "</svg>\n"
            ],
            "text/plain": [
              "<graphviz.dot.Digraph at 0x7fd8d2fb7c90>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "pipeline_model.visualize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"scoring\"></a>\n",
        "## 7. Deploy and Score\n",
        "\n",
        "In this section you will learn how to deploy and score pipeline model as webservice and batch using WML instance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Webservice deployment creation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Preparing an AutoAI Deployment...\n",
            "Published model uid: 014ffcfa-dcf1-4c26-add5-369243b140dd\n",
            "Deploying model 014ffcfa-dcf1-4c26-add5-369243b140dd using V4 client.\n",
            "\n",
            "\n",
            "#######################################################################################\n",
            "\n",
            "Synchronous deployment creation for uid: '014ffcfa-dcf1-4c26-add5-369243b140dd' started\n",
            "\n",
            "#######################################################################################\n",
            "\n",
            "\n",
            "initializing..\n",
            "ready\n",
            "\n",
            "\n",
            "------------------------------------------------------------------------------------------------\n",
            "Successfully finished deployment creation, deployment_uid='e0f705fd-2fcb-437f-bc83-b17d5296b01d'\n",
            "------------------------------------------------------------------------------------------------\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "from ibm_watsonx_ai.deployment import WebService\n",
        "\n",
        "service = WebService(credentials, source_space_id=space_id)\n",
        "\n",
        "service.create(\n",
        "    experiment_run_id=run_id,\n",
        "    model='Pipeline_1', \n",
        "    deployment_name=\"Credit Risk Deployment AutoAI\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Deployment object could be printed to show basic information:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "print(service)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "To show all available information about the deployment use the `.get_params()` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'entity': {'asset': {'id': '014ffcfa-dcf1-4c26-add5-369243b140dd'},\n",
              "  'custom': {},\n",
              "  'deployed_asset_type': 'model',\n",
              "  'hardware_spec': {'id': 'c076e82c-b2a7-4d20-9c0f-1f0c2fdf5a24',\n",
              "   'name': 'M',\n",
              "   'num_nodes': 1},\n",
              "  'hybrid_pipeline_hardware_specs': [{'hardware_spec': {'name': 'S',\n",
              "     'num_nodes': 1},\n",
              "    'node_runtime_id': 'auto_ai.kb'}],\n",
              "  'name': 'Credit Risk Deployment AutoAI',\n",
              "  'online': {},\n",
              "  'space_id': 'fbd2dc3a-634a-46d5-9edd-8faa22d27b8d',\n",
              "  'status': {'online_url': {'url': 'https://cpd-wmlautoai-may12.apps.ocp46wmlautoaai.cp.fyre.ibm.com/ml/v4/deployments/e0f705fd-2fcb-437f-bc83-b17d5296b01d/predictions'},\n",
              "   'state': 'ready'}},\n",
              " 'metadata': {'created_at': '2021-05-20T08:23:53.001Z',\n",
              "  'id': 'e0f705fd-2fcb-437f-bc83-b17d5296b01d',\n",
              "  'modified_at': '2021-05-20T08:23:53.001Z',\n",
              "  'name': 'Credit Risk Deployment AutoAI',\n",
              "  'owner': '1000330999',\n",
              "  'space_id': 'fbd2dc3a-634a-46d5-9edd-8faa22d27b8d'}}"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "service.get_params()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Scoring of webservice\n",
        "You can make scoring request by calling `score()` on deployed pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'predictions': [{'fields': ['prediction', 'probability'],\n",
              "   'values': [['No Risk', [0.6743601502735059, 0.32563984972649407]],\n",
              "    ['No Risk', [0.7892372484049005, 0.21076275159509952]],\n",
              "    ['No Risk', [0.560817617610811, 0.43918238238918905]],\n",
              "    ['No Risk', [0.8068670237279646, 0.1931329762720355]],\n",
              "    ['Risk', [0.3344503240926395, 0.6655496759073605]],\n",
              "    ['Risk', [0.10565678467912125, 0.8943432153208788]],\n",
              "    ['No Risk', [0.7361802572321648, 0.2638197427678351]],\n",
              "    ['No Risk', [0.8315661007516614, 0.1684338992483386]],\n",
              "    ['No Risk', [0.8599408631598607, 0.1400591368401393]],\n",
              "    ['Risk', [0.12661051706745757, 0.8733894829325424]]]}]}"
            ]
          },
          "execution_count": 43,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "predictions = service.score(payload=train_df.drop(['Risk'], axis=1).iloc[:10])\n",
        "predictions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "If you want to work with the web service in an external Python application, you can retrieve the service object:\n",
        " - Initialize the service using `service = WebService(wml_credentials)`\n",
        " - Get deployment_id using the `service.list()` method\n",
        " - Get webservice object using the `service.get('deployment_id')` method\n",
        "\n",
        "After that you can call `service.score()` method."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Deleting deployment\n",
        "You can delete the existing deployment by calling the `service.delete()` command.\n",
        "To list the existing web services you can use `service.list()`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Batch deployment creation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "A batch deployment processes input data from a inline data and return predictions in scoring details or processes from data asset and writes the output to a file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>CheckingStatus</th>\n",
              "      <th>LoanDuration</th>\n",
              "      <th>CreditHistory</th>\n",
              "      <th>LoanPurpose</th>\n",
              "      <th>LoanAmount</th>\n",
              "      <th>ExistingSavings</th>\n",
              "      <th>EmploymentDuration</th>\n",
              "      <th>InstallmentPercent</th>\n",
              "      <th>Sex</th>\n",
              "      <th>OthersOnLoan</th>\n",
              "      <th>CurrentResidenceDuration</th>\n",
              "      <th>OwnsProperty</th>\n",
              "      <th>Age</th>\n",
              "      <th>InstallmentPlans</th>\n",
              "      <th>Housing</th>\n",
              "      <th>ExistingCreditsCount</th>\n",
              "      <th>Job</th>\n",
              "      <th>Dependents</th>\n",
              "      <th>Telephone</th>\n",
              "      <th>ForeignWorker</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0_to_200</td>\n",
              "      <td>31</td>\n",
              "      <td>credits_paid_to_date</td>\n",
              "      <td>other</td>\n",
              "      <td>1889</td>\n",
              "      <td>100_to_500</td>\n",
              "      <td>less_1</td>\n",
              "      <td>3</td>\n",
              "      <td>female</td>\n",
              "      <td>none</td>\n",
              "      <td>3</td>\n",
              "      <td>savings_insurance</td>\n",
              "      <td>32</td>\n",
              "      <td>none</td>\n",
              "      <td>own</td>\n",
              "      <td>1</td>\n",
              "      <td>skilled</td>\n",
              "      <td>1</td>\n",
              "      <td>none</td>\n",
              "      <td>yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>less_0</td>\n",
              "      <td>18</td>\n",
              "      <td>credits_paid_to_date</td>\n",
              "      <td>car_new</td>\n",
              "      <td>462</td>\n",
              "      <td>less_100</td>\n",
              "      <td>1_to_4</td>\n",
              "      <td>2</td>\n",
              "      <td>female</td>\n",
              "      <td>none</td>\n",
              "      <td>2</td>\n",
              "      <td>savings_insurance</td>\n",
              "      <td>37</td>\n",
              "      <td>stores</td>\n",
              "      <td>own</td>\n",
              "      <td>2</td>\n",
              "      <td>skilled</td>\n",
              "      <td>1</td>\n",
              "      <td>none</td>\n",
              "      <td>yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>less_0</td>\n",
              "      <td>15</td>\n",
              "      <td>prior_payments_delayed</td>\n",
              "      <td>furniture</td>\n",
              "      <td>250</td>\n",
              "      <td>less_100</td>\n",
              "      <td>1_to_4</td>\n",
              "      <td>2</td>\n",
              "      <td>male</td>\n",
              "      <td>none</td>\n",
              "      <td>3</td>\n",
              "      <td>real_estate</td>\n",
              "      <td>28</td>\n",
              "      <td>none</td>\n",
              "      <td>own</td>\n",
              "      <td>2</td>\n",
              "      <td>skilled</td>\n",
              "      <td>1</td>\n",
              "      <td>yes</td>\n",
              "      <td>no</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0_to_200</td>\n",
              "      <td>28</td>\n",
              "      <td>credits_paid_to_date</td>\n",
              "      <td>retraining</td>\n",
              "      <td>3693</td>\n",
              "      <td>less_100</td>\n",
              "      <td>greater_7</td>\n",
              "      <td>3</td>\n",
              "      <td>male</td>\n",
              "      <td>none</td>\n",
              "      <td>2</td>\n",
              "      <td>savings_insurance</td>\n",
              "      <td>32</td>\n",
              "      <td>none</td>\n",
              "      <td>own</td>\n",
              "      <td>1</td>\n",
              "      <td>skilled</td>\n",
              "      <td>1</td>\n",
              "      <td>none</td>\n",
              "      <td>yes</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>no_checking</td>\n",
              "      <td>28</td>\n",
              "      <td>prior_payments_delayed</td>\n",
              "      <td>education</td>\n",
              "      <td>6235</td>\n",
              "      <td>500_to_1000</td>\n",
              "      <td>greater_7</td>\n",
              "      <td>3</td>\n",
              "      <td>male</td>\n",
              "      <td>none</td>\n",
              "      <td>3</td>\n",
              "      <td>unknown</td>\n",
              "      <td>57</td>\n",
              "      <td>none</td>\n",
              "      <td>own</td>\n",
              "      <td>2</td>\n",
              "      <td>skilled</td>\n",
              "      <td>1</td>\n",
              "      <td>none</td>\n",
              "      <td>yes</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  CheckingStatus  LoanDuration           CreditHistory LoanPurpose  \\\n",
              "0       0_to_200            31    credits_paid_to_date       other   \n",
              "1         less_0            18    credits_paid_to_date     car_new   \n",
              "2         less_0            15  prior_payments_delayed   furniture   \n",
              "3       0_to_200            28    credits_paid_to_date  retraining   \n",
              "4    no_checking            28  prior_payments_delayed   education   \n",
              "\n",
              "   LoanAmount ExistingSavings EmploymentDuration  InstallmentPercent     Sex  \\\n",
              "0        1889      100_to_500             less_1                   3  female   \n",
              "1         462        less_100             1_to_4                   2  female   \n",
              "2         250        less_100             1_to_4                   2    male   \n",
              "3        3693        less_100          greater_7                   3    male   \n",
              "4        6235     500_to_1000          greater_7                   3    male   \n",
              "\n",
              "  OthersOnLoan  CurrentResidenceDuration       OwnsProperty  Age  \\\n",
              "0         none                         3  savings_insurance   32   \n",
              "1         none                         2  savings_insurance   37   \n",
              "2         none                         3        real_estate   28   \n",
              "3         none                         2  savings_insurance   32   \n",
              "4         none                         3            unknown   57   \n",
              "\n",
              "  InstallmentPlans Housing  ExistingCreditsCount      Job  Dependents  \\\n",
              "0             none     own                     1  skilled           1   \n",
              "1           stores     own                     2  skilled           1   \n",
              "2             none     own                     2  skilled           1   \n",
              "3             none     own                     1  skilled           1   \n",
              "4             none     own                     2  skilled           1   \n",
              "\n",
              "  Telephone ForeignWorker  \n",
              "0      none           yes  \n",
              "1      none           yes  \n",
              "2       yes            no  \n",
              "3      none           yes  \n",
              "4      none           yes  "
            ]
          },
          "execution_count": 44,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "batch_payload_df = train_df.drop(['Risk'], axis=1)[:5]\n",
        "batch_payload_df"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Create batch deployment for `Pipeline_2` created in AutoAI experiment with the `run_id`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Preparing an AutoAI Deployment...\n",
            "Published model uid: 35a98cac-c3b7-448e-9a02-245400c93a82\n",
            "Deploying model 35a98cac-c3b7-448e-9a02-245400c93a82 using V4 client.\n",
            "\n",
            "\n",
            "#######################################################################################\n",
            "\n",
            "Synchronous deployment creation for uid: '35a98cac-c3b7-448e-9a02-245400c93a82' started\n",
            "\n",
            "#######################################################################################\n",
            "\n",
            "\n",
            "ready.\n",
            "\n",
            "\n",
            "------------------------------------------------------------------------------------------------\n",
            "Successfully finished deployment creation, deployment_uid='756558ad-78f1-4d74-9e6d-555aa9cfff21'\n",
            "------------------------------------------------------------------------------------------------\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "from ibm_watsonx_ai.deployment import Batch\n",
        "\n",
        "service_batch = Batch(credentials,source_space_id=space_id)\n",
        "service_batch.create(\n",
        "            experiment_run_id=run_id,\n",
        "            model=\"Pipeline_2\",\n",
        "            deployment_name=\"Credit Risk Batch Deployment AutoAI\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Score batch deployment with inline payload as pandas DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "##########################################################################\n",
            "\n",
            "Synchronous scoring for id: '478b6b5b-a80f-43f5-98f9-63e9da889720' started\n",
            "\n",
            "##########################################################################\n",
            "\n",
            "\n",
            "queued..\n",
            "running\n",
            "completed\n",
            "Scoring job  '478b6b5b-a80f-43f5-98f9-63e9da889720' finished successfully.\n"
          ]
        }
      ],
      "source": [
        "scoring_params = service_batch.run_job(\n",
        "            payload=batch_payload_df,\n",
        "            background_mode=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 47,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'fields': ['prediction', 'probability'],\n",
              "  'values': [['No Risk', [0.7743273377418518, 0.225672647356987]],\n",
              "   ['No Risk', [0.8309863209724426, 0.16901369392871857]],\n",
              "   ['No Risk', [0.8892747759819031, 0.11072523891925812]],\n",
              "   ['No Risk', [0.7456686496734619, 0.2543313801288605]],\n",
              "   ['Risk', [0.22297507524490356, 0.7770249247550964]]]}]"
            ]
          },
          "execution_count": 47,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "scoring_params['entity']['scoring'].get('predictions')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Score batch deployment with payload as connected asset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Simmilary to training use created connection in order to locate tabe in used database."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 48,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Creating data asset...\n",
            "SUCCESS\n"
          ]
        }
      ],
      "source": [
        "from ibm_watsonx_ai.helpers.connections import DeploymentOutputAssetLocation\n",
        "\n",
        "\n",
        "batch_payload_filename = \"credit_risk_batch_payload.csv\"\n",
        "batch_payload_df.to_csv(batch_payload_filename, index=False)\n",
        "\n",
        "asset_details = client.data_assets.create(\n",
        "            name=batch_payload_filename,\n",
        "            file_path=batch_payload_filename)\n",
        "asset_id = client.data_assets.get_id(asset_details)\n",
        "\n",
        "payload_reference = DataConnection(data_asset_id=asset_id)\n",
        "results_reference = DataConnection(\n",
        "            location=DeploymentOutputAssetLocation(name=\"batch_output_credit_risk.csv\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Run scoring job for batch deployment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "##########################################################################\n",
            "\n",
            "Synchronous scoring for id: '80664cc6-3bf8-4fae-9309-f8622972a119' started\n",
            "\n",
            "##########################################################################\n",
            "\n",
            "\n",
            "queued.........\n",
            "running.\n",
            "completed\n",
            "Scoring job  '80664cc6-3bf8-4fae-9309-f8622972a119' finished successfully.\n"
          ]
        }
      ],
      "source": [
        "scoring_params = service_batch.run_job(\n",
        "            payload=[payload_reference],\n",
        "            output_data_reference=results_reference,\n",
        "            background_mode=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Deleting deployment\n",
        "You can delete the existing deployment by calling the `service_batch.delete()` command.\n",
        "To list the existing: \n",
        "- batch services you can use `service_batch.list()`, \n",
        "- scoring jobs you can use `service_batch.list_jobs()`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"cleanup\"></a>\n",
        "## 8. Clean up "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "If you want to clean up all created assets:\n",
        "- experiments\n",
        "- trainings\n",
        "- pipelines\n",
        "- model definitions\n",
        "- models\n",
        "- functions\n",
        "- deployments\n",
        "\n",
        "please follow up this sample [notebook](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd5.1/notebooks/python_sdk/instance-management/Machine%20Learning%20artifacts%20management.ipynb)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"summary\"></a>\n",
        "## 9. Summary and next steps\n",
        "\n",
        "You successfully completed this notebook!.\n",
        " \n",
        "You learned how to use `ibm-watsonx-ai` to run AutoAI experiments. \n",
        "\n",
        "Check out our _<a href=\"https://ibm.github.io/watsonx-ai-python-sdk/samples.html\" target=\"_blank\" rel=\"noopener no referrer\">Online Documentation</a>_ for more samples, tutorials, documentation, how-tos, and blog posts. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Authors\n",
        "\n",
        "**Lukasz Cmielowski**, PhD, is an Automation Architect and Data Scientist at IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.\n",
        "\n",
        "**Amadeusz Masny**, Python Software Developer in Watson Machine Learning at IBM\n",
        "\n",
        "**Kiran Kate**, Senior Software Engineer at IBM Research AI\n",
        "\n",
        "**Martin Hirzel**, Research Staff Member and Manager at IBM Research AI\n",
        "\n",
        "**Jan Sołtysik**, Intern in Watson Machine Learning "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Copyright © 2020-2025 IBM. This notebook and its source code are released under the terms of the MIT License."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}