cpd5.1/notebooks/python_sdk/lifecycle-management/Use python API to automate AutoAI deployment lifecycle.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "# Re-train and re-deploy AutoAI pipelines with `ibm-watsonx-ai`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "This notebook contains the steps and code to demonstrate support of AI Lifecycle features of the AutoAI model in Watson Machine Learning Service in Watson Machine Learning service. It contains steps and code to work with [ibm-watsonx-ai](https://pypi.python.org/pypi/ibm-watsonx-ai) library available in PyPI repository. It also introduces commands for  training, persisting and deploying model, scoring it, updating the model and redeploying it.\n",
        "\n",
        "Some familiarity with Python is helpful. This notebook uses Python 3.11.\n",
        "\n",
        "\n",
        "## Learning goals\n",
        "\n",
        "The learning goals of this notebook are:\n",
        "\n",
        "-  List all deprecated and unsupported deployments.\n",
        "-  Identify AutoAI models that need to be retrained.\n",
        "-  Work with Watson Machine Learning experiments to re-train AutoAI models.\n",
        "-  Persist an updated AutoAI model in Watson Machine Learning repository.\n",
        "-  Redeploy model in-place.\n",
        "-  Score sample records using client library.\n",
        "\n",
        "\n",
        "## Contents\n",
        "\n",
        "This notebook contains the following parts:\n",
        "\n",
        "1. [Setup](#setup)\n",
        "2. [Deployments state check](#state)\n",
        "3. [Identification of model requiring retraining](#identify)\n",
        "4. [Experiment re-run](#run)\n",
        "5. [Persist trained AutoAI model](#persistence)\n",
        "6. [Redeploy and score new version of the model](#redeploy)\n",
        "7. [Clean up](#cleanup)\n",
        "8. [Summary and next steps](#summary)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"setup\"></a>\n",
        "## 1. Set up the environment\n",
        "\n",
        "Before you use the sample code in this notebook, contact with your Cloud Pack for Data administrator and ask for your account credentials."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Install and import the `ibm-watsonx-ai` and dependecies\n",
        "**Note:** `ibm-watsonx-ai` documentation can be found <a href=\"https://ibm.github.io/watsonx-ai-python-sdk/index.html\" target=\"_blank\" rel=\"noopener no referrer\">here</a>."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "!pip install wget | tail -n 1\n",
        "!pip install -U ibm-watsonx-ai | tail -n 1\n",
        "!pip install \"scikit-learn==1.3.2\" | tail -n 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Connection to WML\n",
        "\n",
        "Authenticate the Watson Machine Learning service on IBM Cloud Pack for Data. You need to provide platform `url`, your `username` and `api_key`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "username = 'PASTE YOUR USERNAME HERE'\n",
        "api_key = 'PASTE YOUR API_KEY HERE'\n",
        "url = 'PASTE THE PLATFORM URL HERE'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from ibm_watsonx_ai import Credentials\n",
        "\n",
        "credentials = Credentials(\n",
        "    username=username,\n",
        "    api_key=api_key,\n",
        "    url=url,\n",
        "    instance_id=\"openshift\",\n",
        "    version=\"5.1\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Alternatively you can use `username` and `password` to authenticate WML services.\n",
        "\n",
        "```python\n",
        "credentials = Credentials(\n",
        "    username=***,\n",
        "    password=***,\n",
        "    url=***,\n",
        "    instance_id=\"openshift\",\n",
        "    version=\"5.1\"\n",
        ")\n",
        "\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from ibm_watsonx_ai import APIClient\n",
        "\n",
        "client = APIClient(credentials)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Working with spaces\n",
        "\n",
        "First of all, you need to create a space that will be used for your work. If you do not have space already created, you can use `{PLATFORM_URL}/ml-runtime/spaces?context=icp4data` to create one.\n",
        "\n",
        "- Click New Deployment Space\n",
        "- Create an empty space\n",
        "- Go to space `Settings` tab\n",
        "- Copy `space_id` and paste it below\n",
        "\n",
        "**Tip**: You can also use SDK to prepare the space for your work. More information can be found [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd5.1/notebooks/python_sdk/instance-management/Space%20management.ipynb).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "You can use the `list` method to print all existing spaces."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "client.spaces.list(limit=10)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Extract all spaces `id`'s."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "spaces_ids = [space['metadata']['id'] for space in client.spaces.get_details()['resources']]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"state\"></a>\n",
        "## 2. Deployments state check\n",
        "Iterate over spaces and search for `deprecated` and `unsupported` deployments. Next, identify models requiring re-training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "****** SPACE 448d413d-34e3-4974-a19a-f6a7488cc975 ******\n",
            "{'resources': []}\n",
            "{'resources': []}\n",
            "****** SPACE 4359771d-33c0-4173-9ed5-cdd22897ebfb ******\n",
            "{'resources': []}\n",
            "{'resources': []}\n",
            "****** SPACE 8413e7c3-7483-4568-8b93-b88120d91042 ******\n",
            "{'resources': []}\n",
            "{'resources': []}\n",
            "****** SPACE 71d17a57-8b0d-483c-a928-6791aa21c703 ******\n",
            "{'resources': []}\n",
            "{'resources': []}\n",
            "****** SPACE aaff80ab-d812-48c9-9113-0d0f298c8abe ******\n",
            "{'resources': []}\n",
            "{'resources': []}\n",
            "****** SPACE ac0d6d39-f567-4640-8807-92ca5a17f745 ******\n",
            "{'resources': []}\n",
            "{'resources': []}\n"
          ]
        }
      ],
      "source": [
        "from ibm_watsonx_ai.lifecycle import SpecStates\n",
        "\n",
        "\n",
        "for space_id in spaces_ids:\n",
        "    client.set.default_space(space_id)\n",
        "    print('****** SPACE', space_id, '******')\n",
        "    print(client.deployments.get_details(spec_state=SpecStates.DEPRECATED))\n",
        "    print(client.deployments.get_details(spec_state=SpecStates.UNSUPPORTED))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "You can also list deployments under particular space. The output contains `SPEC_STATE` and `SPEC_REPLACEMENT`. Set the working space."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "'SUCCESS'"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "deployment_space_id = 'PASTE YOUR SPACE ID HERE'\n",
        "client.set.default_space(deployment_space_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "List deployments under this space."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "------------------------------------  --------------------------------------  -----  ------------------------  -------------  ----------  ----------------\n",
            "GUID                                  NAME                                    STATE  CREATED                   ARTIFACT_TYPE  SPEC_STATE  SPEC_REPLACEMENT\n",
            "d4437b42-6e74-49e1-a0d5-f68a8eb03992  AutoAI Credit risk - Online Deployment  ready  2023-05-16T14:05:47.486Z  model          supported\n",
            "------------------------------------  --------------------------------------  -----  ------------------------  -------------  ----------  ----------------\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>GUID</th>\n",
              "      <th>NAME</th>\n",
              "      <th>STATE</th>\n",
              "      <th>CREATED</th>\n",
              "      <th>ARTIFACT_TYPE</th>\n",
              "      <th>SPEC_STATE</th>\n",
              "      <th>SPEC_REPLACEMENT</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>d4437b42-6e74-49e1-a0d5-f68a8eb03992</td>\n",
              "      <td>AutoAI Credit risk - Online Deployment</td>\n",
              "      <td>ready</td>\n",
              "      <td>2023-05-16T14:05:47.486Z</td>\n",
              "      <td>model</td>\n",
              "      <td>supported</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                   GUID  \\\n",
              "0  d4437b42-6e74-49e1-a0d5-f68a8eb03992   \n",
              "\n",
              "                                     NAME  STATE                   CREATED  \\\n",
              "0  AutoAI Credit risk - Online Deployment  ready  2023-05-16T14:05:47.486Z   \n",
              "\n",
              "  ARTIFACT_TYPE SPEC_STATE SPEC_REPLACEMENT  \n",
              "0         model  supported                   "
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "client.deployments.list()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"identify\"></a>\n",
        "## 3. Identification of model requiring retraining\n",
        "Pick up deployment of the AutoAI model you wish to retrain. \n",
        "\n",
        "**Hint**: You can also do that programatically in the loop sequence over spaces check (`Check the state of your deployments` cell).\n",
        "**Hint**: You can also use software_specification information (model details) to identify models and deployments that are not yet deprecated but can be retrained (updated software specification is available)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n"
          ]
        }
      ],
      "source": [
        "deployment_id = 'PASTE YOUR DEPLOYMENT ID HERE'\n",
        "deployed_model_id = client.deployments.get_details(deployment_id)['entity']['asset']['id']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "#### Extract the deployed model's details (including the pipeline information)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "deployed_model_details = client.repository.get_model_details(deployed_model_id)\n",
        "deployed_pipeline_id = deployed_model_details['entity']['pipeline']['id']\n",
        "deployed_pipeline_details = client.repository.get_details(deployed_pipeline_id)\n",
        "experiment_params = deployed_pipeline_details['entity']['document']['pipelines'][0]['nodes'][0]['parameters']\n",
        "optimization_params = experiment_params['optimization']\n",
        "\n",
        "print('Experiment parameters:', json.dumps(experiment_params, indent=3))\n",
        "print('Optimization parameters:', json.dumps(optimization_params, indent=3))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "#### Find the AutoAI experiment runs matching the extracted pipeline\n",
        "\n",
        "Extract the `project_id` where the training took place.\n",
        "\n",
        "**Note:** If the training took place in the space please update accordingly."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "training_project_id = deployed_pipeline_details['metadata']['tags'][0].split('.')[1]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "#### Extract AutoAI experiment `training_id`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "The `training_id` is available in model's details."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "AutoAI experiment training_id found in model details: ce97dafb-fa77-45ed-8603-4f525f6c16b7\n"
          ]
        }
      ],
      "source": [
        "run_id = deployed_model_details['entity']['training_id']\n",
        "print('AutoAI experiment training_id found in model details:', run_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"run\"></a>\n",
        "## 4. Experiment re-run \n",
        "Set the training `project_id` (where data asset resides) to retrain AutoAI models.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from ibm_watsonx_ai.experiment import AutoAI\n",
        "\n",
        "experiment = AutoAI(credentials, project_id=training_project_id)\n",
        "optimizer = experiment.runs.get_optimizer(run_id=run_id)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "User defined (test / holdout) data is not present for this AutoAI experiment.\n",
            "Reason: User specified test data was not present in this experiment. Try to use 'with_holdout_split' parameter for original training_data_references to retrieve test data.\n"
          ]
        }
      ],
      "source": [
        "from ibm_watsonx_ai.utils.autoai.errors import TestDataNotPresent\n",
        "\n",
        "training_data_reference = optimizer.get_data_connections()\n",
        "try:\n",
        "    test_data_reference = optimizer.get_test_data_connections()\n",
        "except TestDataNotPresent:\n",
        "    test_data_reference = None"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training job 1578524e-ab25-467d-b305-acfe24057a7f completed: 100%|████████| [02:24<00:00,  1.45s/it]\n"
          ]
        }
      ],
      "source": [
        "train_details = optimizer.fit(training_data_references = training_data_reference,\n",
        "                              test_data_references=test_data_reference)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Explore experiment's results\n",
        "Connect to finished experiment and preview the results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Enhancements</th>\n",
              "      <th>Estimator</th>\n",
              "      <th>training_accuracy_(optimized)</th>\n",
              "      <th>training_roc_auc</th>\n",
              "      <th>holdout_average_precision</th>\n",
              "      <th>holdout_log_loss</th>\n",
              "      <th>holdout_roc_auc</th>\n",
              "      <th>training_balanced_accuracy</th>\n",
              "      <th>training_f1</th>\n",
              "      <th>holdout_precision</th>\n",
              "      <th>training_average_precision</th>\n",
              "      <th>training_log_loss</th>\n",
              "      <th>holdout_recall</th>\n",
              "      <th>training_precision</th>\n",
              "      <th>holdout_accuracy</th>\n",
              "      <th>holdout_balanced_accuracy</th>\n",
              "      <th>training_recall</th>\n",
              "      <th>holdout_f1</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline Name</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Pipeline_3</th>\n",
              "      <td>HPO, FE</td>\n",
              "      <td>SnapRandomForestClassifier</td>\n",
              "      <td>0.808108</td>\n",
              "      <td>0.854211</td>\n",
              "      <td>0.544314</td>\n",
              "      <td>0.583035</td>\n",
              "      <td>0.764706</td>\n",
              "      <td>0.761896</td>\n",
              "      <td>0.863602</td>\n",
              "      <td>0.833333</td>\n",
              "      <td>0.933068</td>\n",
              "      <td>0.451773</td>\n",
              "      <td>0.882353</td>\n",
              "      <td>0.852453</td>\n",
              "      <td>0.80</td>\n",
              "      <td>0.753676</td>\n",
              "      <td>0.878205</td>\n",
              "      <td>0.857143</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_4</th>\n",
              "      <td>HPO, FE, HPO</td>\n",
              "      <td>SnapRandomForestClassifier</td>\n",
              "      <td>0.808108</td>\n",
              "      <td>0.854211</td>\n",
              "      <td>0.544314</td>\n",
              "      <td>0.583035</td>\n",
              "      <td>0.764706</td>\n",
              "      <td>0.761896</td>\n",
              "      <td>0.863602</td>\n",
              "      <td>0.833333</td>\n",
              "      <td>0.933068</td>\n",
              "      <td>0.451773</td>\n",
              "      <td>0.882353</td>\n",
              "      <td>0.852453</td>\n",
              "      <td>0.80</td>\n",
              "      <td>0.753676</td>\n",
              "      <td>0.878205</td>\n",
              "      <td>0.857143</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_5</th>\n",
              "      <td>HPO, FE, HPO, Ensemble</td>\n",
              "      <td>BatchedTreeEnsembleClassifier(SnapRandomForest...</td>\n",
              "      <td>0.808108</td>\n",
              "      <td>0.854211</td>\n",
              "      <td>0.544314</td>\n",
              "      <td>0.583035</td>\n",
              "      <td>0.764706</td>\n",
              "      <td>0.761896</td>\n",
              "      <td>0.863602</td>\n",
              "      <td>0.833333</td>\n",
              "      <td>0.933068</td>\n",
              "      <td>0.451773</td>\n",
              "      <td>0.882353</td>\n",
              "      <td>0.852453</td>\n",
              "      <td>0.80</td>\n",
              "      <td>0.753676</td>\n",
              "      <td>0.878205</td>\n",
              "      <td>0.857143</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_2</th>\n",
              "      <td>HPO</td>\n",
              "      <td>SnapRandomForestClassifier</td>\n",
              "      <td>0.803664</td>\n",
              "      <td>0.858797</td>\n",
              "      <td>0.549447</td>\n",
              "      <td>0.621004</td>\n",
              "      <td>0.753676</td>\n",
              "      <td>0.750608</td>\n",
              "      <td>0.861756</td>\n",
              "      <td>0.812500</td>\n",
              "      <td>0.936527</td>\n",
              "      <td>0.439293</td>\n",
              "      <td>0.764706</td>\n",
              "      <td>0.843357</td>\n",
              "      <td>0.72</td>\n",
              "      <td>0.694853</td>\n",
              "      <td>0.884615</td>\n",
              "      <td>0.787879</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_6</th>\n",
              "      <td></td>\n",
              "      <td>XGBClassifier</td>\n",
              "      <td>0.776877</td>\n",
              "      <td>0.817903</td>\n",
              "      <td>0.564305</td>\n",
              "      <td>0.891920</td>\n",
              "      <td>0.713235</td>\n",
              "      <td>0.731707</td>\n",
              "      <td>0.840500</td>\n",
              "      <td>0.777778</td>\n",
              "      <td>0.912044</td>\n",
              "      <td>0.637040</td>\n",
              "      <td>0.823529</td>\n",
              "      <td>0.835349</td>\n",
              "      <td>0.72</td>\n",
              "      <td>0.661765</td>\n",
              "      <td>0.846154</td>\n",
              "      <td>0.800000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_9</th>\n",
              "      <td>HPO, FE, HPO</td>\n",
              "      <td>XGBClassifier</td>\n",
              "      <td>0.812553</td>\n",
              "      <td>0.846129</td>\n",
              "      <td>0.550957</td>\n",
              "      <td>0.575589</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>0.757348</td>\n",
              "      <td>0.869416</td>\n",
              "      <td>0.812500</td>\n",
              "      <td>0.923229</td>\n",
              "      <td>0.438852</td>\n",
              "      <td>0.764706</td>\n",
              "      <td>0.843483</td>\n",
              "      <td>0.72</td>\n",
              "      <td>0.694853</td>\n",
              "      <td>0.897436</td>\n",
              "      <td>0.787879</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_10</th>\n",
              "      <td>HPO, FE, HPO, Ensemble</td>\n",
              "      <td>BatchedTreeEnsembleClassifier(XGBClassifier)</td>\n",
              "      <td>0.812553</td>\n",
              "      <td>0.846129</td>\n",
              "      <td>0.550957</td>\n",
              "      <td>0.575589</td>\n",
              "      <td>0.750000</td>\n",
              "      <td>0.757348</td>\n",
              "      <td>0.869416</td>\n",
              "      <td>0.812500</td>\n",
              "      <td>0.923229</td>\n",
              "      <td>0.438852</td>\n",
              "      <td>0.764706</td>\n",
              "      <td>0.843483</td>\n",
              "      <td>0.72</td>\n",
              "      <td>0.694853</td>\n",
              "      <td>0.897436</td>\n",
              "      <td>0.787879</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_1</th>\n",
              "      <td></td>\n",
              "      <td>SnapRandomForestClassifier</td>\n",
              "      <td>0.794775</td>\n",
              "      <td>0.819651</td>\n",
              "      <td>0.562071</td>\n",
              "      <td>0.584794</td>\n",
              "      <td>0.738971</td>\n",
              "      <td>0.748239</td>\n",
              "      <td>0.853594</td>\n",
              "      <td>0.764706</td>\n",
              "      <td>0.902923</td>\n",
              "      <td>0.605309</td>\n",
              "      <td>0.764706</td>\n",
              "      <td>0.844418</td>\n",
              "      <td>0.68</td>\n",
              "      <td>0.632353</td>\n",
              "      <td>0.865385</td>\n",
              "      <td>0.764706</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_7</th>\n",
              "      <td>HPO</td>\n",
              "      <td>XGBClassifier</td>\n",
              "      <td>0.790390</td>\n",
              "      <td>0.790159</td>\n",
              "      <td>0.558832</td>\n",
              "      <td>0.931696</td>\n",
              "      <td>0.727941</td>\n",
              "      <td>0.741651</td>\n",
              "      <td>0.850977</td>\n",
              "      <td>0.800000</td>\n",
              "      <td>0.894378</td>\n",
              "      <td>0.624102</td>\n",
              "      <td>0.705882</td>\n",
              "      <td>0.837639</td>\n",
              "      <td>0.68</td>\n",
              "      <td>0.665441</td>\n",
              "      <td>0.865385</td>\n",
              "      <td>0.750000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Pipeline_8</th>\n",
              "      <td>HPO, FE</td>\n",
              "      <td>XGBClassifier</td>\n",
              "      <td>0.808108</td>\n",
              "      <td>0.828114</td>\n",
              "      <td>0.566836</td>\n",
              "      <td>0.911154</td>\n",
              "      <td>0.713235</td>\n",
              "      <td>0.754143</td>\n",
              "      <td>0.865905</td>\n",
              "      <td>0.764706</td>\n",
              "      <td>0.904703</td>\n",
              "      <td>0.549224</td>\n",
              "      <td>0.764706</td>\n",
              "      <td>0.842430</td>\n",
              "      <td>0.68</td>\n",
              "      <td>0.632353</td>\n",
              "      <td>0.891026</td>\n",
              "      <td>0.764706</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                         Enhancements  \\\n",
              "Pipeline Name                           \n",
              "Pipeline_3                    HPO, FE   \n",
              "Pipeline_4               HPO, FE, HPO   \n",
              "Pipeline_5     HPO, FE, HPO, Ensemble   \n",
              "Pipeline_2                        HPO   \n",
              "Pipeline_6                              \n",
              "Pipeline_9               HPO, FE, HPO   \n",
              "Pipeline_10    HPO, FE, HPO, Ensemble   \n",
              "Pipeline_1                              \n",
              "Pipeline_7                        HPO   \n",
              "Pipeline_8                    HPO, FE   \n",
              "\n",
              "                                                       Estimator  \\\n",
              "Pipeline Name                                                      \n",
              "Pipeline_3                            SnapRandomForestClassifier   \n",
              "Pipeline_4                            SnapRandomForestClassifier   \n",
              "Pipeline_5     BatchedTreeEnsembleClassifier(SnapRandomForest...   \n",
              "Pipeline_2                            SnapRandomForestClassifier   \n",
              "Pipeline_6                                         XGBClassifier   \n",
              "Pipeline_9                                         XGBClassifier   \n",
              "Pipeline_10         BatchedTreeEnsembleClassifier(XGBClassifier)   \n",
              "Pipeline_1                            SnapRandomForestClassifier   \n",
              "Pipeline_7                                         XGBClassifier   \n",
              "Pipeline_8                                         XGBClassifier   \n",
              "\n",
              "               training_accuracy_(optimized)  training_roc_auc  \\\n",
              "Pipeline Name                                                    \n",
              "Pipeline_3                          0.808108          0.854211   \n",
              "Pipeline_4                          0.808108          0.854211   \n",
              "Pipeline_5                          0.808108          0.854211   \n",
              "Pipeline_2                          0.803664          0.858797   \n",
              "Pipeline_6                          0.776877          0.817903   \n",
              "Pipeline_9                          0.812553          0.846129   \n",
              "Pipeline_10                         0.812553          0.846129   \n",
              "Pipeline_1                          0.794775          0.819651   \n",
              "Pipeline_7                          0.790390          0.790159   \n",
              "Pipeline_8                          0.808108          0.828114   \n",
              "\n",
              "               holdout_average_precision  holdout_log_loss  holdout_roc_auc  \\\n",
              "Pipeline Name                                                                 \n",
              "Pipeline_3                      0.544314          0.583035         0.764706   \n",
              "Pipeline_4                      0.544314          0.583035         0.764706   \n",
              "Pipeline_5                      0.544314          0.583035         0.764706   \n",
              "Pipeline_2                      0.549447          0.621004         0.753676   \n",
              "Pipeline_6                      0.564305          0.891920         0.713235   \n",
              "Pipeline_9                      0.550957          0.575589         0.750000   \n",
              "Pipeline_10                     0.550957          0.575589         0.750000   \n",
              "Pipeline_1                      0.562071          0.584794         0.738971   \n",
              "Pipeline_7                      0.558832          0.931696         0.727941   \n",
              "Pipeline_8                      0.566836          0.911154         0.713235   \n",
              "\n",
              "               training_balanced_accuracy  training_f1  holdout_precision  \\\n",
              "Pipeline Name                                                               \n",
              "Pipeline_3                       0.761896     0.863602           0.833333   \n",
              "Pipeline_4                       0.761896     0.863602           0.833333   \n",
              "Pipeline_5                       0.761896     0.863602           0.833333   \n",
              "Pipeline_2                       0.750608     0.861756           0.812500   \n",
              "Pipeline_6                       0.731707     0.840500           0.777778   \n",
              "Pipeline_9                       0.757348     0.869416           0.812500   \n",
              "Pipeline_10                      0.757348     0.869416           0.812500   \n",
              "Pipeline_1                       0.748239     0.853594           0.764706   \n",
              "Pipeline_7                       0.741651     0.850977           0.800000   \n",
              "Pipeline_8                       0.754143     0.865905           0.764706   \n",
              "\n",
              "               training_average_precision  training_log_loss  holdout_recall  \\\n",
              "Pipeline Name                                                                  \n",
              "Pipeline_3                       0.933068           0.451773        0.882353   \n",
              "Pipeline_4                       0.933068           0.451773        0.882353   \n",
              "Pipeline_5                       0.933068           0.451773        0.882353   \n",
              "Pipeline_2                       0.936527           0.439293        0.764706   \n",
              "Pipeline_6                       0.912044           0.637040        0.823529   \n",
              "Pipeline_9                       0.923229           0.438852        0.764706   \n",
              "Pipeline_10                      0.923229           0.438852        0.764706   \n",
              "Pipeline_1                       0.902923           0.605309        0.764706   \n",
              "Pipeline_7                       0.894378           0.624102        0.705882   \n",
              "Pipeline_8                       0.904703           0.549224        0.764706   \n",
              "\n",
              "               training_precision  holdout_accuracy  \\\n",
              "Pipeline Name                                         \n",
              "Pipeline_3               0.852453              0.80   \n",
              "Pipeline_4               0.852453              0.80   \n",
              "Pipeline_5               0.852453              0.80   \n",
              "Pipeline_2               0.843357              0.72   \n",
              "Pipeline_6               0.835349              0.72   \n",
              "Pipeline_9               0.843483              0.72   \n",
              "Pipeline_10              0.843483              0.72   \n",
              "Pipeline_1               0.844418              0.68   \n",
              "Pipeline_7               0.837639              0.68   \n",
              "Pipeline_8               0.842430              0.68   \n",
              "\n",
              "               holdout_balanced_accuracy  training_recall  holdout_f1  \n",
              "Pipeline Name                                                          \n",
              "Pipeline_3                      0.753676         0.878205    0.857143  \n",
              "Pipeline_4                      0.753676         0.878205    0.857143  \n",
              "Pipeline_5                      0.753676         0.878205    0.857143  \n",
              "Pipeline_2                      0.694853         0.884615    0.787879  \n",
              "Pipeline_6                      0.661765         0.846154    0.800000  \n",
              "Pipeline_9                      0.694853         0.897436    0.787879  \n",
              "Pipeline_10                     0.694853         0.897436    0.787879  \n",
              "Pipeline_1                      0.632353         0.865385    0.764706  \n",
              "Pipeline_7                      0.665441         0.865385    0.750000  \n",
              "Pipeline_8                      0.632353         0.891026    0.764706  "
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "optimizer.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Evaluate the best model locally"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Load the model for test purposes.\n",
        "\n",
        "**Hint:** The best model is returned automatically if no `pipeline_name` provided."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;featureunion&#x27;,\n",
              "                 FeatureUnion(transformer_list=[(&#x27;float32_transform_140563054741488&#x27;,\n",
              "                                                 Pipeline(steps=[(&#x27;numpycolumnselector&#x27;,\n",
              "                                                                  NumpyColumnSelector(columns=[0,\n",
              "                                                                                               2,\n",
              "                                                                                               3,\n",
              "                                                                                               5,\n",
              "                                                                                               6,\n",
              "                                                                                               7,\n",
              "                                                                                               8,\n",
              "                                                                                               9,\n",
              "                                                                                               10,\n",
              "                                                                                               11,\n",
              "                                                                                               13,\n",
              "                                                                                               14,\n",
              "                                                                                               15,\n",
              "                                                                                               16,\n",
              "                                                                                               17,\n",
              "                                                                                               18,\n",
              "                                                                                               19])),\n",
              "                                                                 (&#x27;compressstrings&#x27;,\n",
              "                                                                  CompressStrings(compress_type=&#x27;hash&#x27;,\n",
              "                                                                                  dtypes_list=[&#x27;char_str&#x27;,\n",
              "                                                                                               &#x27;char_str&#x27;,\n",
              "                                                                                               &#x27;char_str&#x27;,\n",
              "                                                                                               &#x27;char_str&#x27;,\n",
              "                                                                                               &#x27;char_str&#x27;,\n",
              "                                                                                               &#x27;float_int_n...\n",
              "                 autoai_libs.cognito.transforms.transform_utils.TAM(tans_class = &lt;class &#x27;autoai_libs.cognito.transforms.transform_extras.IsolationForestAnomaly&#x27;&gt;, name = &#x27;isoforestanomaly&#x27;, tgraph = None, apply_all = True, col_names = [&#x27;CheckingStatus&#x27;, &#x27;LoanDuration&#x27;, &#x27;CreditHistory&#x27;, &#x27;LoanPurpose&#x27;, &#x27;LoanAmount&#x27;, &#x27;ExistingSavings&#x27;, &#x27;EmploymentDuration&#x27;, &#x27;InstallmentPercent&#x27;, &#x27;Sex&#x27;, &#x27;OthersOnLoan&#x27;, &#x27;CurrentResidenceDuration&#x27;, &#x27;OwnsProperty&#x27;, &#x27;Age&#x27;, &#x27;InstallmentPlans&#x27;, &#x27;Housing&#x27;, &#x27;ExistingCreditsCount&#x27;, &#x27;Job&#x27;, &#x27;Dependents&#x27;, &#x27;Telephone&#x27;, &#x27;ForeignWorker&#x27;], col_dtypes = [dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;)], col_as_json_objects = None)),\n",
              "                (&#x27;fs1&#x27;,\n",
              "                 autoai_libs.cognito.transforms.transform_utils.FS1(cols_ids_must_keep = range(0, 20), additional_col_count_to_keep = 20, ptype = &#x27;classification&#x27;)),\n",
              "                (&#x27;snaprandomforestclassifier&#x27;,\n",
              "                 SnapRandomForestClassifier(compress_trees=True,\n",
              "                                            gpu_ids=array([0], dtype=uint32),\n",
              "                                            max_depth=5,\n",
              "                                            max_features=0.899349511906677,\n",
              "                                            n_estimators=41,\n",
              "                                            random_state=33))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;featureunion&#x27;,\n",
              "                 FeatureUnion(transformer_list=[(&#x27;float32_transform_140563054741488&#x27;,\n",
              "                                                 Pipeline(steps=[(&#x27;numpycolumnselector&#x27;,\n",
              "                                                                  NumpyColumnSelector(columns=[0,\n",
              "                                                                                               2,\n",
              "                                                                                               3,\n",
              "                                                                                               5,\n",
              "                                                                                               6,\n",
              "                                                                                               7,\n",
              "                                                                                               8,\n",
              "                                                                                               9,\n",
              "                                                                                               10,\n",
              "                                                                                               11,\n",
              "                                                                                               13,\n",
              "                                                                                               14,\n",
              "                                                                                               15,\n",
              "                                                                                               16,\n",
              "                                                                                               17,\n",
              "                                                                                               18,\n",
              "                                                                                               19])),\n",
              "                                                                 (&#x27;compressstrings&#x27;,\n",
              "                                                                  CompressStrings(compress_type=&#x27;hash&#x27;,\n",
              "                                                                                  dtypes_list=[&#x27;char_str&#x27;,\n",
              "                                                                                               &#x27;char_str&#x27;,\n",
              "                                                                                               &#x27;char_str&#x27;,\n",
              "                                                                                               &#x27;char_str&#x27;,\n",
              "                                                                                               &#x27;char_str&#x27;,\n",
              "                                                                                               &#x27;float_int_n...\n",
              "                 autoai_libs.cognito.transforms.transform_utils.TAM(tans_class = &lt;class &#x27;autoai_libs.cognito.transforms.transform_extras.IsolationForestAnomaly&#x27;&gt;, name = &#x27;isoforestanomaly&#x27;, tgraph = None, apply_all = True, col_names = [&#x27;CheckingStatus&#x27;, &#x27;LoanDuration&#x27;, &#x27;CreditHistory&#x27;, &#x27;LoanPurpose&#x27;, &#x27;LoanAmount&#x27;, &#x27;ExistingSavings&#x27;, &#x27;EmploymentDuration&#x27;, &#x27;InstallmentPercent&#x27;, &#x27;Sex&#x27;, &#x27;OthersOnLoan&#x27;, &#x27;CurrentResidenceDuration&#x27;, &#x27;OwnsProperty&#x27;, &#x27;Age&#x27;, &#x27;InstallmentPlans&#x27;, &#x27;Housing&#x27;, &#x27;ExistingCreditsCount&#x27;, &#x27;Job&#x27;, &#x27;Dependents&#x27;, &#x27;Telephone&#x27;, &#x27;ForeignWorker&#x27;], col_dtypes = [dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;)], col_as_json_objects = None)),\n",
              "                (&#x27;fs1&#x27;,\n",
              "                 autoai_libs.cognito.transforms.transform_utils.FS1(cols_ids_must_keep = range(0, 20), additional_col_count_to_keep = 20, ptype = &#x27;classification&#x27;)),\n",
              "                (&#x27;snaprandomforestclassifier&#x27;,\n",
              "                 SnapRandomForestClassifier(compress_trees=True,\n",
              "                                            gpu_ids=array([0], dtype=uint32),\n",
              "                                            max_depth=5,\n",
              "                                            max_features=0.899349511906677,\n",
              "                                            n_estimators=41,\n",
              "                                            random_state=33))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">featureunion: FeatureUnion</label><div class=\"sk-toggleable__content\"><pre>FeatureUnion(transformer_list=[(&#x27;float32_transform_140563054741488&#x27;,\n",
              "                                Pipeline(steps=[(&#x27;numpycolumnselector&#x27;,\n",
              "                                                 NumpyColumnSelector(columns=[0,\n",
              "                                                                              2,\n",
              "                                                                              3,\n",
              "                                                                              5,\n",
              "                                                                              6,\n",
              "                                                                              7,\n",
              "                                                                              8,\n",
              "                                                                              9,\n",
              "                                                                              10,\n",
              "                                                                              11,\n",
              "                                                                              13,\n",
              "                                                                              14,\n",
              "                                                                              15,\n",
              "                                                                              16,\n",
              "                                                                              17,\n",
              "                                                                              18,\n",
              "                                                                              19])),\n",
              "                                                (&#x27;compressstrings&#x27;,\n",
              "                                                 CompressStrings(compress_type=&#x27;hash&#x27;,\n",
              "                                                                 dtypes_list=[&#x27;char_str&#x27;,\n",
              "                                                                              &#x27;char_str&#x27;,\n",
              "                                                                              &#x27;char_str&#x27;,\n",
              "                                                                              &#x27;char_str&#x27;,\n",
              "                                                                              &#x27;char_str&#x27;,\n",
              "                                                                              &#x27;float_int_num&#x27;,\n",
              "                                                                              &#x27;char_str&#x27;,\n",
              "                                                                              &#x27;char_str&#x27;,\n",
              "                                                                              &#x27;float...\n",
              "                                                 FloatStr2Float(dtypes_list=[&#x27;float_int_num&#x27;,\n",
              "                                                                             &#x27;float_int_num&#x27;,\n",
              "                                                                             &#x27;float_int_num&#x27;],\n",
              "                                                                missing_values_reference_list=[])),\n",
              "                                                (&#x27;numpyreplacemissingvalues&#x27;,\n",
              "                                                 NumpyReplaceMissingValues(missing_values=[])),\n",
              "                                                (&#x27;numimputer&#x27;,\n",
              "                                                 NumImputer(missing_values=nan,\n",
              "                                                            strategy=&#x27;median&#x27;)),\n",
              "                                                (&#x27;optstandardscaler&#x27;,\n",
              "                                                 OptStandardScaler(use_scaler_flag=False)),\n",
              "                                                (&#x27;float32_transform&#x27;,\n",
              "                                                 float32_transform())]))])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>float32_transform_140563054741488</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">NumpyColumnSelector</label><div class=\"sk-toggleable__content\"><pre>NumpyColumnSelector(columns=[0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17,\n",
              "                             18, 19])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">CompressStrings</label><div class=\"sk-toggleable__content\"><pre>CompressStrings(compress_type=&#x27;hash&#x27;,\n",
              "                dtypes_list=[&#x27;char_str&#x27;, &#x27;char_str&#x27;, &#x27;char_str&#x27;, &#x27;char_str&#x27;,\n",
              "                             &#x27;char_str&#x27;, &#x27;float_int_num&#x27;, &#x27;char_str&#x27;,\n",
              "                             &#x27;char_str&#x27;, &#x27;float_int_num&#x27;, &#x27;char_str&#x27;,\n",
              "                             &#x27;char_str&#x27;, &#x27;char_str&#x27;, &#x27;float_int_num&#x27;,\n",
              "                             &#x27;char_str&#x27;, &#x27;float_int_num&#x27;, &#x27;char_str&#x27;,\n",
              "                             &#x27;char_str&#x27;],\n",
              "                missing_values_reference_list=[&#x27;&#x27;, &#x27;-&#x27;, &#x27;?&#x27;, nan],\n",
              "                misslist_list=[[], [], [], [], [], [], [], [], [], [], [], [],\n",
              "                               [], [], [], [], []])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">NumpyReplaceMissingValues</label><div class=\"sk-toggleable__content\"><pre>NumpyReplaceMissingValues(missing_values=[])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">NumpyReplaceUnknownValues</label><div class=\"sk-toggleable__content\"><pre>NumpyReplaceUnknownValues(filling_values=nan,\n",
              "                          filling_values_list=[nan, nan, nan, nan, nan, 100001,\n",
              "                                               nan, nan, 100001, nan, nan, nan,\n",
              "                                               100001, nan, 100001, nan, nan],\n",
              "                          known_values_list=[[227259264688753646810077375790908286508,\n",
              "                                              253732214910815238134509288111402486722,\n",
              "                                              303819144345098626554456011496217223575,\n",
              "                                              280353606872939388614315901186094326949],\n",
              "                                             [310861434292724266828...\n",
              "                                             [328286527295663582663365503319902632676,\n",
              "                                              119641707607939038914465000864290288880,\n",
              "                                              283364312271660996400883763491949419861,\n",
              "                                              27741019508977055807423991753468819528],\n",
              "                                             [1.0, 2.0],\n",
              "                                             [68186749286663113704472210246844540664,\n",
              "                                              220736790854050750400968561922076059550],\n",
              "                                             [169662019754859674907370307324476606919,\n",
              "                                              220736790854050750400968561922076059550]],\n",
              "                          missing_values_reference_list=[&#x27;&#x27;, &#x27;-&#x27;, &#x27;?&#x27;, nan])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" ><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">boolean2float</label><div class=\"sk-toggleable__content\"><pre>boolean2float()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-8\" type=\"checkbox\" ><label for=\"sk-estimator-id-8\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">CatImputer</label><div class=\"sk-toggleable__content\"><pre>CatImputer(missing_values=nan, strategy=&#x27;most_frequent&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-9\" type=\"checkbox\" ><label for=\"sk-estimator-id-9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">CatEncoder</label><div class=\"sk-toggleable__content\"><pre>CatEncoder(categories=&#x27;auto&#x27;, dtype=&lt;class &#x27;numpy.float64&#x27;&gt;, encoding=&#x27;ordinal&#x27;,\n",
              "           handle_unknown=&#x27;error&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-10\" type=\"checkbox\" ><label for=\"sk-estimator-id-10\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">float32_transform</label><div class=\"sk-toggleable__content\"><pre>float32_transform()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>float32_transform_140563054740000</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-11\" type=\"checkbox\" ><label for=\"sk-estimator-id-11\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">NumpyColumnSelector</label><div class=\"sk-toggleable__content\"><pre>NumpyColumnSelector(columns=[1, 4, 12])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-12\" type=\"checkbox\" ><label for=\"sk-estimator-id-12\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">FloatStr2Float</label><div class=\"sk-toggleable__content\"><pre>FloatStr2Float(dtypes_list=[&#x27;float_int_num&#x27;, &#x27;float_int_num&#x27;, &#x27;float_int_num&#x27;],\n",
              "               missing_values_reference_list=[])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-13\" type=\"checkbox\" ><label for=\"sk-estimator-id-13\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">NumpyReplaceMissingValues</label><div class=\"sk-toggleable__content\"><pre>NumpyReplaceMissingValues(missing_values=[])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-14\" type=\"checkbox\" ><label for=\"sk-estimator-id-14\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">NumImputer</label><div class=\"sk-toggleable__content\"><pre>NumImputer(missing_values=nan, strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-15\" type=\"checkbox\" ><label for=\"sk-estimator-id-15\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OptStandardScaler</label><div class=\"sk-toggleable__content\"><pre>OptStandardScaler(use_scaler_flag=False)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-16\" type=\"checkbox\" ><label for=\"sk-estimator-id-16\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">float32_transform</label><div class=\"sk-toggleable__content\"><pre>float32_transform()</pre></div></div></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-17\" type=\"checkbox\" ><label for=\"sk-estimator-id-17\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">NumpyPermuteArray</label><div class=\"sk-toggleable__content\"><pre>NumpyPermuteArray(axis=0,\n",
              "                  permutation_indices=[0, 2, 3, 5, 6, 7, 8, 9, 10, 11, 13, 14,\n",
              "                                       15, 16, 17, 18, 19, 1, 4, 12])</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-18\" type=\"checkbox\" ><label for=\"sk-estimator-id-18\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">TAM</label><div class=\"sk-toggleable__content\"><pre>autoai_libs.cognito.transforms.transform_utils.TAM(tans_class = &lt;class &#x27;autoai_libs.cognito.transforms.transform_extras.IsolationForestAnomaly&#x27;&gt;, name = &#x27;isoforestanomaly&#x27;, tgraph = None, apply_all = True, col_names = [&#x27;CheckingStatus&#x27;, &#x27;LoanDuration&#x27;, &#x27;CreditHistory&#x27;, &#x27;LoanPurpose&#x27;, &#x27;LoanAmount&#x27;, &#x27;ExistingSavings&#x27;, &#x27;EmploymentDuration&#x27;, &#x27;InstallmentPercent&#x27;, &#x27;Sex&#x27;, &#x27;OthersOnLoan&#x27;, &#x27;CurrentResidenceDuration&#x27;, &#x27;OwnsProperty&#x27;, &#x27;Age&#x27;, &#x27;InstallmentPlans&#x27;, &#x27;Housing&#x27;, &#x27;ExistingCreditsCount&#x27;, &#x27;Job&#x27;, &#x27;Dependents&#x27;, &#x27;Telephone&#x27;, &#x27;ForeignWorker&#x27;], col_dtypes = [dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;), dtype(&#x27;float32&#x27;)], col_as_json_objects = None)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-19\" type=\"checkbox\" ><label for=\"sk-estimator-id-19\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">FS1</label><div class=\"sk-toggleable__content\"><pre>autoai_libs.cognito.transforms.transform_utils.FS1(cols_ids_must_keep = range(0, 20), additional_col_count_to_keep = 20, ptype = &#x27;classification&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-20\" type=\"checkbox\" ><label for=\"sk-estimator-id-20\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SnapRandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>SnapRandomForestClassifier(compress_trees=True,\n",
              "                           gpu_ids=array([0], dtype=uint32), max_depth=5,\n",
              "                           max_features=0.899349511906677, n_estimators=41,\n",
              "                           random_state=33)</pre></div></div></div></div></div></div></div>"
            ],
            "text/plain": [
              "Pipeline(steps=[('featureunion',\n",
              "                 FeatureUnion(transformer_list=[('float32_transform_140563054741488',\n",
              "                                                 Pipeline(steps=[('numpycolumnselector',\n",
              "                                                                  NumpyColumnSelector(columns=[0,\n",
              "                                                                                               2,\n",
              "                                                                                               3,\n",
              "                                                                                               5,\n",
              "                                                                                               6,\n",
              "                                                                                               7,\n",
              "                                                                                               8,\n",
              "                                                                                               9,\n",
              "                                                                                               10,\n",
              "                                                                                               11,\n",
              "                                                                                               13,\n",
              "                                                                                               14,\n",
              "                                                                                               15,\n",
              "                                                                                               16,\n",
              "                                                                                               17,\n",
              "                                                                                               18,\n",
              "                                                                                               19])),\n",
              "                                                                 ('compressstrings',\n",
              "                                                                  CompressStrings(compress_type='hash',\n",
              "                                                                                  dtypes_list=['char_str',\n",
              "                                                                                               'char_str',\n",
              "                                                                                               'char_str',\n",
              "                                                                                               'char_str',\n",
              "                                                                                               'char_str',\n",
              "                                                                                               'float_int_n...\n",
              "                 autoai_libs.cognito.transforms.transform_utils.TAM(tans_class = <class 'autoai_libs.cognito.transforms.transform_extras.IsolationForestAnomaly'>, name = 'isoforestanomaly', tgraph = None, apply_all = True, col_names = ['CheckingStatus', 'LoanDuration', 'CreditHistory', 'LoanPurpose', 'LoanAmount', 'ExistingSavings', 'EmploymentDuration', 'InstallmentPercent', 'Sex', 'OthersOnLoan', 'CurrentResidenceDuration', 'OwnsProperty', 'Age', 'InstallmentPlans', 'Housing', 'ExistingCreditsCount', 'Job', 'Dependents', 'Telephone', 'ForeignWorker'], col_dtypes = [dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32'), dtype('float32')], col_as_json_objects = None)),\n",
              "                ('fs1',\n",
              "                 autoai_libs.cognito.transforms.transform_utils.FS1(cols_ids_must_keep = range(0, 20), additional_col_count_to_keep = 20, ptype = 'classification')),\n",
              "                ('snaprandomforestclassifier',\n",
              "                 SnapRandomForestClassifier(compress_trees=True,\n",
              "                                            gpu_ids=array([0], dtype=uint32),\n",
              "                                            max_depth=5,\n",
              "                                            max_features=0.899349511906677,\n",
              "                                            n_estimators=41,\n",
              "                                            random_state=33))])"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipeline_name = 'Pipeline_4'\n",
        "pipeline_model = optimizer.get_pipeline(pipeline_name=pipeline_name, astype='sklearn')\n",
        "pipeline_model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "This cell constructs the cell scorer based on the experiment metadata."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import get_scorer\n",
        "\n",
        "scorer = get_scorer(optimization_params['scorer_for_ranking'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "#### Read the train and holdout data.\n",
        "\n",
        "**Hint:** You can also use external test dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "E0516 16:16:20.975787000 140704665543488 tls_security_connector.cc:382] TlsChannelSecurityConnector::cancel_check_peer error: UNKNOWN:Subchannel disconnected {created_time:\"2023-05-16T16:16:20.975767+02:00\"}\n"
          ]
        }
      ],
      "source": [
        "connection = optimizer.get_data_connections()[0]\n",
        "train_X, test_X, train_y, test_y = connection.read(with_holdout_split=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "#### Calculate the score"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0.8\n"
          ]
        }
      ],
      "source": [
        "score = scorer(pipeline_model, test_X.values, test_y.values)\n",
        "print(score)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"persistence\"></a>\n",
        "\n",
        "## 5. Store the model in repository\n",
        "\n",
        "Provide `pipeline_name` and `training_id`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Unsetting the space_id ...\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'SUCCESS'"
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "client.set.default_project(training_project_id)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Re-trained model id 225b11ff-1f52-4d35-959f-9101a1db2560\n"
          ]
        }
      ],
      "source": [
        "model_metadata = {\n",
        "    client.repository.ModelMetaNames.NAME: \"{0} - {1} - {2}\".format(deployed_pipeline_details['metadata']['name'],\n",
        "                                                                    pipeline_name,\n",
        "                                                                    pipeline_model.get_params()['steps'][-1][0])\n",
        "}\n",
        "published_model = client.repository.store_model(model=pipeline_name, meta_props=model_metadata, training_id=train_details['metadata']['id'])\n",
        "updated_model_id = client.repository.get_model_id(published_model)\n",
        "print('Re-trained model id', updated_model_id)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "List stored models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "------------------------------------  ------------------------------------------------------------  ------------------------  --------------  ----------  ----------------\n",
            "ID                                    NAME                                                          CREATED                   TYPE            SPEC_STATE  SPEC_REPLACEMENT\n",
            "225b11ff-1f52-4d35-959f-9101a1db2560  AutoAI Credit risk - Pipeline_4 - snaprandomforestclassifier  2023-05-16T14:16:28.002Z  wml-hybrid_0.1  supported\n",
            "dcee0a44-7f8c-476e-815a-0dcd2befa4df  AutoAI Credit risk - Pipeline_4 - snaprandomforestclassifier  2023-05-16T13:47:31.002Z  wml-hybrid_0.1  supported\n",
            "d2de1575-ded0-42f7-95f6-9b47322469f0  AutoAI Credit risk - Pipeline_4 - snaprandomforestclassifier  2023-05-16T12:16:25.002Z  wml-hybrid_0.1  supported\n",
            "f7fc49ae-798e-42f9-b951-32ef9f866f05  AutoAI Credit risk - P9 XGB Classifier                        2023-05-16T11:54:07.002Z  wml-hybrid_0.1  supported\n",
            "506bf6ed-7ff6-4e02-bc11-9d4b2c640ebc  P10 - Pretrained AutoAI pipeline                              2023-05-15T12:43:59.002Z  wml-hybrid_0.1  supported\n",
            "------------------------------------  ------------------------------------------------------------  ------------------------  --------------  ----------  ----------------\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ID</th>\n",
              "      <th>NAME</th>\n",
              "      <th>CREATED</th>\n",
              "      <th>TYPE</th>\n",
              "      <th>SPEC_STATE</th>\n",
              "      <th>SPEC_REPLACEMENT</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>225b11ff-1f52-4d35-959f-9101a1db2560</td>\n",
              "      <td>AutoAI Credit risk - Pipeline_4 - snaprandomfo...</td>\n",
              "      <td>2023-05-16T14:16:28.002Z</td>\n",
              "      <td>wml-hybrid_0.1</td>\n",
              "      <td>supported</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>dcee0a44-7f8c-476e-815a-0dcd2befa4df</td>\n",
              "      <td>AutoAI Credit risk - Pipeline_4 - snaprandomfo...</td>\n",
              "      <td>2023-05-16T13:47:31.002Z</td>\n",
              "      <td>wml-hybrid_0.1</td>\n",
              "      <td>supported</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>d2de1575-ded0-42f7-95f6-9b47322469f0</td>\n",
              "      <td>AutoAI Credit risk - Pipeline_4 - snaprandomfo...</td>\n",
              "      <td>2023-05-16T12:16:25.002Z</td>\n",
              "      <td>wml-hybrid_0.1</td>\n",
              "      <td>supported</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>f7fc49ae-798e-42f9-b951-32ef9f866f05</td>\n",
              "      <td>AutoAI Credit risk - P9 XGB Classifier</td>\n",
              "      <td>2023-05-16T11:54:07.002Z</td>\n",
              "      <td>wml-hybrid_0.1</td>\n",
              "      <td>supported</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>506bf6ed-7ff6-4e02-bc11-9d4b2c640ebc</td>\n",
              "      <td>P10 - Pretrained AutoAI pipeline</td>\n",
              "      <td>2023-05-15T12:43:59.002Z</td>\n",
              "      <td>wml-hybrid_0.1</td>\n",
              "      <td>supported</td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                     ID  \\\n",
              "0  225b11ff-1f52-4d35-959f-9101a1db2560   \n",
              "1  dcee0a44-7f8c-476e-815a-0dcd2befa4df   \n",
              "2  d2de1575-ded0-42f7-95f6-9b47322469f0   \n",
              "3  f7fc49ae-798e-42f9-b951-32ef9f866f05   \n",
              "4  506bf6ed-7ff6-4e02-bc11-9d4b2c640ebc   \n",
              "\n",
              "                                                NAME  \\\n",
              "0  AutoAI Credit risk - Pipeline_4 - snaprandomfo...   \n",
              "1  AutoAI Credit risk - Pipeline_4 - snaprandomfo...   \n",
              "2  AutoAI Credit risk - Pipeline_4 - snaprandomfo...   \n",
              "3             AutoAI Credit risk - P9 XGB Classifier   \n",
              "4                   P10 - Pretrained AutoAI pipeline   \n",
              "\n",
              "                    CREATED            TYPE SPEC_STATE SPEC_REPLACEMENT  \n",
              "0  2023-05-16T14:16:28.002Z  wml-hybrid_0.1  supported                   \n",
              "1  2023-05-16T13:47:31.002Z  wml-hybrid_0.1  supported                   \n",
              "2  2023-05-16T12:16:25.002Z  wml-hybrid_0.1  supported                   \n",
              "3  2023-05-16T11:54:07.002Z  wml-hybrid_0.1  supported                   \n",
              "4  2023-05-15T12:43:59.002Z  wml-hybrid_0.1  supported                   "
            ]
          },
          "execution_count": 27,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "client.repository.list_models()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"redeploy\"></a>\n",
        "## 6. Redeploy and score new version of the model\n",
        "\n",
        "In this section, you'll learn how to redeploy new version of the model by using the watsonx.ai Client.\n",
        "\n",
        "**Hint:** As a best practice please consider using the test space before moving to production.\n",
        "\n",
        "```\n",
        "promote(asset_id: str, source_project_id: str, target_space_id: str, rev_id: str = None)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Promote model to deployment space"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "promoted_model_id = client.spaces.promote(asset_id=updated_model_id, source_project_id=training_project_id, target_space_id=deployment_space_id)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Check current deployment details before update."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Unsetting the project_id ...\n",
            "Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            "{\n",
            "   \"entity\": {\n",
            "      \"asset\": {\n",
            "         \"id\": \"aeb64510-166c-46d2-8b11-229c1740ea8c\"\n",
            "      },\n",
            "      \"custom\": {},\n",
            "      \"deployed_asset_type\": \"model\",\n",
            "      \"hybrid_pipeline_hardware_specs\": [\n",
            "         {\n",
            "            \"hardware_spec\": {\n",
            "               \"name\": \"S\",\n",
            "               \"num_nodes\": 1\n",
            "            },\n",
            "            \"node_runtime_id\": \"auto_ai.kb\"\n",
            "         }\n",
            "      ],\n",
            "      \"name\": \"AutoAI Credit risk - Online Deployment\",\n",
            "      \"online\": {},\n",
            "      \"space_id\": \"ac0d6d39-f567-4640-8807-92ca5a17f745\",\n",
            "      \"status\": {\n",
            "         \"online_url\": {\n",
            "            \"url\": \"https://cpd-zen.apps.ocp412wmlautoaicpd47x1fips.cp.fyre.ibm.com/ml/v4/deployments/d4437b42-6e74-49e1-a0d5-f68a8eb03992/predictions\"\n",
            "         },\n",
            "         \"serving_urls\": [\n",
            "            \"https://cpd-zen.apps.ocp412wmlautoaicpd47x1fips.cp.fyre.ibm.com/ml/v4/deployments/d4437b42-6e74-49e1-a0d5-f68a8eb03992/predictions\"\n",
            "         ],\n",
            "         \"state\": \"ready\"\n",
            "      }\n",
            "   },\n",
            "   \"metadata\": {\n",
            "      \"created_at\": \"2023-05-16T14:05:47.486Z\",\n",
            "      \"id\": \"d4437b42-6e74-49e1-a0d5-f68a8eb03992\",\n",
            "      \"modified_at\": \"2023-05-16T14:05:47.486Z\",\n",
            "      \"name\": \"AutoAI Credit risk - Online Deployment\",\n",
            "      \"owner\": \"1000331001\",\n",
            "      \"space_id\": \"ac0d6d39-f567-4640-8807-92ca5a17f745\"\n",
            "   },\n",
            "   \"system\": {\n",
            "      \"warnings\": [\n",
            "         {\n",
            "            \"id\": \"Deprecated\",\n",
            "            \"message\": \"online_url is deprecated and will be removed in a future release. Use serving_urls instead.\"\n",
            "         }\n",
            "      ]\n",
            "   }\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "client.set.default_space(deployment_space_id)\n",
        "print(json.dumps(client.deployments.get_details(deployment_id), indent=3))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Update the deployment with new model\n",
        "**Note:** The update is asynchronous."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            "Since ASSET is patched, deployment with new asset id/rev is being started. Monitor the status using deployments.get_details(deployment_uid) api\n"
          ]
        }
      ],
      "source": [
        "metadata = {\n",
        "    client.deployments.ConfigurationMetaNames.ASSET: {\n",
        "        \"id\": promoted_model_id,\n",
        "    }\n",
        "}\n",
        "    \n",
        "updated_deployment = client.deployments.update(deployment_id, changes=metadata)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Wait for the deployment update: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". updating Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            ". ready \n",
            "Deployment update finished with status:  ready\n"
          ]
        }
      ],
      "source": [
        "import time\n",
        "\n",
        "status = None\n",
        "while status not in ['ready', 'failed']:\n",
        "    time.sleep(2)\n",
        "    deployment_details = client.deployments.get_details(deployment_id)\n",
        "    status = deployment_details['entity']['status'].get('state')\n",
        "    print('.', status, end=' ')\n",
        "    \n",
        "print(\"\\nDeployment update finished with status: \", status)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "#### Get updated deployment details"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
            "{\n",
            "  \"entity\": {\n",
            "    \"asset\": {\n",
            "      \"id\": \"d78cc303-f277-42ef-85c0-d5d220b60202\"\n",
            "    },\n",
            "    \"custom\": {},\n",
            "    \"deployed_asset_type\": \"model\",\n",
            "    \"hybrid_pipeline_hardware_specs\": [\n",
            "      {\n",
            "        \"hardware_spec\": {\n",
            "          \"id\": \"e7ed1d6c-2e89-42d7-aed5-863b972c1d2b\",\n",
            "          \"name\": \"S\",\n",
            "          \"num_nodes\": 1\n",
            "        },\n",
            "        \"node_runtime_id\": \"auto_ai.kb\"\n",
            "      }\n",
            "    ],\n",
            "    \"name\": \"AutoAI Credit risk - Online Deployment\",\n",
            "    \"online\": {},\n",
            "    \"space_id\": \"ac0d6d39-f567-4640-8807-92ca5a17f745\",\n",
            "    \"status\": {\n",
            "      \"message\": {\n",
            "        \"level\": \"warning\",\n",
            "        \"text\": \"Successfully patched the asset.\"\n",
            "      },\n",
            "      \"online_url\": {\n",
            "        \"url\": \"https://cpd-zen.apps.ocp412wmlautoaicpd47x1fips.cp.fyre.ibm.com/ml/v4/deployments/d4437b42-6e74-49e1-a0d5-f68a8eb03992/predictions\"\n",
            "      },\n",
            "      \"serving_urls\": [\n",
            "        \"https://cpd-zen.apps.ocp412wmlautoaicpd47x1fips.cp.fyre.ibm.com/ml/v4/deployments/d4437b42-6e74-49e1-a0d5-f68a8eb03992/predictions\"\n",
            "      ],\n",
            "      \"state\": \"ready\"\n",
            "    }\n",
            "  },\n",
            "  \"metadata\": {\n",
            "    \"created_at\": \"2023-05-16T14:05:47.486Z\",\n",
            "    \"id\": \"d4437b42-6e74-49e1-a0d5-f68a8eb03992\",\n",
            "    \"modified_at\": \"2023-05-16T14:17:32.010Z\",\n",
            "    \"name\": \"AutoAI Credit risk - Online Deployment\",\n",
            "    \"owner\": \"1000331001\",\n",
            "    \"space_id\": \"ac0d6d39-f567-4640-8807-92ca5a17f745\"\n",
            "  },\n",
            "  \"system\": {\n",
            "    \"warnings\": [\n",
            "      {\n",
            "        \"id\": \"Deprecated\",\n",
            "        \"message\": \"online_url is deprecated and will be removed in a future release. Use serving_urls instead.\"\n",
            "      }\n",
            "    ]\n",
            "  }\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "print(json.dumps(client.deployments.get_details(deployment_id), indent=2))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Score updated model\n",
        "Create sample payload and score the deployed model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "scoring_payload = {\"input_data\": [{\"values\": test_X[:3]}]}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Use client.deployments.score() method to run scoring."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        }
      },
      "outputs": [],
      "source": [
        "predictions = client.deployments.score(deployment_id, scoring_payload)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "pycharm": {
          "name": "#%%\n"
        },
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"predictions\": [\n",
            "    {\n",
            "      \"fields\": [\n",
            "        \"prediction\",\n",
            "        \"probability\"\n",
            "      ],\n",
            "      \"values\": [\n",
            "        [\n",
            "          \"Risk\",\n",
            "          [\n",
            "            0.1754017922936416,\n",
            "            0.8245982077063584\n",
            "          ]\n",
            "        ],\n",
            "        [\n",
            "          \"Risk\",\n",
            "          [\n",
            "            0.1643842371498666,\n",
            "            0.8356157628501334\n",
            "          ]\n",
            "        ],\n",
            "        [\n",
            "          \"Risk\",\n",
            "          [\n",
            "            0.4907570931969619,\n",
            "            0.5092429068030381\n",
            "          ]\n",
            "        ]\n",
            "      ]\n",
            "    }\n",
            "  ]\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "print(json.dumps(predictions, indent=2))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"cleanup\"></a>\n",
        "## 7. Clean up"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "If you want to clean up all created assets:\n",
        "- experiments\n",
        "- trainings\n",
        "- pipelines\n",
        "- models\n",
        "- deployments\n",
        "\n",
        "please follow up this sample [notebook](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd5.1/notebooks/python_sdk/instance-management/Machine%20Learning%20artifacts%20management.ipynb)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "<a id=\"summary\"></a>\n",
        "## 8. Summary and next steps"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "You successfully completed this notebook! You learned how to use scikit-learn machine learning as well as Watson Machine Learning for model creation and deployment.\n",
        "\n",
        "Check out our _<a href=\"https://ibm.github.io/watsonx-ai-python-sdk/samples.html\" target=\"_blank\" rel=\"noopener no referrer\">Online Documentation</a>_ for more samples, tutorials, documentation, how-tos, and blog posts. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "### Authors\n",
        "\n",
        "Lukasz Cmielowski, PhD, is a Senior Technical Staff Member at IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.\n",
        "\n",
        "Dorota Laczak, Python Software Developer in Watson Machine Learning AutoAI at IBM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {
          "name": "#%% md\n"
        }
      },
      "source": [
        "Copyright © 2023-2025 IBM. This notebook and its source code are released under the terms of the MIT License."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.9"
    },
    "pycharm": {
      "stem_cell": {
        "cell_type": "raw",
        "metadata": {
          "collapsed": false
        },
        "source": []
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}