cpd5.0/notebooks/python_sdk/experiments/autoai/Use AutoAI and timeseries data to predict COVID cases.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use AutoAI and timeseries data to predict COVID19 cases with `ibm-watsonx-ai`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook contains the steps and code to demonstrate support of AutoAI experiments for timeseries data sets in Watson Machine Learning service. It introduces commands for data retrieval, training experiments, persisting pipelines, testing pipelines, deploying pipelines, and scoring.\n",
    "\n",
    "Some familiarity with Python is helpful. This notebook uses Python 3.11.\n",
    "\n",
    "\n",
    "## Learning goals\n",
    "\n",
    "The learning goals of this notebook are:\n",
    "\n",
    "-  Define Watson Machine Learning experiment for timeseries data sets. \n",
    "-  Work with experiments to train AutoAI models.\n",
    "-  Compare trained models quality and select the best one for further deployment.\n",
    "-  Online deployment and score the trained model.\n",
    "\n",
    "\n",
    "## Contents\n",
    "\n",
    "This notebook contains the following parts:\n",
    "\n",
    "1. [Setup](#setup)\n",
    "2. [Timeseries data sets](#ts)\n",
    "3. [Optimizer definition](#definition)\n",
    "4. [Experiment Run](#run)\n",
    "5. [Pipelines comparison](#comparison)\n",
    "6. [Deploy and Score](#scoring)\n",
    "7. [Clean up](#cleanup)\n",
    "8. [Summary and next steps](#summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "## 1. Set up the environment\n",
    "\n",
    "Before you use the sample code in this notebook, you must perform the following setup tasks:\n",
    "\n",
    "-  Contact with your Cloud Pack for Data administrator and ask him for your account credentials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install and import the `ibm-watsonx-ai` and dependecies\n",
    "**Note:** `ibm-watsonx-ai` documentation can be found <a href=\"https://ibm.github.io/watsonx-ai-python-sdk/index.html\" target=\"_blank\" rel=\"noopener no referrer\">here</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install wget | tail -n 1\n",
    "!pip install plotly | tail -n 1\n",
    "!pip install -U ibm-watsonx-ai | tail -n 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connection to WML\n",
    "\n",
    "Authenticate the Watson Machine Learning service on IBM Cloud Pack for Data. You need to provide platform `url`, your `username` and `api_key`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "username = 'PASTE YOUR USERNAME HERE'\n",
    "api_key = 'PASTE YOUR API_KEY HERE'\n",
    "url = 'PASTE THE PLATFORM URL HERE'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watsonx_ai import Credentials\n",
    "\n",
    "credentials = Credentials(\n",
    "    username=username,\n",
    "    api_key=api_key,\n",
    "    url=url,\n",
    "    instance_id=\"openshift\",\n",
    "    version=\"5.0\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively you can use `username` and `password` to authenticate WML services.\n",
    "\n",
    "```python\n",
    "credentials = Credentials(\n",
    "    username=***,\n",
    "    password=***,\n",
    "    url=***,\n",
    "    instance_id=\"openshift\",\n",
    "    version=\"5.0\"\n",
    ")\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watsonx_ai import APIClient\n",
    "\n",
    "client = APIClient(credentials)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with spaces\n",
    "\n",
    "First of all, you need to create a space that will be used for your work. If you do not have space already created, you can use `{PLATFORM_URL}/ml-runtime/spaces?context=icp4data` to create one.\n",
    "\n",
    "- Click New Deployment Space\n",
    "- Create an empty space\n",
    "- Go to space `Settings` tab\n",
    "- Copy `space_id` and paste it below\n",
    "\n",
    "**Tip**: You can also use SDK to prepare the space for your work. More information can be found [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd5.0/notebooks/python_sdk/instance-management/Space%20management.ipynb).\n",
    "\n",
    "**Action**: Assign space ID below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "space_id = 'PASTE YOUR SPACE ID HERE'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the `list` method to print all existing spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.spaces.list(limit=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be able to interact with all resources available in Watson Machine Learning, you need to set the **space** which you will be using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'SUCCESS'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.set.default_space(space_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ts\"></a>\n",
    "## 2. Timeseries data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training data sets\n",
    "\n",
    "Define connection information to training data CSV file.  \n",
    "This example uses the [Poland COVID19 daily confirmed cases](https://raw.githubusercontent.com/LukaszCmielowski/covid-19/master/poland_daily_cases_03_28_2021.csv)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, wget\n",
    "\n",
    "filename = 'poland_daily_cases_03_28_2021.csv'\n",
    "base_url = 'https://raw.githubusercontent.com/LukaszCmielowski/covid-19/master/'\n",
    "\n",
    "if not os.path.isfile(filename): wget.download(base_url + filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating data asset...\n",
      "SUCCESS\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'metadata': {'space_id': 'cbd87244-b000-4279-b991-3cefbf8b1555',\n",
       "  'usage': {'last_updated_at': '2024-04-29T11:39:33Z',\n",
       "   'last_updater_id': '1000330999',\n",
       "   'last_update_time': 1714390773219,\n",
       "   'last_accessed_at': '2024-04-29T11:39:33Z',\n",
       "   'last_access_time': 1714390773219,\n",
       "   'last_accessor_id': '1000330999',\n",
       "   'access_count': 0},\n",
       "  'rov': {'mode': 0,\n",
       "   'collaborator_ids': {},\n",
       "   'member_roles': {'1000330999': {'user_iam_id': '1000330999',\n",
       "     'roles': ['OWNER']}}},\n",
       "  'name': 'poland_daily_cases_03_28_2021.csv',\n",
       "  'description': '',\n",
       "  'asset_type': 'data_asset',\n",
       "  'origin_country': 'us',\n",
       "  'resource_key': 'poland_daily_cases_03_28_2021.csv',\n",
       "  'rating': 0.0,\n",
       "  'total_ratings': 0,\n",
       "  'catalog_id': '6dd61f7f-0608-4288-abe8-3881dce0c404',\n",
       "  'created': 1714390773219,\n",
       "  'created_at': '2024-04-29T11:39:33Z',\n",
       "  'owner_id': '1000330999',\n",
       "  'size': 0,\n",
       "  'version': 2.0,\n",
       "  'asset_state': 'available',\n",
       "  'asset_attributes': ['data_asset'],\n",
       "  'asset_id': '4d0143df-3e12-4422-8ab3-f02c45aae7cf',\n",
       "  'asset_category': 'USER',\n",
       "  'creator_id': '1000330999',\n",
       "  'guid': '4d0143df-3e12-4422-8ab3-f02c45aae7cf',\n",
       "  'href': '/v2/assets/4d0143df-3e12-4422-8ab3-f02c45aae7cf?space_id=cbd87244-b000-4279-b991-3cefbf8b1555',\n",
       "  'last_updated_at': '2024-04-29T11:39:33Z'},\n",
       " 'entity': {'data_asset': {'mime_type': 'text/csv'}}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "asset_details = client.data_assets.create('poland_daily_cases_03_28_2021.csv', filename)\n",
    "asset_details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'4d0143df-3e12-4422-8ab3-f02c45aae7cf'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.data_assets.get_id(asset_details)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watsonx_ai.helpers import DataConnection\n",
    "\n",
    "\n",
    "covid_daily_cases = DataConnection(data_asset_id=client.data_assets.get_id(asset_details))\n",
    "\n",
    "training_data_reference=[covid_daily_cases]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "hovertemplate": "date=%{x}<br>daily_cases=%{y}<extra></extra>",
         "legendgroup": "",
         "line": {
          "color": "#636efa",
          "dash": "solid"
         },
         "marker": {
          "symbol": "circle"
         },
         "mode": "lines",
         "name": "",
         "orientation": "v",
         "showlegend": false,
         "type": "scatter",
         "x": [
          "1/22/2020",
          "1/23/2020",
          "1/24/2020",
          "1/25/2020",
          "1/26/2020",
          "1/27/2020",
          "1/28/2020",
          "1/29/2020",
          "1/30/2020",
          "1/31/2020",
          "2/1/2020",
          "2/2/2020",
          "2/3/2020",
          "2/4/2020",
          "2/5/2020",
          "2/6/2020",
          "2/7/2020",
          "2/8/2020",
          "2/9/2020",
          "2/10/2020",
          "2/11/2020",
          "2/12/2020",
          "2/13/2020",
          "2/14/2020",
          "2/15/2020",
          "2/16/2020",
          "2/17/2020",
          "2/18/2020",
          "2/19/2020",
          "2/20/2020",
          "2/21/2020",
          "2/22/2020",
          "2/23/2020",
          "2/24/2020",
          "2/25/2020",
          "2/26/2020",
          "2/27/2020",
          "2/28/2020",
          "2/29/2020",
          "3/1/2020",
          "3/2/2020",
          "3/3/2020",
          "3/4/2020",
          "3/5/2020",
          "3/6/2020",
          "3/7/2020",
          "3/8/2020",
          "3/9/2020",
          "3/10/2020",
          "3/11/2020",
          "3/12/2020",
          "3/13/2020",
          "3/14/2020",
          "3/15/2020",
          "3/16/2020",
          "3/17/2020",
          "3/18/2020",
          "3/19/2020",
          "3/20/2020",
          "3/21/2020",
          "3/22/2020",
          "3/23/2020",
          "3/24/2020",
          "3/25/2020",
          "3/26/2020",
          "3/27/2020",
          "3/28/2020",
          "3/29/2020",
          "3/30/2020",
          "3/31/2020",
          "4/1/2020",
          "4/2/2020",
          "4/3/2020",
          "4/4/2020",
          "4/5/2020",
          "4/6/2020",
          "4/7/2020",
          "4/8/2020",
          "4/9/2020",
          "4/10/2020",
          "4/11/2020",
          "4/12/2020",
          "4/13/2020",
          "4/14/2020",
          "4/15/2020",
          "4/16/2020",
          "4/17/2020",
          "4/18/2020",
          "4/19/2020",
          "4/20/2020",
          "4/21/2020",
          "4/22/2020",
          "4/23/2020",
          "4/24/2020",
          "4/25/2020",
          "4/26/2020",
          "4/27/2020",
          "4/28/2020",
          "4/29/2020",
          "4/30/2020",
          "5/1/2020",
          "5/2/2020",
          "5/3/2020",
          "5/4/2020",
          "5/5/2020",
          "5/6/2020",
          "5/7/2020",
          "5/8/2020",
          "5/9/2020",
          "5/10/2020",
          "5/11/2020",
          "5/12/2020",
          "5/13/2020",
          "5/14/2020",
          "5/15/2020",
          "5/16/2020",
          "5/17/2020",
          "5/18/2020",
          "5/19/2020",
          "5/20/2020",
          "5/21/2020",
          "5/22/2020",
          "5/23/2020",
          "5/24/2020",
          "5/25/2020",
          "5/26/2020",
          "5/27/2020",
          "5/28/2020",
          "5/29/2020",
          "5/30/2020",
          "5/31/2020",
          "6/1/2020",
          "6/2/2020",
          "6/3/2020",
          "6/4/2020",
          "6/5/2020",
          "6/6/2020",
          "6/7/2020",
          "6/8/2020",
          "6/9/2020",
          "6/10/2020",
          "6/11/2020",
          "6/12/2020",
          "6/13/2020",
          "6/14/2020",
          "6/15/2020",
          "6/16/2020",
          "6/17/2020",
          "6/18/2020",
          "6/19/2020",
          "6/20/2020",
          "6/21/2020",
          "6/22/2020",
          "6/23/2020",
          "6/24/2020",
          "6/25/2020",
          "6/26/2020",
          "6/27/2020",
          "6/28/2020",
          "6/29/2020",
          "6/30/2020",
          "7/1/2020",
          "7/2/2020",
          "7/3/2020",
          "7/4/2020",
          "7/5/2020",
          "7/6/2020",
          "7/7/2020",
          "7/8/2020",
          "7/9/2020",
          "7/10/2020",
          "7/11/2020",
          "7/12/2020",
          "7/13/2020",
          "7/14/2020",
          "7/15/2020",
          "7/16/2020",
          "7/17/2020",
          "7/18/2020",
          "7/19/2020",
          "7/20/2020",
          "7/21/2020",
          "7/22/2020",
          "7/23/2020",
          "7/24/2020",
          "7/25/2020",
          "7/26/2020",
          "7/27/2020",
          "7/28/2020",
          "7/29/2020",
          "7/30/2020",
          "7/31/2020",
          "8/1/2020",
          "8/2/2020",
          "8/3/2020",
          "8/4/2020",
          "8/5/2020",
          "8/6/2020",
          "8/7/2020",
          "8/8/2020",
          "8/9/2020",
          "8/10/2020",
          "8/11/2020",
          "8/12/2020",
          "8/13/2020",
          "8/14/2020",
          "8/15/2020",
          "8/16/2020",
          "8/17/2020",
          "8/18/2020",
          "8/19/2020",
          "8/20/2020",
          "8/21/2020",
          "8/22/2020",
          "8/23/2020",
          "8/24/2020",
          "8/25/2020",
          "8/26/2020",
          "8/27/2020",
          "8/28/2020",
          "8/29/2020",
          "8/30/2020",
          "8/31/2020",
          "9/1/2020",
          "9/2/2020",
          "9/3/2020",
          "9/4/2020",
          "9/5/2020",
          "9/6/2020",
          "9/7/2020",
          "9/8/2020",
          "9/9/2020",
          "9/10/2020",
          "9/11/2020",
          "9/12/2020",
          "9/13/2020",
          "9/14/2020",
          "9/15/2020",
          "9/16/2020",
          "9/17/2020",
          "9/18/2020",
          "9/19/2020",
          "9/20/2020",
          "9/21/2020",
          "9/22/2020",
          "9/23/2020",
          "9/24/2020",
          "9/25/2020",
          "9/26/2020",
          "9/27/2020",
          "9/28/2020",
          "9/29/2020",
          "9/30/2020",
          "10/1/2020",
          "10/2/2020",
          "10/3/2020",
          "10/4/2020",
          "10/5/2020",
          "10/6/2020",
          "10/7/2020",
          "10/8/2020",
          "10/9/2020",
          "10/10/2020",
          "10/11/2020",
          "10/12/2020",
          "10/13/2020",
          "10/14/2020",
          "10/15/2020",
          "10/16/2020",
          "10/17/2020",
          "10/18/2020",
          "10/19/2020",
          "10/20/2020",
          "10/21/2020",
          "10/22/2020",
          "10/23/2020",
          "10/24/2020",
          "10/25/2020",
          "10/26/2020",
          "10/27/2020",
          "10/28/2020",
          "10/29/2020",
          "10/30/2020",
          "10/31/2020",
          "11/1/2020",
          "11/2/2020",
          "11/3/2020",
          "11/4/2020",
          "11/5/2020",
          "11/6/2020",
          "11/7/2020",
          "11/8/2020",
          "11/9/2020",
          "11/10/2020",
          "11/11/2020",
          "11/12/2020",
          "11/13/2020",
          "11/14/2020",
          "11/15/2020",
          "11/16/2020",
          "11/17/2020",
          "11/18/2020",
          "11/19/2020",
          "11/20/2020",
          "11/21/2020",
          "11/22/2020",
          "11/23/2020",
          "11/24/2020",
          "11/25/2020",
          "11/26/2020",
          "11/27/2020",
          "11/28/2020",
          "11/29/2020",
          "11/30/2020",
          "12/1/2020",
          "12/2/2020",
          "12/3/2020",
          "12/4/2020",
          "12/5/2020",
          "12/6/2020",
          "12/7/2020",
          "12/8/2020",
          "12/9/2020",
          "12/10/2020",
          "12/11/2020",
          "12/12/2020",
          "12/13/2020",
          "12/14/2020",
          "12/15/2020",
          "12/16/2020",
          "12/17/2020",
          "12/18/2020",
          "12/19/2020",
          "12/20/2020",
          "12/21/2020",
          "12/22/2020",
          "12/23/2020",
          "12/24/2020",
          "12/25/2020",
          "12/26/2020",
          "12/27/2020",
          "12/28/2020",
          "12/29/2020",
          "12/30/2020",
          "12/31/2020",
          "1/1/2021",
          "1/2/2021",
          "1/3/2021",
          "1/4/2021",
          "1/5/2021",
          "1/6/2021",
          "1/7/2021",
          "1/8/2021",
          "1/9/2021",
          "1/10/2021",
          "1/11/2021",
          "1/12/2021",
          "1/13/2021",
          "1/14/2021",
          "1/15/2021",
          "1/16/2021",
          "1/17/2021",
          "1/18/2021",
          "1/19/2021",
          "1/20/2021",
          "1/21/2021",
          "1/22/2021",
          "1/23/2021",
          "1/24/2021",
          "1/25/2021",
          "1/26/2021",
          "1/27/2021",
          "1/28/2021",
          "1/29/2021",
          "1/30/2021",
          "1/31/2021",
          "2/1/2021",
          "2/2/2021",
          "2/3/2021",
          "2/4/2021",
          "2/5/2021",
          "2/6/2021",
          "2/7/2021",
          "2/8/2021",
          "2/9/2021",
          "2/10/2021",
          "2/11/2021",
          "2/12/2021",
          "2/13/2021",
          "2/14/2021",
          "2/15/2021",
          "2/16/2021",
          "2/17/2021",
          "2/18/2021",
          "2/19/2021",
          "2/20/2021",
          "2/21/2021",
          "2/22/2021",
          "2/23/2021",
          "2/24/2021",
          "2/25/2021",
          "2/26/2021",
          "2/27/2021",
          "2/28/2021",
          "3/1/2021",
          "3/2/2021",
          "3/3/2021",
          "3/4/2021",
          "3/5/2021",
          "3/6/2021",
          "3/7/2021",
          "3/8/2021",
          "3/9/2021",
          "3/10/2021",
          "3/11/2021",
          "3/12/2021",
          "3/13/2021",
          "3/14/2021",
          "3/15/2021",
          "3/16/2021",
          "3/17/2021",
          "3/18/2021",
          "3/19/2021",
          "3/20/2021",
          "3/21/2021",
          "3/22/2021",
          "3/23/2021",
          "3/24/2021",
          "3/25/2021",
          "3/26/2021",
          "3/27/2021",
          "3/28/2021"
         ],
         "xaxis": "x",
         "y": [
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          0,
          1,
          0,
          4,
          0,
          6,
          5,
          6,
          9,
          18,
          19,
          35,
          16,
          58,
          61,
          13,
          104,
          70,
          111,
          98,
          115,
          152,
          150,
          170,
          168,
          249,
          224,
          193,
          256,
          243,
          392,
          437,
          244,
          475,
          311,
          435,
          357,
          370,
          380,
          401,
          318,
          260,
          268,
          380,
          336,
          461,
          363,
          545,
          306,
          263,
          313,
          342,
          381,
          381,
          344,
          285,
          316,
          422,
          237,
          228,
          270,
          318,
          313,
          425,
          309,
          307,
          319,
          285,
          345,
          330,
          595,
          283,
          411,
          401,
          241,
          272,
          356,
          383,
          471,
          404,
          476,
          312,
          395,
          305,
          443,
          399,
          352,
          330,
          416,
          215,
          379,
          230,
          292,
          361,
          362,
          576,
          575,
          599,
          400,
          282,
          359,
          376,
          440,
          375,
          396,
          407,
          506,
          314,
          301,
          304,
          311,
          296,
          300,
          294,
          298,
          276,
          319,
          193,
          247,
          239,
          382,
          371,
          259,
          314,
          231,
          205,
          257,
          277,
          262,
          265,
          305,
          370,
          299,
          267,
          264,
          333,
          353,
          339,
          358,
          279,
          399,
          380,
          418,
          458,
          584,
          443,
          337,
          502,
          512,
          615,
          657,
          658,
          548,
          575,
          680,
          640,
          726,
          809,
          843,
          624,
          619,
          551,
          715,
          811,
          832,
          771,
          594,
          595,
          597,
          735,
          767,
          903,
          900,
          581,
          548,
          763,
          729,
          887,
          791,
          759,
          631,
          502,
          550,
          595,
          612,
          691,
          567,
          437,
          302,
          400,
          421,
          506,
          594,
          603,
          502,
          377,
          605,
          600,
          837,
          757,
          1002,
          910,
          748,
          711,
          974,
          1136,
          1587,
          1584,
          1350,
          1306,
          1326,
          1552,
          1967,
          2292,
          2367,
          1934,
          2006,
          2236,
          3003,
          4280,
          4739,
          5300,
          4178,
          4394,
          5068,
          6526,
          8099,
          7705,
          9622,
          8536,
          7482,
          9291,
          10040,
          12107,
          13632,
          13628,
          11742,
          10241,
          16300,
          18820,
          20156,
          21629,
          21897,
          17171,
          15578,
          19364,
          24692,
          27143,
          27086,
          27875,
          24785,
          21713,
          25454,
          25221,
          22683,
          24051,
          25571,
          21854,
          20816,
          19152,
          19883,
          23975,
          22464,
          24213,
          17856,
          15002,
          32733,
          15356,
          16690,
          17304,
          15177,
          11482,
          5736,
          9113,
          13823,
          14863,
          13236,
          12427,
          9176,
          4421,
          8310,
          12166,
          13750,
          13105,
          11499,
          8976,
          4896,
          6874,
          12455,
          11953,
          11010,
          11246,
          8590,
          4633,
          7192,
          12358,
          13115,
          9081,
          4878,
          3842,
          3211,
          7624,
          12780,
          13464,
          10896,
          7006,
          5782,
          4385,
          7596,
          14220,
          12119,
          8763,
          10744,
          9133,
          4863,
          5394,
          9126,
          9457,
          7958,
          7292,
          5970,
          3332,
          4890,
          6943,
          7008,
          6693,
          6431,
          4566,
          2674,
          4603,
          6790,
          7153,
          6145,
          5864,
          4711,
          2504,
          4326,
          6801,
          6495,
          6053,
          5966,
          4725,
          2431,
          3999,
          6960,
          7013,
          6378,
          6585,
          5334,
          2542,
          5176,
          8699,
          9074,
          8772,
          8509,
          7040,
          3891,
          6304,
          12147,
          12143,
          11536,
          12097,
          10101,
          4786,
          7936,
          15698,
          15253,
          15831,
          14855,
          13569,
          6169,
          9953,
          17277,
          21111,
          18873,
          21063,
          17272,
          10895,
          14394,
          25053,
          27274,
          25996,
          26456,
          21850,
          14579,
          16740,
          30802,
          34150,
          35145,
          31759,
          29266
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "legend": {
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "date"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "daily_cases"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import plotly.express as px\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(filename)\n",
    "fig = px.line(df, x='date', y=\"daily_cases\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"definition\"></a>\n",
    "## 3. Optimizer definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizer configuration\n",
    "\n",
    "Provide the input information for AutoAI optimizer:\n",
    "- `name` - experiment name\n",
    "- `prediction_type` - type of the problem\n",
    "- `prediction_columns` - target columns names\n",
    "- `scoring` - optimization metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "from ibm_watsonx_ai.experiment import AutoAI\n",
    "\n",
    "experiment = AutoAI(credentials, space_id=space_id)\n",
    "forecast_window = 7\n",
    "\n",
    "pipeline_optimizer = experiment.optimizer(\n",
    "            name=\"COVID19 PL timeseries\",\n",
    "            prediction_type=AutoAI.PredictionType.FORECASTING,\n",
    "            prediction_columns=['daily_cases'],\n",
    "            timestamp_column_name=0,\n",
    "            scoring=AutoAI.Metrics.R2_SCORE,\n",
    "            max_number_of_estimators=1,\n",
    "            backtest_num=4,\n",
    "            lookback_window=5,\n",
    "            forecast_window=forecast_window,\n",
    "            holdout_size=5\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Configuration parameters can be retrieved via `pipeline_optimizer.get_params()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"run\"></a>\n",
    "## 4. Experiment run\n",
    "\n",
    "Call the `fit()` method to trigger the AutoAI experiment. You can either use interactive mode (synchronous job) or background mode (asychronous job) by specifying `background_model=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training job dacaa0a5-03b5-4f10-9403-5e8295b02aad completed: 100%|████████| [00:56<00:00,  1.77it/s]\n"
     ]
    }
   ],
   "source": [
    "fit_details = pipeline_optimizer.fit(training_data_reference=training_data_reference, background_mode=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the `get_run_status()` method to monitor AutoAI jobs in background mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'completed'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_optimizer.get_run_status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"comparison\"></a>\n",
    "## 5. Pipelines comparison\n",
    "\n",
    "You can list trained pipelines and evaluation metrics information in\n",
    "the form of a Pandas DataFrame by calling the `summary()` method. You can\n",
    "use the DataFrame to compare all discovered pipelines and select the one\n",
    "you like for further deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Enhancements</th>\n",
       "      <th>Estimator</th>\n",
       "      <th>Winner</th>\n",
       "      <th>validation_symmetric_mean_absolute_percentage_error</th>\n",
       "      <th>holdout_avg_r2</th>\n",
       "      <th>holdout_avg_mean_absolute_error</th>\n",
       "      <th>holdout_avg_root_mean_squared_error</th>\n",
       "      <th>holdout_avg_symmetric_mean_absolute_percentage_error</th>\n",
       "      <th>holdout_mean_absolute_error</th>\n",
       "      <th>holdout_root_mean_squared_error</th>\n",
       "      <th>holdout_symmetric_mean_absolute_percentage_error</th>\n",
       "      <th>holdout_r2</th>\n",
       "      <th>backtest_avg_r2</th>\n",
       "      <th>backtest_avg_mean_absolute_error</th>\n",
       "      <th>backtest_avg_root_mean_squared_error</th>\n",
       "      <th>backtest_avg_symmetric_mean_absolute_percentage_error</th>\n",
       "      <th>backtest_mean_absolute_error</th>\n",
       "      <th>backtest_root_mean_squared_error</th>\n",
       "      <th>backtest_symmetric_mean_absolute_percentage_error</th>\n",
       "      <th>backtest_r2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pipeline Name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Pipeline_6</th>\n",
       "      <td>HPO, FE</td>\n",
       "      <td>Ensembler</td>\n",
       "      <td>True</td>\n",
       "      <td>28.031457</td>\n",
       "      <td>-21.05751</td>\n",
       "      <td>9994.500147</td>\n",
       "      <td>10123.154738</td>\n",
       "      <td>36.81724</td>\n",
       "      <td>9994.500147</td>\n",
       "      <td>10123.154738</td>\n",
       "      <td>36.81724</td>\n",
       "      <td>-21.05751</td>\n",
       "      <td>0.042897</td>\n",
       "      <td>3538.269264</td>\n",
       "      <td>4011.517396</td>\n",
       "      <td>22.172355</td>\n",
       "      <td>3538.269264</td>\n",
       "      <td>4011.517396</td>\n",
       "      <td>22.172355</td>\n",
       "      <td>0.042897</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Enhancements  Estimator  Winner  \\\n",
       "Pipeline Name                                   \n",
       "Pipeline_6         HPO, FE  Ensembler    True   \n",
       "\n",
       "               validation_symmetric_mean_absolute_percentage_error  \\\n",
       "Pipeline Name                                                        \n",
       "Pipeline_6                                             28.031457     \n",
       "\n",
       "               holdout_avg_r2  holdout_avg_mean_absolute_error  \\\n",
       "Pipeline Name                                                    \n",
       "Pipeline_6          -21.05751                      9994.500147   \n",
       "\n",
       "               holdout_avg_root_mean_squared_error  \\\n",
       "Pipeline Name                                        \n",
       "Pipeline_6                            10123.154738   \n",
       "\n",
       "               holdout_avg_symmetric_mean_absolute_percentage_error  \\\n",
       "Pipeline Name                                                         \n",
       "Pipeline_6                                              36.81724      \n",
       "\n",
       "               holdout_mean_absolute_error  holdout_root_mean_squared_error  \\\n",
       "Pipeline Name                                                                 \n",
       "Pipeline_6                     9994.500147                     10123.154738   \n",
       "\n",
       "               holdout_symmetric_mean_absolute_percentage_error  holdout_r2  \\\n",
       "Pipeline Name                                                                 \n",
       "Pipeline_6                                             36.81724   -21.05751   \n",
       "\n",
       "               backtest_avg_r2  backtest_avg_mean_absolute_error  \\\n",
       "Pipeline Name                                                      \n",
       "Pipeline_6            0.042897                       3538.269264   \n",
       "\n",
       "               backtest_avg_root_mean_squared_error  \\\n",
       "Pipeline Name                                         \n",
       "Pipeline_6                              4011.517396   \n",
       "\n",
       "               backtest_avg_symmetric_mean_absolute_percentage_error  \\\n",
       "Pipeline Name                                                          \n",
       "Pipeline_6                                             22.172355       \n",
       "\n",
       "               backtest_mean_absolute_error  backtest_root_mean_squared_error  \\\n",
       "Pipeline Name                                                                   \n",
       "Pipeline_6                      3538.269264                       4011.517396   \n",
       "\n",
       "               backtest_symmetric_mean_absolute_percentage_error  backtest_r2  \n",
       "Pipeline Name                                                                  \n",
       "Pipeline_6                                             22.172355     0.042897  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary = pipeline_optimizer.summary()\n",
    "summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check pipeline details by calling `get_pipeline_details(pipeline_name='Pipeline_8')`. By default details of best pipeline are returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best pipeline is: Pipeline_6\n"
     ]
    }
   ],
   "source": [
    "best_pipeline_name = summary[summary.Winner==True].index.values[0]\n",
    "print('Best pipeline is:', best_pipeline_name)\n",
    "\n",
    "pipeline_details = pipeline_optimizer.get_pipeline_details()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Holdout data visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "hovertemplate": "variable=observed_values<br>time=%{x|%B %d, %Y}<br>value=%{y}<extra></extra>",
         "legendgroup": "observed_values",
         "line": {
          "color": "#636efa",
          "dash": "solid"
         },
         "marker": {
          "symbol": "circle"
         },
         "mode": "lines",
         "name": "observed_values",
         "orientation": "v",
         "showlegend": true,
         "type": "scatter",
         "x": [
          "2021-03-19",
          "2021-03-20",
          "2021-03-21",
          "2021-03-22",
          "2021-03-23"
         ],
         "xaxis": "x",
         "y": [
          25996,
          26456,
          21850,
          14579,
          16740
         ],
         "yaxis": "y"
        },
        {
         "hovertemplate": "variable=predicted_values<br>time=%{x|%B %d, %Y}<br>value=%{y}<extra></extra>",
         "legendgroup": "predicted_values",
         "line": {
          "color": "#EF553B",
          "dash": "solid"
         },
         "marker": {
          "symbol": "circle"
         },
         "mode": "lines",
         "name": "predicted_values",
         "orientation": "v",
         "showlegend": true,
         "type": "scatter",
         "x": [
          "2021-03-19",
          "2021-03-20",
          "2021-03-21",
          "2021-03-22",
          "2021-03-23"
         ],
         "xaxis": "x",
         "y": [
          24163.672718013193,
          21826.1187622461,
          18993.628424420873,
          15513.583285188495,
          17177.66852610146
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "legend": {
         "title": {
          "text": "variable"
         },
         "tracegroupgap": 0
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Holdout data"
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "dtick": "M1",
         "tickformat": "%b\n%Y",
         "title": {
          "text": "time"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "value"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from datetime import datetime, timedelta\n",
    "import numpy as np\n",
    "\n",
    "visualization = pipeline_details['visualization']['backtest']['iterations'][-1]\n",
    "holdout_dates = visualization['time']\n",
    "holdout_predictions = visualization['predicted_values'][0]\n",
    "observed_values = visualization['observed_values'][0]\n",
    "holdout_df = pd.DataFrame({'time':holdout_dates, 'observed_values':observed_values, 'predicted_values': holdout_predictions})\n",
    "fig = px.line(holdout_df, x=\"time\", y=holdout_df.columns, hover_data={\"time\": \"|%B %d, %Y\"}, title='Holdout data')\n",
    "fig.update_xaxes(dtick=\"M1\", tickformat=\"%b\\n%Y\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"scoring\"></a>\n",
    "## 6. Deploy and Score\n",
    "\n",
    "In this section you will learn how to deploy and score pipeline model as online deployment using WML instance.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Online deployment creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing an AutoAI Deployment...\n",
      "Published model uid: 6c7b68b2-87e9-48c8-a2fd-e564568f428d\n",
      "Deploying model 6c7b68b2-87e9-48c8-a2fd-e564568f428d using V4 client.\n",
      "\n",
      "\n",
      "######################################################################################\n",
      "\n",
      "Synchronous deployment creation for id: '6c7b68b2-87e9-48c8-a2fd-e564568f428d' started\n",
      "\n",
      "######################################################################################\n",
      "\n",
      "\n",
      "initializing\n",
      "Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.\n",
      "......\n",
      "ready\n",
      "\n",
      "\n",
      "-----------------------------------------------------------------------------------------------\n",
      "Successfully finished deployment creation, deployment_id='d0fbcd82-d125-4961-9cd9-33d9823423b0'\n",
      "-----------------------------------------------------------------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from ibm_watsonx_ai.deployment import WebService\n",
    "\n",
    "service = WebService(credentials, source_space_id=space_id)\n",
    "\n",
    "service.create(\n",
    "    experiment_run_id=pipeline_optimizer.get_run_details()['metadata']['id'],\n",
    "    model=best_pipeline_name, \n",
    "    deployment_name=\"COVID19 for Poland\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "To show all available information about the deployment use the `.get_params()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "service.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scoring\n",
    "\n",
    "You can use the `score` method to get predictions for defined forecasting window. You can either send payload records or use empty list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predictions for next 7 days:\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'predictions': [{'fields': ['prediction'],\n",
       "   'values': [[[29333.44280453762]],\n",
       "    [[29957.455192354864]],\n",
       "    [[31766.405211835852]],\n",
       "    [[34318.30390937023]],\n",
       "    [[35107.35033785305]],\n",
       "    [[32564.84119370212]],\n",
       "    [[29853.500820049674]]]}]}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = service.score(payload=pd.DataFrame({'daily_cases' : []}))\n",
    "print('predictions for next 7 days:\\n')\n",
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or you could send payload with new obeservations:\n",
    "\n",
    "```\n",
    "predictions = service.score(payload=pd.DataFrame({'daily_cases' : df.iloc[-10:, 1].values.tolist()}))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualization of predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "hovertemplate": "variable=observed_values<br>time=%{x|%B %d, %Y}<br>value=%{y}<extra></extra>",
         "legendgroup": "observed_values",
         "line": {
          "color": "#636efa",
          "dash": "solid"
         },
         "marker": {
          "symbol": "circle"
         },
         "mode": "lines",
         "name": "observed_values",
         "orientation": "v",
         "showlegend": true,
         "type": "scatter",
         "x": [
          "2021-03-19",
          "2021-03-20",
          "2021-03-21",
          "2021-03-22",
          "2021-03-23",
          "2021-03-24",
          "2021-03-25",
          "2021-03-26",
          "2021-03-27",
          "2021-03-28",
          "2021-03-29",
          "2021-03-30"
         ],
         "xaxis": "x",
         "y": [
          25996,
          26456,
          21850,
          14579,
          16740,
          null,
          null,
          null,
          null,
          null,
          null,
          null
         ],
         "yaxis": "y"
        },
        {
         "hovertemplate": "variable=predicted_values<br>time=%{x|%B %d, %Y}<br>value=%{y}<extra></extra>",
         "legendgroup": "predicted_values",
         "line": {
          "color": "#EF553B",
          "dash": "solid"
         },
         "marker": {
          "symbol": "circle"
         },
         "mode": "lines",
         "name": "predicted_values",
         "orientation": "v",
         "showlegend": true,
         "type": "scatter",
         "x": [
          "2021-03-19",
          "2021-03-20",
          "2021-03-21",
          "2021-03-22",
          "2021-03-23",
          "2021-03-24",
          "2021-03-25",
          "2021-03-26",
          "2021-03-27",
          "2021-03-28",
          "2021-03-29",
          "2021-03-30"
         ],
         "xaxis": "x",
         "y": [
          24163.672718013193,
          21826.1187622461,
          18993.628424420873,
          15513.583285188495,
          17177.66852610146,
          29333.44280453762,
          29957.455192354864,
          31766.405211835852,
          34318.30390937023,
          35107.35033785305,
          32564.84119370212,
          29853.500820049674
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "legend": {
         "title": {
          "text": "variable"
         },
         "tracegroupgap": 0
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "title": {
         "text": "Forecast data"
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "dtick": "M1",
         "tickformat": "%b\n%Y",
         "title": {
          "text": "time"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "value"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "last_date = datetime.strptime(holdout_df.tail(1).time.values.tolist()[0],'%Y-%m-%d')\n",
    "prediction_dates = [(last_date + timedelta(days=1 + i)).date() for i in range(forecast_window)]\n",
    "prediction_values = [pred[0][0] for pred in  predictions['predictions'][0]['values']]\n",
    "pred_df = pd.DataFrame({'time' : holdout_dates + prediction_dates,\n",
    "                        'observed_values' : observed_values + [np.NAN for i in range(forecast_window)],\n",
    "                        'predicted_values' : holdout_predictions + prediction_values})\n",
    "\n",
    "fig = px.line(pred_df, x=\"time\", y=pred_df.columns, hover_data={\"time\": \"|%B %d, %Y\"}, title='Forecast data')\n",
    "fig.update_xaxes(dtick=\"M1\", tickformat=\"%b\\n%Y\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"cleanup\"></a>\n",
    "## 7. Clean up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to clean up all created assets:\n",
    "- experiments\n",
    "- trainings\n",
    "- pipelines\n",
    "- model definitions\n",
    "- models\n",
    "- functions\n",
    "- deployments\n",
    "\n",
    "please follow up this sample [notebook](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd5.0/notebooks/python_sdk/instance-management/Machine%20Learning%20artifacts%20management.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"summary\"></a>\n",
    "## 8. Summary and next steps\n",
    "\n",
    "You successfully completed this notebook!.\n",
    " \n",
    "You learned how to use `watsonx.ai` to run AutoAI experiments. \n",
    " \n",
    "Check out our _<a href=\"https://ibm.github.io/watsonx-ai-python-sdk/samples.html\" target=\"_blank\" rel=\"noopener no referrer\">Online Documentation</a>_ for more samples, tutorials, documentation, how-tos, and blog posts. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authors\n",
    "\n",
    "**Lukasz Cmielowski**, PhD, is an Automation Architect and Data Scientist at IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable knowledge.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2020-2025 IBM. This notebook and its source code are released under the terms of the MIT License."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "rt241_v2",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}