diabetes-classification-model-pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Creating an Azure Machine Learning Pipeline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "You can perform the various steps required to ingest data, train a model, and register the model individually by using the Azure ML SDK to run script-based experiments. However, in an enterprise environment it is common to encapsulate the sequence of discrete steps required to build a machine learning solution into a *pipeline* that can be run on one or more compute targets, either on-demand by a user, from an automated build process, or on a schedule.\n",
    "\n",
    "In this lab, you'll bring together all of these elements to create a simple pipeline that trains and registers a model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Connect to Your Workspace\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "The first thing you need to do is to connect to your workspace using the Azure ML SDK.\n",
    "\n",
    "> **Note**: If the authenticated session with your Azure subscription has expired since you completed the previous exercise, you'll be prompted to reauthenticate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ready to use Azure ML 1.40.0 to work with azureml-workspace\n"
     ]
    }
   ],
   "source": [
    "import azureml.core\n",
    "from azureml.core import Workspace\n",
    "\n",
    "# Load the workspace from the saved config file\n",
    "ws = Workspace.from_config()\n",
    "print('Ready to use Azure ML {} to work with {}'.format(azureml.core.VERSION, ws.name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ws.datasets\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Prepare the Training Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "You can use local data files to train a model, but when running training workloads automatically on cloud-based compute, it makes more sense to store the data centrally in the cloud and ingest it into the training script wherever it happens to be running.\n",
    "\n",
    "In this lab, you'll upload the training data to a *datastore* and define a *dataset* that can be used to access the data from a training script. For simplicity, you'll upload the data to the default datastore for your Azure Machine Learning workspace - this is an Azure Storage blob container that was created when you provisioned the workspace. In a real solution, you'd likely register a datastore that references the cloud location where you typically store your data. You'll then create a *tabular* dataset that references the CSV files you uploaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "workspaceblobstore\n",
      "Uploading an estimated of 2 files\n",
      "Uploading data/diabetes.csv\n",
      "Uploaded data/diabetes.csv, 1 files out of an estimated total of 2\n",
      "Uploading data/diabetes2.csv\n",
      "Uploaded data/diabetes2.csv, 2 files out of an estimated total of 2\n",
      "Uploaded 2 files\n",
      "Dataset registered.\n"
     ]
    }
   ],
   "source": [
    "from azureml.core import Dataset\n",
    "\n",
    "default_ds = ws.get_default_datastore()\n",
    "print(default_ds.name)\n",
    "\n",
    "if 'diabetes dataset' not in ws.datasets:\n",
    "    default_ds.upload_files(files=['data/diabetes.csv', 'data/diabetes2.csv'], # Upload the diabetes csv files in /data\n",
    "                        target_path='diabetes-data/', # Put it in a folder path in the datastore\n",
    "                        overwrite=True, # Replace existing files of the same name\n",
    "                        show_progress=True)\n",
    "\n",
    "    #Create a tabular dataset from the path on the datastore (this may take a short while)\n",
    "    tab_data_set = Dataset.Tabular.from_delimited_files(path=(default_ds, 'diabetes-data/*.csv'))\n",
    "\n",
    "    # Register the tabular dataset\n",
    "    try:\n",
    "        tab_data_set = tab_data_set.register(workspace=ws, \n",
    "                                name='diabetes dataset',\n",
    "                                description='diabetes data',\n",
    "                                tags = {'format':'CSV'},\n",
    "                                create_new_version=True)\n",
    "        print('Dataset registered.')\n",
    "    except Exception as ex:\n",
    "        print(ex)\n",
    "else:\n",
    "    print('Dataset already registered.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Create Scripts for Pipeline Steps\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Now you're ready to start work on your pipeline. Pipelines consist of one or more *steps*, which can be Python scripts, or specialized steps like an Auto ML training estimator or a data transfer step that copies data from one location to another. Each step can run in its own compute context.\n",
    "\n",
    "In this exercise, you'll build a simple pipeline that contains an estimator step (to train a model) and a Python script step (to register the trained model). Start by creating a folder to contain the scripts for each step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "diabetes_model_pipeline\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "# Create a folder for the pipeline step files\n",
    "experiment_folder = 'diabetes_model_pipeline'\n",
    "os.makedirs(experiment_folder, exist_ok=True)\n",
    "\n",
    "print(experiment_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "The first step will use an estimator to run a training script. The code in the following cell creates this script for you. Note that the script includes a parameter named **output_folder**, which references the folder where the trained model should be saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting diabetes_model_pipeline/train_diabetes.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $experiment_folder/train_diabetes.py\n",
    "# Import libraries\n",
    "import os\n",
    "from azureml.core import Run\n",
    "import argparse\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import joblib\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "# Get parameters\n",
    "parser = argparse.ArgumentParser()\n",
    "parser.add_argument('--output_folder', type=str, dest='output_folder', default=\"diabetes_model_pipeline\", help='output folder')\n",
    "args = parser.parse_args()\n",
    "output_folder = args.output_folder\n",
    "\n",
    "# Get the experiment run context\n",
    "run = Run.get_context()\n",
    "\n",
    "# load the diabetes data (passed as an input dataset)\n",
    "print(\"Loading Data...\")\n",
    "diabetes = run.input_datasets['diabetes_train'].to_pandas_dataframe()\n",
    "\n",
    "# Separate features and labels\n",
    "X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values\n",
    "\n",
    "# Split data into training set and test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)\n",
    "\n",
    "# Train adecision tree model\n",
    "print('Training a decision tree model')\n",
    "model = DecisionTreeClassifier().fit(X_train, y_train)\n",
    "\n",
    "# calculate accuracy\n",
    "y_hat = model.predict(X_test)\n",
    "acc = np.average(y_hat == y_test)\n",
    "print('Accuracy:', acc)\n",
    "run.log('Accuracy', np.float(acc))\n",
    "\n",
    "# calculate AUC\n",
    "y_scores = model.predict_proba(X_test)\n",
    "auc = roc_auc_score(y_test,y_scores[:,1])\n",
    "print('AUC: ' + str(auc))\n",
    "run.log('AUC', np.float(auc))\n",
    "\n",
    "# Save the trained model\n",
    "os.makedirs(output_folder, exist_ok=True)\n",
    "output_path = output_folder + \"/model.pkl\"\n",
    "joblib.dump(value=model, filename=output_path)\n",
    "\n",
    "run.complete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %%writefile $experiment_folder/train_diabetes.py\n",
    "# Import libraries\n",
    "import os\n",
    "from azureml.core import Run\n",
    "import argparse\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import joblib\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.metrics import roc_auc_score\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Get parameters\n",
    "parser = argparse.ArgumentParser()\n",
    "parser.add_argument('--output_folder', type=str, dest='output_folder', default=\"diabetes_model\", help='output folder')\n",
    "args, unknown = parser.parse_known_args()\n",
    "output_folder = args.output_folder\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Get the experiment run context\n",
    "run = Run.get_context()\n",
    "ws = Workspace.from_config() if not hasattr(\n",
    "    run, 'experiment') else run.experiment.workspace\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading Data...\n"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'_OfflineRun' object has no attribute 'input_datasets'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-12-e760be78c410>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mazureml\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcore\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrun\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0m_OfflineRun\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"Loading Data...\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m \u001b[0mdiabetes\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrun\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0minput_datasets\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'diabetes_train'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mto_pandas_dataframe\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m: '_OfflineRun' object has no attribute 'input_datasets'"
     ]
    }
   ],
   "source": [
    "# load the diabetes data (passed as an input dataset)\n",
    "\n",
    "print(\"Loading Data...\")\n",
    "diabetes = run.input_datasets['diabetes_train'].to_pandas_dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Separate features and labels\n",
    "X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Split data into training set and test set\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Train adecision tree model\n",
    "print('Training a decision tree model')\n",
    "model = DecisionTreeClassifier().fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# calculate accuracy\n",
    "y_hat = model.predict(X_test)\n",
    "acc = np.average(y_hat == y_test)\n",
    "print('Accuracy:', acc)\n",
    "run.log('Accuracy', np.float(acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# calculate AUC\n",
    "y_scores = model.predict_proba(X_test)\n",
    "auc = roc_auc_score(y_test,y_scores[:,1])\n",
    "print('AUC: ' + str(auc))\n",
    "run.log('AUC', np.float(auc))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Save the trained model\n",
    "os.makedirs(output_folder, exist_ok=True)\n",
    "output_path = output_folder + \"/model.pkl\"\n",
    "joblib.dump(value=model, filename=output_path)\n",
    "\n",
    "run.complete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "The script for the second step of the pipeline will load the model from where it was saved, and then register it in the workspace. It includes a single **model_folder** parameter that contains the path to the folder where the model was saved by the previous step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing diabetes_model_pipeline/register_diabetes.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $experiment_folder/register_diabetes.py\n",
    "# Import libraries\n",
    "import argparse\n",
    "import joblib\n",
    "from azureml.core import Workspace, Model, Run\n",
    "\n",
    "# Get parameters\n",
    "parser = argparse.ArgumentParser()\n",
    "parser.add_argument('--model_folder', type=str, dest='model_folder', default=\"diabetes_model\", help='model location')\n",
    "args = parser.parse_args()\n",
    "model_folder = args.model_folder\n",
    "\n",
    "# Get the experiment run context\n",
    "run = Run.get_context()\n",
    "\n",
    "# load the model\n",
    "print(\"Loading model from \" + model_folder)\n",
    "model_file = model_folder + \"/model.pkl\"\n",
    "model = joblib.load(model_file)\n",
    "\n",
    "Model.register(workspace=run.experiment.workspace,\n",
    "               model_path = model_file,\n",
    "               model_name = 'diabetes_model',\n",
    "               tags={'Training context':'Pipeline'})\n",
    "\n",
    "run.complete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Prepare a Compute Environment for the Pipeline Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "The pipeline will eventually be published and run on-demand, so it needs a compute environment in which to run. In this exercise, you'll use the same compute for both steps, but it's important to realize that each step is run independently; so you could specify different compute contexts for each step if appropriate.\n",
    "\n",
    "First, you need a compute target. In this case, you create an Azure Machine Learning compute cluster in your workspace (or use an existing one if you have created it previously).\n",
    "\n",
    "> **Important**: Change *your-compute-cluster* to the unique name for your compute cluster in the code below before running it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "InProgress.\n",
      "SucceededProvisioning operation finished, operation \"Succeeded\"\n",
      "Succeeded\n",
      "AmlCompute wait for completion finished\n",
      "\n",
      "Minimum number of nodes requested have been provisioned\n"
     ]
    }
   ],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "cluster_name = \"ovo-azureml-sdk\"\n",
    "\n",
    "# Verify that cluster exists\n",
    "try:\n",
    "    pipeline_cluster = ComputeTarget(workspace=ws, name=cluster_name)\n",
    "    print('Found existing cluster, use it.')\n",
    "except ComputeTargetException:\n",
    "    # If not, create it\n",
    "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',\n",
    "                                                           max_nodes=4,\n",
    "                                                           idle_seconds_before_scaledown=1800)\n",
    "    pipeline_cluster = ComputeTarget.create(ws, cluster_name, compute_config)\n",
    "\n",
    "pipeline_cluster.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "The compute will require a Python environment with the necessary package dependencies installed, so we'll create a run configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Run configuration created.\n"
     ]
    }
   ],
   "source": [
    "from azureml.core import Environment\n",
    "from azureml.core.conda_dependencies import CondaDependencies\n",
    "from azureml.core.runconfig import RunConfiguration, DockerConfiguration\n",
    "\n",
    "# Create a Python environment for the experiment\n",
    "diabetes_env = Environment.from_pip_requirements(\n",
    "    \"diabetes-pipeline-env\", \"requirements.txt\")\n",
    "diabetes_env.python.user_managed_dependencies = True # Let Azure ML manage dependencies\n",
    "# diabetes_env.docker.enabled = True # Use a docker container\n",
    "\n",
    "# Create a set of package dependencies\n",
    "# diabetes_packages = CondaDependencies.create(conda_packages=['scikit-learn'],\n",
    "                                            #  pip_packages=['azureml-sdk', 'pandas', 'numpy'])\n",
    "\n",
    "# Add the dependencies to the environment\n",
    "# diabetes_env.python.conda_dependencies = diabetes_packages\n",
    "\n",
    "# Register the environment (just in case you want to use it again)\n",
    "diabetes_env.register(workspace=ws)\n",
    "registered_env = Environment.get(ws, 'diabetes-pipeline-env')\n",
    "\n",
    "# Create a new runconfig object for the pipeline\n",
    "pipeline_run_config = RunConfiguration()\n",
    "docker_configuration = DockerConfiguration(use_docker=True)\n",
    "pipeline_run_config.docker = docker_configuration\n",
    "\n",
    "# Use the compute you created above. \n",
    "pipeline_run_config.target = pipeline_cluster\n",
    "\n",
    "# Assign the environment to the run configuration\n",
    "pipeline_run_config.environment = registered_env\n",
    "\n",
    "print (\"Run configuration created.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Create and Run a Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Now you're ready to define and run the pipeline.\n",
    "\n",
    "- First you need to define the steps for the pipeline, and any data references that need to passed between them. \n",
    "- In this case, the first step must write the model to a folder that can be read from by the second step. \n",
    "- Since the steps will be run on remote compute (and in fact, could each be run on different compute), the folder path must be passed as a data reference to a location in a datastore within the workspace. \n",
    "- The **PipelineData** object is a special kind of data reference that is used to pass data from the output of one pipeline step to the input of another, creating a dependency between them. \n",
    "- You'll create one and use it as the output for the first step and the input for the second step. \n",
    "- Note that you also need to pass it as a script argument so your code can access the datastore location referenced by the data reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline steps defined\n"
     ]
    }
   ],
   "source": [
    "from azureml.core import ScriptRunConfig\n",
    "from azureml.pipeline.core import PipelineData\n",
    "from azureml.pipeline.steps import PythonScriptStep, CommandStep\n",
    "from azureml.train.estimator import Estimator\n",
    "\n",
    "# Get the training dataset\n",
    "diabetes_ds = ws.datasets.get(\"diabetes dataset\")\n",
    "\n",
    "# Create a PipelineData (temporary Data Reference) for the model folder\n",
    "model_folder = PipelineData(\"model_folder\", datastore=ws.get_default_datastore())\n",
    "\n",
    "estimator = ScriptRunConfig(source_directory=experiment_folder,\n",
    "                        compute_target = pipeline_cluster,\n",
    "                        environment=pipeline_run_config.environment,\n",
    "                        script='train_diabetes.py')\n",
    "\n",
    "# Step 1, run the estimator to train the model\n",
    "train_step = CommandStep(name = \"Train Model\",\n",
    "                           source_directory=experiment_folder,\n",
    "                           command=\"python train_diabetes.py --output_folder {output_folder}\",\n",
    "                           inputs=[diabetes_ds.as_named_input('diabetes_train')],\n",
    "                           outputs=[model_folder],\n",
    "                           compute_target = pipeline_cluster,\n",
    "                           allow_reuse = True)\n",
    "\n",
    "# Step 2, run the model registration script\n",
    "register_step = PythonScriptStep(name = \"Register Model\",\n",
    "                                source_directory = experiment_folder,\n",
    "                                script_name = \"register_diabetes.py\",\n",
    "                                arguments = ['--model_folder', model_folder],\n",
    "                                inputs=[model_folder],\n",
    "                                compute_target = pipeline_cluster,\n",
    "                                runconfig = pipeline_run_config,\n",
    "                                allow_reuse = True)\n",
    "\n",
    "print(\"Pipeline steps defined\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "OK, now you're ready to build the pipeline from the steps you've defined and run it as an experiment.\n",
    "\n",
    "> **Note**: This may take a while. The training cluster must be started and configured with the Python environment before the scripts can be run. Now might be a good time to take a coffee break!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline is built.\n",
      "Created step Train Model [b65dbdde][fbacaf40-1a37-4cb1-8dff-4013f24579fd], (This step will run and generate new outputs)\n",
      "Created step Register Model [24243692][37c434a0-3c9c-462f-b463-970b14de9ea3], (This step will run and generate new outputs)\n",
      "Submitted PipelineRun caf1fcfb-fd01-451e-adf5-0e786795a77a\n",
      "Link to Azure Machine Learning Portal: https://ml.azure.com/runs/caf1fcfb-fd01-451e-adf5-0e786795a77a?wsid=/subscriptions/8ef9d38c-cd92-4cea-b46c-0527c93d4acb/resourcegroups/azure-ml-sdk-project/workspaces/azureml-workspace&tid=babcc559-b87a-427b-9175-f021a70345c0\n",
      "Pipeline submitted for execution.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a0327f022fa548509088b6119ef4656d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_PipelineWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Failed\", \"workbench_run_details_uri\": \"https://ml.azure.com/runs/caf1fcfb-fd01-451e-adf5-0e786795a77a?wsid=/subscriptions/8ef9d38c-cd92-4cea-b46c-0527c93d4acb/resourcegroups/azure-ml-sdk-project/workspaces/azureml-workspace&tid=babcc559-b87a-427b-9175-f021a70345c0\", \"run_id\": \"caf1fcfb-fd01-451e-adf5-0e786795a77a\", \"run_properties\": {\"run_id\": \"caf1fcfb-fd01-451e-adf5-0e786795a77a\", \"created_utc\": \"2022-04-09T18:00:00.570259Z\", \"properties\": {\"azureml.runsource\": \"azureml.PipelineRun\", \"runSource\": \"SDK\", \"runType\": \"SDK\", \"azureml.parameters\": \"{}\", \"azureml.continue_on_step_failure\": \"False\", \"azureml.pipelineComponent\": \"pipelinerun\"}, \"tags\": {}, \"end_time_utc\": \"2022-04-09T18:00:39.185415Z\", \"status\": \"Failed\", \"log_files\": {\"logs/azureml/executionlogs.txt\": \"https://azuremlworkspa4632104135.blob.core.windows.net/azureml/ExperimentRun/dcid.caf1fcfb-fd01-451e-adf5-0e786795a77a/logs/azureml/executionlogs.txt?sv=2019-07-07&sr=b&sig=7fd7qMRxSXGaYSKqVkSAL6KE5GOvfSrIeZsiIMRkw%2FY%3D&skoid=f5abfce9-8646-42e6-99ea-e8297745b595&sktid=babcc559-b87a-427b-9175-f021a70345c0&skt=2022-04-09T17%3A26%3A08Z&ske=2022-04-11T01%3A36%3A08Z&sks=b&skv=2019-07-07&st=2022-04-09T17%3A55%3A20Z&se=2022-04-10T02%3A05%3A20Z&sp=r\", \"logs/azureml/stderrlogs.txt\": \"https://azuremlworkspa4632104135.blob.core.windows.net/azureml/ExperimentRun/dcid.caf1fcfb-fd01-451e-adf5-0e786795a77a/logs/azureml/stderrlogs.txt?sv=2019-07-07&sr=b&sig=WMyZSNfk58ZbkrvMnJZUmbfffl5XicuKiM0hNESonw0%3D&skoid=f5abfce9-8646-42e6-99ea-e8297745b595&sktid=babcc559-b87a-427b-9175-f021a70345c0&skt=2022-04-09T17%3A26%3A08Z&ske=2022-04-11T01%3A36%3A08Z&sks=b&skv=2019-07-07&st=2022-04-09T17%3A55%3A20Z&se=2022-04-10T02%3A05%3A20Z&sp=r\", \"logs/azureml/stdoutlogs.txt\": \"https://azuremlworkspa4632104135.blob.core.windows.net/azureml/ExperimentRun/dcid.caf1fcfb-fd01-451e-adf5-0e786795a77a/logs/azureml/stdoutlogs.txt?sv=2019-07-07&sr=b&sig=L4U1wJBBBjAHPcCHCJCoNAKnM8wcRwH8YxeRbC5rdNg%3D&skoid=f5abfce9-8646-42e6-99ea-e8297745b595&sktid=babcc559-b87a-427b-9175-f021a70345c0&skt=2022-04-09T17%3A26%3A08Z&ske=2022-04-11T01%3A36%3A08Z&sks=b&skv=2019-07-07&st=2022-04-09T17%3A55%3A20Z&se=2022-04-10T02%3A05%3A20Z&sp=r\"}, \"log_groups\": [[\"logs/azureml/executionlogs.txt\", \"logs/azureml/stderrlogs.txt\", \"logs/azureml/stdoutlogs.txt\"]], \"run_duration\": \"0:00:38\", \"run_number\": \"1649527200\", \"run_queued_details\": {\"status\": \"Failed\", \"details\": null}}, \"child_runs\": [{\"run_id\": \"\", \"name\": \"Train Model\", \"status\": \"NotStarted\", \"start_time\": \"\", \"created_time\": \"\", \"end_time\": \"\", \"duration\": \"\"}, {\"run_id\": \"\", \"name\": \"Register Model\", \"status\": \"NotStarted\", \"start_time\": \"\", \"created_time\": \"\", \"end_time\": \"\", \"duration\": \"\"}], \"children_metrics\": {\"categories\": null, \"series\": null, \"metricName\": null}, \"run_metrics\": [], \"run_logs\": \"[2022-04-09 18:00:05Z] Submitting 1 runs, first five are: b65dbdde:294c2c98-cbd4-4921-a7de-564f263f1b51\\n[2022-04-09 18:00:39Z] Execution of experiment failed, update experiment status and cancel running nodes.\\n\", \"graph\": {\"datasource_nodes\": {\"59a3574c\": {\"node_id\": \"59a3574c\", \"name\": \"diabetes dataset\"}}, \"module_nodes\": {\"b65dbdde\": {\"node_id\": \"b65dbdde\", \"name\": \"Train Model\", \"status\": \"NotStarted\"}, \"24243692\": {\"node_id\": \"24243692\", \"name\": \"Register Model\", \"status\": \"NotStarted\"}}, \"edges\": [{\"source_node_id\": \"59a3574c\", \"source_node_name\": \"diabetes dataset\", \"source_name\": \"data\", \"target_name\": \"diabetes_train\", \"dst_node_id\": \"b65dbdde\", \"dst_node_name\": \"Train Model\"}, {\"source_node_id\": \"b65dbdde\", \"source_node_name\": \"Train Model\", \"source_name\": \"model_folder\", \"target_name\": \"model_folder\", \"dst_node_id\": \"24243692\", \"dst_node_name\": \"Register Model\"}], \"child_runs\": [{\"run_id\": \"\", \"name\": \"Train Model\", \"status\": \"NotStarted\", \"start_time\": \"\", \"created_time\": \"\", \"end_time\": \"\", \"duration\": \"\"}, {\"run_id\": \"\", \"name\": \"Register Model\", \"status\": \"NotStarted\", \"start_time\": \"\", \"created_time\": \"\", \"end_time\": \"\", \"duration\": \"\"}]}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.40.0\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PipelineRunId: caf1fcfb-fd01-451e-adf5-0e786795a77a\n",
      "Link to Azure Machine Learning Portal: https://ml.azure.com/runs/caf1fcfb-fd01-451e-adf5-0e786795a77a?wsid=/subscriptions/8ef9d38c-cd92-4cea-b46c-0527c93d4acb/resourcegroups/azure-ml-sdk-project/workspaces/azureml-workspace&tid=babcc559-b87a-427b-9175-f021a70345c0\n",
      "PipelineRun Status: Running\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Expected a StepRun object but received <class 'azureml.core.run.Run'> instead.\n",
      "This usually indicates a package conflict with one of the dependencies of azureml-core or azureml-pipeline-core.\n",
      "Please check for package conflicts in your python environment\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "PipelineRun Execution Summary\n",
      "==============================\n",
      "PipelineRun Status: Failed\n",
      "{'runId': 'caf1fcfb-fd01-451e-adf5-0e786795a77a', 'status': 'Failed', 'startTimeUtc': '2022-04-09T18:00:04.68406Z', 'endTimeUtc': '2022-04-09T18:00:39.185415Z', 'services': {}, 'properties': {'azureml.runsource': 'azureml.PipelineRun', 'runSource': 'SDK', 'runType': 'SDK', 'azureml.parameters': '{}', 'azureml.continue_on_step_failure': 'False', 'azureml.pipelineComponent': 'pipelinerun'}, 'inputDatasets': [], 'outputDatasets': [], 'logFiles': {'logs/azureml/executionlogs.txt': 'https://azuremlworkspa4632104135.blob.core.windows.net/azureml/ExperimentRun/dcid.caf1fcfb-fd01-451e-adf5-0e786795a77a/logs/azureml/executionlogs.txt?sv=2019-07-07&sr=b&sig=Smp7fyjrzE3jZtR%2FZZUV83t2w6l3USjQs%2B%2BYSXwhj%2Bs%3D&skoid=f5abfce9-8646-42e6-99ea-e8297745b595&sktid=babcc559-b87a-427b-9175-f021a70345c0&skt=2022-04-09T17%3A26%3A08Z&ske=2022-04-11T01%3A36%3A08Z&sks=b&skv=2019-07-07&st=2022-04-09T17%3A50%3A08Z&se=2022-04-10T02%3A00%3A08Z&sp=r', 'logs/azureml/stderrlogs.txt': 'https://azuremlworkspa4632104135.blob.core.windows.net/azureml/ExperimentRun/dcid.caf1fcfb-fd01-451e-adf5-0e786795a77a/logs/azureml/stderrlogs.txt?sv=2019-07-07&sr=b&sig=HCNBv9a7tiBMBVL70wU5%2F0WSqyNwMXFsMOiaWRXOQpA%3D&skoid=f5abfce9-8646-42e6-99ea-e8297745b595&sktid=babcc559-b87a-427b-9175-f021a70345c0&skt=2022-04-09T17%3A26%3A08Z&ske=2022-04-11T01%3A36%3A08Z&sks=b&skv=2019-07-07&st=2022-04-09T17%3A50%3A08Z&se=2022-04-10T02%3A00%3A08Z&sp=r', 'logs/azureml/stdoutlogs.txt': 'https://azuremlworkspa4632104135.blob.core.windows.net/azureml/ExperimentRun/dcid.caf1fcfb-fd01-451e-adf5-0e786795a77a/logs/azureml/stdoutlogs.txt?sv=2019-07-07&sr=b&sig=W9jLptAz0jdMMO9tTkZeVihVB37e3y5448TxDe6P25Q%3D&skoid=f5abfce9-8646-42e6-99ea-e8297745b595&sktid=babcc559-b87a-427b-9175-f021a70345c0&skt=2022-04-09T17%3A26%3A08Z&ske=2022-04-11T01%3A36%3A08Z&sks=b&skv=2019-07-07&st=2022-04-09T17%3A50%3A08Z&se=2022-04-10T02%3A00%3A08Z&sp=r'}, 'submittedBy': 'Ovo Okpubuluku'}\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Failed'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from azureml.core import Experiment\n",
    "from azureml.pipeline.core import Pipeline\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "# Construct the pipeline\n",
    "pipeline_steps = [train_step, register_step]\n",
    "pipeline = Pipeline(workspace = ws, steps=pipeline_steps)\n",
    "print(\"Pipeline is built.\")\n",
    "\n",
    "# Create an experiment and run the pipeline\n",
    "experiment = Experiment(workspace = ws, name = 'diabetes-training-pipeline')\n",
    "pipeline_run = experiment.submit(pipeline, regenerate_outputs=True)\n",
    "print(\"Pipeline submitted for execution.\")\n",
    "\n",
    "RunDetails(pipeline_run).show()\n",
    "pipeline_run.wait_for_completion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "The widget above shows details of the pipeline as it runs. You can also monitor pipeline runs in the **Experiments** page in [Azure Machine Learning studio](https://ml.azure.com).\n",
    "\n",
    "When the pipeline has finished, a new model should be registered with a *Training context* tag indicating it was trained in a pipeline. Run the following code to verify this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from azureml.core import Model\n",
    "\n",
    "for model in Model.list(ws):\n",
    "    print(model.name, 'version:', model.version)\n",
    "    for tag_name in model.tags:\n",
    "        tag = model.tags[tag_name]\n",
    "        print ('\\t',tag_name, ':', tag)\n",
    "    for prop_name in model.properties:\n",
    "        prop = model.properties[prop_name]\n",
    "        print ('\\t',prop_name, ':', prop)\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Publish the Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Now that you've created a pipeline and verified it works, you can publish it as a REST service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "published_pipeline = pipeline.publish(name=\"Diabetes_Training_Pipeline\",\n",
    "                                      description=\"Trains diabetes model\",\n",
    "                                      version=\"1.0\")\n",
    "rest_endpoint = published_pipeline.endpoint\n",
    "print(rest_endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "To use the endpoint, client applications need to make a REST call over HTTP. This request must be authenticated, so an authorization header is required. A real application would require a service principal with which to be authenticated, but to test this out, we'll use the authorization header from your current connection to your Azure workspace, which you can get using the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from azureml.core.authentication import InteractiveLoginAuthentication\n",
    "\n",
    "interactive_auth = InteractiveLoginAuthentication()\n",
    "auth_header = interactive_auth.get_authentication_header()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Now you're ready to call the REST interface. The pipeline runs asynchronously, so you'll get an identifier back, which you can use to track the pipeline experiment as it runs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "experiment_name = 'Run-diabetes-pipeline'\n",
    "\n",
    "response = requests.post(rest_endpoint, \n",
    "                         headers=auth_header, \n",
    "                         json={\"ExperimentName\": experiment_name})\n",
    "run_id = response.json()[\"Id\"]\n",
    "run_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Since you have the run ID, you can use the **RunDetails** widget to view the experiment as it runs.\n",
    "\n",
    "> **Note**: The pipeline should complete quickly, because each step was configured to allow output reuse. This was done primarily for convenience and to save time in this example. In reality, you'd likely want the first step to run every time in case the data has changed, and trigger the subsequent steps only if the output from step one changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from azureml.pipeline.core.run import PipelineRun\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "published_pipeline_run = PipelineRun(ws.experiments[experiment_name], run_id)\n",
    "RunDetails(published_pipeline_run).show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}