sdk/python/responsible-ai/vision/responsibleaidashboard-automl-object-detection-fridge-private-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "# Object Detection scenario with RAI Dashboard\n",
    "\n",
    "##### This notebook demonstrates how to\n",
    "\n",
    "##### 1. Upload a private mltable dataset to AzureML\n",
    "##### 2. Fine-tune an **AutoML Object Detection model (fasterrcnn_resnet50_fpn)** on the OD Fridge Dataset using an **individual run of an AzureML training job** and then\n",
    "##### 3. Evaluate the AutoML model on the dataset using the RAI Vision Dashboard created using another AzureML pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "##### Depending on your current Python environment, you may need to install the following packages\n",
    "\n",
    "```\n",
    "pip install azure-identity\n",
    "pip install azure-ai-ml\n",
    "pip install mlflow\n",
    "pip install azureml-mlflow\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "gather": {
     "logged": 1686235424383
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# !pip install -U azure-identity\n",
    "# !pip install -U azure-ai-ml\n",
    "# !pip install -U mlflow\n",
    "# !pip install -U azureml-mlflow\n",
    "# !pip install -U mltable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "###### First, we need to specify the version of the RAI components which are available in the workspace. This was specified when the components were uploaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "gather": {
     "logged": 1686846410392
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "version_string = \"0.0.17\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "###### We also need to give the name of the compute cluster we want to use in AzureML. Later in this notebook, we will create it if it does not already exist:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "gather": {
     "logged": 1686846412047
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# Compute cluster to run the AutoML training job\n",
    "train_compute_name = \"gpu-cluster-nc6-v3\"\n",
    "\n",
    "# Compute cluster to visualize and interact with the RAI Dashboard\n",
    "rai_compute_name = \"cpucluster\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "# 1. Connect to Azure Machine Learning Workspace\n",
    "###### The [workspace](https://learn.microsoft.com/en-us/azure/machine-learning/concept-workspace?view=azureml-api-2) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## 1.1. Import the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "gather": {
     "logged": 1686846457572
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# Import required libraries\n",
    "from azure.identity import DefaultAzureCredential\n",
    "from azure.ai.ml import MLClient\n",
    "\n",
    "from azure.ai.ml.automl import SearchSpace, ObjectDetectionPrimaryMetrics\n",
    "from azure.ai.ml import automl, dsl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enter details of your AML workspace\n",
    "subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "resource_group = \"<RESOURCE_GROUP>\"\n",
    "workspace = \"<AML_WORKSPACE_NAME>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## 1.2. Configure workspace details and get a handle to the workspace\n",
    "\n",
    "###### To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the default [default azure authentication](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. Check the [configuration notebook](https://github.com/Azure/azureml-examples/blob/d8070da9dec29978e9322e727d1740f5486aa56e/sdk/python/jobs/configuration.ipynb) for more details on how to configure credentials and connect to a workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "gather": {
     "logged": 1686846461097
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# Handle to the workspace\n",
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "try:\n",
    "    credential = DefaultAzureCredential()\n",
    "    ml_client = MLClient(\n",
    "        credential=credential,\n",
    "        subscription_id=subscription_id,\n",
    "        resource_group_name=resource_group,\n",
    "        workspace_name=workspace,\n",
    "    )\n",
    "except Exception:\n",
    "    # If in compute instance we can get the config automatically\n",
    "    from azureml.core import Workspace\n",
    "\n",
    "    workspace = Workspace.from_config()\n",
    "    workspace.write_config()\n",
    "    ml_client = MLClient.from_config(\n",
    "        credential=DefaultAzureCredential(exclude_shared_token_cache_credential=True),\n",
    "        logging_enable=True,\n",
    "    )\n",
    "\n",
    "print(ml_client)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "# 2. Compute target setup\n",
    "\n",
    "###### You will need to provide a [Compute Target](https://docs.microsoft.com/en-us/azure/machine-learning/concept-azure-machine-learning-architecture#computes) that will be used for your AutoML model training. AutoML models for image tasks require [GPU SKUs](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-gpu) such as the ones from the NC, NCv2, NCv3, ND, NDv2 and NCasT4 series. We recommend using the NCsv3-series (with v100 GPUs) for faster training. Using a compute target with a multi-GPU VM SKU will leverage the multiple GPUs to speed up training. Additionally, setting up a compute target with multiple nodes will allow for faster model training by leveraging parallelism, when tuning hyperparameters for your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "gather": {
     "logged": 1686846463639
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import AmlCompute\n",
    "from azure.core.exceptions import ResourceNotFoundError\n",
    "\n",
    "try:\n",
    "    _ = ml_client.compute.get(train_compute_name)\n",
    "    print(\"Found existing compute target.\")\n",
    "except ResourceNotFoundError:\n",
    "    print(\"Creating a new compute target...\")\n",
    "    compute_config = AmlCompute(\n",
    "        name=train_compute_name,\n",
    "        type=\"amlcompute\",\n",
    "        size=\"Standard_NC6s_v3\",\n",
    "        idle_time_before_scale_down=120,\n",
    "        min_instances=0,\n",
    "        max_instances=4,\n",
    "    )\n",
    "    ml_client.begin_create_or_update(compute_config).result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "gather": {
     "logged": 1686846466881
    },
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import AmlCompute\n",
    "\n",
    "all_compute_names = [x.name for x in ml_client.compute.list()]\n",
    "\n",
    "if rai_compute_name in all_compute_names:\n",
    "    print(f\"Found existing compute: {rai_compute_name}\")\n",
    "else:\n",
    "    rai_compute_config = AmlCompute(\n",
    "        name=rai_compute_name,\n",
    "        size=\"STANDARD_DS3_V2\",\n",
    "        min_instances=0,\n",
    "        max_instances=4,\n",
    "        idle_time_before_scale_down=3600,\n",
    "    )\n",
    "    ml_client.compute.begin_create_or_update(rai_compute_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. MLTable with input Training Data\n",
    "\n",
    "###### In order to generate models for computer vision tasks with automated machine learning, you need to bring labeled image data as input for model training in the form of an MLTable. You can create an MLTable from labeled training data in JSONL format. If your labeled training data is in a different format (like, pascal VOC or COCO), you can use a conversion script to first convert it to JSONL, and then create an MLTable. Alternatively, you can use Azure Machine Learning's [data labeling tool](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-image-labeling-projects) to manually label images, and export the labeled data to use for training your AutoML model.\n",
    "\n",
    "###### In this notebook, we use a toy dataset called Fridge Objects, which consists of 128 images of 4 labels of beverage container {`can`, `carton`, `milk bottle`, `water bottle`} photos taken on different backgrounds.\n",
    "\n",
    "###### All images in this notebook are hosted in [this repository](https://github.com/microsoft/computervision-recipes) and are made available under the [MIT license](https://github.com/microsoft/computervision-recipes/blob/master/LICENSE)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1. Download the Data\n",
    "###### We first download and unzip the data locally. By default, the data would be downloaded in `./data` folder in current directory. \n",
    "###### If you prefer to download the data at a different location, update it in `dataset_parent_dir = ...` in the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import urllib\n",
    "from zipfile import ZipFile\n",
    "\n",
    "# Change to a different location if you prefer\n",
    "dataset_parent_dir = \"./data\"\n",
    "\n",
    "# create data folder if it doesnt exist.\n",
    "os.makedirs(dataset_parent_dir, exist_ok=True)\n",
    "\n",
    "# download data\n",
    "download_url = \"https://publictestdatasets.blob.core.windows.net/computervision/odFridgeObjects.zip\"\n",
    "\n",
    "# Extract current dataset name from dataset url\n",
    "dataset_name = os.path.split(download_url)[-1].split(\".\")[0]\n",
    "# Get dataset path for later use\n",
    "dataset_dir = os.path.join(dataset_parent_dir, dataset_name)\n",
    "\n",
    "# Get the data zip file path\n",
    "data_file = os.path.join(dataset_parent_dir, f\"{dataset_name}.zip\")\n",
    "\n",
    "# Download the dataset\n",
    "urllib.request.urlretrieve(download_url, filename=data_file)\n",
    "\n",
    "# extract files\n",
    "with ZipFile(data_file, \"r\") as zip:\n",
    "    print(\"extracting files...\")\n",
    "    zip.extractall(path=dataset_parent_dir)\n",
    "    print(\"done\")\n",
    "# delete zip file\n",
    "os.remove(data_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a sample image from this dataset:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "\n",
    "sample_image = os.path.join(dataset_dir, \"images\", \"31.jpg\")\n",
    "Image(filename=sample_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2. Upload the images to Datastore through an AML Data asset (URI Folder)\n",
    "\n",
    "###### In order to use the data for training in Azure ML, we upload it to our default Azure Blob Storage of our  Azure ML Workspace.\n",
    "\n",
    "###### [Check this notebook for AML data asset example](https://github.com/Azure/azureml-examples/blob/d8070da9dec29978e9322e727d1740f5486aa56e/sdk/python/assets/data/data.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uploading image files by creating a 'data asset URI FOLDER':\n",
    "\n",
    "from azure.ai.ml.entities import Data\n",
    "from azure.ai.ml.constants import AssetTypes, InputOutputModes\n",
    "from azure.ai.ml import Input\n",
    "\n",
    "my_data = Data(\n",
    "    path=dataset_dir,\n",
    "    type=AssetTypes.URI_FOLDER,\n",
    "    description=\"Fridge-items images Object detection\",\n",
    "    name=\"fridge-items-images-object-detection\",\n",
    ")\n",
    "\n",
    "uri_folder_data_asset = ml_client.data.create_or_update(my_data)\n",
    "\n",
    "print(uri_folder_data_asset)\n",
    "print(\"\")\n",
    "print(\"Path to folder in Blob Storage:\")\n",
    "print(uri_folder_data_asset.path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3. Convert the downloaded data to JSONL\n",
    "\n",
    "###### In this example, the fridge object dataset is annotated in Pascal VOC format, where each image corresponds to an xml file. Each xml file contains information on where its corresponding image file is located and also contains information about the bounding boxes and the object labels. \n",
    "\n",
    "###### For documentation on preparing the datasets beyond this notebook, please refer to the [documentation on how to prepare datasets](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-prepare-datasets-for-automl-images).\n",
    "\n",
    "\n",
    "###### In order to use this data to create an AzureML MLTable, we first need to convert it to the required JSONL format. The following script is creating two `.jsonl` files (one for training and one for validation) in the corresponding MLTable folder. The train / validation ratio corresponds to 20% of the data going into the validation file. For further details on jsonl file used for image classification task in automated ml, please refer to the [data schema documentation for image object-detection task](https://learn.microsoft.com/en-us/azure/machine-learning/reference-automl-images-schema#object-detection)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First generate JSONL files\n",
    "\n",
    "###### The JSONL Conversion helpers require pycocotools and simplification packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install pycocotools==2.0.6\n",
    "%pip install simplification==0.6.11\n",
    "%pip install scikit-image==0.19.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "# use the jsonl-conversion files from automl examples folder\n",
    "sys.path.insert(0, \"../../jobs/automl-standalone-jobs/jsonl-conversion/\")\n",
    "from base_jsonl_converter import write_json_lines\n",
    "from voc_jsonl_converter import VOCJSONLConverter\n",
    "\n",
    "base_url = os.path.join(uri_folder_data_asset.path, \"images/\")\n",
    "converter = VOCJSONLConverter(base_url, os.path.join(dataset_dir, \"annotations\"))\n",
    "jsonl_annotations = os.path.join(dataset_dir, \"annotations_voc.jsonl\")\n",
    "write_json_lines(converter, jsonl_annotations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Then split into train and validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# We'll copy each JSONL file within its related MLTable folder\n",
    "training_mltable_path = os.path.join(dataset_parent_dir, \"training-mltable-folder\")\n",
    "validation_mltable_path = os.path.join(dataset_parent_dir, \"validation-mltable-folder\")\n",
    "\n",
    "# First, let's create the folders if they don't exist\n",
    "os.makedirs(training_mltable_path, exist_ok=True)\n",
    "os.makedirs(validation_mltable_path, exist_ok=True)\n",
    "\n",
    "train_validation_ratio = 5\n",
    "\n",
    "# Path to the training and validation files\n",
    "train_annotations_file = os.path.join(training_mltable_path, \"train_annotations.jsonl\")\n",
    "validation_annotations_file = os.path.join(\n",
    "    validation_mltable_path, \"validation_annotations.jsonl\"\n",
    ")\n",
    "\n",
    "with open(jsonl_annotations, \"r\") as annot_f:\n",
    "    json_lines = annot_f.readlines()\n",
    "\n",
    "index = 0\n",
    "with open(train_annotations_file, \"w\") as train_f:\n",
    "    with open(validation_annotations_file, \"w\") as validation_f:\n",
    "        for json_line in json_lines:\n",
    "            if index % train_validation_ratio == 0:\n",
    "                # validation annotation\n",
    "                validation_f.write(json_line)\n",
    "            else:\n",
    "                # train annotation\n",
    "                train_f.write(json_line)\n",
    "            index += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4. Convert annotation file from COCO to JSONL\n",
    "###### If you want to try with a dataset in COCO format, see the example shown in Section 2 (specifically 2.4) of this [AutoML training notebook](https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/automl-standalone-jobs/automl-image-object-detection-task-fridge-items/automl-image-object-detection-task-fridge-items.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize bounding boxes\n",
    "###### Please refer to the \"Visualize data\" section in the following [tutorial](https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-auto-train-image-models#visualize-data) to see how to easily visualize your ground truth bounding boxes before starting to train."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5. Create MLTable data input\n",
    "###### Create MLTable data input using the jsonl files created above.\n",
    "\n",
    "###### For documentation on creating your own MLTable assets for jobs beyond this notebook, please refer to below resources\n",
    "###### - [MLTable YAML Schema](https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-mltable) - covers how to write MLTable YAML, which is required for each MLTable asset.\n",
    "###### - [Create MLTable data asset](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-data-assets?tabs=Python-SDK#create-a-mltable-data-asset) - covers how to create MLTable data asset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_ml_table_file(filename):\n",
    "    \"\"\"Create ML Table definition\"\"\"\n",
    "\n",
    "    return (\n",
    "        \"paths:\\n\"\n",
    "        \"  - file: ./{0}\\n\"\n",
    "        \"transformations:\\n\"\n",
    "        \"  - read_json_lines:\\n\"\n",
    "        \"        encoding: utf8\\n\"\n",
    "        \"        invalid_lines: error\\n\"\n",
    "        \"        include_path_column: false\\n\"\n",
    "        \"  - convert_column_types:\\n\"\n",
    "        \"      - columns: image_url\\n\"\n",
    "        \"        column_type: stream_info\"\n",
    "    ).format(filename)\n",
    "\n",
    "\n",
    "def save_ml_table_file(output_path, mltable_file_contents):\n",
    "    with open(os.path.join(output_path, \"MLTable\"), \"w\") as f:\n",
    "        f.write(mltable_file_contents)\n",
    "\n",
    "\n",
    "# Create and save train mltable\n",
    "train_mltable_file_contents = create_ml_table_file(\n",
    "    os.path.basename(train_annotations_file)\n",
    ")\n",
    "save_ml_table_file(training_mltable_path, train_mltable_file_contents)\n",
    "\n",
    "# Save train and validation mltable\n",
    "validation_mltable_file_contents = create_ml_table_file(\n",
    "    os.path.basename(validation_annotations_file)\n",
    ")\n",
    "save_ml_table_file(validation_mltable_path, validation_mltable_file_contents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training MLTable defined locally, with local data to be uploaded\n",
    "my_training_data_input = Input(type=AssetTypes.MLTABLE, path=training_mltable_path)\n",
    "\n",
    "# Validation MLTable defined locally, with local data to be uploaded\n",
    "my_validation_data_input = Input(type=AssetTypes.MLTABLE, path=validation_mltable_path)\n",
    "\n",
    "# WITH REMOTE PATH: If available already in the cloud/workspace-blob-store\n",
    "# my_training_data_input = Input(type=AssetTypes.MLTABLE, path=\"azureml://datastores/workspaceblobstore/paths/vision-classification/train\")\n",
    "# my_validation_data_input = Input(type=AssetTypes.MLTABLE, path=\"azureml://datastores/workspaceblobstore/paths/vision-classification/valid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Specify the name of the label/target column containing the classes and bbox information. (NOTE: This variable is used later in the RAI Dashboard section of the notebook as well.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_column_name = \"label\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Configure and run the AutoML for Images Object Detection training job\n",
    "\n",
    "###### AutoML allows you to easily train models for Image Classification, Object Detection & Instance Segmentation on your image data. You can control the model algorithm and hyperparameters to be used, perform a sweep over a manually specified hyperparameter space, or the system can automatically perform a hyperparameter sweep for you.\n",
    "\n",
    "###### To explore different training options with AutoML, have a look at Section of 4 of this [example notebook that demonstrates how to train AutoML models for Object Detection.](https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/automl-standalone-jobs/automl-image-object-detection-task-fridge-items/automl-image-object-detection-task-fridge-items.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Individual AutoML training run\n",
    "\n",
    "###### You can launch individual runs to explore model algorithms; we provide sensible default hyperparameters for each algorithm. You can also launch individual runs for the same model algorithm and different hyperparameter combinations. The model algorithm is specified using the model_name parameter. Please refer to the [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models?tabs=CLI-v2#configure-model-algorithms-and-hyperparameters) for the list of supported model algorithms.\n",
    "\n",
    "###### The following function can be used to configure AutoML jobs for individual runs:\n",
    "### set_training_parameters() function parameters:\n",
    "###### This is an optional configuration method to configure fixed settings or parameters that don't change during the parameter space sweep. Some of the key parameters of this function are:\n",
    "\n",
    "###### - `model_name` - The name of the ML algorithm that we want to use in training job. Please refer to this [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models?tabs=CLI-v2#supported-model-algorithms) for supported model algorithm.\n",
    "###### - `number_of_epochs` - The number of training epochs. It must be positive integer (default value is 15).\n",
    "###### - `layers_to_freeze` - The number of layers to freeze in model for transfer learning. It must be a positive integer (default value is 0).\n",
    "###### - `early_stopping` - It enable early stopping logic during training, It must be boolean value (default is True).   \n",
    "###### - `optimizer` - Type of optimizer to use in training. It must be either sgd, adam, adamw (default is sgd).\n",
    "###### - `distributed` - It enable distributed training if compute target contain multiple GPUs. It must be boolean value (default is True).\n",
    "\n",
    "###### If you wish to use the default hyperparameter values for a given algorithm (say `yolov5`), you can specify the job for your AutoML Image runs as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# general job parameters\n",
    "exp_name = \"dpv2-odfridge-automl-training\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the AutoML job with the related factory-function.\n",
    "image_object_detection_job = automl.image_object_detection(\n",
    "    compute=train_compute_name,\n",
    "    experiment_name=exp_name,\n",
    "    training_data=my_training_data_input,\n",
    "    validation_data=my_validation_data_input,\n",
    "    target_column_name=target_column_name,\n",
    "    primary_metric=ObjectDetectionPrimaryMetrics.MEAN_AVERAGE_PRECISION,\n",
    "    tags={\"data\": \"ODFridge\", \"model type\": \"AutoML\"},\n",
    ")\n",
    "\n",
    "# Set limits\n",
    "image_object_detection_job.set_limits(timeout_minutes=60)\n",
    "\n",
    "# Pass the fixed settings or parameters\n",
    "image_object_detection_job.set_training_parameters(\n",
    "    model_name=\"fasterrcnn_resnet50_fpn\",\n",
    "    early_stopping=1,\n",
    "    number_of_epochs=1,\n",
    "    learning_rate=0.09,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Submit the AutoML job\n",
    "returned_job = ml_client.jobs.create_or_update(image_object_detection_job)\n",
    "\n",
    "print(f\"Created job: {returned_job}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.jobs.stream(returned_job.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Retrieve the Best Trial (Best Model's trial/run) and Register best model\n",
    "###### Use the MLFLowClient to access the results (such as Models, Artifacts, Metrics) of a previously completed AutoML Trial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 Initialize MLFlow Client and Obtain the tracking URI for MLFlow\n",
    "\n",
    "###### The models and artifacts that are produced by AutoML can be accessed via the MLFlow interface.\n",
    "###### Initialize the MLFlow client here, and set the backend as Azure ML, via. the MLFlow Client.\n",
    "\n",
    "###### IMPORTANT, you need to have installed the latest MLFlow packages with:\n",
    "\n",
    "    pip install azureml-mlflow\n",
    "\n",
    "    pip install mlflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mlflow\n",
    "\n",
    "# Obtain the tracking URL from MLClient\n",
    "MLFLOW_TRACKING_URI = ml_client.workspaces.get(\n",
    "    name=ml_client.workspace_name\n",
    ").mlflow_tracking_uri\n",
    "\n",
    "print(MLFLOW_TRACKING_URI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the MLFLOW TRACKING URI\n",
    "mlflow.set_tracking_uri(MLFLOW_TRACKING_URI)\n",
    "print(f\"\\nCurrent tracking uri: {mlflow.get_tracking_uri()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlflow.tracking.client import MlflowClient\n",
    "\n",
    "# Initialize MLFlow client\n",
    "mlflow_client = MlflowClient()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Get the AutoML parent Job and Get the AutoML best child run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_name = returned_job.name\n",
    "\n",
    "# # Example if providing an specific Job name/ID\n",
    "# job_name = \"happy_yam_40fq53m7c2\" #\"ashy_net_gdd31zf2fq\"\n",
    "\n",
    "# Get the parent run\n",
    "mlflow_parent_run = mlflow_client.get_run(job_name)\n",
    "\n",
    "print(\"Parent Run: \")\n",
    "print(mlflow_parent_run)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print parent run tags. 'automl_best_child_run_id' tag should be there.\n",
    "print(mlflow_parent_run.data.tags.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the best model's child run\n",
    "\n",
    "best_child_run_id = mlflow_parent_run.data.tags[\"automl_best_child_run_id\"]\n",
    "print(f\"Found best child run id: {best_child_run_id}\")\n",
    "\n",
    "best_run = mlflow_client.get_run(best_child_run_id)\n",
    "\n",
    "print(\"Best child run: \")\n",
    "print(best_run)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Access the results (such as Models, Artifacts, Metrics) of a previously completed AutoML Run.\n",
    "pd.DataFrame(best_run.data.metrics, index=[0]).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3 Download the best model locally\n",
    "###### Access the results (such as Models, Artifacts, Metrics) of a previously completed AutoML Run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create local folder\n",
    "import os\n",
    "\n",
    "local_dir = \"./artifact_downloads\"\n",
    "if not os.path.exists(local_dir):\n",
    "    os.mkdir(local_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download run's artifacts/outputs\n",
    "local_path = mlflow_client.download_artifacts(\n",
    "    best_run.info.run_id, \"outputs\", local_dir\n",
    ")\n",
    "print(f\"Artifacts downloaded in: {local_path}\")\n",
    "print(f\"Artifacts: {os.listdir(local_path)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "mlflow_model_dir = os.path.join(local_dir, \"outputs\", \"mlflow-model\")\n",
    "\n",
    "# Show the contents of the MLFlow model folder\n",
    "os.listdir(mlflow_model_dir)\n",
    "\n",
    "# You should see a list of files such as the following:\n",
    "# ['artifacts', 'conda.yaml', 'MLmodel', 'python_env.yaml', 'python_model.pkl', 'requirements.txt']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.4 Register Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import required libraries\n",
    "from azure.ai.ml.entities import Model\n",
    "\n",
    "model_name = \"automl-fasterrcnn-odfridge-model\"\n",
    "\n",
    "model = Model(\n",
    "    path=f\"azureml://jobs/{best_run.info.run_id}/outputs/artifacts/outputs/mlflow-model/\",\n",
    "    name=model_name,\n",
    "    description=\"AutoML FasterRCNN model trained on OD Fridge Dataset\",\n",
    "    type=AssetTypes.MLFLOW_MODEL,\n",
    ")\n",
    "\n",
    "# for downloaded file\n",
    "# model = Model(\n",
    "#     path=mlflow_model_dir,\n",
    "#     name=model_name,\n",
    "#     description=\"AutoML FasterRCNN model trained on OD Fridge Dataset\",\n",
    "#     type=AssetTypes.MLFLOW_MODEL,\n",
    "# )\n",
    "\n",
    "registered_model = ml_client.models.create_or_update(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "registered_model.id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5.5 (Optional) If you already have a registered AutoML Object Detection model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model_name = \"automl-fasterrcnn-odfridge-model\"\n",
    "# version_number = 4\n",
    "\n",
    "# registered_model = ml_client.models.get(model_name, version=version_number)\n",
    "# print(registered_model.name, registered_model.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Specify the name of the label/target column containing the classes and bbox information. (NOTE: This variable is used later in the RAI Dashboard section of the notebook as well.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# target_column_name = \"label\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Creating the RAI Vision Insights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Now that we have our model, we can generate RAI Vision insights for it. We will need the `id` of the registered model, which will be as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expected_model_id = f\"{registered_model.name}:{registered_model.version}\"\n",
    "azureml_model_id = f\"azureml:{expected_model_id}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1 Prepare the data you want to analyze with the RAI Dashboard\n",
    "#### NOTE: This example notebook demonstrates how to use a private OD Fridge Dataset in AzureML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from urllib import urlretrieve\n",
    "except ImportError:\n",
    "    from urllib.request import urlretrieve\n",
    "\n",
    "# from enum import Enum\n",
    "import xml.etree.ElementTree as ET\n",
    "import pandas as pd\n",
    "from zipfile import ZipFile\n",
    "\n",
    "from azure.ai.ml import Input\n",
    "from azure.ai.ml.constants import AssetTypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fridge_test_mltable = Input(type=AssetTypes.MLTABLE, path=validation_mltable_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.3 Load the RAI Components\n",
    "###### Next, we load the RAI components, so that we can construct a pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "registry_name = \"azureml\"\n",
    "credential = DefaultAzureCredential()\n",
    "\n",
    "ml_client_registry = MLClient(\n",
    "    credential=credential,\n",
    "    subscription_id=ml_client.subscription_id,\n",
    "    resource_group_name=ml_client.resource_group_name,\n",
    "    registry_name=registry_name,\n",
    ")\n",
    "\n",
    "rai_vision_insights_component = ml_client_registry.components.get(\n",
    "    name=\"rai_vision_insights\", version=version_string\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.4 Constructing the pipeline in sdk\n",
    "###### We can now specify our pipeline. Complex objects (such as lists of column names) have to be converted to JSON strings before being passed to the components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from azure.ai.ml import Input\n",
    "from azure.ai.ml.constants import AssetTypes\n",
    "\n",
    "\n",
    "@dsl.pipeline(\n",
    "    compute=rai_compute_name,\n",
    "    description=\"Example RAI computation on Fridge data with AutoML FasterRCNN Object Detection model\",\n",
    "    experiment_name=f\"RAI_Fridge_Example_RAIInsights_Computation_{expected_model_id}\",\n",
    ")\n",
    "def rai_fridge_object_detection_pipeline(target_column_name, test_data, classes):\n",
    "    # Initiate the RAIInsights\n",
    "    rai_image_job = rai_vision_insights_component(\n",
    "        task_type=\"object_detection\",\n",
    "        model_info=expected_model_id,\n",
    "        model_input=Input(type=AssetTypes.MLFLOW_MODEL, path=azureml_model_id),\n",
    "        test_dataset=test_data,\n",
    "        target_column_name=target_column_name,\n",
    "        classes=classes,\n",
    "        model_type=\"pyfunc\",\n",
    "        precompute_explanation=True,\n",
    "        dataset_type=\"private\",\n",
    "        enable_error_analysis=False,\n",
    "        maximum_rows_for_test_dataset=5000,\n",
    "        num_masks=300,\n",
    "        mask_res=4,\n",
    "    )\n",
    "    rai_image_job.set_limits(timeout=24000)\n",
    "\n",
    "    rai_image_job.outputs.dashboard.mode = \"upload\"\n",
    "    rai_image_job.outputs.ux_json.mode = \"upload\"\n",
    "\n",
    "    return {\n",
    "        \"dashboard\": rai_image_job.outputs.dashboard,\n",
    "        \"ux_json\": rai_image_job.outputs.ux_json,\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Next, we define the pipeline object itself, and ensure that the outputs will be available for download:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "from azure.ai.ml import Output\n",
    "\n",
    "insights_pipeline_job = rai_fridge_object_detection_pipeline(\n",
    "    target_column_name=target_column_name,\n",
    "    test_data=fridge_test_mltable,\n",
    "    classes='[\"can\", \"carton\", \"milk_bottle\", \"water_bottle\"]',  # Ensure the class order matches that of used while creating the jsonl file for the test data\n",
    ")\n",
    "\n",
    "rand_path = str(uuid.uuid4())\n",
    "insights_pipeline_job.outputs.dashboard = Output(\n",
    "    path=f\"azureml://datastores/workspaceblobstore/paths/{rand_path}/dashboard/\",\n",
    "    mode=\"upload\",\n",
    "    type=\"uri_folder\",\n",
    ")\n",
    "insights_pipeline_job.outputs.ux_json = Output(\n",
    "    path=f\"azureml://datastores/workspaceblobstore/paths/{rand_path}/ux_json/\",\n",
    "    mode=\"upload\",\n",
    "    type=\"uri_folder\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### And submit the pipeline to AzureML for execution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Submit the RAI Vision Insights job\n",
    "returned_job = ml_client.jobs.create_or_update(insights_pipeline_job)\n",
    "\n",
    "print(f\"Created job: {returned_job}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.jobs.stream(returned_job.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### The dashboard should appear in the AzureML portal in the registered model view. The following cell computes the expected URI:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_id = ml_client._operation_scope.subscription_id\n",
    "rg_name = ml_client._operation_scope.resource_group_name\n",
    "ws_name = ml_client.workspace_name\n",
    "\n",
    "expected_uri = f\"https://ml.azure.com/model/{expected_model_id}/model_analysis?wsid=/subscriptions/{sub_id}/resourcegroups/{rg_name}/workspaces/{ws_name}\"\n",
    "\n",
    "print(f\"Please visit {expected_uri} to see your analysis\")"
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python310-sdkv2"
  },
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  },
  "microsoft": {
   "host": {
    "AzureML": {
     "notebookHasBeenCompleted": true
    }
   },
   "ms_spell_check": {
    "ms_spell_check_language": "en"
   }
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}