sdk/python/foundation-models/system/finetune/Llama-notebooks/multinode-text-classification/emotion-detection-llama-multinode-serverless.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Classification with distributed training (MultiNode) - Emotion Detection using Serverless Compute \n",
    "\n",
    "This sample shows how use `text-classification` components from the `azureml` system registry to fine tune a model to detect emotions using emotion dataset. [Serverless compute](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-serverless-compute?view=azureml-api-2&tabs=python) is used to fine-tune the model. **This notebook is dedicated to illustrate how to use multinode training with finetune components**. We then deploy the fine tuned model to an online endpoint for real time inference. The model is trained on tiny sample of the dataset with a small number of epochs to illustrate the fine tuning approach.\n",
    "\n",
    "### Training data\n",
    "We will use the [emotion](https://huggingface.co/datasets/dair-ai/emotion) dataset.\n",
    "\n",
    "### Model\n",
    "This notebook is curated for `Llama` models for text-classification, Llama models are picked from `azureml-meta` registry. In this notebook we finetune with Llama-2-7b model, if you want to finetune with other variants like 13b or 70b, you can use this notebook with probably different SKUs.\n",
    "\n",
    "### Outline\n",
    "* Pick a model to fine tune.\n",
    "* Pick and explore training data.\n",
    "* Configure the fine tuning job.\n",
    "* Run the fine tuning job.\n",
    "* Review training and evaluation metrics. \n",
    "* Register the fine tuned model. \n",
    "* Deploy the fine tuned model for real time inference.\n",
    "* Clean up resources. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Setup pre-requisites\n",
    "* Install dependencies\n",
    "* Connect to AzureML Workspace. Learn more at [set up SDK authentication](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-setup-authentication?tabs=sdk). Replace  `<WORKSPACE_NAME>`, `<RESOURCE_GROUP>` and `<SUBSCRIPTION_ID>` below.\n",
    "* Connect to `azureml` system registry\n",
    "* Set an optional experiment name\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install dependencies by running below cell. This is not an optional step if running in a new environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install azure-ai-ml\n",
    "%pip install azure-identity\n",
    "%pip install datasets==2.9.0\n",
    "%pip install mlflow\n",
    "%pip install azureml-mlflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import (\n",
    "    DefaultAzureCredential,\n",
    "    InteractiveBrowserCredential,\n",
    "    ClientSecretCredential,\n",
    ")\n",
    "from azure.ai.ml.entities import AmlCompute\n",
    "import time\n",
    "\n",
    "try:\n",
    "    credential = DefaultAzureCredential()\n",
    "    credential.get_token(\"https://management.azure.com/.default\")\n",
    "except Exception as ex:\n",
    "    credential = InteractiveBrowserCredential()\n",
    "\n",
    "try:\n",
    "    workspace_ml_client = MLClient.from_config(credential=credential)\n",
    "except:\n",
    "    workspace_ml_client = MLClient(\n",
    "        credential,\n",
    "        subscription_id=\"<SUBSCRIPTION_ID>\",\n",
    "        resource_group_name=\"<RESOURCE_GROUP>\",\n",
    "        workspace_name=\"<WORKSPACE_NAME>\",\n",
    "    )\n",
    "\n",
    "# the models, fine tuning pipelines and environments are available in the AzureML system registry, \"azureml\"\n",
    "registry_ml_client = MLClient(credential, registry_name=\"azureml\")\n",
    "registry_ml_client_meta = MLClient(credential, registry_name=\"azureml-meta\")\n",
    "\n",
    "experiment_name = \"llama-text-classification-emotion-detection\"\n",
    "\n",
    "# generating a unique timestamp that can be used for names and versions that need to be unique\n",
    "timestamp = str(int(time.time()))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Pick a foundation model to fine tune\n",
    "\n",
    "Models that support `fill-mask` tasks are good candidates to fine tune for `text-classification`. You can browse these models in the Model Catalog in the AzureML Studio, filtering by the `fill-mask` task. In this example, we use the `bert-base-uncased` model. If you have opened this notebook for a different model, replace the model name and version accordingly. \n",
    "\n",
    "Note the model id property of the model. This will be passed as input to the fine tuning job. This is also available as the `Asset ID` field in model details page in AzureML Studio Model Catalog. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"Llama-2-7b\"\n",
    "foundation_model = registry_ml_client_meta.models.get(model_name, label=\"latest\")\n",
    "print(\n",
    "    \"\\n\\nUsing model name: {0}, version: {1}, id: {2} for fine tuning\".format(\n",
    "        foundation_model.name, foundation_model.version, foundation_model.id\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Pick the dataset for fine-tuning the model\n",
    "\n",
    "We use the [emotion](https://huggingface.co/datasets/dair-ai/emotion) dataset. The next few cells show basic data preparation for fine tuning:\n",
    "* Visualize some data rows\n",
    "* Replace numerical categories in data with the actual string labels. This mapping is available in the [./emotion-dataset/label.json](./emotion-dataset/label.json). This step is needed if you want string labels such as `anger`, `joy`, etc. returned when scoring the model. If you skip this step, the model will return numerical categories such as 0, 1, 2, etc. and you will have to map them to what the category represents yourself. \n",
    "* We want this sample to run quickly, so save smaller `train`, `validation` and `test` files containing 10% of the original. This means the fine tuned model will have lower accuracy, hence it should not be put to real-world use. \n",
    "\n",
    "##### Here is an example of how the data should look like\n",
    "\n",
    "Single text classification requires the training data to include at least 2 fields – one for ‘Sentence1’ and ‘Label’ like in this example. Sentence 2 can be left blank in this case. The below examples are from Emotion dataset. \n",
    "\n",
    "| Text (Sentence1) | Label (Label) |\n",
    "| :- | :- |\n",
    "| i feel so blessed to be able to share it with you all | joy | \n",
    "| i feel intimidated nervous and overwhelmed and i shake like a leaf | fear | \n",
    "\n",
    " \n",
    "\n",
    "Text pair classification, where you have two sentences to be classified (e.g., sentence entailment) will need the training data to have 3 fields – for ‘Sentence1’, ‘Sentence2’ and ‘Label’ like in this example. The below examples are from Microsoft Research Paraphrase Corpus dataset. \n",
    "\n",
    "| Text1 (Sentence 1) | Text2 (Sentence 2) | Label_text (Label) |\n",
    "| :- | :- | :- |\n",
    "| Amrozi accused his brother , whom he called \" the witness \" , of deliberately distorting his evidence . | Referring to him as only \" the witness \" , Amrozi accused his brother of deliberately distorting his evidence . | equivalent |\n",
    "| Yucaipa owned Dominick 's before selling the chain to Safeway in 1998 for $ 2.5 billion . | Yucaipa bought Dominick 's in 1995 for \\$ 693 million and sold it to Safeway for \\$ 1.8 billion in 1998 . | not equivalent |\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# download the dataset using the helper script. This needs datasets library: https://pypi.org/project/datasets/\n",
    "import os\n",
    "\n",
    "exit_status = os.system(\"python ./download-dataset.py --download_dir emotion-dataset\")\n",
    "if exit_status != 0:\n",
    "    raise Exception(\"Error downloading dataset\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the ./emotion-dataset/train.jsonl file into a pandas dataframe and show the first 5 rows\n",
    "import pandas as pd\n",
    "\n",
    "pd.set_option(\n",
    "    \"display.max_colwidth\", 0\n",
    ")  # set the max column width to 0 to display the full text\n",
    "df = pd.read_json(\"./emotion-dataset/train.jsonl\", lines=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the id2label json element of the ./emotion-dataset/label.json file into pandas table with keys as 'label' column of int64 type and values as 'label_string' column as string type\n",
    "import json\n",
    "\n",
    "with open(\"./emotion-dataset/label.json\") as f:\n",
    "    id2label = json.load(f)\n",
    "    id2label = id2label[\"id2label\"]\n",
    "    label_df = pd.DataFrame.from_dict(\n",
    "        id2label, orient=\"index\", columns=[\"label_string\"]\n",
    "    )\n",
    "    label_df[\"label\"] = label_df.index.astype(\"int64\")\n",
    "    label_df = label_df[[\"label\", \"label_string\"]]\n",
    "label_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load test.jsonl, train.jsonl and validation.jsonl form the ./emotion-dataset folder into pandas dataframes\n",
    "test_df = pd.read_json(\"./emotion-dataset/test.jsonl\", lines=True)\n",
    "train_df = pd.read_json(\"./emotion-dataset/train.jsonl\", lines=True)\n",
    "validation_df = pd.read_json(\"./emotion-dataset/validation.jsonl\", lines=True)\n",
    "# join the train, validation and test dataframes with the id2label dataframe to get the label_string column\n",
    "train_df = train_df.merge(label_df, on=\"label\", how=\"left\")\n",
    "validation_df = validation_df.merge(label_df, on=\"label\", how=\"left\")\n",
    "test_df = test_df.merge(label_df, on=\"label\", how=\"left\")\n",
    "# show the first 5 rows of the train dataframe\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save 10% of the rows from the train, validation and test dataframes into files with small_ prefix in the ./emotion-dataset folder\n",
    "frac = 1\n",
    "train_df.sample(frac=frac).to_json(\n",
    "    \"./emotion-dataset/small_train.jsonl\", orient=\"records\", lines=True\n",
    ")\n",
    "validation_df.sample(frac=frac).to_json(\n",
    "    \"./emotion-dataset/small_validation.jsonl\", orient=\"records\", lines=True\n",
    ")\n",
    "test_df.sample(frac=frac).to_json(\n",
    "    \"./emotion-dataset/small_test.jsonl\", orient=\"records\", lines=True\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Submit the fine tuning job using the the model and data as inputs\n",
    " \n",
    "Create the job that uses the `text-classification` pipeline component. [Learn more](https://github.com/Azure/azureml-assets/blob/main/training/finetune_acft_hf_nlp/components/pipeline_components/text_classification/README.md) about all the parameters supported for fine tuning."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define finetune parameters\n",
    "\n",
    "Finetune parameters can be grouped into 2 categories - training parameters, optimization parameters\n",
    "\n",
    "Training parameters define the training aspects such as - \n",
    "1. the optimizer, scheduler to use\n",
    "2. the metric to optimize the finetune\n",
    "3. number of training steps and the batch size\n",
    "and so on\n",
    "\n",
    "Optimization parameters help in optimizing the GPU memory and effectively using the compute resources. Below are few of the parameters that belong to this category. _The optimization parameters differs for each model and are packaged with the model to handle these variations._\n",
    "1. enable the deepspeed, ORT and LoRA\n",
    "2. enable mixed precision training\n",
    "2. enable multi-node training "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ast\n",
    "\n",
    "# Training parameters\n",
    "training_parameters = dict(\n",
    "    num_train_epochs=3,\n",
    "    per_device_train_batch_size=1,\n",
    "    per_device_eval_batch_size=1,\n",
    "    learning_rate=2e-5,\n",
    "    metric_for_best_model=\"f1_macro\",\n",
    ")\n",
    "print(f\"The following training parameters are enabled - {training_parameters}\")\n",
    "\n",
    "# Optimization parameters - As these parameters are packaged with the model itself, lets retrieve those parameters\n",
    "if \"model_specific_defaults\" in foundation_model.tags:\n",
    "    optimization_parameters = ast.literal_eval(\n",
    "        foundation_model.tags[\"model_specific_defaults\"]\n",
    "    )  # convert string to python dict\n",
    "else:\n",
    "    optimization_parameters = dict(\n",
    "        apply_lora=\"true\", apply_deepspeed=\"true\", apply_ort=\"true\"\n",
    "    )\n",
    "print(f\"The following optimizations are enabled - {optimization_parameters}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Enable MULTINODE training**\n",
    "\n",
    "You can set the parameter `num_nodes_finetune` to finetune on required number of nodes.\n",
    "Default value is 1 which is same as single node training.\n",
    "If set to a value greater than 1, it will use as many number of nodes as requested\n",
    "\n",
    "Things to take care for multinode training:\n",
    "1. parameter `number_of_gpu_to_use_finetuning` should be set to number of gpus to use per node. for eg. if you are using ND40rs_v2 SKU which contains 8 gpu per node and you set num_nodes_finetune=2, you still need to set number_of_gpu_to_use_finetuning=8 and **not** 16.\n",
    "2. Make sure you have enough quota, and your compute is scalable to required number of gpus. The job will wait to acquire given number of nodes before it starts\n",
    "3. Set output modes to mount instead of upload. `pytorch_model_folder` output of finetune component needs to be changed to mount mode instead of default upload mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# PARAMETER for enabling multinode training\n",
    "# Running distributed training on 2 nodes\n",
    "num_nodes = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml.dsl import pipeline\n",
    "from azure.ai.ml.entities import CommandComponent, PipelineComponent, Job, Component\n",
    "from azure.ai.ml import PyTorchDistribution, Input\n",
    "from azure.ai.ml.constants import InputOutputModes\n",
    "\n",
    "# fetch the pipeline component\n",
    "pipeline_component_func = registry_ml_client.components.get(\n",
    "    name=\"text_classification_pipeline\", label=\"latest\"\n",
    ")\n",
    "\n",
    "\n",
    "# define the pipeline job\n",
    "@pipeline()\n",
    "def create_pipeline():\n",
    "    text_classification_pipeline = pipeline_component_func(\n",
    "        # specify the foundation model available in the azureml system registry id identified in step #3\n",
    "        mlflow_model_path=foundation_model.id,\n",
    "        # huggingface_id = 'bert-base-uncased', # if you want to use a huggingface model, uncomment this line and comment the above line\n",
    "        instance_type_finetune=\"Standard_NC24s_v3\",\n",
    "        instance_type_model_import=\"Standard_E4s_v3\",\n",
    "        instance_type_preprocess=\"Standard_E4s_v3\",\n",
    "        instance_type_model_evalutation=\"Standard_NC24s_v3\",\n",
    "        # map the dataset splits to parameters\n",
    "        train_file_path=Input(\n",
    "            type=\"uri_file\", path=\"./emotion-dataset/small_train.jsonl\"\n",
    "        ),\n",
    "        validation_file_path=Input(\n",
    "            type=\"uri_file\", path=\"./emotion-dataset/small_validation.jsonl\"\n",
    "        ),\n",
    "        test_file_path=Input(\n",
    "            type=\"uri_file\", path=\"./emotion-dataset/small_test.jsonl\"\n",
    "        ),\n",
    "        evaluation_config=Input(\n",
    "            type=\"uri_file\", path=\"./text-classification-config.json\"\n",
    "        ),\n",
    "        # The following parameters map to the dataset fields\n",
    "        sentence1_key=\"text\",\n",
    "        label_key=\"label_string\",\n",
    "        # Training settings\n",
    "        number_of_gpu_to_use_finetuning=4,  # set to the number of GPUs available in the compute\n",
    "        num_nodes_finetune=num_nodes,  # set to number of nodes to be used for distributed training\n",
    "        **training_parameters,\n",
    "        **optimization_parameters\n",
    "    )\n",
    "\n",
    "    # Set output mode to mount instead of upload, multinode setting does not work with upload mode\n",
    "    text_classification_pipeline.outputs[\n",
    "        \"pytorch_model_folder\"\n",
    "    ].mode = InputOutputModes.RW_MOUNT\n",
    "    text_classification_pipeline.outputs[\n",
    "        \"mlflow_model_folder\"\n",
    "    ].mode = InputOutputModes.RW_MOUNT\n",
    "\n",
    "    return {\n",
    "        # map the output of the fine tuning job to the output of pipeline job so that we can easily register the fine tuned model\n",
    "        # registering the model is required to deploy the model to an online or batch endpoint\n",
    "        \"trained_model\": text_classification_pipeline.outputs.mlflow_model_folder\n",
    "    }\n",
    "\n",
    "\n",
    "pipeline_object = create_pipeline()\n",
    "\n",
    "# don't use cached results from previous jobs\n",
    "pipeline_object.settings.force_rerun = True\n",
    "\n",
    "# set continue on step failure to False\n",
    "pipeline_object.settings.continue_on_step_failure = False"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Submit the job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# submit the pipeline job\n",
    "pipeline_job = workspace_ml_client.jobs.create_or_update(\n",
    "    pipeline_object, experiment_name=experiment_name\n",
    ")\n",
    "# wait for the pipeline job to complete\n",
    "workspace_ml_client.jobs.stream(pipeline_job.name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Review training and evaluation metrics\n",
    "Viewing the job in AzureML studio is the best way to analyze logs, metrics and outputs of jobs. You can create custom charts and compare metics across different jobs. See https://learn.microsoft.com/en-us/azure/machine-learning/how-to-log-view-metrics?tabs=interactive#view-jobsruns-information-in-the-studio to learn more. \n",
    "\n",
    "However, we may need to access and review metrics programmatically for which we will use MLflow, which is the recommended client for logging and querying metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mlflow, json\n",
    "\n",
    "mlflow_tracking_uri = workspace_ml_client.workspaces.get(\n",
    "    workspace_ml_client.workspace_name\n",
    ").mlflow_tracking_uri\n",
    "mlflow.set_tracking_uri(mlflow_tracking_uri)\n",
    "# concat 'tags.mlflow.rootRunId=' and pipeline_job.name in single quotes as filter variable\n",
    "filter = \"tags.mlflow.rootRunId='\" + pipeline_job.name + \"'\"\n",
    "runs = mlflow.search_runs(\n",
    "    experiment_names=[experiment_name], filter_string=filter, output_format=\"list\"\n",
    ")\n",
    "training_run = None\n",
    "evaluation_run = None\n",
    "# get the training and evaluation runs.\n",
    "# using a hacky way till 'Bug 2320997: not able to show eval metrics in FT notebooks - mlflow client now showing display names' is fixed\n",
    "for run in runs:\n",
    "    # check if run.data.metrics.epoch exists\n",
    "    if \"epoch\" in run.data.metrics:\n",
    "        training_run = run\n",
    "    # else, check if run.data.metrics.accuracy exists\n",
    "    elif \"accuracy\" in run.data.metrics:\n",
    "        evaluation_run = run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if training_run:\n",
    "    print(\"Training metrics:\\n\\n\")\n",
    "    print(json.dumps(training_run.data.metrics, indent=2))\n",
    "else:\n",
    "    print(\"No Training job found\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if evaluation_run:\n",
    "    print(\"Evaluation metrics:\\n\\n\")\n",
    "    print(json.dumps(evaluation_run.data.metrics, indent=2))\n",
    "else:\n",
    "    print(\"No Evaluation job found\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Register the fine tuned model with the workspace\n",
    "\n",
    "We will register the model from the output of the fine tuning job. This will track lineage between the fine tuned model and the fine tuning job. The fine tuning job, further, tracks lineage to the foundation model, data and training code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import Model\n",
    "from azure.ai.ml.constants import AssetTypes\n",
    "\n",
    "# check if the `trained_model` output is available\n",
    "print(\"pipeline job outputs: \", workspace_ml_client.jobs.get(pipeline_job.name).outputs)\n",
    "\n",
    "# fetch the model from pipeline job output - not working, hence fetching from fine tune child job\n",
    "model_path_from_job = \"azureml://jobs/{0}/outputs/{1}\".format(\n",
    "    pipeline_job.name, \"trained_model\"\n",
    ")\n",
    "\n",
    "finetuned_model_name = model_name + \"-emotion-detection\"\n",
    "finetuned_model_name = finetuned_model_name.replace(\"/\", \"-\")\n",
    "print(\"path to register model: \", model_path_from_job)\n",
    "prepare_to_register_model = Model(\n",
    "    path=model_path_from_job,\n",
    "    type=AssetTypes.MLFLOW_MODEL,\n",
    "    name=finetuned_model_name,\n",
    "    version=timestamp,  # use timestamp as version to avoid version conflict\n",
    "    description=model_name + \" fine tuned model for emotion detection\",\n",
    ")\n",
    "print(\"prepare to register model: \\n\", prepare_to_register_model)\n",
    "# register the model from pipeline job output\n",
    "registered_model = workspace_ml_client.models.create_or_update(\n",
    "    prepare_to_register_model\n",
    ")\n",
    "print(\"registered model: \\n\", registered_model)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Deploy the fine tuned model to an online endpoint\n",
    "Online endpoints give a durable REST API that can be used to integrate with applications that need to use the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time, sys\n",
    "from azure.ai.ml.entities import (\n",
    "    ManagedOnlineEndpoint,\n",
    "    ManagedOnlineDeployment,\n",
    "    ProbeSettings,\n",
    ")\n",
    "\n",
    "# Create online endpoint - endpoint names need to be unique in a region, hence using timestamp to create unique endpoint name\n",
    "\n",
    "online_endpoint_name = \"emotion-\" + timestamp\n",
    "# create an online endpoint\n",
    "endpoint = ManagedOnlineEndpoint(\n",
    "    name=online_endpoint_name,\n",
    "    description=\"Online endpoint for \"\n",
    "    + registered_model.name\n",
    "    + \", fine tuned model for emotion detection\",\n",
    "    auth_mode=\"key\",\n",
    ")\n",
    "workspace_ml_client.begin_create_or_update(endpoint).wait()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can find here the list of SKU's supported for deployment - [Managed online endpoints SKU list](https://learn.microsoft.com/en-us/azure/machine-learning/reference-managed-online-endpoints-vm-sku-list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a deployment\n",
    "demo_deployment = ManagedOnlineDeployment(\n",
    "    name=\"demo\",\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    model=registered_model.id,\n",
    "    instance_type=\"Standard_E64s_v3\",\n",
    "    instance_count=1,\n",
    "    liveness_probe=ProbeSettings(initial_delay=600),\n",
    ")\n",
    "workspace_ml_client.online_deployments.begin_create_or_update(demo_deployment).wait()\n",
    "endpoint.traffic = {\"demo\": 100}\n",
    "workspace_ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Test the endpoint with sample data\n",
    "\n",
    "We will fetch some sample data from the test dataset and submit to online endpoint for inference. We will then show the display the scored labels alongside the ground truth labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read ./emotion-dataset/small_test.jsonl into a pandas dataframe\n",
    "test_df = pd.read_json(\"./emotion-dataset/small_test.jsonl\", lines=True)\n",
    "# take 5 random samples\n",
    "test_df = test_df.sample(n=5)\n",
    "# rebuild index\n",
    "test_df.reset_index(drop=True, inplace=True)\n",
    "# rename the label_string column to ground_truth_label\n",
    "test_df = test_df.rename(columns={\"label_string\": \"ground_truth_label\"})\n",
    "test_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a json object with the key as \"inputs\" and value as a list of values from the text column of the test dataframe\n",
    "test_df_copy = test_df[[\"text\"]]\n",
    "test_df_copy = test_df_copy.rename(columns={\"text\": \"input_string\"})\n",
    "test_json = {\"input_data\": test_df_copy.to_dict(\"split\")}\n",
    "# save the json object to a file named sample_score.json in the ./emotion-dataset folder\n",
    "with open(\"./emotion-dataset/sample_score.json\", \"w\") as f:\n",
    "    json.dump(test_json, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# score the sample_score.json file using the online endpoint with the azureml endpoint invoke method\n",
    "response = workspace_ml_client.online_endpoints.invoke(\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    deployment_name=\"demo\",\n",
    "    request_file=\"./emotion-dataset/sample_score.json\",\n",
    ")\n",
    "print(\"raw response: \\n\", response, \"\\n\")\n",
    "# convert the response to a pandas dataframe and rename the label column as scored_label\n",
    "response_df = pd.read_json(response)\n",
    "response_df = response_df.rename(columns={0: \"scored_label\"})\n",
    "response_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# merge the test dataframe and the response dataframe on the index\n",
    "merged_df = pd.merge(test_df, response_df, left_index=True, right_index=True)\n",
    "merged_df.head(5)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Delete the online endpoint\n",
    "Don't forget to delete the online endpoint, else you will leave the billing meter running for the compute used by the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workspace_ml_client.online_endpoints.begin_delete(name=online_endpoint_name).wait()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}