deploy-to-databox-edge/deploy-to-edge-using-sdk.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy Azure ML Models to IoT Edge Devices - Using Python SDK\n",
    "You can use the Azure Machine Learning Python SDK to deploy models to IoT Edge devices if your IoT Hub is in the same subscription as your Azure ML Workspace. Otherwise, you will need to deploy using the Azure IoT CLI or the Azure Portal. \n",
    "\n",
    "### Prerequisites\n",
    "* An IoT Edge Device (specifically, the Data Box Edge machine with Compute VM enabled)\n",
    "    * Follow [steps 1 - 5a](https://docs.microsoft.com/en-us/azure/databox-online/data-box-edge-deploy-prep) to set up your Data Box Edge machine and enable the Linux Compute VM and an associated IoT Hub. This IoT Hub will deploy the AccelContainerImage to the Compute VM. **Note:** In order to use the Azure ML SDK to deploy your model to your IoT Hub, you must create your Data Box Edge resource in the same subscription as your Azure ML Workspace.\n",
    "* An Azure ML model registered OR a ContainerImage successfully created in your Workspace.\n",
    "    * For an Accelerated model, you can follow [this notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/accelerated-models/accelerated-models-quickstart.ipynb) through step 3. Step 4 and Step 5 are optional, and will be completed behind the scenes in deployment if not done.\n",
    "\n",
    "### Steps\n",
    "1. Retrieve your Azure ML Workspace \n",
    "* Retrieve your IoT Hub's connection string\n",
    "* Attach your IoT Hub as Azure ML Compute\n",
    "* Configure Azure ML module\n",
    "* Configure Sample Client module\n",
    "* Deploy your Model or Image\n",
    "* Stop Sample Client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Retrieve your Azure ML Workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Workspace\n",
    "ws = Workspace.from_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Retrieve your IoT Hub's connection string\n",
    "\n",
    "You do not need to follow this if you do not want to install the Azure CLI and are willing to access the IoT Hub connection string via the Azure Portal.\n",
    "\n",
    "You can install the necessary CLI packages in this notebook by running the below cell, if not already installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "pip install azure-cli # Install Azure CLI\n",
    "az extension add --name azure-cli-iot-ext # Add IoT CLI extension\n",
    "pip install -U jupyter_console # Fix Jupyter dependencies overridden by Azure CLI installation\n",
    "pip install docker # Install Docker\n",
    "pip install azureml-accel-models[cpu]\n",
    "az login # Log into Azure account"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set your Data Box Edge's resource group, IoT Hub name, and IoT Edge device name. If you're using the Azure ML SDK, we assume as a prerequisite that the subscription of your IoT Hub is the same as the Azure ML Workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the resource group where your IoT Hub resides. This can be different from your Azure ML Workspace.\n",
    "resource_group = ws.resource_group\n",
    "\n",
    "# Set the subscription where your IoT Hub resides. This should be the same as your Azure ML Workspace\n",
    "subscription_id = ws.subscription_id\n",
    "\n",
    "# Your Data Box Edge IoT Hub name\n",
    "iot_hub_name = \"dummy-dbe\"\n",
    "# Your Data Box Edge IoT Edge Device\n",
    "iot_device_id = \"computevm\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set subscription and resource group defaults, then get connection string for IoT Hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!az account set --subscription $subscription_id\n",
    "!az configure --defaults group=$resource_group\n",
    "ret = !az iot hub show-connection-string -n $iot_hub_name -o tsv\n",
    "connection_string = ret[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Attach your IoT Hub as Azure ML Compute\n",
    "\n",
    "If your IoT Hub is not attached to the Azure ML Workspace, you can attach it by running the below command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.core.compute import IotHubCompute\n",
    "\n",
    "# If you haven't attached your Data Box Edge's IoT Hub as compute\n",
    "config = IotHubCompute.attach_configuration(name=iot_hub_name, resource_group=resource_group, connection_string=connection_string)\n",
    "iothub_compute = IotHubCompute.attach(ws, iot_device_id, config)\n",
    "iothub_compute.wait_for_completion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or if you've already attached your IoT Hub as compute, run the below command to pull the existing Azure ML IotHubCompute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from azureml.contrib.core.compute import IotHubCompute\n",
    "\n",
    "# iothub_compute = IotHubCompute(ws, iot_device_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List Azure ML Compute Targets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import ComputeTarget\n",
    "\n",
    "compute_targets = ComputeTarget.list(ws)\n",
    "for t in compute_targets: \n",
    "    if t.type == \"IotHub\":\n",
    "        print(\"IotHub '{}' has provisioning state '{}'.\".format(t.name, t.provisioning_state))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Configure Azure ML module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below code provides information for the AccelContainerImage to run on the FPGA-enabled Data Box Edge. \n",
    "\n",
    "The ``container_config`` for accelerated models allows access to the FPGA device and exposes the port 50051. You can use the default ``container_config`` provided in the code.\n",
    "\n",
    "For CPU models, use the following ``container_config``, adjusting the port as you like:\n",
    "\n",
    "```\n",
    "container_config = '{ \\\n",
    "  \"HostConfig\": { \\\n",
    "    \"PortBindings\": { \\\n",
    "      \"80/tcp\": [ \\\n",
    "        { \\\n",
    "          \"HostPort\": \"80\" \\\n",
    "        } \\\n",
    "      ] \\\n",
    "    } \\\n",
    "  } \\\n",
    "}'\n",
    "```\n",
    "\n",
    "The ``routes`` are the default IoT Edge deployment routes which route any messages sent from modules to the IoT Hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.core.webservice import IotWebservice, IotBaseModuleSettings, IotModuleSettings\n",
    "\n",
    "#Pick a module name\n",
    "module_name = \"resnet50-host-2\"\n",
    "\n",
    "container_config = '{ \\\n",
    "  \"HostConfig\": { \\\n",
    "    \"Binds\": [ \\\n",
    "      \"/etc/hosts:/etc/hosts\" \\\n",
    "    ], \\\n",
    "    \"Privileged\": true, \\\n",
    "    \"Devices\": [ \\\n",
    "      { \\\n",
    "        \"PathOnHost\": \"/dev/catapult0\", \\\n",
    "        \"PathInContainer\": \"/dev/catapult0\" \\\n",
    "      }, \\\n",
    "      { \\\n",
    "        \"PathOnHost\": \"/dev/catapult1\", \\\n",
    "        \"PathInContainer\": \"/dev/catapult1\" \\\n",
    "      } \\\n",
    "    ], \\\n",
    "    \"PortBindings\": { \\\n",
    "      \"50051/tcp\": [ \\\n",
    "        { \\\n",
    "          \"HostPort\": \"50051\" \\\n",
    "        } \\\n",
    "      ] \\\n",
    "    } \\\n",
    "  } \\\n",
    "}'\n",
    "\n",
    "routes = {\n",
    "    \"route\": \"FROM /messages/* INTO \"\n",
    "}\n",
    "\n",
    "# Here, we define the Azure ML module with the container_config options above\n",
    "aml_module = IotBaseModuleSettings(name=module_name, create_option=container_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Configure Sample Client module\n",
    "\n",
    "You can also deploy modules that are not created through Azure ML, for example to deploy a client."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build sample client image and push to your ACR\n",
    "The Docker image built from the /sample-client directory will work with the ResNet50 image created in Quickstart. To create a sample client that works with the Image created from transfer learning, update the line to below. The /sample-client uses ImageNet pictures for inferencing; the /sample-client-tl handles the response differently and uses images of cats and dogs. \n",
    "\n",
    "```\n",
    "client_app = \"sample-client-tl\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import ContainerRegistry\n",
    "from azure.mgmt.containerregistry import ContainerRegistryManagementClient\n",
    "from azure.mgmt import containerregistry\n",
    "\n",
    "acr = ws.get_details().get(\"containerRegistry\")\n",
    "address = acr.split(\"/\")[-1]\n",
    "\n",
    "# The default sample client works with the Docker image created from Quickstart\n",
    "client_app = \"sample-client\"\n",
    "client_version = 1\n",
    "client_url = \"{}.azurecr.io/{}:{}\".format(address, client_app, client_version)\n",
    "\n",
    "print(address, client_app, client_url, sep = '\\n')\n",
    "\n",
    "# Here, we use Azure CLI to sign into the ACR. Otherwise, you can sign in manually using docker login and your ACR credentials\n",
    "# !docker login $address -u $username -p $password\n",
    "!az acr login --name $address --subscription $subscription_id --resource-group $resource_group\n",
    "!docker build -t $client_url -f ./Dockerfile ./$client_app\n",
    "!docker push $client_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client_env = { \n",
    "    \"DEVICE_CONNECTION_STRING\": connection_string\n",
    "}\n",
    "\n",
    "client_container_config = '{  \\\n",
    "    \"Tty\": true, \\\n",
    "    \"Cmd\": [ \\\n",
    "        \"--input-tensors\", \\\n",
    "        \"Placeholder:0\", \\\n",
    "        \"--output-tensors\", \\\n",
    "        \"classifier/resnet_v1_50/predictions/Softmax:0\", \\\n",
    "        \"--wait\", \\\n",
    "        \"10\", \\\n",
    "    ] \\\n",
    "}'\n",
    "\n",
    "aml_client = IotModuleSettings(name='azureml-client',\n",
    "                               image_location='{}:1.0'.format(client_url),\n",
    "                               env=client_env,\n",
    "                               create_option=client_container_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Deploy your Model or Image\n",
    "\n",
    "There are two scenarios for deployment, depending on whether or not your model is packaged into the AccelContainerImage. \n",
    "\n",
    "If you haven't created an AccelContainerImage yet, you can deploy from the model by specifying the inference_config. This will create an AccelContainerImage (or ContainerImage for CPU) behind the scenes each time. That AccelContainerImage is stored in your Azure Container Registery (ACR) associated with the Azure ML Workspace.\n",
    "\n",
    "#### a. Deploy from a Model that needs to be converted\n",
    "For deploying Accelerated Models, we will create an [AccelInferenceConfig](https://docs.microsoft.com/en-us/python/api/azureml-accel-models/azureml.accel.accelinferenceconfig?view=azure-ml-py) that will be used to create the AccelContainerImage behind the scenes. In the case of a model that is meant for CPU only, refer to [documentation](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.model.inferenceconfig?view=azure-ml-py) to create an InferenceConfig to create a regular ContainerImage.\n",
    "\n",
    "For AccelInferenceConfig, if your model is not yet converted to the Accelerated Onnx format, you can provide the ``input_tensor`` and ``output_tensor``. In the case that your model has already been converted, then you can create an empty ``AccelInferenceConfig()`` object.\n",
    "\n",
    "Then define the deployment manifest and the model you want to deploy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.accel import AccelInferenceConfig\n",
    "from azureml.core.model import Model\n",
    "\n",
    "# The input and output tensors are available in the Azure ML Hardware Accelerated notebooks. \n",
    "# These are the defaults from the Resnet50 Quickstart notebook.\n",
    "inference_config = AccelInferenceConfig(input_tensor=\"Placeholder:0\", output_tensor=\"classifier/resnet_v1_50/predictions/Softmax:0\")\n",
    "\n",
    "# Create deployment manifest\n",
    "deploy_config = IotWebservice.deploy_configuration(device_id=iot_device_id, routes=routes, aml_module=aml_module, external_modules=[aml_client])\n",
    "\n",
    "# Get the model you want to deploy. This can be the CPU version of the model, \n",
    "# an Accelerated Model before conversion, or an Accelerated Model after conversion.\n",
    "model_name = \"resnet50\"\n",
    "registered_model = Model(ws, name=model_name)\n",
    "\n",
    "# Deploy from model, using module_name as your IotWebservice name\n",
    "iot_service_name = module_name\n",
    "\n",
    "iot_service = Model.deploy(ws, iot_service_name, [registered_model], inference_config, deploy_config, iothub_compute)\n",
    "iot_service.wait_for_deployment()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### b. Deploy from ContainerImage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you've already created an AccelContainerImage with the converted model that you want to run, then you can use the cell below to pull that image and deploy it. This image is stored in your Azure Container Registry (ACR) associated with your Azure ML Workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Image, Webservice\n",
    "from azureml.contrib.core.webservice import IotWebservice\n",
    "from azureml.accel import AccelContainerImage\n",
    "\n",
    "# Then we define the deployment manifest for our IoT Edge device with the aml_module and routes\n",
    "deploy_config = IotWebservice.deploy_configuration(device_id=iot_device_id, routes=routes, aml_module=aml_module, external_modules=[aml_client])\n",
    "\n",
    "# Deploy from latest version of image, using module_name as your IotWebservice name\n",
    "image_name = \"resnet50-image\"\n",
    "iot_service_name = module_name\n",
    "\n",
    "# Can specify version=x, otherwise will grab latest\n",
    "image = Image(ws, image_name) \n",
    "iot_service = IotWebservice.deploy_from_image(ws, iot_service_name, image, deploy_config, iothub_compute)\n",
    "iot_service.wait_for_deployment()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Stop Sample Client\n",
    "First, delete the previous iot_service, then re-deploy without the sample client so that it doesn't use power for inferencing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iot_service.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When creating the IotWebservice deployment configuration, we will leave off the external module from the deployment manifest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Image, Webservice\n",
    "from azureml.contrib.core.webservice import IotWebservice\n",
    "from azureml.accel import AccelContainerImage\n",
    "\n",
    "# This time, we will leave off the external module from the deployment manifest\n",
    "deploy_config = IotWebservice.deploy_configuration(device_id=iot_device_id, routes=routes, aml_module=aml_module)\n",
    "\n",
    "# Deploy from latest version of image, using module_name as your IotWebservice name\n",
    "image_name = \"resnet50-image\"\n",
    "iot_service_name = module_name\n",
    "\n",
    "# Can specify version=x, otherwise will grab latest\n",
    "image = Image(ws, image_name) \n",
    "iot_service = IotWebservice.deploy_from_image(ws, iot_service_name, image, deploy_config, iothub_compute)\n",
    "iot_service.wait_for_deployment()"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "paledger"
   }
  ],
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}