Meetup11-Lab1-BatchAI-SingleNode-MultiGPU.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LAB 1 - Batch AI with TensorFlow MultiGPU\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This example uses the MNIST dataset to demonstrate how to train a convolutional neural network (CNN) on a GPU cluster. This recipe is running on a single node.\n",
    "\n",
    "## Details\n",
    "\n",
    "- For demonstration purposes, MNIST dataset and ConvNet_MNIST.py will be deployed at Azure File Share;\n",
    "- Standard output of the job and the model will be stored on Azure File Share;\n",
    "- MNIST dataset (http://yann.lecun.com/exdb/mnist/) has been preprocessed by usign install_mnist.py available at [here](https://batchaisamples.blob.core.windows.net/samples/mnist_dataset.zip?st=2017-09-29T18%3A29%3A00Z&se=2099-12-31T08%3A00%3A00Z&sp=rl&sv=2016-05-31&sr=c&sig=PmhL%2BYnYAyNTZr1DM2JySvrI12e%2F4wZNIwCtf7TRI%2BM%3D)\n",
    "- The original CNTK example (https://github.com/Microsoft/CNTK/blob/master/Examples/Image/Classification/ConvNet/Python/ConvNet_MNIST.py) has been modified to accept CNTK dataset and model locations via command line arguments and available here [ConvNet_MNIST.py](/recipes/CNTK/Python/CNTK-GPU-Python/ConvNet_MNIST.py). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "\n",
    "### Create in Azure the Resource Groups and Storage Accounts needed.\n",
    "```\n",
    "> ssh sshuser@YOUR.VM.IP.ADDRESS\n",
    "> az login\n",
    "> az group create --name batchai_rg  --location eastus\n",
    "> az storage account create --location eastus --name batchaipablo --resource-group batchai_rg --sku Standard_LRS\n",
    "> az storage account keys list --account-name batchaipablo --resource-group batchai_rg -o table\n",
    "> az ad sp create-for-rbac --name MyAppSvcPppl --password Passw0rd\n",
    "> az storage account keys list --account-name batchaipablo --resource-group batchai_rg\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Configuration and Create Batch AI client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "bfa11f00-8866-4051-bbfe-a9646e004910"
    }
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "from datetime import datetime\n",
    "import os\n",
    "import sys\n",
    "import zipfile\n",
    "\n",
    "from azure.storage.file import FileService\n",
    "import azure.mgmt.batchai.models as models\n",
    "\n",
    "# utilities.py contains helper functions\n",
    "import utilities\n",
    "\n",
    "# Resource Group\n",
    "location = 'eastus'\n",
    "resource_group = 'batchai_rg'\n",
    "\n",
    "# credentials used for authentication\n",
    "client_id = 'ec0640c7-61fa-4662-bce4-8a3e931939ac'\n",
    "secret = 'Passw0rd'\n",
    "token_uri = 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/token'\n",
    "subscription_id = 'b1395605-1fe9-4af4-b3ff-82a4725a3791'\n",
    "\n",
    "# credentials used for storage\n",
    "storage_account_name = 'batchaipablo'\n",
    "storage_account_key = 'y59heteYEbw5nTLBB/b7rj3jUphvs2Iwslg4AsXFSb4G7ZLgJUep4AuccSmST7I3E8Zw4BaUloebK+VyKmGpog=='\n",
    "\n",
    "# specify the credentials used to remote login your GPU node\n",
    "admin_user_name = 'sshuser'\n",
    "admin_user_password = 'Passw0rd.1!!'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.common.credentials import ServicePrincipalCredentials\n",
    "import azure.mgmt.batchai as batchai\n",
    "import azure.mgmt.batchai.models as models\n",
    "\n",
    "creds = ServicePrincipalCredentials(client_id=client_id, secret=secret, token_uri=token_uri)\n",
    "\n",
    "client = batchai.BatchAIManagementClient(credentials=creds,subscription_id=subscription_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create File Share\n",
    "\n",
    "For this example we will create a new File Share with name `batchaisample` under your storage account.\n",
    "\n",
    "**Note** You don't need to create new file share for every cluster. We are doing this in this sample to simplify resource management for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "azure_file_share_name = 'batchailab1'\n",
    "service = FileService(storage_account_name, storage_account_key)\n",
    "service.create_share(azure_file_share_name, fail_on_exist=False)\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Compute Cluster\n",
    "\n",
    "- For this example we will use a gpu cluster of `STANDARD_NC6` nodes. Number of nodes in the cluster is configured with `nodes_count` variable;\n",
    "- We will mount file share at folder with name `external`. Full path of this folder on a computer node will be `$AZ_BATCHAI_MOUNT_ROOT/external`;\n",
    "- We will call the cluster `nc6`;\n",
    "\n",
    "So, the cluster will have the following parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "azure_file_share = 'external'\n",
    "nodes_count = 1\n",
    "cluster_name = 'nc12'\n",
    "vmsize = \"Standard_NC12\" \n",
    "\n",
    "volumes = models.MountVolumes(\n",
    "    azure_file_shares=[\n",
    "        models.AzureFileShareReference(\n",
    "            account_name=storage_account_name,\n",
    "            credentials=models.AzureStorageCredentialsInfo(\n",
    "                account_key=storage_account_key),\n",
    "            azure_file_url = 'https://{0}.file.core.windows.net/{1}'.format(\n",
    "                storage_account_name, azure_file_share_name),\n",
    "            relative_mount_path=azure_file_share)\n",
    "    ]\n",
    ")\n",
    "\n",
    "parameters = models.ClusterCreateParameters(\n",
    "    location=location,\n",
    "    vm_size=vmsize,\n",
    "    virtual_machine_configuration=models.VirtualMachineConfiguration(\n",
    "        image_reference=models.ImageReference(\n",
    "            publisher=\"microsoft-ads\",\n",
    "            offer=\"linux-data-science-vm-ubuntu\",\n",
    "            sku=\"linuxdsvmubuntu\",\n",
    "            version=\"latest\")),    \n",
    "    user_account_settings=models.UserAccountSettings(\n",
    "        admin_user_name=admin_user_name,\n",
    "        admin_user_password=admin_user_password),\n",
    "    scale_settings=models.ScaleSettings(\n",
    "        manual=models.ManualScaleSettings(target_node_count=nodes_count)\n",
    "    ),\n",
    "    node_setup=models.NodeSetup(\n",
    "        mount_volumes=volumes,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Compute Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster = client.clusters.create(resource_group, cluster_name, parameters).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor Cluster Creation\n",
    "\n",
    "Monitor the just created cluster. utilities.py contains a helper function to print out detail status of the cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster state: AllocationState.resizing Target: 1; Allocated: 0; Idle: 0; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n"
     ]
    }
   ],
   "source": [
    "cluster = client.clusters.get(resource_group, cluster_name)\n",
    "utilities.print_cluster_status(cluster)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Deploy MNIST Dataset\n",
    "\n",
    "For demonstration purposes, we will download preprocessed MNIST dataset to the current directory and upload it to file share directory named `mnist_dataset`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download and Extract MNIST Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "mnist_dataset_url = 'https://batchaisamples.blob.core.windows.net/samples/mnist_dataset.zip?st=2017-09-29T18%3A29%3A00Z&se=2099-12-31T08%3A00%3A00Z&sp=rl&sv=2016-05-31&sr=c&sig=PmhL%2BYnYAyNTZr1DM2JySvrI12e%2F4wZNIwCtf7TRI%2BM%3D'\n",
    "if not os.path.exists('Train-28x28_cntk_text.txt') or not os.path.exists('Test-28x28_cntk_text.txt'):\n",
    "    utilities.download_file(mnist_dataset_url, 'mnist_dataset.zip')\n",
    "    print('Extracting MNIST dataset...')\n",
    "    with zipfile.ZipFile('mnist_dataset.zip', 'r') as z:\n",
    "        z.extractall('.')\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create File Share and Upload MNIST Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mnist_dataset_directory = 'mnist_dataset'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are multiple ways to create folders and upload files into Azure File Share - you can use [Azure Portal](https://ms.portal.azure.com), [Storage Explorer](http://storageexplorer.com/), [Azure CLI2](/azure-cli-extension) or Azure SDK for your preferable programming language.\n",
    "In this example we will use Azure SDK for python to copy files into file share."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally on large datasets it is better to use the Azure CLI, with this command\n",
    "\n",
    "> AzCopy /Source:https://myaccount1.blob.core.windows.net/mycontainer/ /Dest:https://myaccount2.file.core.windows.net/myfileshare/ /SourceKey:key1 /DestKey:key2 /S"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "service = FileService(storage_account_name, storage_account_key)\n",
    "service.create_directory(\n",
    "    azure_file_share_name, mnist_dataset_directory,\n",
    "    fail_on_exist=False)\n",
    "# Since uploading can take significant time, let's check first if the\n",
    "# file has been uploaded already.\n",
    "for f in ['Train-28x28_cntk_text.txt', 'Test-28x28_cntk_text.txt']:\n",
    "    if service.exists(azure_file_share_name, mnist_dataset_directory, f):\n",
    "        continue\n",
    "    service.create_file_from_path(\n",
    "        azure_file_share_name, mnist_dataset_directory, f, f)\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy Sample Script and Configure the Input Directories\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- CNTK: For each job we will create a folder containing a copy of ConvNet_MNIST.py\n",
    "- Tensorflow: For each job we will create a folder containing a copy of convolutional.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "# Uncomment this if you want to use CNTK\n",
    "#cntk_script_path = \"cntk_samples\"\n",
    "#service = FileService(storage_account_name, storage_account_key)\n",
    "#service.create_directory(\n",
    "#    azure_file_share_name, cntk_script_path, fail_on_exist=False)\n",
    "#service.create_file_from_path(\n",
    "#    azure_file_share_name, cntk_script_path, 'ConvNet_MNIST.py', 'ConvNet_MNIST.py')\n",
    "\n",
    "mnist_script_directory = 'tensorflow_samples'\n",
    "service = FileService(storage_account_name, storage_account_key)\n",
    "service.create_directory(\n",
    "    azure_file_share_name, mnist_script_directory, fail_on_exist=False)\n",
    "service.create_file_from_path(\n",
    "    azure_file_share_name, mnist_script_directory, 'convolutional.py', 'convolutional.py')\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The job needs to know where to find ConvNet_MNIST.py and input MNIST dataset. We will create two input directories for this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#input_directories = [\n",
    "#    models.InputDirectory(\n",
    "#        id='SCRIPT',\n",
    "#        path='$AZ_BATCHAI_MOUNT_ROOT/{0}/{1}'.format(azure_file_share, cntk_script_path)),\n",
    "#    models.InputDirectory(\n",
    "#        id='DATASET',\n",
    "#        path='$AZ_BATCHAI_MOUNT_ROOT/{0}/{1}'.format(azure_file_share, mnist_dataset_directory))]\n",
    "\n",
    "\n",
    "input_directories = [\n",
    "    models.InputDirectory(\n",
    "        id='SCRIPT',\n",
    "        path='$AZ_BATCHAI_MOUNT_ROOT/{0}/{1}'.format(azure_file_share, mnist_script_directory))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The job will be able to reference those directories using ```$AZ_BATCHAI_INPUT_SCRIPT``` and ```$AZ_BATCHAI_INPUT_DATASET``` environment variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Output Directories\n",
    "We will store standard and error output of the job in File Share:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "std_output_path_prefix = \"$AZ_BATCHAI_MOUNT_ROOT/{0}\".format(azure_file_share)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The job will be able to reference this directory as `$AZ_BATCHAI_OUTPUT_MODEL` and we will be able to enumerate files in this directory using `MODEL` id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "output_directories = [\n",
    "    models.OutputDirectory(\n",
    "        id='MODEL',\n",
    "        path_prefix='$AZ_BATCHAI_MOUNT_ROOT/{0}'.format(azure_file_share),\n",
    "        path_suffix=\"Models\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Job\n",
    "- The job will use `microsoft/cntk:2.1-gpu-python3.5-cuda8.0-cudnn6.0` container.\n",
    "- Will use configured previously input and output directories;\n",
    "- Will run modified ConvNet_MNIST.py providing MNIST Dataset path as the first parameter and desired mode output as the second one.\n",
    "- By removing container_settings, the job will be ran on the host VMs if you are using DSVM.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "job_name = datetime.utcnow().strftime(\"cntk_%m_%d_%Y_%H%M%S\")\n",
    "parameters = models.job_create_parameters.JobCreateParameters(\n",
    "     location=location,\n",
    "     cluster=models.ResourceId(cluster.id),\n",
    "     node_count=nodes_count,\n",
    "     input_directories=input_directories,\n",
    "     std_out_err_path_prefix=std_output_path_prefix,\n",
    "     output_directories=output_directories,\n",
    "     #container_settings=models.ContainerSettings(\n",
    "     #    models.ImageSourceRegistry(image='microsoft/cntk:2.1-gpu-python3.5-cuda8.0-cudnn6.0')),\n",
    "     container_settings=models.ContainerSettings(\n",
    "         models.ImageSourceRegistry(image='tensorflow/tensorflow:1.1.0-gpu')),\n",
    "     #cntk_settings = models.CNTKsettings(\n",
    "     #    python_script_file_path='$AZ_BATCHAI_INPUT_SCRIPT/ConvNet_MNIST.py',\n",
    "     #    command_line_args='$AZ_BATCHAI_INPUT_DATASET $AZ_BATCHAI_OUTPUT_MODEL')\n",
    "     tensor_flow_settings=models.TensorFlowSettings(\n",
    "     python_script_file_path='$AZ_BATCHAI_INPUT_SCRIPT/convolutional.py',\n",
    "     master_command_line_args=\"-p\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a training Job and wait for Job completion\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created Job: cntk_02_13_2018_231620\n"
     ]
    }
   ],
   "source": [
    "job = client.jobs.create(resource_group, job_name, parameters).result()\n",
    "print('Created Job: {}'.format(job_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait for Job to Finish\n",
    "The job will start running when the cluster will have enought idle nodes. The following code waits for job to start running printing the cluster state. During job run, the code prints current content of stdout.\n",
    "\n",
    "**Note** Execution may take several minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster state: AllocationState.steady Target: 1; Allocated: 1; Idle: 1; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: succeeded ExitCode: 0\n",
      "Waiting for job output to become available...\n",
      "2018-02-13 23:20:56.437213: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 23:20:56.441945: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 23:20:56.447544: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 23:20:56.453561: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 23:20:56.459702: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.\n",
      "2018-02-13 23:20:56.928987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties: \n",
      "name: Tesla K80\n",
      "major: 3 minor: 7 memoryClockRate (GHz) 0.8235\n",
      "pciBusID 640b:00:00.0\n",
      "Total memory: 11.17GiB\n",
      "Free memory: 11.10GiB\n",
      "2018-02-13 23:20:56.934061: W tensorflow/stream_executor/cuda/cuda_driver.cc:485] creating context when one is currently active; existing: 0x30713c0\n",
      "2018-02-13 23:20:57.400221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 1 with properties: \n",
      "name: Tesla K80\n",
      "major: 3 minor: 7 memoryClockRate (GHz) 0.8235\n",
      "pciBusID afcc:00:00.0\n",
      "Total memory: 11.17GiB\n",
      "Free memory: 11.10GiB\n",
      "2018-02-13 23:20:57.404895: I tensorflow/core/common_runtime/gpu/gpu_device.cc:779] Peer access not supported between device ordinals 0 and 1\n",
      "2018-02-13 23:20:57.410360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:779] Peer access not supported between device ordinals 1 and 0\n",
      "2018-02-13 23:20:57.415348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 1 \n",
      "2018-02-13 23:20:57.453594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y N \n",
      "2018-02-13 23:20:57.469494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 1:   N Y \n",
      "2018-02-13 23:20:57.484901: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: 640b:00:00.0)\n",
      "2018-02-13 23:20:57.508594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: Tesla K80, pci bus id: afcc:00:00.0)\n",
      "Job state: succeeded ExitCode: 0\n"
     ]
    }
   ],
   "source": [
    "#utilities.wait_for_job_completion(client, resource_group, job_name, cluster_name, 'stdOuterr', 'stderr.txt')\n",
    "utilities.wait_for_job_completion(client, resource_group, job_name, cluster_name, 'stdOuterr', 'stderr-wk-0.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster state: AllocationState.steady Target: 1; Allocated: 1; Idle: 1; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n",
      "Job state: succeeded ExitCode: 0\n",
      "Waiting for job output to become available...\n",
      "Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.\n",
      "Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.\n",
      "Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.\n",
      "Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.\n",
      "Extracting data/train-images-idx3-ubyte.gz\n",
      "Extracting data/train-labels-idx1-ubyte.gz\n",
      "Extracting data/t10k-images-idx3-ubyte.gz\n",
      "Extracting data/t10k-labels-idx1-ubyte.gz\n",
      "Initialized!\n",
      "Step 0 (epoch 0.00), 27.4 ms\n",
      "Minibatch loss: 8.334, learning rate: 0.010000\n",
      "Minibatch error: 85.9%\n",
      "Validation error: 84.6%\n",
      "Step 100 (epoch 0.12), 17.0 ms\n",
      "Minibatch loss: 3.266, learning rate: 0.010000\n",
      "Minibatch error: 4.7%\n",
      "Validation error: 7.4%\n",
      "Step 200 (epoch 0.23), 12.5 ms\n",
      "Minibatch loss: 3.363, learning rate: 0.010000\n",
      "Minibatch error: 10.9%\n",
      "Validation error: 4.5%\n",
      "Step 300 (epoch 0.35), 12.2 ms\n",
      "Minibatch loss: 3.155, learning rate: 0.010000\n",
      "Minibatch error: 6.2%\n",
      "Validation error: 2.9%\n",
      "Step 400 (epoch 0.47), 11.9 ms\n",
      "Minibatch loss: 3.215, learning rate: 0.010000\n",
      "Minibatch error: 6.2%\n",
      "Validation error: 2.8%\n",
      "Step 500 (epoch 0.58), 13.4 ms\n",
      "Minibatch loss: 3.153, learning rate: 0.010000\n",
      "Minibatch error: 6.2%\n",
      "Validation error: 2.5%\n",
      "Step 600 (epoch 0.70), 15.2 ms\n",
      "Minibatch loss: 3.126, learning rate: 0.010000\n",
      "Minibatch error: 3.1%\n",
      "Validation error: 2.1%\n",
      "Step 700 (epoch 0.81), 19.8 ms\n",
      "Minibatch loss: 2.958, learning rate: 0.010000\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 2.1%\n",
      "Step 800 (epoch 0.93), 13.4 ms\n",
      "Minibatch loss: 3.067, learning rate: 0.010000\n",
      "Minibatch error: 7.8%\n",
      "Validation error: 1.9%\n",
      "Step 900 (epoch 1.05), 13.0 ms\n",
      "Minibatch loss: 2.900, learning rate: 0.009500\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.7%\n",
      "Step 1000 (epoch 1.16), 13.3 ms\n",
      "Minibatch loss: 2.890, learning rate: 0.009500\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.8%\n",
      "Step 1100 (epoch 1.28), 13.6 ms\n",
      "Minibatch loss: 2.823, learning rate: 0.009500\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.5%\n",
      "Step 1200 (epoch 1.40), 12.1 ms\n",
      "Minibatch loss: 2.934, learning rate: 0.009500\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.5%\n",
      "Step 1300 (epoch 1.51), 13.1 ms\n",
      "Minibatch loss: 2.790, learning rate: 0.009500\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.8%\n",
      "Step 1400 (epoch 1.63), 13.5 ms\n",
      "Minibatch loss: 2.822, learning rate: 0.009500\n",
      "Minibatch error: 3.1%\n",
      "Validation error: 1.7%\n",
      "Step 1500 (epoch 1.75), 12.3 ms\n",
      "Minibatch loss: 2.861, learning rate: 0.009500\n",
      "Minibatch error: 3.1%\n",
      "Validation error: 1.4%\n",
      "Step 1600 (epoch 1.86), 12.8 ms\n",
      "Minibatch loss: 2.721, learning rate: 0.009500\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.4%\n",
      "Step 1700 (epoch 1.98), 12.0 ms\n",
      "Minibatch loss: 2.658, learning rate: 0.009500\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.6%\n",
      "Step 1800 (epoch 2.09), 12.4 ms\n",
      "Minibatch loss: 2.662, learning rate: 0.009025\n",
      "Minibatch error: 3.1%\n",
      "Validation error: 1.3%\n",
      "Step 1900 (epoch 2.21), 11.9 ms\n",
      "Minibatch loss: 2.635, learning rate: 0.009025\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.3%\n",
      "Step 2000 (epoch 2.33), 11.8 ms\n",
      "Minibatch loss: 2.593, learning rate: 0.009025\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.2%\n",
      "Step 2100 (epoch 2.44), 12.6 ms\n",
      "Minibatch loss: 2.571, learning rate: 0.009025\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.1%\n",
      "Step 2200 (epoch 2.56), 12.0 ms\n",
      "Minibatch loss: 2.571, learning rate: 0.009025\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.2%\n",
      "Step 2300 (epoch 2.68), 12.0 ms\n",
      "Minibatch loss: 2.593, learning rate: 0.009025\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.2%\n",
      "Step 2400 (epoch 2.79), 12.2 ms\n",
      "Minibatch loss: 2.522, learning rate: 0.009025\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.2%\n",
      "Step 2500 (epoch 2.91), 12.2 ms\n",
      "Minibatch loss: 2.467, learning rate: 0.009025\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.1%\n",
      "Step 2600 (epoch 3.03), 12.0 ms\n",
      "Minibatch loss: 2.457, learning rate: 0.008574\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.3%\n",
      "Step 2700 (epoch 3.14), 13.0 ms\n",
      "Minibatch loss: 2.511, learning rate: 0.008574\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.2%\n",
      "Step 2800 (epoch 3.26), 12.6 ms\n",
      "Minibatch loss: 2.505, learning rate: 0.008574\n",
      "Minibatch error: 3.1%\n",
      "Validation error: 1.2%\n",
      "Step 2900 (epoch 3.37), 12.0 ms\n",
      "Minibatch loss: 2.531, learning rate: 0.008574\n",
      "Minibatch error: 3.1%\n",
      "Validation error: 1.1%\n",
      "Step 3000 (epoch 3.49), 11.9 ms\n",
      "Minibatch loss: 2.398, learning rate: 0.008574\n",
      "Minibatch error: 3.1%\n",
      "Validation error: 0.8%\n",
      "Step 3100 (epoch 3.61), 12.9 ms\n",
      "Minibatch loss: 2.376, learning rate: 0.008574\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 3200 (epoch 3.72), 12.6 ms\n",
      "Minibatch loss: 2.341, learning rate: 0.008574\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.1%\n",
      "Step 3300 (epoch 3.84), 13.0 ms\n",
      "Minibatch loss: 2.326, learning rate: 0.008574\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.1%\n",
      "Step 3400 (epoch 3.96), 12.5 ms\n",
      "Minibatch loss: 2.294, learning rate: 0.008574\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.0%\n",
      "Step 3500 (epoch 4.07), 12.1 ms\n",
      "Minibatch loss: 2.279, learning rate: 0.008145\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.0%\n",
      "Step 3600 (epoch 4.19), 11.9 ms\n",
      "Minibatch loss: 2.251, learning rate: 0.008145\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.0%\n",
      "Step 3700 (epoch 4.31), 13.0 ms\n",
      "Minibatch loss: 2.228, learning rate: 0.008145\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.0%\n",
      "Step 3800 (epoch 4.42), 12.0 ms\n",
      "Minibatch loss: 2.215, learning rate: 0.008145\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 3900 (epoch 4.54), 11.9 ms\n",
      "Minibatch loss: 2.243, learning rate: 0.008145\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.0%\n",
      "Step 4000 (epoch 4.65), 12.3 ms\n",
      "Minibatch loss: 2.263, learning rate: 0.008145\n",
      "Minibatch error: 3.1%\n",
      "Validation error: 1.0%\n",
      "Step 4100 (epoch 4.77), 11.7 ms\n",
      "Minibatch loss: 2.171, learning rate: 0.008145\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.0%\n",
      "Step 4200 (epoch 4.89), 12.4 ms\n",
      "Minibatch loss: 2.156, learning rate: 0.008145\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.0%\n",
      "Step 4300 (epoch 5.00), 12.5 ms\n",
      "Minibatch loss: 2.166, learning rate: 0.007738\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.0%\n",
      "Step 4400 (epoch 5.12), 12.0 ms\n",
      "Minibatch loss: 2.128, learning rate: 0.007738\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.0%\n",
      "Step 4500 (epoch 5.24), 12.7 ms\n",
      "Minibatch loss: 2.175, learning rate: 0.007738\n",
      "Minibatch error: 3.1%\n",
      "Validation error: 0.9%\n",
      "Step 4600 (epoch 5.35), 12.8 ms\n",
      "Minibatch loss: 2.080, learning rate: 0.007738\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 4700 (epoch 5.47), 12.2 ms\n",
      "Minibatch loss: 2.079, learning rate: 0.007738\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.0%\n",
      "Step 4800 (epoch 5.59), 12.5 ms\n",
      "Minibatch loss: 2.066, learning rate: 0.007738\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.0%\n",
      "Step 4900 (epoch 5.70), 12.1 ms\n",
      "Minibatch loss: 2.068, learning rate: 0.007738\n",
      "Minibatch error: 3.1%\n",
      "Validation error: 1.0%\n",
      "Step 5000 (epoch 5.82), 11.9 ms\n",
      "Minibatch loss: 2.123, learning rate: 0.007738\n",
      "Minibatch error: 4.7%\n",
      "Validation error: 0.9%\n",
      "Step 5100 (epoch 5.93), 12.6 ms\n",
      "Minibatch loss: 2.016, learning rate: 0.007738\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 1.0%\n",
      "Step 5200 (epoch 6.05), 12.6 ms\n",
      "Minibatch loss: 2.044, learning rate: 0.007351\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 0.9%\n",
      "Step 5300 (epoch 6.17), 12.9 ms\n",
      "Minibatch loss: 1.972, learning rate: 0.007351\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.0%\n",
      "Step 5400 (epoch 6.28), 12.3 ms\n",
      "Minibatch loss: 1.962, learning rate: 0.007351\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 5500 (epoch 6.40), 12.8 ms\n",
      "Minibatch loss: 1.953, learning rate: 0.007351\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.8%\n",
      "Step 5600 (epoch 6.52), 12.3 ms\n",
      "Minibatch loss: 1.930, learning rate: 0.007351\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 5700 (epoch 6.63), 12.2 ms\n",
      "Minibatch loss: 1.911, learning rate: 0.007351\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.8%\n",
      "Step 5800 (epoch 6.75), 13.2 ms\n",
      "Minibatch loss: 1.898, learning rate: 0.007351\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 5900 (epoch 6.87), 12.8 ms\n",
      "Minibatch loss: 1.892, learning rate: 0.007351\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.8%\n",
      "Step 6000 (epoch 6.98), 12.0 ms\n",
      "Minibatch loss: 1.899, learning rate: 0.007351\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 6100 (epoch 7.10), 12.7 ms\n",
      "Minibatch loss: 1.869, learning rate: 0.006983\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 6200 (epoch 7.21), 11.8 ms\n",
      "Minibatch loss: 1.843, learning rate: 0.006983\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.8%\n",
      "Step 6300 (epoch 7.33), 14.9 ms\n",
      "Minibatch loss: 1.844, learning rate: 0.006983\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 6400 (epoch 7.45), 12.4 ms\n",
      "Minibatch loss: 1.841, learning rate: 0.006983\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 0.9%\n",
      "Step 6500 (epoch 7.56), 12.0 ms\n",
      "Minibatch loss: 1.807, learning rate: 0.006983\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.8%\n",
      "Step 6600 (epoch 7.68), 11.8 ms\n",
      "Minibatch loss: 1.840, learning rate: 0.006983\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 0.8%\n",
      "Step 6700 (epoch 7.80), 12.7 ms\n",
      "Minibatch loss: 1.782, learning rate: 0.006983\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 1.0%\n",
      "Step 6800 (epoch 7.91), 13.1 ms\n",
      "Minibatch loss: 1.771, learning rate: 0.006983\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 6900 (epoch 8.03), 12.1 ms\n",
      "Minibatch loss: 1.758, learning rate: 0.006634\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 7000 (epoch 8.15), 11.9 ms\n",
      "Minibatch loss: 1.747, learning rate: 0.006634\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 7100 (epoch 8.26), 12.3 ms\n",
      "Minibatch loss: 1.734, learning rate: 0.006634\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.8%\n",
      "Step 7200 (epoch 8.38), 13.0 ms\n",
      "Minibatch loss: 1.733, learning rate: 0.006634\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 7300 (epoch 8.49), 13.1 ms\n",
      "Minibatch loss: 1.781, learning rate: 0.006634\n",
      "Minibatch error: 3.1%\n",
      "Validation error: 0.8%\n",
      "Step 7400 (epoch 8.61), 12.3 ms\n",
      "Minibatch loss: 1.700, learning rate: 0.006634\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.7%\n",
      "Step 7500 (epoch 8.73), 12.1 ms\n",
      "Minibatch loss: 1.693, learning rate: 0.006634\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.8%\n",
      "Step 7600 (epoch 8.84), 12.6 ms\n",
      "Minibatch loss: 1.772, learning rate: 0.006634\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 0.9%\n",
      "Step 7700 (epoch 8.96), 12.1 ms\n",
      "Minibatch loss: 1.666, learning rate: 0.006634\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 7800 (epoch 9.08), 11.9 ms\n",
      "Minibatch loss: 1.657, learning rate: 0.006302\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 7900 (epoch 9.19), 13.8 ms\n",
      "Minibatch loss: 1.647, learning rate: 0.006302\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.8%\n",
      "Step 8000 (epoch 9.31), 11.7 ms\n",
      "Minibatch loss: 1.649, learning rate: 0.006302\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.9%\n",
      "Step 8100 (epoch 9.43), 12.0 ms\n",
      "Minibatch loss: 1.639, learning rate: 0.006302\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.8%\n",
      "Step 8200 (epoch 9.54), 12.1 ms\n",
      "Minibatch loss: 1.619, learning rate: 0.006302\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.8%\n",
      "Step 8300 (epoch 9.66), 12.0 ms\n",
      "Minibatch loss: 1.612, learning rate: 0.006302\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.8%\n",
      "Step 8400 (epoch 9.77), 12.4 ms\n",
      "Minibatch loss: 1.598, learning rate: 0.006302\n",
      "Minibatch error: 0.0%\n",
      "Validation error: 0.8%\n",
      "Step 8500 (epoch 9.89), 12.6 ms\n",
      "Minibatch loss: 1.617, learning rate: 0.006302\n",
      "Minibatch error: 1.6%\n",
      "Validation error: 0.8%\n",
      "Test error: 0.9%\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Job state: succeeded ExitCode: 0\n"
     ]
    }
   ],
   "source": [
    "utilities.wait_for_job_completion(client, resource_group, job_name, cluster_name, 'stdOuterr', 'stdout-wk-0.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Download stdout.txt and stderr.txt files for the Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading https://batchaipablo.file.core.windows.net/batchailab1/b1395605-1fe9-4af4-b3ff-82a4725a3791/batchai_rg/jobs/cntk_02_13_2018_231620/6dc40e3b-61a3-402d-9a16-32667c669a70/stderr-wk-0.txt?sv=2016-05-31&sr=f&sig=VWloNYk7WTt2V3EAAyhTAd9677wzoXLkXCtGgltZYEo%3D&se=2018-02-14T00%3A24%3A31Z&sp=rl ...Done\n",
      "Downloading https://batchaipablo.file.core.windows.net/batchailab1/b1395605-1fe9-4af4-b3ff-82a4725a3791/batchai_rg/jobs/cntk_02_13_2018_231620/6dc40e3b-61a3-402d-9a16-32667c669a70/stdout-wk-0.txt?sv=2016-05-31&sr=f&sig=rVvrLbHwuHI7smfJ%2FP9hRhMPA0gQFUEg7sfiE4eyyrI%3D&se=2018-02-14T00%3A24%3A31Z&sp=rl ...Done\n",
      "All files downloaded\n"
     ]
    }
   ],
   "source": [
    "files = client.jobs.list_output_files(resource_group, job_name, models.JobsListOutputFilesOptions(\"stdOuterr\")) \n",
    "for f in list(files):\n",
    "    utilities.download_file(f.download_url, f.name)\n",
    "print(\"All files downloaded\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = client.jobs.delete(resource_group, job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Cluster\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the cluster using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "_ = client.clusters.delete(resource_group, cluster_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Delete File Share\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the file share completely with all files using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "service = FileService(storage_account_name, storage_account_key)\n",
    "service.delete_share(azure_file_share_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}