Meetup13-End-to-End-ML-Pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align=\"center\">End to End Machine Learning Pipeline</h1>\n",
    "<h2 align=\"center\">Ingestion, ETL, Modeling, Operationalization</h2>\n",
    "<h3 align=\"center\">Meetup DFW Data & AI - Microsoft</h3>\n",
    "## Setting Up Environment "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Deploy the Linux Deep Learning VM in Azure "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to complete this notebook, you must deploy a Linux DLVM (Deep Learning VM) in your azure subscription. Click [HERE](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/microsoft-ads.dsvm-deep-learning?tab=Overview), then click on GET IT NOW and once you are directed to the Azure portal click CREATE.\n",
    "\n",
    "**In Basics blade:**<br>\n",
    "**Name:** meetupdsvmgpu <br>\n",
    "**Select OS:** Linux <br>\n",
    "**Username:** sshuser<br>\n",
    "**Password:** Passw0rd.1!!<br>\n",
    "**Resource Group:** meetupdsvmgpu_rg <br>\n",
    "**Location:** Pick among East US or West US 2<br>\n",
    "\n",
    "**In Settings blade:**\n",
    "Leave as is\n",
    "\n",
    "The Deep Learning Virtual Machine (DLVM) is a specially configured variant of the Data Science Virtual Machine(DSVM) to make it easier to use GPU-based VM instances for training deep learning models. It is supported on Windows 2016, or the Ubuntu Data Science Virtual Machine and shares the same core VM images (and hence all the rich toolset) as the DSVM. We also provide end-to-end AI samples for image and text understanding. The deep learning virtual machine also makes the rich set of tools and samples on the DSVM more easily discoverable. In terms of the tooling, the Deep Learning Virtual Machine provides several popular deep learning frameworks, tools to acquire and pre-process image, textual data. \n",
    "\n",
    "\n",
    "The DLVM contains several tools for AI including popular GPU editions of deep learning frameworks like Microsoft Cognitive Toolkit, TensorFlow, Keras, Caffe2, Chainer; tools to acquire and pre-process image, textual data, tools for data science modeling and development activities such as Microsoft R Server Developer Edition, Anaconda Python, Jupyter notebooks for Python and R, IDEs for Python and R , SQL database and many other data science and ML tools. \n",
    "\n",
    "\n",
    "The DLVM runs on Azure GPU NC-series VM instances. These GPUs use discrete device assignment, resulting in performance close to bare-metal, and are well-suited to deep learning problems.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) SSH into the VM and git clone the meetup repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "$ ssh sshuser@YOUR.VM.IP.ADDRESS\n",
    "\n",
    "$ cd notebooks\n",
    "\n",
    "$ git clone https://github.com/pablomarin/Meetups-Data-AI-DFW.git\n",
    "\n",
    "$ az login\n",
    "\n",
    "# make sure you pick the right subscription to work with (IsDefault)\n",
    "\n",
    "$ az account list --output table\n",
    "\n",
    "# If the subscription you want to work with is not default, then pick it by doing:\n",
    "\n",
    "$ az account set --subscription \"YOUR SUB ID\"\n",
    "\n",
    "# create a new conda environment with Python 3.6, numpy and cython\n",
    "$ conda create -n azml Python=3.6 cython numpy\n",
    "\n",
    "# Activate the conde environment\n",
    "$ source activate azml\n",
    "\n",
    "# install azure-cli\n",
    "(azml) $ pip install azure-cli sklearn tensorflow-gpu cntk-gpu keras matplotlib pandas seaborn livelossplot\n",
    "\n",
    "# install or update azureml meta-package\n",
    "(azml) $ pip install --upgrade azureml-sdk\n",
    "\n",
    "# add azml conda environment as a new Jupyter Kernel\n",
    "(azml) $ python -m ipykernel install --user --name azml --display-name \"AzureML\"\n",
    "\n",
    "# Now change the kernel on the LAB notebook, to AzureML\n",
    "\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Open the Jupyter notebook from your VM on your local browser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> https://YOUR.VM.IP.ADDRESS:8000 <br>\n",
    "> Login with your VM username and password<br>\n",
    "> Go to the ***Meetups-Data-AI-DFW folder***<br>\n",
    "> Open the Notebook:***Meetup13-End-to-End-ML-Pipeline.ipynb***<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Architecture of what we are going to do today\n",
    "\n",
    "<div style=\"width:100%; margin-left:auto; margin-right:auto; margin-bottom:5px; margin-top:17px;\">\n",
    "<img src=\"image/AML-Pipeline.png\" alt=\"IMAGE\" /><br>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to cover: <br>\n",
    "- How to extract time series data from an On-Premise SQL Server using Azure Data Factory\n",
    "- Perform ETL (Extract, Transform and Load) and Data Analysis using DSVM\n",
    "- Azure Machine Learning SDK (private preview), to track Machine Learning experiments and putting model into Production"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) What is Azure Data Factory?\n",
    "Source: https://azure.microsoft.com/en-us/services/data-factory/<br>\n",
    "\n",
    "Create, schedule, and manage your data integration at scale with Azure Data Factory—a hybrid data integration (ETL) service. Work with data wherever it lives, in the cloud or on-premises, with enterprise-grade security.\n",
    "\n",
    "Accelerate your data integration projects by taking advantage of over 70 available [data source connectors](https://docs.microsoft.com/en-us/azure/data-factory/v1/data-factory-data-movement-activities). Use the graphical user interface to build and manage your data pipelines. Transform raw data into finished, shaped data ready for consumption by business intelligence tools or custom applications. Easily lift your SQL Server Integration Services (SSIS) packages to Azure, and let Azure Data Factory manage your resources for you so you can increase productivity and lower your total cost of ownership (TCO).\n",
    "\n",
    "\n",
    "### What can you do with Azure Data Factory?\n",
    "\n",
    "Integrate structured and unstructured data from on-premises and cloud data sources, transform data in big data and data warehouse engines, then load your data marts into Azure SQL Data Warehouse for business analytics.\n",
    "\n",
    "\n",
    "<div style=\"width:100%; margin-left:auto; margin-right:auto; margin-bottom:5px; margin-top:17px;\">\n",
    "<img src=\"https://azurecomcdn.azureedge.net/mediahandler/acomblog/media/Default/blog/29515ffa-a917-4ebb-b326-f6655cab3bda.png\" alt=\"IMAGE\" /><br>\n",
    "</div>\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## 2) Where to do ETL and Data Analysis?\n",
    "\n",
    "### Small Data:  Data Science VM. \n",
    "Source: https://aka.ms/dsvm/alltools\n",
    "\n",
    "<div style=\"width:100%; margin-left:auto; margin-right:auto; margin-bottom:5px; margin-top:17px;\">\n",
    "<img src=\"https://azurecomcdn.azureedge.net/cvt-569abacd3eadc3d5032bef3afa7f751fe0d36630318be7d55ce9732a37e0c116/images/page/services/virtual-machines/data-science-virtual-machines/diagram-1.jpg\" alt=\"IMAGE\" /><br>\n",
    "</div>\n",
    "\n",
    "\n",
    "### Big Data: Azure Databricks\n",
    "Source: https://azure.microsoft.com/en-us/services/databricks/\n",
    "\n",
    "Azure Databricks is an Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform. Designed with the founders of Apache Spark, Databricks is integrated with Azure to provide one-click setup, streamlined workflows, and an interactive workspace that enables collaboration between data scientists, data engineers, and business analysts.\n",
    "\n",
    "<div style=\"width:100%; margin-left:auto; margin-right:auto; margin-bottom:5px; margin-top:17px;\">\n",
    "<img src=\"https://docs.microsoft.com/en-us/azure/azure-databricks/media/what-is-azure-databricks/azure-databricks-overview.png\" alt=\"IMAGE\" /><br>\n",
    "</div>\n",
    "\n",
    "\n",
    "## 3) What is the Azure Machine Learning python SDK?\n",
    "\n",
    "Azure Machine Learning service (preview) is an integrated, end-to-end data science and advanced analytics solution for professional data scientists to prepare data, develop experiments, and deploy models at cloud scale.<br>\n",
    "\n",
    "<div style=\"width:100%; margin-left:auto; margin-right:auto; margin-bottom:5px; margin-top:17px;\">\n",
    "<img src=\"./image/aml-taxonomy.png\" alt=\"IMAGE\" /><br>\n",
    "</div>\n",
    "\n",
    "\n",
    "<div style=\"width:100%; margin-left:auto; margin-right:auto; margin-bottom:5px; margin-top:17px;\">\n",
    "<img src=\"https://docs.microsoft.com/en-us/azure/machine-learning/desktop-workbench/media/model-management-overview/modelmanagementworkflow.png\" alt=\"IMAGE\" /><br>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, enough theory, let's run some code.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 4) LAB 1 - End-To-End ML Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Click [here](Meetup13-Lab-EndToEndMLPipelime.ipynb) to open the notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}