Lesson2_AWS_Big_Data_Collection.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Lesson2-AWS-Big-Data-Collection.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "c_Id55m6Jsbu",
        "2iIcfuKHoSog",
        "v7OC_Fh5QV9A",
        "DJxHWoBmcY_4",
        "WESUX507e4jg",
        "bDlM65z5nM0i",
        "ewIs7l_AptvW",
        "PlVk0ly3m8SH",
        "oWeNDZCtp6NP",
        "Hc2I3g2xnH3Y",
        "ET060N0qos-o",
        "c69G5pk8Dh7Z",
        "LYQ8rXU-Doab",
        "czV9D-QSIJGp",
        "UQ6yU_Ryo-ZB",
        "fXpNEXuP6RYH"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/paiml/awsbigdata/blob/master/Lesson2_AWS_Big_Data_Collection.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "chymwgvAoGP9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Lesson2:  Collection"
      ]
    },
    {
      "metadata": {
        "id": "c_Id55m6Jsbu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Pragmatic AI Labs\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "e5p96AqpSDZa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![alt text](https://paiml.com/images/logo_with_slogan_white_background.png)\n",
        "\n",
        "This notebook was produced by [Pragmatic AI Labs](https://paiml.com/).  You can continue learning about these topics by:\n",
        "\n",
        "*   Buying a copy of [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](http://www.informit.com/store/pragmatic-ai-an-introduction-to-cloud-based-machine-9780134863863) from Informit.\n",
        "*   Buying a copy of  [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.amazon.com/Pragmatic-AI-Introduction-Cloud-Based-Learning/dp/0134863860) from Amazon\n",
        "*   Reading an online copy of [Pragmatic AI:Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.safaribooksonline.com/library/view/pragmatic-ai-an/9780134863924/)\n",
        "*  Watching video [Essential Machine Learning and AI with Python and Jupyter Notebook-Video-SafariOnline](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118) on Safari Books Online.\n",
        "* Watching video [AWS Certified Machine Learning-Speciality](https://learning.oreilly.com/videos/aws-certified-machine/9780135556597)\n",
        "* Purchasing video [Essential Machine Learning and AI with Python and Jupyter Notebook- Purchase Video](http://www.informit.com/store/essential-machine-learning-and-ai-with-python-and-jupyter-9780135261095)\n",
        "*   Viewing more content at [noahgift.com](https://noahgift.com/)\n"
      ]
    },
    {
      "metadata": {
        "id": "2iIcfuKHoSog",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Lesson 2.1 Determine the operational characteristics of the collection system"
      ]
    },
    {
      "metadata": {
        "id": "8oYv9z6P6Ukj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Data Ingestion Concepts"
      ]
    },
    {
      "metadata": {
        "id": "v7OC_Fh5QV9A",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Data Lakes"
      ]
    },
    {
      "metadata": {
        "id": "Nt6VTvFVQb-D",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Central Repository** for all data at any scale"
      ]
    },
    {
      "metadata": {
        "id": "8iyQzWR3amgh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![data_lake](https://user-images.githubusercontent.com/58792/49777724-8aef8300-fcb6-11e8-981e-96d14498a801.png)"
      ]
    },
    {
      "metadata": {
        "id": "DJxHWoBmcY_4",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### AWS Lake Formation"
      ]
    },
    {
      "metadata": {
        "id": "X5iz82w_dCJ_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* New Service Announced at Reinvent 2018\n",
        "* Build a secure lake in days...**not months**\n",
        "* Enforce security policies\n",
        "* Gain and manage insights"
      ]
    },
    {
      "metadata": {
        "id": "4GwDZDvXchGJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![aws_lake](https://user-images.githubusercontent.com/58792/49777834-f9ccdc00-fcb6-11e8-84a0-7295a0c69a15.png)"
      ]
    },
    {
      "metadata": {
        "id": "WESUX507e4jg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### AWS Batch (BATCH)"
      ]
    },
    {
      "metadata": {
        "id": "lk-T-dSsfHC0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Example could be Financial Service Trade Analysis"
      ]
    },
    {
      "metadata": {
        "id": "AQfrQvV7h2IA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![financial_services_trade](https://user-images.githubusercontent.com/58792/49778503-64334b80-fcba-11e8-85e7-dcdbfe473cd9.png)"
      ]
    },
    {
      "metadata": {
        "id": "bDlM65z5nM0i",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Using AWS Batch for ML Jobs\n",
        "\n",
        "* *[Watch Video Lesson 11.6:  Use AWS Batch for ML Jobs](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118/9780135261118-EMLA_01_11_06)*\n"
      ]
    },
    {
      "metadata": {
        "id": "gn6fah4M4Sa1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "https://aws.amazon.com/batch/\n",
        "\n",
        "![alt text](https://d1.awsstatic.com/Test%20Images/Kate%20Test%20Images/Dilithium-Diagrams_Visual-Effects-Rendering.ad9c0479c3772c67953e96ef8ae76a5095373d81.png)\n",
        "\n",
        "\n",
        "Example submissions tool\n",
        "\n",
        "```python\n",
        "@cli.group()\n",
        "def run():\n",
        "    \"\"\"Run AWS Batch\"\"\"\n",
        "\n",
        "@run.command(\"submit\")\n",
        "@click.option(\"--queue\", default=\"first-run-job-queue\", help=\"Batch Queue\")\n",
        "@click.option(\"--jobname\", default=\"1\", help=\"Name of Job\")\n",
        "@click.option(\"--jobdef\", default=\"test\", help=\"Job Definition\")\n",
        "@click.option(\"--cmd\", default=[\"uname\"], help=\"Container Override Commands\")\n",
        "def submit(queue, jobname, jobdef, cmd):\n",
        "    \"\"\"Submit a job\"\"\"\n",
        "\n",
        "    result = submit_job(\n",
        "        job_name=jobname,\n",
        "        job_queue=queue,\n",
        "        job_definition=jobdef,\n",
        "        command=cmd\n",
        "    )\n",
        "    click.echo(\"CLI:  Run Job Called\")\n",
        "    return result\n",
        "```"
      ]
    },
    {
      "metadata": {
        "id": "ewIs7l_AptvW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Lambda (EVENTS)"
      ]
    },
    {
      "metadata": {
        "id": "dX_F3p8ipyJu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "* Serverless\n",
        "*   Used in most if not all ML Platforms\n",
        " - DeepLense\n",
        " - Sagemaker\n",
        " - S3 Events\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "PlVk0ly3m8SH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Starting development with AWS Python Lambda development with Chalice\n",
        "\n",
        "* *[Watch Video Lesson 11.3:  Use AWS Lambda development with Chalice](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118/9780135261118-EMLA_01_11_03)*\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "RS4-D7b72XwE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "***Demo on Sagemaker Terminal***\n",
        "\n",
        "https://github.com/aws/chalice\n",
        "\n",
        "*Hello World Example:*\n",
        "\n",
        "```python\n",
        "$ pip install chalice\n",
        "$ chalice new-project helloworld && cd helloworld\n",
        "$ cat app.py\n",
        "\n",
        "from chalice import Chalice\n",
        "\n",
        "app = Chalice(app_name=\"helloworld\")\n",
        "\n",
        "@app.route(\"/\")\n",
        "def index():\n",
        "    return {\"hello\": \"world\"}\n",
        "\n",
        "$ chalice deploy\n",
        "...\n",
        "https://endpoint/dev\n",
        "\n",
        "$ curl https://endpoint/api\n",
        "{\"hello\": \"world\"}\n",
        "```\n",
        "\n",
        "References:\n",
        "\n",
        "[Serverless Web Scraping Project](https://github.com/noahgift/web_scraping_python)"
      ]
    },
    {
      "metadata": {
        "id": "oWeNDZCtp6NP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### [Demo] Deploying Hello World Lambda Function"
      ]
    },
    {
      "metadata": {
        "id": "N8HXfU4GqUcq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "Hc2I3g2xnH3Y",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Using Step functions with AWS\n",
        "\n",
        "* *[Watch Video Lesson 11.5:  Use AWS Step Functions](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118/9780135261118-EMLA_01_11_05)*"
      ]
    },
    {
      "metadata": {
        "id": "316vRqUj3ELe",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "https://aws.amazon.com/step-functions/\n",
        "\n",
        "![Step Functions](https://d1.awsstatic.com/product-marketing/Step%20Functions/AmazonCloudWatchUpdated4.a57e968b08739e170aa504feed8db3761de21e60.png)\n",
        "\n",
        "Example Project:\n",
        "\n",
        "https://github.com/noahgift/web_scraping_python"
      ]
    },
    {
      "metadata": {
        "id": "rExHzr2wqYgy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "[Demo] Step Function"
      ]
    },
    {
      "metadata": {
        "id": "oHl-TEsdoZ5J",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "SZec7zeooabh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Lesson 2.2 Select a collection system that handles the frequency of data change and type of data being ingested"
      ]
    },
    {
      "metadata": {
        "id": "UbvGoIR2ofQM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "qP7Q3bZvoiIR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Lesson 2.3 Identify the properties that need to be enforced by the collection system: order, data structure, metadata, etc."
      ]
    },
    {
      "metadata": {
        "id": "F9Qu60AjoiL3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "lmZ7yW7FonZd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Lesson 2.4 Explain the durability and availability characteristics for the collection approach"
      ]
    },
    {
      "metadata": {
        "id": "KUnc9dM2oncQ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "ET060N0qos-o",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Lesson 2.5 Learn AWS Kinesis Streams"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "TweeLlwR9iFR"
      },
      "cell_type": "markdown",
      "source": [
        "#### Kinesis Ad Tech Pipeline\n",
        "\n",
        "![Ad Tech Pipeline](https://user-images.githubusercontent.com/58792/49441021-285c2b80-f77a-11e8-82e2-da9006dc4c6d.png)"
      ]
    },
    {
      "metadata": {
        "colab_type": "text",
        "id": "RSpPmOll9iFO"
      },
      "cell_type": "markdown",
      "source": [
        "#### Kinesis IoT\n",
        "\n",
        "![Kinesis IoT](https://user-images.githubusercontent.com/58792/49441101-5e011480-f77a-11e8-9727-4f7706361a08.png)"
      ]
    },
    {
      "metadata": {
        "id": "c69G5pk8Dh7Z",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Kinesis (STREAMING)"
      ]
    },
    {
      "metadata": {
        "id": "GaH-jrqEIRtp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Solves Three Key Problems**\n",
        "\n",
        "\n",
        "\n",
        "*   Time-series Analytics\n",
        "*   Real-time Dashboards\n",
        "*   Real-time Metrics\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "1skrZ0TuDpVq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "##### Kinesis Analytics Workflow\n",
        "![Kinesis Analytics](https://user-images.githubusercontent.com/58792/49440264-02ce2280-f778-11e8-9d7e-149819e74807.png)"
      ]
    },
    {
      "metadata": {
        "id": "LYQ8rXU-Doab",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Kinesis Real-Time Log Analytics Example\n",
        "\n",
        "![Real-Time Log Analytics](https://user-images.githubusercontent.com/58792/49440433-7cfea700-f778-11e8-8cd5-55999cb7713c.png)"
      ]
    },
    {
      "metadata": {
        "id": "htJ-ngnWotB8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "czV9D-QSIJGp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### [Demo] Kinesis"
      ]
    },
    {
      "metadata": {
        "id": "Vq64AVvuILKx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "UQ6yU_Ryo-ZB",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Lesson 2.6 Learn AWS Kinesis Firehose"
      ]
    },
    {
      "metadata": {
        "id": "fXpNEXuP6RYH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Kinesis Features"
      ]
    },
    {
      "metadata": {
        "id": "LV-LMaYT-wuO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "[Kinesis FAQ](https://aws.amazon.com/kinesis/data-streams/faqs/)\n",
        "\n",
        "* Processes Data in Real-Time\n",
        "* Can process hundreds of TBs an hour\n",
        "* Example inputs are:  \n",
        " - logs\n",
        " - financial transactions\n",
        " * Streaming Data"
      ]
    },
    {
      "metadata": {
        "id": "Tnye93h6h0jn",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip install -q sensible"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "gCR4VAgV75DC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import boto3\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dQVPZr-c799V",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import asyncio\n",
        "import time\n",
        "import datetime\n",
        "import uuid\n",
        "import boto3\n",
        "import json\n",
        "from sensible.loginit import logger\n",
        "\n",
        "LOG = logger(__name__)\n",
        "\n",
        "def firehose_client(region_name=\"us-east-1\"):\n",
        "    \"\"\"Kinesis Firehose client\"\"\"\n",
        "\n",
        "    firehose_conn = boto3.client(\"firehose\", region_name=region_name)\n",
        "    extra_msg = {\"region_name\": region_name, \"aws_service\": \"firehose\"}\n",
        "    LOG.info(\"firehose connection initiated\", extra=extra_msg)\n",
        "    return firehose_conn\n",
        "\n",
        "async def put_record(data,\n",
        "            client,\n",
        "            delivery_stream_name=\"aws-ml-cert\"):\n",
        "    \"\"\"\n",
        "    See this:\n",
        "        http://boto3.readthedocs.io/en/latest/reference/services/\n",
        "        firehose.html#Firehose.Client.put_record\n",
        "    \"\"\"\n",
        "    extra_msg = {\"aws_service\": \"firehose\"}\n",
        "    LOG.info(f\"Pushing record to firehose: {data}\", extra=extra_msg)\n",
        "    response = client.put_record(\n",
        "        DeliveryStreamName=delivery_stream_name,\n",
        "        Record={\n",
        "            'Data': data\n",
        "        }\n",
        "    )\n",
        "    return response\n",
        "\n",
        "\n",
        "def gen_uuid_events():\n",
        "    \"\"\"Creates a time stamped UUID based event\"\"\"\n",
        "\n",
        "    current_time = 'test-{date:%Y-%m-%d %H:%M:%S}'.format(date=datetime.datetime.now())\n",
        "    event_id = str(uuid.uuid4())\n",
        "    event = {event_id:current_time}\n",
        "    return json.dumps(event)\n",
        "\n",
        "def send_async_firehose_events(count=100):\n",
        "    \"\"\"Async sends events to firehose\"\"\"\n",
        "\n",
        "    start = time.time() \n",
        "    client = firehose_client()\n",
        "    extra_msg = {\"aws_service\": \"firehose\"}\n",
        "    loop = asyncio.get_event_loop()\n",
        "    tasks = []\n",
        "    LOG.info(f\"sending aysnc events TOTAL {count}\",extra=extra_msg)\n",
        "    num = 0\n",
        "    for _ in range(count):\n",
        "        tasks.append(asyncio.ensure_future(put_record(gen_uuid_events(), client)))\n",
        "        LOG.info(f\"sending aysnc events: COUNT {num}/{count}\")\n",
        "        num +=1\n",
        "    loop.run_until_complete(asyncio.wait(tasks))\n",
        "    loop.close()\n",
        "    end = time.time()  \n",
        "    LOG.info(\"Total time: {}\".format(end - start))\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "nhWXJvwc8hNp",
        "colab_type": "code",
        "outputId": "6f5cd3d1-bb57-4381-812e-bd9dfad79071",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 428
        }
      },
      "cell_type": "code",
      "source": [
        "send_async_firehose_events(10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2018-12-14 18:49:58,211 - __main__ - INFO - firehose connection initiated\n",
            "2018-12-14 18:49:58,213 - __main__ - INFO - sending aysnc events TOTAL 10\n",
            "2018-12-14 18:49:58,214 - __main__ - INFO - sending aysnc events: COUNT 0/10\n",
            "2018-12-14 18:49:58,216 - __main__ - INFO - sending aysnc events: COUNT 1/10\n",
            "2018-12-14 18:49:58,220 - __main__ - INFO - sending aysnc events: COUNT 2/10\n",
            "2018-12-14 18:49:58,221 - __main__ - INFO - sending aysnc events: COUNT 3/10\n",
            "2018-12-14 18:49:58,225 - __main__ - INFO - sending aysnc events: COUNT 4/10\n",
            "2018-12-14 18:49:58,228 - __main__ - INFO - sending aysnc events: COUNT 5/10\n",
            "2018-12-14 18:49:58,231 - __main__ - INFO - sending aysnc events: COUNT 6/10\n",
            "2018-12-14 18:49:58,233 - __main__ - INFO - sending aysnc events: COUNT 7/10\n",
            "2018-12-14 18:49:58,236 - __main__ - INFO - sending aysnc events: COUNT 8/10\n",
            "2018-12-14 18:49:58,237 - __main__ - INFO - sending aysnc events: COUNT 9/10\n",
            "2018-12-14 18:49:58,242 - __main__ - INFO - Pushing record to firehose: {\"23bacac0-6eff-410f-b655-a7faa122b68f\": \"test-2018-12-14 18:49:58\"}\n",
            "2018-12-14 18:49:58,364 - __main__ - INFO - Pushing record to firehose: {\"03b15c89-e0a2-46b8-b8d3-cdd71e2ab140\": \"test-2018-12-14 18:49:58\"}\n",
            "2018-12-14 18:49:58,404 - __main__ - INFO - Pushing record to firehose: {\"5ff296d3-f43a-4e69-b61b-624603a354b4\": \"test-2018-12-14 18:49:58\"}\n",
            "2018-12-14 18:49:58,448 - __main__ - INFO - Pushing record to firehose: {\"7ca78179-7ff7-49d1-bd6c-08ca5569737c\": \"test-2018-12-14 18:49:58\"}\n",
            "2018-12-14 18:49:58,486 - __main__ - INFO - Pushing record to firehose: {\"554e587b-2d59-4e4f-95a3-75e4fb129c60\": \"test-2018-12-14 18:49:58\"}\n",
            "2018-12-14 18:49:58,511 - __main__ - INFO - Pushing record to firehose: {\"30532d9e-6f3a-4366-b6ab-3bf9291c4e0b\": \"test-2018-12-14 18:49:58\"}\n",
            "2018-12-14 18:49:58,548 - __main__ - INFO - Pushing record to firehose: {\"99cdff7d-32d1-424b-886e-9e93bb5e5a03\": \"test-2018-12-14 18:49:58\"}\n",
            "2018-12-14 18:49:58,575 - __main__ - INFO - Pushing record to firehose: {\"8ea0c709-a8a5-4bde-9454-c6dd5691d58a\": \"test-2018-12-14 18:49:58\"}\n",
            "2018-12-14 18:49:58,610 - __main__ - INFO - Pushing record to firehose: {\"99ce9671-7971-415c-9c64-ac2c82f9f478\": \"test-2018-12-14 18:49:58\"}\n",
            "2018-12-14 18:49:58,643 - __main__ - INFO - Pushing record to firehose: {\"dc515833-8474-4377-9514-0f6d6fcfa403\": \"test-2018-12-14 18:49:58\"}\n",
            "2018-12-14 18:49:58,677 - __main__ - INFO - Total time: 0.5094583034515381\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "u106eYBmo-cc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "c-kI49DvpDN9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Lesson 2.7 Use SQS"
      ]
    },
    {
      "metadata": {
        "id": "PbbEnGH3pDRC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    },
    {
      "metadata": {
        "id": "oj_ilFnIpIOa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Lesson 2.8 Create Data Pipelines"
      ]
    },
    {
      "metadata": {
        "id": "uRJ77H4YpIRJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    }
  ]
}