013_parallelization_and_gpu_analytics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/rzl-ds/gu511/blob/master/013_parallelization_and_gpu_analytics.ipynb\" target=\"_parent\">\n",
    "    <img src=\"https://colab.research.google.com/assets/colab-badge.svg\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# parallelization and `gpu` analytics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## the back-story: parallelization\n",
    "\n",
    "the code we write is (generally speaking) sequential.\n",
    "\n",
    "```python\n",
    "x = 100\n",
    "y = x * 4\n",
    "print(x + y)\n",
    "for i in range(x):\n",
    "    print(i ** 2)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "programming this way matches our way of reading and writing, and the way we tend to break down problems logically. we write out an ordered list of deterministic steps we want our computer to execute and the order we want them to execute those steps and we let 'er rip\n",
    "\n",
    "at the lowest levels, the machinery implementing our programs is also performing those basic computational actions sequentially as well. these calculations are sent to the central processing unit (`cpu`) and that unit assigns that calculation to one of it's little local workers (a \"core\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "sometimes, though, a thing we tell the computer to do is kind of a dud, and the `cpu` / core sits waiting on some *other* thing (loading something from RAM, talking to another part of the computer, etc) to complete before it can go on.\n",
    "\n",
    "computers *hate* this stuff. they are incredibly impatient. they're *busy*, don't you get that?\n",
    "\n",
    "it's really just that they want to please, don't take it too personally"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and it turns out that we actually do this... a lot. especially in data science, where many of the things we do are *embarrassingly parallel* -- that is, we are asking a computer to do simple things over and over and over again with slightly different parameters or conditions. some examples:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**hyperparameter optimization**: for my model, try each of these N hyperparameter sets\n",
    "\n",
    "```py\n",
    "for hyperparams in hyperparameter_list:\n",
    "    clf = sklearn.mymodel(**hyperparams)\n",
    "    clf.fit(train, test)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**k-fold cross validation**: for each fold, train a model\n",
    "\n",
    "```python\n",
    "kf = sklearn.model_selection.KFold(n_splits=10)\n",
    "for train, test in kf.split(X):\n",
    "    clf = sklearn.mymodel()\n",
    "    clf.fit(train, test)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**vectorized computations**: I need to calculate a gradient w.r.t. each of these N dimensions\n",
    "\n",
    "```python\n",
    "for i in range(w):\n",
    "    grad[i] = gradient_wrt(loss, i, w)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**many tasks on the same data**: for each of these 1000 random forests, train on this same dataset\n",
    "\n",
    "```python\n",
    "for forest in random_forest:\n",
    "    forest.fit(train, test)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**one task on many pieces of data**: for each of these million `mnist` digit images, calculate the loss of my convolutional neural net model\n",
    "\n",
    "```python\n",
    "for (image, label) in mnist_images:\n",
    "    total_loss += my_loss(mymodel, image, loss)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**iterations**: after you get my cnn loss, backpropagate changes to my coefficients and do it all over again... a few million times\n",
    "\n",
    "```python\n",
    "for i in range(1_000_000):\n",
    "    mymodel.forward_prop()\n",
    "    mymodel.backward_prop()\n",
    "    mymodel.summary()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**every matrix multiplication step**: there are a few of these\n",
    "\n",
    "```python\n",
    "for epoch in epochs:\n",
    "    for batch in batches:\n",
    "        for layer in layers:\n",
    "            for node in layer:\n",
    "                batch * node.weights + node.biases\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**example preparation masochism**: multiply this linear regression matrix against the `iris` dataset every day for the rest of my life"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "of course, the problem isn't new nor unique to data scientists, so over time computer engineers have dedicated tons of time and resources to making extremely efficient use of our precious and limited computational resources. this has included:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### bit-level parallelism\n",
    "\n",
    "increasing the number of bits a single computation can act on. this makes numeric computations with \"large\" numbers much faster (or equivalently, makes the definition of \"large\" much larger)\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <th>8 bits</th>\n",
    "        <th>16 bits</th>\n",
    "        <th>32 bits</th>\n",
    "        <th>64 bits</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td><img width=\"50px\" src=\"https://i.pinimg.com/originals/6e/45/f8/6e45f874ba6f11d8ae1f0974be496f76.jpg\"></td>\n",
    "        <td><img width=\"200px\" src=\"https://i.ytimg.com/vi/2r6T0vNN2dY/hqdefault.jpg\"></td>\n",
    "        <td><img width=\"200px\" src=\"https://videochums.com/article/digging-up-the-mega-man-legends-series.jpg\"></td>\n",
    "        <td><img width=\"400px\" src=\"https://hips.hearstapps.com/hmg-prod.s3.amazonaws.com/images/screen-shot-2018-10-03-at-5-46-34-pm-1538603301.png?resize=480:*\"></td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### instruction level parallelism\n",
    "\n",
    "if we break up every computation (aka \"thing the `cpu` can do\") into distinct parts (e.g. fetch instructions, decode them, execute them, store results in memory, and write the results to a local register), and have different physical components of the processor do each of those tasks, we can process several computations at a time by\n",
    "\n",
    "+ fetch the 1st instruction\n",
    "+ decode the 1st, fetch the 2nd\n",
    "+ execute the 1st, decode the 2nd, fetch the 3rd\n",
    "+ ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##### task parallelism\n",
    "\n",
    "increase the number of cores you have to work with (multiprocessing), or create cores that are internally capable of working on multiple process at the same time (multithreading)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we can multithread pretty easily in `python`. let's look at the performance of a function with random waiting when we call it sequentially vs when we have parallel threads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import time\n",
    "\n",
    "def make_rest_request():\n",
    "    r = .01 * random.random()\n",
    "    time.sleep(r)\n",
    "    return r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "for i in range(1_000):\n",
    "    make_rest_request()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and now let's see what happens if we *thread* those calculations (that is, tell our `cpu` to work on both things at the same time instead of sequentially)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import threading\n",
    "import queue"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# create threads\n",
    "threads = [threading.Thread(target=make_rest_request)\n",
    "           for i in range(1_000)]\n",
    "# start them together, then \"end\" them together with .join\n",
    "for t in threads:\n",
    "    t.start()\n",
    "\n",
    "for t in threads:\n",
    "    t.join()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "perhaps the most illustrative fact here is in the output from `%%time` -- let's compare the results\n",
    "\n",
    "| measurement | no threads | threads |\n",
    "|-|-|-|\n",
    "| total cpu time (ms) | 48.4 | 130 |\n",
    "| wall time (ms) | 6000 | 150 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the \"total cpu time\" measurement is telling us exactly how much time our computer spent performing calculations for us whereas the wall time is the time we human users waited from start to finish. notice that our threading method took almost 3 times as long as the non-threading *from the cpu's perspective* (overhead of creating the `threading.Thread` objects, mostly), but because they were executed in parallel there is almost no difference between that time and how long we (the user) wait for the results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "note that in the above we didn't actually *collect* the values returned by our `make_rest_request` function. to do that, we need to do something a *little* more complicated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "def make_rest_request_and_add_to_queue(q):\n",
    "    q.put(make_rest_request())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "q = queue.Queue()\n",
    "threads = [threading.Thread(target=make_rest_request_and_add_to_queue, args=(q,))\n",
    "           for i in range(1_000)]\n",
    "\n",
    "for t in threads:\n",
    "    t.start()\n",
    "\n",
    "for t in threads:\n",
    "    t.join()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "results = []\n",
    "while not q.empty():\n",
    "    results.append(q.get())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "len(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "results[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## single-machine limits\n",
    "\n",
    "the advances above occurred slowly but surely from the 1960s on into the 2010s, and were focused on squeezing every last drop of performance out of single `cpu`s. people were also trying to make those `cpu`s faster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and they have had exponential success... up until very recently\n",
    "\n",
    "<br><img src=\"http://www.extremetech.com/wp-content/uploads/2012/02/CPU-Scaling.jpg\" width=\"800px\"></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### go tall\n",
    "\n",
    "for a long time, our computer's ability to generate parallelizable tasks far outstripped the capabilities of even the fastest `cpu`s. as a result, the answer to whether or not we needed more `cpu` power was pretty much always the same:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<div align=\"center\"><img src=\"https://i.imgur.com/0fWjvQ0.png\" width=\"500px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### go wide\n",
    "\n",
    "having more cores on my single computer's processor was and still is great -- I'm able to do everything faster.\n",
    "\n",
    "but what if I had two computers...\n",
    "\n",
    "<br><div align=\"center\"><img src=\"http://i208.photobucket.com/albums/bb106/Allscifi/Sherlock/Sherlock17.png\" width=\"800px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the decision to scale *outward* instead of *upward* in computational resources has been the driving force behind a ton of the most powerful advances in computational methods. the move to \"the cloud\" has largely been about this exact concept: I can dynamically update my resources (size of storage, or number of `cpu` cores, and therefore number of simultaneous computations I can execute) by adding additional networked computers.\n",
    "\n",
    "we will revisit this approach in the next lecture (`hadoop` and `spark`, in particular), so we won't belabor it now"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "but there is one other option"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## go game\n",
    "\n",
    "<br><div align=\"center\"><img src=\"https://mygaming.co.za/news/wp-content/uploads/2017/06/AcerForGaming-01.jpg\" width=\"900px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the increasingly graphical nature of computations -- frame-by-frame rendering of screens for video games, desktops, mobile phones, webpages, etc -- meant that `cpu`s were spending a lot of their time performing these specific niche computations. this incentives engineers and companies (e.g. nvidia) to sink research and development into constructing extremely efficient hardware and software for offloading that subset of calculations to a new processing unit, and the `g`raphical `p`rocessing `u`nits (or `gpu`) was born."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the basic idea is pretty simple: take any *extremely common* operation `x` and get it off of the cpu's plate by creating an `x`pu that is \"dumb\" (it only knows how to do `x`) but *extremely fast* at doing `x`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "because the actions that the `gpu` performs are much simpler, it is possible for the `gpu` architecture to be much simpler as well. this allows us to put **a lot** more cores on one `gpu`:\n",
    "\n",
    "<br><div align=\"center\"><img src=\"https://frontline.nextron.no/Content/Images/CPUcoresVsGPUcores_web.png\" width=\"900px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and the performance shows:\n",
    "\n",
    "<br><div align=\"center\"><img src=\"https://www.karlrupp.net/wp-content/uploads/2013/06/flops-per-cycle-sp.png\" width=\"1000px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "in addition, the actual cost of `gpu` analytics -- while not cheap by any stretch -- is considerably less:\n",
    "\n",
    "<br><div align=\"center\"><img src=\"https://cdn-images-1.medium.com/max/2000/1*Hsu_MSC58ZR2Dl7QvT8Ycg.png\" width=\"1000px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "back to what we can do with `gpu`s, though. take, for example the 3D world created in a video game. internally, a single frame is represented by a handful of points (vectors in some space) and rules for drawing the frame based on those vectors.\n",
    "\n",
    "frame-to-frame changes (e.g. moves made via the controller) are translations, rescalings, or rotations of those vectors, and each of those actions can (and is) expressed as a matrix multiplication\n",
    "\n",
    "<br><div align=\"center\"><img src=\"https://www.alanzucconi.com/wp-content/uploads/2016/02/2D_affine_transformation_matrix.svg_.png\" width=\"500px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "there are a few other things I can think of that are basically just matrix multiplication:\n",
    "\n",
    "<br><div align=\"center\"><img src=\"https://cdn-images-1.medium.com/max/1600/1*UKIHA2AHtB9WPG-KrfwSZg.png\" width=\"500px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "to be serious, though, `gpu`s excel at exactly the sorts of things we tend to do in data science and analytics:\n",
    "\n",
    "+ floating point arithmetic\n",
    "+ dense linear algebra\n",
    "+ doing the same calculation on many different data points (`S`ingle `I`nstruction `M`ultiple `D`ata, `SIMD`)\n",
    "    + training or predicting on many records\n",
    "    + monte carlo simulations\n",
    "    + hyperparameter tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the current age of `gpu` data science was kicked off in the late 2000s, in particular with [this paper](https://ai.stanford.edu/~ang/papers/icml09-LargeScaleUnsupervisedDeepLearningGPU.pdf) in which Andrew Ng et al. demonstrated a 70x speed up on state of the art unsupervised learning methods when using `gpu`s instead of `cpu`s.\n",
    "\n",
    "for deep learning, 70x times faster doesn't necessarily excite as much as the ability to train or predict on 70x as much **data** in the allocated time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">PAUSE FOR ZOOM BREAK</div>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## applications and frameworks\n",
    "\n",
    "there have been quite a few software products put out that attempt to leverage `gpu` performance for analytical tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### `gpu` databases\n",
    "\n",
    "many of the parallelization actions we talked about in our database lectures are reproducible with `gpu`s. for example, we could take a large aggregation query, distributing portions of the underlying data to different `gpu` cores, performing those sub-calculations in parallel, and then aggregating."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the current players in the `gpu` database field are\n",
    "\n",
    "+ in memory (*extremely* fast, all records in data up to several Terabytes)\n",
    "    + [`kinetica`](https://www.kinetica.com/gpu-database/)\n",
    "    + [`omnisci` (formerly `mapd`)](https://www.omnisci.com/)\n",
    "    + [`brytlyt`](https://www.brytlyt.com/) (this is actually in `gpu` ram, so even faster in theory)\n",
    "+ on disk (still very fast, but can handle data size beyond memory limits)\n",
    "    + [`sqream db`](https://sqream.com/product/)\n",
    "    + [`blazingdb`](https://blazingdb.com/#/)\n",
    "+ nosql\n",
    "    + [`blazegraph` aka amazon `neptune`](https://www.blazegraph.com/), a high-performance graph database that implements `gpu` acceleration (i.e. isn't *wholly* a `gpu` database)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### bitcoin mining\n",
    "\n",
    "I won't go into this much, but much of the recent demand driving research in `gpu`s has been driven by bitcoin mining. if you're interested in starting, for the low-low price of only 30K USD you can get this beautiful rig off etsy of all places:\n",
    "\n",
    "<a href=\"https://www.etsy.com/listing/595797685/exclusive-titan-rtx-8gpu-rig-worlds-most\"><img src=\"https://i.etsystatic.com/11426206/r/il/59f0ca/1485448756/il_794xN.1485448756_4ok1.jpg\" width=\"800px\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### analytics frameworks\n",
    "\n",
    "the real reason we are talking about `gpu` analytics is the ability to utilize `gpu`s in our code (e.g. instructing our deep neural net to be trained on our `gpu`.\n",
    "\n",
    "there is a lot of software required to convert our extremely high-level `python` code down to instructions consumable by your `gpu`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "*it doesn't **have** to be `nvidia` products, but they currently dominate the market*\n",
    "\n",
    "<br><div align=\"center\"><img src=\"http://drive.google.com/uc?export=view&id=1M3LZQRI8nfCscnyL_h7xjKi4i9e8lo1t\" width=\"1000px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "unless you are planning on writing your code in `c++` (and writing it better than google's world-class fleet of software engineers), you are probably going to spend most of your deep learning programming time in the `tensorflow` and `keras` region. we will cover these two libraries in more detail in the next lecture.\n",
    "\n",
    "for now it suffices to say that there are a fair number of moving pieces between you and a `gpu` (just as there are between you and a `cpu`!) and any analytics framework you work with will sit somewhere along this spectrum."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "for example:\n",
    "\n",
    "+ [`gunrock`](https://gunrock.github.io/docs/), a graph algorithm execution library using `gpu` acceleration. you can write `c++` or relatively inconvenient `python` code to perform graph calculations on **enormous** graphs. current state of the art\n",
    "+ [`tensorflow`](https://www.tensorflow.org/), google's `python` `api` for graph calculation execution will compile down to various different lower-level `gpu` libraries (as well as `tpu` libraries)\n",
    "+ [`pytorch`](https://pytorch.org/), an open-source alternative (to `tensorflow`) framework for deep learning\n",
    "+ [`apache mxnet`](https://mxnet.apache.org/), another open-source alternative (to `tensorflow`) framework for deep learning, emphasizing distributed computing in addition to `gpu` analytics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## actually using `gpu`s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "let's start with the most basic question -- do you already have a `gpu`? and if so, what kind is it?\n",
    "\n",
    "the answer to the first question is yes for all of you\n",
    "\n",
    "the answer to the second question is almost certainly \"not the kind we're talking about here\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "almost every laptop has what is called an \"integrated\" `gpu` -- a `gpu` that is physically attached to the `cpu` and included as part of the overall `cpu` infrastructure. this is what handles all graphics calculations for your `cpu` but is a far cry from the much more powerful `gpu`s built by `nvidia` et al."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**<div align=\"center\">exercise: determine if you have a `gpu` on your laptop</div>**\n",
    "\n",
    "+ OS is `windows`\n",
    "    + pre-10: [check your device manager](https://www.addictivetips.com/windows-tips/check-dedicated-gpu/) for a second \"display adapter\"\n",
    "    + 10: you can [do the same as above](https://www.addictivetips.com/windows-tips/check-dedicated-gpu/), or you can get more info with [the directx diagnostic tool](https://www.techjunkie.com/check-graphics-card-windows-10/)\n",
    "+ OS is `mac`\n",
    "    + launch the \"System information\" app, select \"Graphics/Displays\", and [see what items you might have](https://s3.amazonaws.com/quantstart/media/images/qs-cuda-1-0004.png) (you probably only have one)\n",
    "    + if you have a `gpu` with `bus` type \"Built-In\", you have what is called an integrated `gpu` -- it's built into the `cpu` itself. this isn't what we're looking for\n",
    "+ OS is `*nix`\n",
    "    + `lspci | grep ' VGA ' | cut -d\" \" -f 1 | xargs -i lspci -v -s {}`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "most of us won't have a truly amazing dedicated `gpu` on our personal laptop. getting to the state of the art is not cheap! even a relatively straight-forward laptop containing a number of [current deep-learning-capable `gpu`s](https://developer.nvidia.com/cuda-gpus) will typically cost [several thousand dollars](https://www.amazon.com/s?k=laptop+nvidia+gpu&i=electronics) -- though that number is going down daily"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "fortunately we don't need to buy a `gpu` machine outright -- we can just rent one from `aws`.\n",
    "\n",
    "go to https://aws.amazon.com/ec2/instance-types/ and check out the \"accelerated computing\" options -- specifically types P2, P3, and P4\n",
    "\n",
    "*note: p4 new as of 1 week ago: https://aws.amazon.com/blogs/aws/new-gpu-equipped-ec2-p4-instances-for-machine-learning-hpc/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the main differences are the type of card used, whether or not there is a \"link\", and the memory (system and `gpu`). just to compare the cards:\n",
    "\n",
    "| `ec2` type | `gpu` card | `gpu` cores | tensor cores | tflops (double precision), higher is better |\n",
    "|-|-|-|-|-|\n",
    "| p2 | NVIDIA K80 | 2496 | 0 | 1.87 |\n",
    "| p3 | NVIDIA Tesla V100 | 5120 | 640 | 7 |\n",
    "| p4 | NVIDIA A100 Tensor Core | 6912 | 432 | 9.7 |\n",
    "\n",
    "+ \"tensor cores\" are cores specifically built to accelerate specific common operations in deep learning algorithms\n",
    "+ `p4` architecture supposedly doubles the throughput of tensor cores, so those 432 are as effective as approx 864 cores in the v100 architecture\n",
    "+ the jump from `p2` to `p3` is more significant than the jump from `p3` to `p4` (imo)\n",
    "\n",
    "*note: performance numbers taken from [here](https://www.microway.com/knowledge-center-articles/comparison-of-nvidia-geforce-gpus-and-nvidia-tesla-gpus/) and [here](https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth/)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "given the different card qualities, the actual makeup and price of the instance types is\n",
    "\n",
    "| Model | GPUs | GPU Mem (GiB) | vCPU | Mem (GiB) | link | on demand USD / hr | spot USD / hr | fractional savings |\n",
    "|-|-|-|-|-|-|-|-|-|\n",
    "| p2.xlarge | 1 | 12 | 4 | 61 | none | 0.9 | 0.27 | 0.70 |\n",
    "| p3.2xlarge | 1 | 16 | 8 | 61 | none | 3.06 | 0.918 | 0.70 |\n",
    "| p2.8xlarge | 8 | 96 | 32 | 488 | none | 7.2 | 2.16 | 0.70 |\n",
    "| p3.8xlarge | 4 | 64 | 32 | 244 | NVLink | 12.24 | 3.672 | 0.70 |\n",
    "| p2.16xlarge | 16 | 192 | 64 | 732 | none | 14.4 | 4.32 | 0.70 |\n",
    "| p3.16xlarge | 8 | 128 | 64 | 488 | NVLink | 24.48 | 7.344 | 0.70 |\n",
    "| p3dn.24xlarge | 8 | 256 | 96 | 768 | NVLink | 31.212 | 9.3636 | 0.70 |\n",
    "| p4d.24xlarge | 8 | 320 | 96 | 1152 | NVSwitch | 32.7726 | 9.8318 | 0.70 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "we will spin these up, but given the cost we will be very selective about exactly when and why we do! and we will **shut them down** as soon as possible when we do!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `tpu`s\n",
    "\n",
    "google thinks `gpu`s are cool and all, but they kind of have their own thing going"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "back in 2013, google recognized that its demand for `gpu` calculations was growing too fast (in their blog, they suggest they predicted they needed to *double* their capacity, which is wild). the way they chose to address this was by designing their *own* processor -- an alternative to `cpu`s or `gpu`s -- which was not *accidentally* good at deep learning applications like a `gpu`, but was *intentionally* good at it.\n",
    "\n",
    "they called their processor a `t`ensor `p`rocessing `u`nit, or `tpu`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "this type of hardware designed to perform a very context-specific set of calculations is an `A`pplication-`S`pecific `I`ntegrated `C`ircuit, or [`ASIC`](https://en.wikipedia.org/wiki/Application-specific_integrated_circuit). in the case of `tpu`s, the entire unit has separate components each of which only do:\n",
    "\n",
    "1. matrix multiplication / convolution\n",
    "1. aggregation of matrices\n",
    "1. *activation* functions (this is an essential step in neural networks; we will cover in the deep learning lecture)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "the results I'm about to show come from [a google blog post](https://cloud.google.com/blog/products/gcp/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu), so *caveat emptor*, but they are pretty compelling (and use some truly awesome units)\n",
    "\n",
    "<br><div align=\"center\"><img src=\"https://storage.googleapis.com/gweb-cloudblog-publish/images/tpu-3gpcs.max-1200x1200.PNG\" width=\"800px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "and also:\n",
    "\n",
    "<br><div align=\"center\"><img src=\"https://storage.googleapis.com/gweb-cloudblog-publish/original_images/tpu-6tlel.PNG\" width=\"800px\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "for the interested, here are some useful links\n",
    "\n",
    "+ https://kids.kiddle.co/Central_processing_unit (the eli5 version of CPUs, extremely helpful for starting out!)\n",
    "+ https://www.geeksforgeeks.org/multithreading-python-set-1/\n",
    "+ https://en.wikipedia.org/wiki/Parallel_computing\n",
    "    + https://en.wikipedia.org/wiki/Central_processing_unit#Parallelism\n",
    "    + https://en.wikipedia.org/wiki/Multithreading_(computer_architecture)\n",
    "+ https://scicomp.stackexchange.com/questions/943/what-kinds-of-problems-lend-themselves-well-to-gpu-computing\n",
    "+ https://www.quora.com/What-kind-of-math-is-a-graphics-card-better-at-than-a-CPU\n",
    "+ https://blog.inten.to/hardware-for-deep-learning-part-3-gpu-8906c1644664"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<strong><em><div align=\"center\">beware scrambled `gpu`s, they don't work as expected</div></em></strong>\n",
    "<div align=\"center\"><img src=\"https://cdn.statically.io/img/blazepress.com/.image/c_fit,h_600,w_600/MTI4OTkzMzA5OTE2OTYwMDE4/funny-pug-pet-confessions-11.jpg?quality=100&f=auto\" width=\"500px\"></div>\n",
    "\n",
    "# END OF LECTURE\n",
    "\n",
    "next lecture: [deep learning](014_deep_learning.ipynb)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}