03-Serving-Tensorflow-Locally.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serving TensorFlow Models (locally)\n",
    "\n",
    "In this notebook, we'll look at how to serve the Model we created in the previous notebook. This will give us the ability to share a Model with other users.\n",
    "\n",
    "TensorFlow helpfully includes a \"TensorFlow Serving\" server, that is specialized to serving TensorFlow models, and optimized for use in production. It's robust, and has the ability to serve more than one model, and also versions of a given model\n",
    "\n",
    "In order to use it though, we need to have our model saved in a different format. We touched on this at the end of the last notebook, but it works on SavedModel format models.\n",
    "\n",
    "Luckily, we don't have to lose the portability of HDF5 models, since we can convert between the two formats easily.\n",
    "\n",
    "After doing that, we'll then serve the model with TF Serving from a docker container, and use a supplied webapp to send a hand(or cursor)-drawn model to the served model and see its predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start, let's load the model from before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import tensorflow as tf\n",
    "\n",
    "# Load our saved model\n",
    "model = tf.keras.models.load_model('mnist-model.h5')\n",
    "\n",
    "# Save it out in the SavedModel directory format\n",
    "tf.contrib.saved_model.save_keras_model(model, 'mnist-model')\n",
    "# Note that at time of writing (TF 1.13), the call is listed as deprecated in TF 2.0\n",
    "# replaced with tf.keras.experimental.export_saved_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will create an output directory, which we can inspect\n",
    "\n",
    "It contains a subdirectory for this specific version of the model, and inside several files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -R ./mnist-model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the format that TF Serving expects, and is passed as an argument to the container. We can also examine some data about it using the saved_model_cli command\n",
    "\n",
    "Here we can see that it accepts tensors with a shape (-1, 28, 28, 1). The -1 indicates that the shape is undefined, so we can pass an array of 1 to N tensors with the (28, 28, 1) shape and outputs a single 10-element tensor for each prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!saved_model_cli show --dir \"mnist-model/$(ls mnist-model)\" --tag_set serve --signature_def serving_default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running TF Serving in a container\n",
    "\n",
    "Now on to setting up TensorFlow Serving\n",
    "\n",
    "We run it in a docker container with port 8501 exposed. Port 8501 is the HTTP API\n",
    "\n",
    "In this case, we mount our mnist-model directory as a volume at /models/mnist-model in the container. This simplifies getting the SavedModel into the container. It also supports other storage systems like S3\n",
    "\n",
    "We also need to pass the name of the model (the directory name of the SavedModel) as an environment variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!docker run -d --rm -p 8501:8501 \\\n",
    "    -v \"$(pwd)/mnist-model:/models/mnist-model:Z\" \\\n",
    "    -e MODEL_NAME=mnist-model \\\n",
    "    tensorflow/serving"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF Serving's API\n",
    "\n",
    "TF serving serves a HTTP API, which we can query\n",
    "\n",
    "There are 3 endpoints that we really care about:\n",
    "* GET /v1/models/mnist-model - This is where we can see the model is served from. We can call several methods from it\n",
    "* GET /v1/models/mnist-model/metadata - This shows us metadata about the model\n",
    "* POST /v1/models/mnist-model:predict - This is what we POST a JSON 4d tensor to in order to get a label for each tensor\n",
    "\n",
    "Note that simply querying / on the server doesn't tell you much, giving just a 404"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl localhost:8501/v1/models/mnist-model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# As we can see, this tells us about the input and output shapes\n",
    "!curl localhost:8501/v1/models/mnist-model/metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the webapp\n",
    "\n",
    "Now for the POST, instead of showing it normally, it'd be better to do something nice and interactive. For that purpose, there's an included flask app that we can run in order to show the power of the model interactively.\n",
    "\n",
    "Running this cell will hang indefinitely while the Flask server runs, you'll need to kill it manually from kernel > interrupt in the menus above\n",
    "\n",
    "When ready, go to localhost:5000 in the browser (or click the link in the output) to use the app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!TF_SERVING_URL=\"http://localhost:8501\" python Resources/flask-mnist/flask-mnist.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some tips for the webapp\n",
    "\n",
    "* The image is not automatically centered, try to center it as much as possible\n",
    "* The image isn't scaled, you will want to try to scale it to take up most of the canvas"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}