time_series.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "time_series.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hamzafarooq/time_series/blob/master/time_series.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2Pmxv2ioyCRw"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "colab_type": "code",
        "id": "b-2ShX25yNWf",
        "colab": {}
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pa49bUnKyRgF"
      },
      "source": [
        "# Time series forecasting"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GU8C5qm_4vZb"
      },
      "source": [
        "This tutorial is an introduction to time series forecasting using Recurrent Neural Networks (RNNs). This is covered in two parts: first, you will forecast a univariate time series, then you will forecast a multivariate time series."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7rZnJaGTWQw0",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "import matplotlib as mpl\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import os\n",
        "import pandas as pd\n",
        "\n",
        "mpl.rcParams['figure.figsize'] = (8, 6)\n",
        "mpl.rcParams['axes.grid'] = False"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "TokBlnUhWFw9"
      },
      "source": [
        "## Iowa Dataset\n",
        "This tutorial uses a <a href=\"https://console.cloud.google.com/marketplace/details/iowa-department-of-commerce/iowa-liquor-sales\" class=\"external\">Iowa Liquor Retails Sales</a>.\n",
        "\n",
        "This dataset contains every wholesale purchase of liquor in the State of Iowa by retailers for sale to individuals since January 1, 2012. The State of Iowa controls the wholesale distribution of liquor intended for retail sale, which means this dataset offers a complete view of retail liquor sales in the entire state. The dataset contains every wholesale order of liquor by all grocery stores, liquor stores, convenience stores, etc., with details about the store and location, the exact liquor brand and size, and the number of bottles ordered."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "SeTJb51SKs_W",
        "colab": {}
      },
      "source": [
        "from google.colab import auth\n",
        "auth.authenticate_user()\n",
        "print('Authenticated')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "cLh05lDpJ4wj",
        "colab": {}
      },
      "source": [
        "# Save output in a variable `df`\n",
        "\n",
        "%%bigquery --project predictiondeployment df\n",
        "SELECT \n",
        "  * \n",
        "FROM `bigquery-public-data.iowa_liquor_sales.sales`\n",
        "where store_number  = '2633'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VdbOWXiTWM2T"
      },
      "source": [
        "Let's take a glance at the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ojHE-iCCWIhz",
        "colab": {}
      },
      "source": [
        "df.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rsbUCqS7p69J",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df.describe()\n",
        "from datetime import datetime"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vbLFfG7tqNA3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_single_item_aggregate =df[['date','sale_dollars']]\n",
        "df_single_item_aggregate['date'] = pd.to_datetime(df_single_item_aggregate['date'])\n",
        "#print(type(date_object))\n",
        "#print(date_object) "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D1CEddh4qw_c",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_single_item_aggregate = df_single_item_aggregate.groupby(['date']).sum().rename_axis('date')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "47LioiQWsr-C",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_single_item_aggregate"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ny8GvzBlGOhj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df.date.max()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7-o3Guc8Edq-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "r = pd.date_range(start=df.date.min(), end=df.date.max())\n",
        "r\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0B7hFxsjGcCl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_single_2633=df_single_item_aggregate.set_index('date').reindex(r).fillna(0.0).rename_axis('date')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_1_yxQ75yrAK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_single_2633"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ku9DwHRdIIXE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df_single_2633['sale_dollars']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ym6pFaQcIBVL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from numpy import nan\n",
        "df_single_2633['sale_dollars'] = df_single_2633['sale_dollars'].replace(0, nan)\n",
        "# fill missing values with mean column values\n",
        "df_single_2633.fillna(df_single_2633.mean(), inplace=True)\n",
        "# count the number of NaN values in each column\n",
        "print(df_single_2633.isnull().sum())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gWpZ_CvrJUmb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "df_single_2633"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZIFtGuUqTvBR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import statsmodels.api as sm\n",
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "import matplotlib.pyplot as plt\n",
        "import xgboost as xgb\n",
        "from sklearn.metrics import mean_squared_error, mean_absolute_error\n",
        "import imageio\n",
        "import os\n",
        "from statsmodels.graphics.tsaplots import plot_acf"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WoOEZZkhT8XV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def split_data(data, split_date):\n",
        "    return data[data.index <= split_date].copy(), \\\n",
        "           data[data.index >  split_date].copy()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kOsMKEuH68p-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train, test = split_data(df_single_item_aggregate, '2019-03-01')\n",
        "\n",
        "plt.figure(figsize=(200,10))\n",
        "plt.xlabel('time')\n",
        "plt.ylabel('close')\n",
        "plt.plot(train.index,train)\n",
        "plt.plot(test.index,test)\n",
        "plt.show()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_VCzq15Qov5Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qfbpcV0MWQzl"
      },
      "source": [
        "As you can see above, an observation is recorded every 10 minutes. This means that, for a single hour, you will have 6 observations. Similarly, a single day will contain 144 (6x24) observations. \n",
        "\n",
        "Given a specific time, let's say you want to predict the temperature 6 hours in the future. In order to make this prediction, you choose to use 5 days of observations. Thus, you would create a window containing the last 720(5x144) observations to train the model. Many such configurations are possible, making this dataset a good one to experiment with.\n",
        "\n",
        "The function below returns the above described windows of time for the model to train on. The parameter `history_size` is the size of the past window of information. The `target_size` is how far in the future does the model need to learn to predict. The `target_size` is the label that needs to be predicted."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7AoxQuTrWIbi",
        "colab": {}
      },
      "source": [
        "def univariate_data(dataset, start_index, end_index, history_size, target_size):\n",
        "  data = []\n",
        "  labels = []\n",
        "\n",
        "  start_index = start_index + history_size\n",
        "  if end_index is None:\n",
        "    end_index = len(dataset) - target_size\n",
        "\n",
        "  for i in range(start_index, end_index):\n",
        "    indices = range(i-history_size, i)\n",
        "    # Reshape data from (history_size,) to (history_size, 1)\n",
        "    data.append(np.reshape(dataset[indices], (history_size, 1)))\n",
        "    labels.append(dataset[i+target_size])\n",
        "  return np.array(data), np.array(labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qoFJZmXBaxCc"
      },
      "source": [
        "In both the following tutorials, the first 300,000 rows of the data will be the training dataset, and there remaining will be the validation dataset. This amounts to ~2100 days worth of training data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ia-MPAHxbInX",
        "colab": {}
      },
      "source": [
        "TRAIN_SPLIT = 210\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "EowWDtaNnH1y"
      },
      "source": [
        "Setting seed to ensure reproducibility."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "-x-GgENynHdx",
        "colab": {}
      },
      "source": [
        "tf.random.set_seed(13)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8YEwr-NoWUpV"
      },
      "source": [
        "## Part 1: Forecast a univariate time series\n",
        "First, you will train a model using only a single feature (temperature), and use it to make predictions for that value in the future.\n",
        "\n",
        "Let's first extract only the temperature from the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "nbdcnm1_WIY9",
        "colab": {}
      },
      "source": [
        "uni_data = df_single_2633['sale_dollars']\n",
        "uni_data.index = df_single_2633['date']\n",
        "uni_data.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aQB-46MyWZMm"
      },
      "source": [
        "Let's observe how this data looks across time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ftOExwAqWXSU",
        "colab": {}
      },
      "source": [
        "uni_data.plot(subplots=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ejSEiDqBWXQa",
        "colab": {}
      },
      "source": [
        "uni_data = uni_data.values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-eFckdUUHWmT"
      },
      "source": [
        "It is important to scale features before training a neural network. Standardization is a common way of doing this scaling by subtracting the mean and dividing by the standard deviation of each feature.You could also use a `tf.keras.utils.normalize` method that rescales the values into a range of [0,1]."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mxbIic5TMlxx"
      },
      "source": [
        "Note: The mean and standard deviation should only be computed using the training data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Eji6njXvHusN",
        "colab": {}
      },
      "source": [
        "uni_train_mean = uni_data[:TRAIN_SPLIT].mean()\n",
        "uni_train_std = uni_data[:TRAIN_SPLIT].std()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "clzVRAyexNMV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "uni_data"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8Gob1YJYH0cH"
      },
      "source": [
        "Let's standardize the data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "BO55yRD6H0Dx",
        "colab": {}
      },
      "source": [
        "uni_data = (uni_data-uni_train_mean)/uni_train_std"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "gn8A_nrccKtn"
      },
      "source": [
        "Let's now create the data for the univariate model. For part 1, the model will be given the last 20 recorded temperature observations, and needs to learn to predict the temperature at the next time step. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "aJJ-T49vWXOZ",
        "colab": {}
      },
      "source": [
        "univariate_past_history = 10\n",
        "univariate_future_target = 0\n",
        "\n",
        "x_train_uni, y_train_uni = univariate_data(uni_data, 0, TRAIN_SPLIT,\n",
        "                                           univariate_past_history,\n",
        "                                           univariate_future_target)\n",
        "x_val_uni, y_val_uni = univariate_data(uni_data, TRAIN_SPLIT, None,\n",
        "                                       univariate_past_history,\n",
        "                                       univariate_future_target)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aWpVMENsdp0N"
      },
      "source": [
        "This is what the `univariate_data` function returns."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "feDd95XFdz5H",
        "colab": {}
      },
      "source": [
        "print ('Single window of past history')\n",
        "print (x_train_uni[0])\n",
        "print ('\\n Target sales to predict')\n",
        "print (y_train_uni[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hni3Jt9OMR1_"
      },
      "source": [
        "Now that the data has been created, let's take a look at a single example. The information given to the network is given in blue, and it must predict the value at the red cross."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "qVukM9dRipop",
        "colab": {}
      },
      "source": [
        "def create_time_steps(length):\n",
        "  return list(range(-length, 0))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "QQeGvh7cWXMR",
        "colab": {}
      },
      "source": [
        "def show_plot(plot_data, delta, title):\n",
        "  labels = ['History', 'True Future', 'Model Prediction']\n",
        "  marker = ['.-', 'rx', 'go']\n",
        "  time_steps = create_time_steps(plot_data[0].shape[0])\n",
        "  if delta:\n",
        "    future = delta\n",
        "  else:\n",
        "    future = 0\n",
        "\n",
        "  plt.title(title)\n",
        "  for i, x in enumerate(plot_data):\n",
        "    if i:\n",
        "      plt.plot(future, plot_data[i], marker[i], markersize=10,\n",
        "               label=labels[i])\n",
        "    else:\n",
        "      plt.plot(time_steps, plot_data[i].flatten(), marker[i], label=labels[i])\n",
        "  plt.legend()\n",
        "  plt.xlim([time_steps[0], (future+5)*2])\n",
        "  plt.xlabel('Time-Step')\n",
        "  return plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Pd05iV-UWXKL",
        "colab": {}
      },
      "source": [
        "show_plot([x_train_uni[0], y_train_uni[0]], 0, 'Sample Example')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "b5rUJ_2YMWzG"
      },
      "source": [
        "### Baseline\n",
        "Before proceeding to train a model, let's first set a simple baseline. Given an input point, the baseline method looks at all the history and predicts the next point to be the average of the last 20 observations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "P9nYWcxMMWnr",
        "colab": {}
      },
      "source": [
        "def baseline(history):\n",
        "  return np.mean(history)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "KMcdFYKQMWlm",
        "colab": {}
      },
      "source": [
        "show_plot([x_train_uni[0], y_train_uni[0], baseline(x_train_uni[0])], 0,\n",
        "           'Baseline Prediction Example')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "067m6t8cMakb"
      },
      "source": [
        "Let's see if you can beat this baseline using a recurrent neural network."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "H4crpOcoMlSe"
      },
      "source": [
        "### Recurrent neural network\n",
        "\n",
        "A Recurrent Neural Network (RNN) is a type of neural network well-suited to time series data. RNNs process a time series step-by-step, maintaining an internal state summarizing the information they've seen so far. For more details, read the [RNN tutorial](https://www.tensorflow.org/tutorials/sequences/recurrent). In this tutorial, you will use a specialized RNN layer called Long Short Term Memory ([LSTM](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/LSTM))\n",
        "\n",
        "Let's now use `tf.data` to shuffle, batch, and cache the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kk-evkrmMWh9",
        "colab": {}
      },
      "source": [
        "BATCH_SIZE = 256\n",
        "BUFFER_SIZE = 10000\n",
        "\n",
        "train_univariate = tf.data.Dataset.from_tensor_slices((x_train_uni, y_train_uni))\n",
        "train_univariate = train_univariate.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()\n",
        "\n",
        "val_univariate = tf.data.Dataset.from_tensor_slices((x_val_uni, y_val_uni))\n",
        "val_univariate = val_univariate.batch(BATCH_SIZE).repeat()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "n2AmKkyVS5Ht"
      },
      "source": [
        "The following visualisation should help you understand how the data is represented after batching.\n",
        "\n",
        "![Time Series](https://github.com/tensorflow/docs/blob/master/site/en/tutorials/structured_data/images/time_series.png?raw=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4nagdTRNfPuZ"
      },
      "source": [
        "You will see the LSTM requires the input shape of the data it is being given."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IDbpHosCMWZO",
        "colab": {}
      },
      "source": [
        "simple_lstm_model = tf.keras.models.Sequential([\n",
        "    tf.keras.layers.LSTM(8, input_shape=x_train_uni.shape[-2:]),\n",
        "    tf.keras.layers.Dense(1)\n",
        "])\n",
        "\n",
        "simple_lstm_model.compile(optimizer='adam', loss='mae')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "NOGZtDAqMtSi"
      },
      "source": [
        "Let's make a sample prediction, to check the output of the model. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "2mPZbIKCMtLR",
        "colab": {}
      },
      "source": [
        "for x, y in val_univariate.take(1):\n",
        "    print(simple_lstm_model.predict(x).shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QYz6RN_mMyau"
      },
      "source": [
        "Let's train the model now. Due to the large size of the dataset, in the interest of saving time, each epoch will only run for 200 steps, instead of the complete training data as normally done."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "0opH9xi5MtIk",
        "colab": {}
      },
      "source": [
        "EVALUATION_INTERVAL = 200\n",
        "EPOCHS = 10\n",
        "\n",
        "simple_lstm_model.fit(train_univariate, epochs=EPOCHS,\n",
        "                      steps_per_epoch=EVALUATION_INTERVAL,\n",
        "                      validation_data=val_univariate, validation_steps=50)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "euyPo_lyNryZ"
      },
      "source": [
        "#### Predict using the simple LSTM model\n",
        "Now that you have trained your simple LSTM, let's try and make a few predictions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "S2rRLrs8MtGU",
        "colab": {}
      },
      "source": [
        "for x, y in val_univariate.take(3):\n",
        "  plot = show_plot([x[0].numpy(), y[0].numpy(),\n",
        "                    simple_lstm_model.predict(x)[0]], 0, 'Simple LSTM model')\n",
        "  plot.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Q-AVEJyRNvt0"
      },
      "source": [
        "This looks better than the baseline. Now that you have seen the basics, let's move on to part two, where you will work with a multivariate time series."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "VlJYi3_HXcw8"
      },
      "source": [
        "## Part 2: Forecast a multivariate time series"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hoxNZ2GM7DPm"
      },
      "source": [
        "The original dataset contains fourteen features. For simplicity, this section considers only three of the original fourteen. The features used are air temperature, atmospheric pressure, and air density. \n",
        "\n",
        "To use more features, add their names to this list."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "DphrB7bxSNDd",
        "colab": {}
      },
      "source": [
        "features_considered = ['p (mbar)', 'T (degC)', 'rho (g/m**3)']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "IfQUSiJfUpXJ",
        "colab": {}
      },
      "source": [
        "features = df[features_considered]\n",
        "features.index = df['Date Time']\n",
        "features.head()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qSfhTZi5r15R"
      },
      "source": [
        "Let's have a look at how each of these features vary across time."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "QdgC8zvGr21X",
        "colab": {}
      },
      "source": [
        "features.plot(subplots=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "cqStgZ-O1b3_"
      },
      "source": [
        "As mentioned, the first step will be to standardize the dataset using the mean and standard deviation of the training data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "W7VuNIwfHRHx",
        "colab": {}
      },
      "source": [
        "dataset = features.values\n",
        "data_mean = dataset[:TRAIN_SPLIT].mean(axis=0)\n",
        "data_std = dataset[:TRAIN_SPLIT].std(axis=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "eJUeWDqploCt",
        "colab": {}
      },
      "source": [
        "dataset = (dataset-data_mean)/data_std"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LyuGuJUgjUK3"
      },
      "source": [
        "### Single step model\n",
        "In a single step setup, the model learns to predict a single point in the future based on some history provided.\n",
        "\n",
        "The below function performs the same windowing task as above, however, here it samples the past observation based on the step size given."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "d-rVX4d3OF86",
        "colab": {}
      },
      "source": [
        "def multivariate_data(dataset, target, start_index, end_index, history_size,\n",
        "                      target_size, step, single_step=False):\n",
        "  data = []\n",
        "  labels = []\n",
        "\n",
        "  start_index = start_index + history_size\n",
        "  if end_index is None:\n",
        "    end_index = len(dataset) - target_size\n",
        "\n",
        "  for i in range(start_index, end_index):\n",
        "    indices = range(i-history_size, i, step)\n",
        "    data.append(dataset[indices])\n",
        "\n",
        "    if single_step:\n",
        "      labels.append(target[i+target_size])\n",
        "    else:\n",
        "      labels.append(target[i:i+target_size])\n",
        "\n",
        "  return np.array(data), np.array(labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "HWVGYwbN2ITI"
      },
      "source": [
        "In this tutorial, the network is shown data from the last five (5) days, i.e. 720 observations that are sampled every hour. The sampling is done every one hour since a drastic change is not expected within 60 minutes. Thus, 120 observation represent history of the last five days.  For the single step prediction model, the label for a datapoint is the temperature 12 hours into the future. In order to create a label for this, the temperature after 72(12*6) observations is used."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "HlhVGzPhmMYI",
        "colab": {}
      },
      "source": [
        "past_history = 720\n",
        "future_target = 72\n",
        "STEP = 6\n",
        "\n",
        "x_train_single, y_train_single = multivariate_data(dataset, dataset[:, 1], 0,\n",
        "                                                   TRAIN_SPLIT, past_history,\n",
        "                                                   future_target, STEP,\n",
        "                                                   single_step=True)\n",
        "x_val_single, y_val_single = multivariate_data(dataset, dataset[:, 1],\n",
        "                                               TRAIN_SPLIT, None, past_history,\n",
        "                                               future_target, STEP,\n",
        "                                               single_step=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CamMObrwPhnp"
      },
      "source": [
        "Let's look at a single data-point.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "_tVKm-ZIPls0",
        "colab": {}
      },
      "source": [
        "print ('Single window of past history : {}'.format(x_train_single[0].shape))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "eCWG4xgQ3O6E",
        "colab": {}
      },
      "source": [
        "train_data_single = tf.data.Dataset.from_tensor_slices((x_train_single, y_train_single))\n",
        "train_data_single = train_data_single.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()\n",
        "\n",
        "val_data_single = tf.data.Dataset.from_tensor_slices((x_val_single, y_val_single))\n",
        "val_data_single = val_data_single.batch(BATCH_SIZE).repeat()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "0aWec9_nlxBl",
        "colab": {}
      },
      "source": [
        "single_step_model = tf.keras.models.Sequential()\n",
        "single_step_model.add(tf.keras.layers.LSTM(32,\n",
        "                                           input_shape=x_train_single.shape[-2:]))\n",
        "single_step_model.add(tf.keras.layers.Dense(1))\n",
        "\n",
        "single_step_model.compile(optimizer=tf.keras.optimizers.RMSprop(), loss='mae')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oYhUfWjwOPFN"
      },
      "source": [
        "Let's check out a sample prediction."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "yY7FodHVOPsH",
        "colab": {}
      },
      "source": [
        "for x, y in val_data_single.take(1):\n",
        "  print(single_step_model.predict(x).shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "U0jnt2l2mwkl",
        "colab": {}
      },
      "source": [
        "single_step_history = single_step_model.fit(train_data_single, epochs=EPOCHS,\n",
        "                                            steps_per_epoch=EVALUATION_INTERVAL,\n",
        "                                            validation_data=val_data_single,\n",
        "                                            validation_steps=50)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "-ZAdeAnP5c72",
        "colab": {}
      },
      "source": [
        "def plot_train_history(history, title):\n",
        "  loss = history.history['loss']\n",
        "  val_loss = history.history['val_loss']\n",
        "\n",
        "  epochs = range(len(loss))\n",
        "\n",
        "  plt.figure()\n",
        "\n",
        "  plt.plot(epochs, loss, 'b', label='Training loss')\n",
        "  plt.plot(epochs, val_loss, 'r', label='Validation loss')\n",
        "  plt.title(title)\n",
        "  plt.legend()\n",
        "\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "l8lBKA-z5yYV",
        "colab": {}
      },
      "source": [
        "plot_train_history(single_step_history,\n",
        "                   'Single Step Training and validation loss')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "DfjrGAlEUp7i"
      },
      "source": [
        "#### Predict a single step future\n",
        "Now that the model is trained, let's make a few sample predictions. The model is given the history of three features over the past five days sampled every hour (120 data-points), since the goal is to predict the temperature, the plot only displays the past temperature. The prediction is made one day into the future (hence the gap between the history and prediction). "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "h1qmPLLVUpuN",
        "colab": {}
      },
      "source": [
        "for x, y in val_data_single.take(3):\n",
        "  plot = show_plot([x[0][:, 1].numpy(), y[0].numpy(),\n",
        "                    single_step_model.predict(x)[0]], 12,\n",
        "                   'Single Step Prediction')\n",
        "  plot.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2GnE087bJYSu"
      },
      "source": [
        "### Multi-Step model\n",
        "In a multi-step prediction model, given a past history, the model needs to learn to predict a range of future values. Thus, unlike a single step model, where only a single future point is predicted, a multi-step model predict a sequence of the future.\n",
        "\n",
        "For the multi-step model, the training data again consists of recordings over the past five days sampled every hour. However, here, the model needs to learn to predict the temperature for the next 12 hours. Since an obversation is taken every 10 minutes, the output is 72 predictions. For this task, the dataset needs to be prepared accordingly, thus the first step is just to create it again, but with a different target window."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kZCk9fqyJZqX",
        "colab": {}
      },
      "source": [
        "future_target = 72\n",
        "x_train_multi, y_train_multi = multivariate_data(dataset, dataset[:, 1], 0,\n",
        "                                                 TRAIN_SPLIT, past_history,\n",
        "                                                 future_target, STEP)\n",
        "x_val_multi, y_val_multi = multivariate_data(dataset, dataset[:, 1],\n",
        "                                             TRAIN_SPLIT, None, past_history,\n",
        "                                             future_target, STEP)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LImXPwAGRtWy"
      },
      "source": [
        "Let's check out a sample data-point."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "SpWDcBkQRwS-",
        "colab": {}
      },
      "source": [
        "print ('Single window of past history : {}'.format(x_train_multi[0].shape))\n",
        "print ('\\n Target temperature to predict : {}'.format(y_train_multi[0].shape))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "cjR4PJArMOpA",
        "colab": {}
      },
      "source": [
        "train_data_multi = tf.data.Dataset.from_tensor_slices((x_train_multi, y_train_multi))\n",
        "train_data_multi = train_data_multi.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()\n",
        "\n",
        "val_data_multi = tf.data.Dataset.from_tensor_slices((x_val_multi, y_val_multi))\n",
        "val_data_multi = val_data_multi.batch(BATCH_SIZE).repeat()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "IZcg8FWpSG8K"
      },
      "source": [
        "Plotting a sample data-point."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ksXKVbwBV7D3",
        "colab": {}
      },
      "source": [
        "def multi_step_plot(history, true_future, prediction):\n",
        "  plt.figure(figsize=(12, 6))\n",
        "  num_in = create_time_steps(len(history))\n",
        "  num_out = len(true_future)\n",
        "\n",
        "  plt.plot(num_in, np.array(history[:, 1]), label='History')\n",
        "  plt.plot(np.arange(num_out)/STEP, np.array(true_future), 'bo',\n",
        "           label='True Future')\n",
        "  if prediction.any():\n",
        "    plt.plot(np.arange(num_out)/STEP, np.array(prediction), 'ro',\n",
        "             label='Predicted Future')\n",
        "  plt.legend(loc='upper left')\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LCQKetflZRMF"
      },
      "source": [
        "In this plot and subsequent similar plots, the history and the future data are sampled every hour."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "R6G8bacQR4w2",
        "colab": {}
      },
      "source": [
        "for x, y in train_data_multi.take(1):\n",
        "  multi_step_plot(x[0], y[0], np.array([0]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XOjz8DzZ4HFS"
      },
      "source": [
        "Since the task here is a bit more complicated than the previous task, the model now consists of two LSTM layers. Finally, since 72 predictions are made, the dense layer outputs 72 predictions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "byAl0NKSNBP6",
        "colab": {}
      },
      "source": [
        "multi_step_model = tf.keras.models.Sequential()\n",
        "multi_step_model.add(tf.keras.layers.LSTM(32,\n",
        "                                          return_sequences=True,\n",
        "                                          input_shape=x_train_multi.shape[-2:]))\n",
        "multi_step_model.add(tf.keras.layers.LSTM(16, activation='relu'))\n",
        "multi_step_model.add(tf.keras.layers.Dense(72))\n",
        "\n",
        "multi_step_model.compile(optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0), loss='mae')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "UvB7zBqVSMyl"
      },
      "source": [
        "Let's see how the model predicts before it trains."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "13_ZWvB9SRlZ",
        "colab": {}
      },
      "source": [
        "for x, y in val_data_multi.take(1):\n",
        "  print (multi_step_model.predict(x).shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7uwOhXo3Oems",
        "colab": {}
      },
      "source": [
        "multi_step_history = multi_step_model.fit(train_data_multi, epochs=EPOCHS,\n",
        "                                          steps_per_epoch=EVALUATION_INTERVAL,\n",
        "                                          validation_data=val_data_multi,\n",
        "                                          validation_steps=50)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "UKfQoBjQ5l7U",
        "colab": {}
      },
      "source": [
        "plot_train_history(multi_step_history, 'Multi-Step Training and validation loss')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oDg94-yq4pas"
      },
      "source": [
        "#### Predict a multi-step future\n",
        "Let's now have a look at how well your network has learnt to predict the future."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "dt22wq6fyIBU",
        "colab": {}
      },
      "source": [
        "for x, y in val_data_multi.take(3):\n",
        "  multi_step_plot(x[0], y[0], multi_step_model.predict(x)[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pOzaIRYBhqwg"
      },
      "source": [
        "## Next steps\n",
        "This tutorial was a quick introduction to time series forecasting using an RNN. You may now try to predict the stock market and become a billionaire.\n",
        "\n",
        "In addition, you may also write a generator to yield data (instead of the uni/multivariate_data function), which would be more memory efficient. You may also check out this [time series windowing](https://www.tensorflow.org/guide/data#time_series_windowing) guide and use it in this tutorial.\n",
        "\n",
        "For further understanding, you may read Chapter 15 of [Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/), 2nd Edition and Chapter 6 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python)."
      ]
    }
  ]
}