Quiz_CNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dilated CNN model\n",
    "\n",
    "In this notebook, we demonstrate how to:\n",
    "- prepare time series data for training a Convolutional Neural Network (CNN) forecasting model\n",
    "- get data in the required shape for the keras API\n",
    "- implement a CNN model in keras to predict 3 steps ahead (time *t+1* to *t+1*) in the time series\n",
    "- enable early stopping to reduce the likelihood of model overfitting\n",
    "- evaluate the model on a test dataset\n",
    "\n",
    "The data in this example is taken from the GEFCom2014 forecasting competition<sup>1</sup>. It consists of 3 years of hourly electricity load and temperature values between 2012 and 2014. The task is to forecast future values of electricity load. In this example, we show how to forecast one time step ahead, using historical load data only.\n",
    "\n",
    "<sup>1</sup>Tao Hong, Pierre Pinson, Shu Fan, Hamidreza Zareipour, Alberto Troccoli and Rob J. Hyndman, \"Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond\", International Journal of Forecasting, vol.32, no.3, pp 896-913, July-September, 2016."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import warnings\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import datetime as dt\n",
    "from collections import UserDict\n",
    "from IPython.display import Image\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "%matplotlib inline\n",
    "\n",
    "from common.utils import load_data, mape, TimeSeriesTensor, create_evaluation_df\n",
    "\n",
    "pd.options.display.float_format = '{:,.2f}'.format\n",
    "np.set_printoptions(precision=2)\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the data from csv into a Pandas dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = 'data/'\n",
    "energy = load_data(data_dir)\n",
    "energy.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create train, validation and test sets\n",
    "\n",
    "We separate our dataset into train, validation and test sets. We train the model on the train set. The validation set is used to evaluate the model after each training epoch and ensure that the model is not overfitting the training data. After the model has finished training, we evaluate the model on the test set. We must ensure that the validation set and test set cover a later period in time from the training set, to ensure that the model does not gain from information from future time periods.\n",
    "\n",
    "We will allocate the period 1st November 2014 to 31st December 2014 to the test set. The period 1st September 2014 to 31st October is allocated to validation set. All other time periods are available for the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_start_dt = '2014-09-01 00:00:00'\n",
    "test_start_dt = '2014-11-01 00:00:00'\n",
    "\n",
    "energy.plot(y=['load', 'temp'], subplots=True, figsize=(15, 8), fontsize=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load and temperature in first week of July 2014"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "energy['2014-07-01':'2014-07-07'].plot(y=['load', 'temp'], subplots=True, figsize=(15, 8), fontsize=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation\n",
    "\n",
    "For this example, we will set *T=24*. This means that the input for each sample is a vector of the prevous 24 hours of the energy load.\n",
    "\n",
    "*HORIZON=3* specifies that we have a forecasting horizon of 3 (*t+1* to *t+3*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "T = 24\n",
    "HORIZON = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data preparation - training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create training dataset with load and temp features\n",
    "train = energy.copy()[energy.index < valid_start_dt][['load', 'temp']]\n",
    "\n",
    "# Fit a scaler for the y values\n",
    "y_scaler = MinMaxScaler()\n",
    "y_scaler.fit(train[['load']])\n",
    "\n",
    "# Also scale the input features data (load and temp values)\n",
    "X_scaler = MinMaxScaler()\n",
    "train[['load', 'temp']] = X_scaler.fit_transform(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the TimeSeriesTensor convenience class to:\n",
    "1. Shift the values of the time series to create a Pandas dataframe containing all the data for a single training example\n",
    "2. Discard any samples with missing values\n",
    "3. Transform this Pandas dataframe into a numpy array of shape (samples, time steps, features) for input into Keras\n",
    "\n",
    "The class takes the following parameters:\n",
    "\n",
    "- **dataset**: original time series\n",
    "- **H**: the forecast horizon\n",
    "- **tensor_structure**: a dictionary discribing the tensor structure in the form { 'tensor_name' : (range(max_backward_shift, max_forward_shift), [feature, feature, ...] ) }\n",
    "- **freq**: time series frequency\n",
    "- **drop_incomplete**: (Boolean) whether to drop incomplete samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor_structure = {'X':(range(-T+1, 1), ['load', 'temp'])}\n",
    "train_inputs = TimeSeriesTensor(dataset=train,\n",
    "                            target='load',\n",
    "                            H=HORIZON,\n",
    "                            tensor_structure=tensor_structure,\n",
    "                            freq='H',\n",
    "                            drop_incomplete=True)\n",
    "train_inputs.dataframe.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = train_inputs['X']\n",
    "y_train = train_inputs['target']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "X_train[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data preparation - validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "look_back_dt = dt.datetime.strptime(valid_start_dt, '%Y-%m-%d %H:%M:%S') - dt.timedelta(hours=T-1)\n",
    "valid = energy.copy()[(energy.index >=look_back_dt) & (energy.index < test_start_dt)][['load', 'temp']]\n",
    "valid[['load', 'temp']] = X_scaler.transform(valid)\n",
    "valid_inputs = TimeSeriesTensor(valid, 'load', HORIZON, tensor_structure)\n",
    "y_valid = valid_inputs['target']\n",
    "X_valid = valid_inputs['X']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_valid.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_valid.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quiz: Implement multivariate CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Model, Sequential\n",
    "from keras.layers import Conv1D, Dense, Flatten\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fill in your code below and replace the question marks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement your CNN model with the data prepared above and the following requirements:\n",
    "1. Use 2 features: past load and temperature\n",
    "2. Stack 5 convolutional layers of kernel width 2 with dilation rates 1, 2, 4, 8, 16\n",
    "3. Use 5 filters in each layer\n",
    "4. Train for 10 epochs\n",
    "5. Batch size 32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "LATENT_DIM = ?\n",
    "KERNEL_SIZE = ?\n",
    "BATCH_SIZE = ?\n",
    "EPOCHS = ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill in your code to replace the question mark\n",
    "# Hint: there is a parameter you need to add when stacking multiple RNN layers\n",
    "model = Sequential()\n",
    "?\n",
    "?\n",
    "?\n",
    "?\n",
    "?\n",
    "?\n",
    "?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you done, run the rest of the notebook to check if your model works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='RMSprop', loss='mse')\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify the early stopping criteria. We **monitor** the validation loss (in this case the mean squared error) on the validation set after each training epoch. If the validation loss has not improved by **min_delta** after **patience** epochs, we stop the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "earlystop = EarlyStopping(monitor='val_loss', min_delta=0, patience=5)\n",
    "\n",
    "history = model.fit(X_train,\n",
    "                    y_train,\n",
    "                    batch_size=BATCH_SIZE,\n",
    "                    epochs=EPOCHS,\n",
    "                    validation_data=(X_valid, y_valid),\n",
    "                    callbacks=[earlystop],\n",
    "                    verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_df = pd.DataFrame.from_dict({'train_loss':history.history['loss'], 'val_loss':history.history['val_loss']})\n",
    "plot_df.plot(logy=True, figsize=(10,10), fontsize=12)\n",
    "plt.xlabel('epoch', fontsize=12)\n",
    "plt.ylabel('loss', fontsize=12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "look_back_dt = dt.datetime.strptime(test_start_dt, '%Y-%m-%d %H:%M:%S') - dt.timedelta(hours=T-1)\n",
    "test = energy.copy()[test_start_dt:][['load', 'temp']]\n",
    "test[['load', 'temp']] = X_scaler.transform(test)\n",
    "test_inputs = TimeSeriesTensor(test, 'load', HORIZON, tensor_structure)\n",
    "X_test = test_inputs['X']\n",
    "y_test = test_inputs['target']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(X_test)\n",
    "eval_df = create_evaluation_df(predictions, test_inputs, HORIZON, y_scaler)\n",
    "eval_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_df['APE'] = (eval_df['prediction'] - eval_df['actual']).abs() / eval_df['actual']\n",
    "eval_df.groupby('h')['APE'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_df = eval_df[(eval_df.timestamp<'2014-11-08') & (eval_df.h=='t+1')][['timestamp', 'actual']]\n",
    "for t in range(1, HORIZON+1):\n",
    "    plot_df['t+'+str(t)] = eval_df[(eval_df.timestamp<'2014-11-08') & (eval_df.h=='t+'+str(t))]['prediction'].values\n",
    "\n",
    "fig = plt.figure(figsize=(15, 8))\n",
    "ax = plt.plot(plot_df['timestamp'], plot_df['actual'], color='red', linewidth=4.0)\n",
    "ax = fig.add_subplot(111)\n",
    "ax.plot(plot_df['timestamp'], plot_df['t+1'], color='blue', linewidth=4.0, alpha=0.75)\n",
    "ax.plot(plot_df['timestamp'], plot_df['t+2'], color='blue', linewidth=3.0, alpha=0.5)\n",
    "ax.plot(plot_df['timestamp'], plot_df['t+3'], color='blue', linewidth=2.0, alpha=0.25)\n",
    "plt.xlabel('timestamp', fontsize=12)\n",
    "plt.ylabel('load', fontsize=12)\n",
    "ax.legend(loc='best')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dnntutorial",
   "language": "python",
   "name": "dnntutorial"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}