2_BuildingOurFirstActivityRecognitionModels.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "2_BuildingOurFirstActivityRecognitionModels.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "roiOtqvWuWvZ",
        "hzuECCvj2a5j",
        "yEQ06O13w_Qx",
        "DUuZI1JSWZIn"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "roiOtqvWuWvZ"
      },
      "source": [
        "# 0. Preliminaries"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n8fOXj7-_R6q"
      },
      "source": [
        "## Drive integration\n",
        "You have to connect to your drive each time you start the execution of a colab notebook. It supposes that you have performed the steps provided in [Lab 1](https://colab.research.google.com/github/institut-galilee/2021-ml-iot-labs/blob/main/1_DiscoveringTheSHLDataset.ipynb)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MF9FpCOfekYH"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v96bIVFA6gYE"
      },
      "source": [
        "%cd /content/drive/MyDrive/ml-iot/2021-ml-iot-labs/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "URQaTn-W5HZJ"
      },
      "source": [
        "## Pull latest updates from the central repository\n",
        "If something has been pulled from the repository, you have to restart the execution environment using \"**Ctrl+M .**\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8RvJ1rnk5Gkm"
      },
      "source": [
        "!git pull"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZiV8eUPxMkin"
      },
      "source": [
        "## Create (or re-create) a sample\n",
        "It is safer, even if this was already done during Lab 1"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "85g_UVgBTQVx"
      },
      "source": [
        "!rm -r generated/sample\n",
        "!rm -r generated/0.1\n",
        "!mkdir -p generated/sample/train/Torso\n",
        "!cp -r generated/tmp/validation/ generated/sample/validation"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q0h5RXPoNDIs"
      },
      "source": [
        "import sample\n",
        "\n",
        "idxs = sample.build_sample(student_id=2334)\n",
        "sample.save_index(idxs, 'generated/sample/2021-01-sample.pickle')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fd29QFpAWqLP"
      },
      "source": [
        "# load data and visualize its format\n",
        "from dataset import DataReader\n",
        "\n",
        "# when run for the first time, this may take a while!\n",
        "train = DataReader(what='train', train_frames=15618)\n",
        "valid = DataReader(what='validation')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ig7YR7PfdjfV"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "## Check if the Data Was Loaded Correctly\n",
        "\n",
        "**Ex. 1** take the time to explore the data structure that contains the SHL data and its contents: number of train and test points, their array size, etc.\n",
        "Take also a look at the labels.\n",
        "You can use `len()` and `arrrayName.shape()`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RdYc-RI0Xc4U"
      },
      "source": [
        "train.X['Torso']['Acc_x'][-1]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hzuECCvj2a5j"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "# 1. Building our First Activity Recognition Models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JdpWzdIbXXoG"
      },
      "source": [
        "Building a neural network in general requires configuring the layers of the model, then compiling the model.  The basic building block of a neural network is thelayer.   Layers  extract  representations  from  the  data  fed  into  them.   Hopefully,  these  representations  are  meaningful  for  the  problem  at  hand.   Most  of deep learning consists of chaining together simple layers.  Most layers, such as `tf.keras.layers.Dense` or `tf.keras.layers.Conv1d`, have parameters that are learned during training.\n",
        "The network that we will construct during this lab is depicted in the following figure.\n",
        "<p align=\"center\">\n",
        "<img src=\"https://user-images.githubusercontent.com/8298445/106061343-8c2dd500-60f5-11eb-8226-ca703437ffcb.png\" height=\"600\"/>\n",
        "</p>\n",
        "\n",
        "\n",
        "The **first** part of the network (starting from the bottom) consists of input layers `tf.keras.Input` with shape 500 which corresponds to the size of the examples from the SHL dataset.\n",
        "Note that we will start with a simple model that takes as input only a single channel (Acc_x) and later on we will leverage the remaining channels and modalities (Gyr, Mag) via different sensor fusion schemes.\n",
        "\n",
        "<!--\n",
        "<p align=\"center\">\n",
        "<img src=\"https://user-images.githubusercontent.com/8298445/103903120-c78f4200-50fb-11eb-8a38-8f50f790c6fe.png\" height=\"200px\"/>\n",
        "</p>\n",
        "-->\n",
        "\n",
        "\n",
        "The **second** part of the network consists of a sequence of two `tf.keras.layers.Dense` layers.   These  are  densely  connected,  or  fully  connected, neural layers.  The first Dense layer has 128 nodes (or neurons).  The second (and last) layer is a 8-node softmax layer that returns an array of 8 probability scores that sum to 1.  Each node contains a score that indicates the probability that the current example belongs to one of the 8 classes.\n",
        "\n",
        "In this exercise, we don't explain the reasons of defining a neural network with this architecture. So, to define  a  network  compatible  with  our  data,  we  should define an input layer with the same size as the input data and an output corresponding the output data.\n",
        "\n",
        "<p align=\"center\">\n",
        "<img src=\"https://user-images.githubusercontent.com/8298445/103903117-c5c57e80-50fb-11eb-986a-e3de1b32b051.png\" height=\"200px\"/>\n",
        "</p>\n",
        "\n",
        "\n",
        "**Side notes about the 1-Dimension Convolution**\n",
        "\n",
        "Here you can see an illustration of the process of convolution between two signals (top) along with its result (bottom).\n",
        "\n",
        "<p align=\"center\">\n",
        "<img src=\"https://user-images.githubusercontent.com/8298445/104822951-b4d8e380-5846-11eb-9c16-dc7b2fbcd610.gif\" height=\"200px\">\n",
        "</p>\n",
        "\n",
        "More formally, given two functions f and g, the convolution can be written as follows. \n",
        "Check out [this](https://fr.wikipedia.org/wiki/Produit_de_convolution) entry in wikipedia to find out more.\n",
        "<p align=\"center\">\n",
        "<img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/09e0e66216b29a0ccd3d60a6e0f9aba3ce6fb4b2\">\n",
        "</p>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "edNbxKVVXVD4"
      },
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "\n",
        "# TensorFlow and tf.keras\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "\n",
        "# Helper libraries\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "print(tf.__version__)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XAAWSsqd0LcY"
      },
      "source": [
        "**Ex. 2 (code provided)** Build the network as shown above.\n",
        "\n",
        "Note that we use a single channel which is `Acc_x` (the x dimension of accelerometer) of the Torso. So we first specify these.\n",
        "\n",
        "We will be using `tf.keras.Input` to handle the sensory input.\n",
        "\n",
        "We will be using (for now, a single) `tf.keras.Conv1D` to extract useful features from the sensory input. These are followed by a `tf.keras.MaxPooling1D` layer used to select the most prominent filter responses in a feature map. (As mentioned, it is not important to understand every aspect of the network now but if you are interested you can check out [this](https://machinelearningmastery.com/pooling-layers-for-convolutional-neural-networks) introduction to pooling layers for convolutional neural networks).\n",
        "\n",
        "we will be using `tf.keras.Dense` to perform the categorization of the input signals into their corresponding class.\n",
        "\n",
        "\n",
        "If you have any doubt, please check out the documentation of the layers by hovering on it in the code or by following these links:\n",
        "* [Conv1D](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv1D)\n",
        "* [MaxPooling1D](https://www.tensorflow.org/api_docs/python/tf/keras/layers/MaxPool1D)\n",
        "* [GlobalMaxPooling1D](https://www.tensorflow.org/api_docs/python/tf/keras/layers/GlobalMaxPool1D)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xy6Q_PPKeRLr"
      },
      "source": [
        "position='Torso'\n",
        "channel='Acc_x'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aUpNcIDC0J-7"
      },
      "source": [
        "# 3D tensor with shape: (batch_size, steps, input_dim)\n",
        "ts = keras.Input(shape=(500,), name=position+'_'+channel)\n",
        "x = keras.layers.Reshape((500, 1))(ts)\n",
        "\n",
        "x = keras.layers.Conv1D(\n",
        "    filters=32,\n",
        "    kernel_size=7,\n",
        "    strides=2,\n",
        "    padding='valid',\n",
        "    activation='relu',\n",
        "    input_shape=(None, 500, 1),\n",
        "    name=position+'/'+channel+'/Conv1d/layer_0')(x)\n",
        "x = keras.layers.MaxPooling1D()(x)\n",
        "\n",
        "x = keras.layers.GlobalMaxPooling1D()(x)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rMttaSrnQbej"
      },
      "source": [
        "x = keras.layers.Dense(\n",
        "    units=128,\n",
        "    activation='relu',\n",
        "    name='Dense/layer_1')(x)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "INf6oykkQerR"
      },
      "source": [
        "class_output = keras.layers.Dense(\n",
        "    units=8,\n",
        "    activation='softmax',\n",
        "    name='class_output')(x)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lKtWC6PkQvXM"
      },
      "source": [
        "model = keras.Model(inputs=ts, outputs=class_output)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SI8olURfQxDc"
      },
      "source": [
        "keras.utils.plot_model(model, 'first_model.png', show_shapes=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rxpc7sQjU6xh"
      },
      "source": [
        "**Ex. 3 (code provided)** Before the model is ready for training,  it needs a few more settings. These are added during the model's compile step (We will explain the following parts in details during the future sections.  Use them here as an example, because it is necessary for NN trainings):\n",
        "- Loss function: This measures how accurate the model is during training. You  want  to  minimize  this  function  to  ”steer”  the  model  in  the  right direction;\n",
        "- Optimizer: This is how the model is updated based on the data it sees and its loss function;\n",
        "- Metrics: Used  to  monitor  the  training  and  testing  steps.   The  following example uses accuracy, the fraction of the images that are correctly classified."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "35hh0_2-eTuW"
      },
      "source": [
        "model.compile(\n",
        "    optimizer=keras.optimizers.Adam(learning_rate=0.0001, amsgrad=True),\n",
        "    loss=[keras.losses.CategoricalCrossentropy(from_logits=True)],\n",
        "    metrics=['acc'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mmrhCNP9eeoD"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SUm7MPkyeuvH"
      },
      "source": [
        "**Ex. 4 (code provided)** Format the data according to the naming convention used to define the Input layers (i.e. `ts = keras.Input(shape=(500,), name=position+'_'+channel)`). This will be re-used later when we will leverage the remaining channels and modalities."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oV6NNgfgRoJh"
      },
      "source": [
        "def format_data(train, valid):\n",
        "\n",
        "    train_dict = {}\n",
        "    valid_dict = {}\n",
        "    for chan in [channel]:\n",
        "        train_dict[position + '_' + channel] = train.X[position][channel]\n",
        "        valid_dict[position + '_' + channel] = valid.X[position][channel]\n",
        "\n",
        "    train_labels = train.y[:, 0] - 1  # classes should be from 0 to num_classes-1\n",
        "    valid_labels = valid.y[:, 0] - 1\n",
        "\n",
        "    return train_dict, train_labels, valid_dict, valid_labels"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ViF9tyO5SHq9"
      },
      "source": [
        "train_dict, train_labels, valid_dict, valid_labels = format_data(train, valid)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rCMJW77GVJFb"
      },
      "source": [
        "**Ex. 5 (code provided)** Training the neural network model requires thefollowing steps:\n",
        "- Feed the training data to the model.  In this example, the training data isin the train images and trainlabels arrays.\n",
        "- The model learns to associate images and labels.\n",
        "- You ask the model to make predictions about a test set, in this example,the test images array.  Verify that the predictions match the labels fromthe test labels array."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "heFFl-OMeaMy"
      },
      "source": [
        "history = model.fit(\n",
        "    train_dict,\n",
        "    keras.utils.to_categorical(train_labels, num_classes=8),\n",
        "    batch_size=512,\n",
        "    epochs=50,\n",
        "    shuffle=True\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5LlZCt0OmNqn"
      },
      "source": [
        "**Ex. 6** Plot the evolution of the training loss as a function of the training epochs. Hint: the loss can be found in the dictionnary `history.history[????]` returned by `model.fit()` above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RFkNGTQ8YC7l"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# b+ is for \"blue cross\"\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IAwxuoA9nmf0"
      },
      "source": [
        "**Ex. 7 (code provided)** Check out the distribution of the model's weights."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "gf-_91lheqre",
        "outputId": "288dd087-5613-42fe-ce2c-011453a748d2"
      },
      "source": [
        "model.weights\n",
        "\n",
        "weights, bias = model.get_layer(position+'/'+channel+'/Conv1d/layer_0').get_weights()\n",
        "plt.hist(tf.keras.backend.eval(weights).ravel())\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAM7UlEQVR4nO3db4xl9V3H8fen/LHGVgF3sm6AddpCNJjYxYxrDaathSqFRGhCVJLSNSGZPihJG/tkUx5YjQ+o2uKTpnErpKvpX2kJm1KrsJLUJi11liJl2bRLyVbBZXexYKk11YWvD+ZsOpk/e8/M/Te/3fcrmcy9554755sf8M7lzLl3UlVIktrzimkPIEnaGAMuSY0y4JLUKAMuSY0y4JLUqHMnebAtW7bU7OzsJA8pSc07cODAc1U1s3z7RAM+OzvLwsLCJA8pSc1L8t3VtnsKRZIaZcAlqVEGXJIaZcAlqVEGXJIaZcAlqVEGXJIaZcAlqVEGXJIaNdF3YqoNs7vvn9qxj9xx/dSOLbXGV+CS1CgDLkmNGhjwJK9M8vUk/5rkYJI/7ra/JsnDSZ5M8pkk549/XEnSKX1egf8IeEtVvR7YAVyb5A3AB4E7q+oy4Hng1vGNKUlabmDAa9EPurvndV8FvAW4p9u+F7hxLBNKklbV6yqUJOcAB4DLgI8A3wFeqKqT3S5PAxev8dx5YB5g+/btw847FdO6KsMrMiSdTq9fYlbVS1W1A7gE2An8Yt8DVNWeqpqrqrmZmRV/UEKStEHrugqlql4AHgJ+HbggyalX8JcAz4x4NknSafS5CmUmyQXd7Z8E3gocYjHkN3W77QLuG9eQkqSV+pwD3wbs7c6DvwL4bFV9IckTwKeT/CnwDeCuMc4pSVpmYMCr6jHgylW2P8Xi+XBJ0hT4TkxJapQBl6RGGXBJapQBl6RGGXBJapQBl6RGGXBJapQBl6RGGXBJapQBl6RGGXBJapQBl6RGGXBJapQBl6RGGXBJapQBl6RGGXBJapQBl6RGGXBJapQBl6RGGXBJatTAv0ovabxmd98/leMeueP6qRxXo+MrcElqlAGXpEYZcElq1MCAJ7k0yUNJnkhyMMl7uu0fSPJMkke7r+vGP64k6ZQ+v8Q8Cbyvqh5J8mrgQJIHusfurKq/GN94kqS1DAx4VR0Fjna3X0xyCLh43INJkk5vXZcRJpkFrgQeBq4CbkvyTmCBxVfpz6/ynHlgHmD79u1Djnt2mdblZZLa0PuXmEleBXwOeG9VfR/4KPA6YAeLr9A/tNrzqmpPVc1V1dzMzMwIRpYkQc+AJzmPxXh/oqo+D1BVx6rqpap6GfgYsHN8Y0qSlutzFUqAu4BDVfXhJdu3Ldnt7cDjox9PkrSWPufArwJuAb6Z5NFu2/uBm5PsAAo4ArxrLBNKklbV5yqUrwBZ5aEvjn4cSVJffpiVpInzA7xGw7fSS1KjDLgkNcqAS1KjDLgkNcqAS1KjDLgkNcqAS1KjDLgkNcqAS1KjDLgkNcqAS1KjDLgkNcqAS1KjDLgkNcqAS1KjDLgkNcqAS1KjDLgkNcqAS1KjDLgkNco/aiydpab1h4U1Or4Cl6RGGXBJatTAgCe5NMlDSZ5IcjDJe7rtFyV5IMnh7vuF4x9XknRKn1fgJ4H3VdUVwBuAdye5AtgN7K+qy4H93X1J0oQMDHhVHa2qR7rbLwKHgIuBG4C93W57gRvHNaQkaaV1XYWSZBa4EngY2FpVR7uHngW2rvGceWAeYPv27Rud09+YnyX85yz11/uXmEleBXwOeG9VfX/pY1VVQK32vKraU1VzVTU3MzMz1LCSpB/rFfAk57EY709U1ee7zceSbOse3wYcH8+IkqTV9LkKJcBdwKGq+vCSh/YBu7rbu4D7Rj+eJGktfc6BXwXcAnwzyaPdtvcDdwCfTXIr8F3gd8czoiRpNQMDXlVfAbLGw1ePdhxJUl++E1OSGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRAwOe5O4kx5M8vmTbB5I8k+TR7uu68Y4pSVquzyvwjwPXrrL9zqra0X19cbRjSZIGGRjwqvoy8L0JzCJJWodhzoHfluSx7hTLhSObSJLUy0YD/lHgdcAO4CjwobV2TDKfZCHJwokTJzZ4OEnSchsKeFUdq6qXqupl4GPAztPsu6eq5qpqbmZmZqNzSpKW2VDAk2xbcvftwONr7StJGo9zB+2Q5FPAm4EtSZ4G/gh4c5IdQAFHgHeNcUZJ0ioGBryqbl5l811jmEWStA6+E1OSGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGmXAJalRBlySGjUw4EnuTnI8yeNLtl2U5IEkh7vvF453TEnScn1egX8cuHbZtt3A/qq6HNjf3ZckTdDAgFfVl4HvLdt8A7C3u70XuHHEc0mSBtjoOfCtVXW0u/0ssHWtHZPMJ1lIsnDixIkNHk6StNzQv8SsqgLqNI/vqaq5qpqbmZkZ9nCSpM5GA34syTaA7vvx0Y0kSepjowHfB+zqbu8C7hvNOJKkvvpcRvgp4KvALyR5OsmtwB3AW5McBq7p7kuSJujcQTtU1c1rPHT1iGeRJK2D78SUpEYZcElqlAGXpEYZcElqlAGXpEYZcElqlAGXpEYZcElqlAGXpEYZcElqlAGXpEYZcElqlAGXpEYZcElq1MCPk5WkM8Xs7vunduwjd1w/8p/pK3BJapQBl6RGGXBJapQBl6RGGXBJapQBl6RGGXBJapQBl6RGGXBJatRQ78RMcgR4EXgJOFlVc6MYSpI02CjeSv+bVfXcCH6OJGkdPIUiSY0aNuAF/GOSA0nmV9shyXyShSQLJ06cGPJwkqRThg34b1TVrwBvA96d5I3Ld6iqPVU1V1VzMzMzQx5OknTKUAGvqme678eBe4GdoxhKkjTYhgOe5KeSvPrUbeC3gMdHNZgk6fSGuQplK3BvklM/55NV9aWRTCVJGmjDAa+qp4DXj3AWSdI6eBmhJDXKgEtSowy4JDXKgEtSowy4JDXKgEtSowy4JDXKgEtSowy4JDXKgEtSowy4JDXKgEtSowy4JDXKgEtSowy4JDXKgEtSowy4JDXKgEtSowy4JDXKgEtSowy4JDXKgEtSowy4JDXKgEtSowy4JDVqqIAnuTbJt5I8mWT3qIaSJA224YAnOQf4CPA24Arg5iRXjGowSdLpDfMKfCfwZFU9VVX/C3wauGE0Y0mSBjl3iOdeDPz7kvtPA7+2fKck88B8d/cHSb61zuNsAZ7b0IRnNtdlJddkJddkpamsST441NN/frWNwwS8l6raA+zZ6POTLFTV3AhHOiO4Liu5Jiu5JiudSWsyzCmUZ4BLl9y/pNsmSZqAYQL+L8DlSV6T5Hzg94F9oxlLkjTIhk+hVNXJJLcB/wCcA9xdVQdHNtmPbfj0yxnOdVnJNVnJNVnpjFmTVNW0Z5AkbYDvxJSkRhlwSWrUpgt4kouSPJDkcPf9wlX22ZHkq0kOJnksye9NY9ZJ6rMu3X5fSvJCki9MesZJGfQRDkl+IslnuscfTjI7+Sknq8eavDHJI0lOJrlpGjNOWo81+cMkT3QN2Z9k1WutN7NNF3BgN7C/qi4H9nf3l/sh8M6q+iXgWuAvk1wwwRmnoc+6APw5cMvEppqwnh/hcCvwfFVdBtwJDPcWik2u55r8G/AHwCcnO9109FyTbwBzVfXLwD3An012yuFtxoDfAOztbu8Fbly+Q1V9u6oOd7f/AzgOzExswukYuC4AVbUfeHFSQ01Bn49wWLpW9wBXJ8kEZ5y0gWtSVUeq6jHg5WkMOAV91uShqvphd/drLL6XpSmbMeBbq+pod/tZYOvpdk6yEzgf+M64B5uyda3LGWy1j3C4eK19quok8F/Az05kuunosyZnm/Wuya3A3491ojEY+1vpV5PkQeDnVnno9qV3qqqSrHmdY5JtwN8Cu6qq+VcWo1oXSf0leQcwB7xp2rOs11QCXlXXrPVYkmNJtlXV0S7Qx9fY76eB+4Hbq+prYxp1okaxLmeBPh/hcGqfp5OcC/wM8J+TGW8q/FiLlXqtSZJrWHyB9Kaq+tGEZhuZzXgKZR+wq7u9C7hv+Q7dW/fvBf6mqu6Z4GzTNHBdzhJ9PsJh6VrdBPxTndnvWPNjLVYauCZJrgT+CvidqmrzBVFVbaovFs9V7gcOAw8CF3Xb54C/7m6/A/g/4NElXzumPfu016W7/8/ACeB/WDzv99vTnn0Ma3Ed8G0Wf+9xe7ftT1j8DxHglcDfAU8CXwdeO+2ZN8Ga/Gr378N/s/h/IwenPfMmWJMHgWNLGrJv2jOv98u30ktSozbjKRRJUg8GXJIaZcAlqVEGXJIaZcAlqVEGXJIaZcAlqVH/D3cpwYl3JdDGAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3iBbyoiTqMzE"
      },
      "source": [
        "**Ex. 8** If you want to visualize the evolution of the weights distribution before and after training, you have to save a version of the model we defined above (in a new variable) before training and compare it to the weights of the trained model after training. Plot the two distributions (before and after) in the same figure."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O08XDeaMqXit"
      },
      "source": [
        "# plot weights distribution before and after training"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ru-nTiqlVooT"
      },
      "source": [
        "**Ex. 9** The model predicts a label for each image in the testing set.\n",
        "Check the shape of the predictions. Print the first, second  and  third  element  of  the  predicted  test  sets.   You  can  see  that  each element contains 8 values indicating a probability of each label.  Choose the maximum one using `np.argmax()` function.  Compare the predicted label of the first three elements with their predicted labels.  How many are correct?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tB29xQPee0_V"
      },
      "source": [
        "# perform prediction on the entire validation set\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "idmfEMgSak6c"
      },
      "source": [
        "# print the shape of the predictions returned by the model\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eST_HRp1fFv1"
      },
      "source": [
        "# compare the predicted label of the first three elements with their predicted labels\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gw-DOVMh-cB9"
      },
      "source": [
        "**Ex. 10** Grab a single element from the test set such as `valid_dict['Torso_Acc_x'][5]`.  Send it to the `model.predict()` and check what will happen.  Why?  Correct it.  (hint:  you can use `expand_dims()` from the `numpy` library)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ClMgwK5qjQf5"
      },
      "source": [
        "# prediction of a single example\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C7rQbrqre6PL"
      },
      "source": [
        "### Visualizing the predictions\n",
        "\n",
        "**Ex. 11** Using the `plotly`-based code you saw during the first lab, we want you to write code that can display the predictions along with the sensory signals. This can provide interesting insights about the behavior of your machine learning model.\n",
        "You should get something like what is depicted in the following figure but you can design alternative ways for visualizing the predictions which can help you analyze the behavior of your model:\n",
        "<p align=\"center\">\n",
        "<img src=\"https://user-images.githubusercontent.com/8298445/106063587-b5039980-60f8-11eb-9a9f-e4306a035de6.png\" height=\"500px\"/>\n",
        "</p>\n",
        "\n",
        "In this figure, the examples that are correctly classified are colored in green and those incorrectly classified are colored in red.\n",
        "Add also a text that will appear when we hover above an example and display the true label of the example and the one predicted by your model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cM95fKT7gSiT"
      },
      "source": [
        "# display sensory signals along with model's predictions\n",
        "from IPython.display import HTML\n",
        "import plotly.express as px\n",
        "import pandas as pd\n",
        "import plotly.graph_objects as go\n",
        "\n",
        "\n",
        "fig = go.Figure()\n",
        "for i in range(100):\n",
        "    # select an example\n",
        "    # get a its prediction\n",
        "    fig.add_trace(\n",
        "        go.Scatter(\n",
        "            # parameters to go.Scatter\n",
        "        )\n",
        "    )\n",
        "HTML(fig.to_html())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LunsnH-JCWff"
      },
      "source": [
        "### Playing with the Network's Hyperparameters\n",
        "\n",
        "**Ex. 12** Keep the input and output sizes and try to change the hyperparameters of your model, i.e. `kernel_size`, `filters`, `stride`, etc. in **Ex. 2** and observe their effects of your changes on the performances. Save your different runs and try to analyze and document your observations."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l9VWRIisqjXi"
      },
      "source": [
        "### Stacking additional 1d convolution layers\n",
        "\n",
        "**Ex. 13** till now, we used networks containing a single `tf.keras.Conv1D` layer on top of the input signal. Try to stack additional `tf.keras.Conv1D` and see the effects."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yEQ06O13w_Qx"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "# 2. Leveraging other channels and the remaining modalities\n",
        "\n",
        "We built previously a model using a single channel, i.e. Acc_x. We can however use the other channels as well in order to improve the performances of our model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HsoMpH_d2u2d"
      },
      "source": [
        "**Ex. 1** Construct the first part of a network reproducing the input scheme illustrated in the following figure where we leverage the whole input modalities (Acc, Gyr, and Mag).\n",
        "\n",
        "**Hint:** you can iterate over the input channels using `DataReader.channels`. Make sure you store the `keras.Input()` corresponding to each channel (in a list for example) in order to use it in the final definition of the model, i.e. `model = keras.Model(inputs=ts, outputs=class_output)`. Make sure also that you put a concatenate layer in the right location in order to reproduce exactly the depicted architecture.\n",
        "\n",
        "<p align=\"center\">\n",
        "<img src=\"https://user-images.githubusercontent.com/8298445/103903120-c78f4200-50fb-11eb-8a38-8f50f790c6fe.png\" height=\"200px\"/>\n",
        "</p>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e93hy7OP0wZb"
      },
      "source": [
        "xs = []\n",
        "inputs = []\n",
        "\n",
        "for _, channel in DataReader.channels.items():\n",
        "    # 3D tensor with shape: (batch_size, steps, input_dim)\n",
        "    ts = keras.Input(shape=(500,), name=position+'_'+channel)\n",
        "\n",
        "    # keep track of the inputs\n",
        "\n",
        "######\n",
        "x = keras.layers.concatenate(xs)\n",
        "######\n",
        "\n",
        "# convolution layers"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U8Xstuzr4cwq"
      },
      "source": [
        "**Ex. 2** Construct the second part of the network (This part does not differ from the first models we constructed earlier in the first part of the lab)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TP1fx4oO2hOU"
      },
      "source": [
        "# a first dense layer\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7WltO2_B2iBr"
      },
      "source": [
        "# a second dense layer which corresponds to the output of the network\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5TiH4lte5GZJ"
      },
      "source": [
        "**Ex. 3** Build the model using `keras.Model()`.\n",
        "Make sure that all inputs are considered. If it is not the case, go back to **Ex. 1**: maybe you did not keep track of all the inputs. Plot the model to verify this."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MQPbAzTP2kzH"
      },
      "source": [
        "# build the model\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2qWOfnWe2soM"
      },
      "source": [
        "# plot the model\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ev0syAL45TZm"
      },
      "source": [
        "**Ex. 4** Train the model by defining an optimizer and calling `fit()` (reload data if necessary)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "II3RVgan5h-s"
      },
      "source": [
        "# optimizer\n",
        "\n",
        "# fit"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bal3IkOr5p89"
      },
      "source": [
        "**Ex. 5** Plot the evolution of the training loss"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eWMNvgZK6GrF"
      },
      "source": [
        "# plot"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bYpOu9186IgR"
      },
      "source": [
        "**Ex. 6** Make predictions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n5CRQYz36tPR"
      },
      "source": [
        "# preditions"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zet-INWn7so1"
      },
      "source": [
        "**Ex. 7** Use the visualization tool you constructed earlier to plot the predictions. Check which parts are impacted (both positively and negatively) by using the whole modalities."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n2P8Nbi26u-J"
      },
      "source": [
        "**Ex. 8** Analyze the results and document your observations."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DUuZI1JSWZIn"
      },
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "# 3. Exploring other fusion schemes\n",
        "\n",
        "We build previously models that consider the inputs as flat sensory inputs. In other words, all inputs are considered to be at the same level. We refer to these models as _grouped modalities_ in contrast to _split modalities_ and _split channels_ that we will see in the following.\n",
        "Indeed, we can process each channel or modality separatly before merging or fusing them. This is what we will try to implement in the following."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_2nl3kTRXpzp"
      },
      "source": [
        "## Split Channels\n",
        "\n",
        "Consider the architecture depicted in the figure below. This architecture implements a fusion scheme that we refer to as _split channels_. In this fusion scheme, channels are processed individually by the `Conv1d~ReLU~MaxPool` blocks before being concatenated and fed into a dense layer. \n",
        "\n",
        "<p align=\"center\">\n",
        "<img src=\"https://user-images.githubusercontent.com/8298445/103903118-c65e1500-50fb-11eb-992e-140186f086f6.png\" height=\"200px\"/>\n",
        "</p>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "StugQShtYEaM"
      },
      "source": [
        "**Ex. 1** Construct the first part of a network reproducing the input scheme illustrated in the figure above. Hint: iterate over the channels provided in `DataReader.channels`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FmVJX4Lynyd_"
      },
      "source": [
        "xs = []\n",
        "inputs = []\n",
        "\n",
        "for _, channel in DataReader.channels.items():\n",
        "    # 3D tensor with shape: (batch_size, steps, input_dim)\n",
        "    ts = keras.Input(shape=(500,), name=position+'_'+channel)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VrueRXE2sKK7"
      },
      "source": [
        "**TODO**\n",
        "Concatenate the output from the different channel-dedicated `Conv1d~ReLU~MaxPool` blocks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N6YJc9e0sIR3"
      },
      "source": [
        "x = keras.layers.concatenate(xs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yDiWOjPSDyrD"
      },
      "source": [
        "**Ex. 2** Construct the second part of the network (This part does not differ from the first models we constructed earlier in the first part of the lab)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8pyciy7WtA7-"
      },
      "source": [
        "# a first dense layer"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xz0DB1vztDVV"
      },
      "source": [
        "# a second dense layer"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CzfPqCHx_SL9"
      },
      "source": [
        "**Ex. 3** Build the model using `keras.Model()`.\n",
        "Make sure that all inputs are considered. If it is not the case, go back to **Ex. 1**: maybe you did not keep track of all the inputs. Plot the model to verify this. You should get something like the following:\n",
        "\n",
        "<p align=\"center\">\n",
        "<img src=\"https://user-images.githubusercontent.com/8298445/106086965-63243900-6123-11eb-916f-e65fee9f6cb6.png\" height=\"200px\" title=\"model_SplitChannels\"/>\n",
        "</p>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BB9vVRPXvKFL"
      },
      "source": [
        "# build the model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R-1CHW2ttP-n"
      },
      "source": [
        "# plot the model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "72Pil-mw_YKf"
      },
      "source": [
        "**Ex. 4** Train the model by defining an optimizer and calling `fit()` (reload data if necessary)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "En1-wffK_gs9"
      },
      "source": [
        "# optimizer\n",
        "\n",
        "# fit"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_F7LmTfa_gtn"
      },
      "source": [
        "**Ex. 5** Plot the evolution of the training loss"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NWfziJRw_gtn"
      },
      "source": [
        "# plot"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fXFu81Jz_gto"
      },
      "source": [
        "**Ex. 6** Make predictions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "832zf0JY_gto"
      },
      "source": [
        "# preditions"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SniB4F1S_gtp"
      },
      "source": [
        "**Ex. 7** Use the visualization tool you constructed earlier to plot the predictions. Check which parts are impacted (both positively and negatively) by using the whole modalities."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RWChicQx_gtp"
      },
      "source": [
        "**Ex. 8** Analyze the results and document your observations."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zCGb3QvqX0Bd"
      },
      "source": [
        "## Split modalities\n",
        "\n",
        "Consider the architecture depicted in the figure below. This architecture implements a fusion scheme that we refer to as _split modalities_. In this fusion scheme, the modalities (each of which consists of 3 channels) are processed individually by the `Conv1d~ReLU~MaxPool` blocks before being concatenated and fed into a dense layer.\n",
        "\n",
        "<p align=\"center\">\n",
        "<img src=\"https://user-images.githubusercontent.com/8298445/103903119-c6f6ab80-50fb-11eb-8408-48730cd27a47.png\" height=\"200px\"/>\n",
        "</p>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kZT_o_N1ZRig"
      },
      "source": [
        "**Ex. 9** Construct the first part of a network reproducing the input scheme illustrated in the figure above.\n",
        "Hint: Iterate over the modalities provided in `DataReader.modalities`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dJ8SiJT7YcSq"
      },
      "source": [
        "xs = []\n",
        "inputs = []\n",
        "mods = []\n",
        "\n",
        "for modality in DataReader.modalities:\n",
        "    xs = []\n",
        "    # modality = DataReader.channel_to_modality(channel)\n",
        "    channels = filter(lambda chan: chan[1].startswith(modality), DataReader.channels.items())\n",
        "    for _, channel in channels:\n",
        "\n",
        "        # 3D tensor with shape: (batch_size, steps, input_dim)\n",
        "        ts = keras.Input(shape=(500,), name=position+'_'+channel)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F8lAuJkwG4x-"
      },
      "source": [
        "**Ex. 10** Construct the second part of the network (This part does not differ from the first models we constructed earlier in the first part of the lab)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sIFSK86J2269"
      },
      "source": [
        "# a first dense layer"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FNbSPrOC8XMd"
      },
      "source": [
        "# a second dense layer"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MmiodTCjG4yr"
      },
      "source": [
        "**Ex. 11** Build the model using `keras.Model()`.\n",
        "Make sure that all inputs are considered. If it is not the case, go back to **Ex. 9**: maybe you did not keep track of all the inputs. Plot the model to verify this. You should get something like the following:\n",
        "\n",
        "<p align=\"center\">\n",
        "<img src=\"https://user-images.githubusercontent.com/8298445/106086967-63bccf80-6123-11eb-9950-c37c55fdd23b.png\" height=\"200px\" title=\"model_SplitModalities\"/>\n",
        "</p>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J9NlVS8R8aKM"
      },
      "source": [
        "# build the model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i4kU7uxu8fe0"
      },
      "source": [
        "# plot the model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9E57K9IjA8Xo"
      },
      "source": [
        "**Ex. 12** Train the model by defining an optimizer and calling `fit()` (reload data if necessary)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pGNitYkCA8Xr"
      },
      "source": [
        "# optimizer\n",
        "\n",
        "# fit"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3jFd3Vz5A8Xs"
      },
      "source": [
        "**Ex. 13** Plot the evolution of the training loss"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W4azFNL9A8Xt"
      },
      "source": [
        "# plot"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "750a9KwEA8Xu"
      },
      "source": [
        "**Ex. 14** Make predictions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EHJ6jRrKA8Xv"
      },
      "source": [
        "# preditions"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-NNH-3mFA8Xw"
      },
      "source": [
        "**Ex. 15** Use the visualization tool you constructed earlier to plot the predictions. Check which parts are impacted (both positively and negatively) by using the whole modalities."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NKM3A83WA8Xy"
      },
      "source": [
        "**Ex. 16** Analyze the results and document your observations."
      ]
    }
  ]
}