rnn.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "rnn.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyOWCo0gn0e/TqAPJR8hRQ6X",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hikmatfarhat-ndu/CSC645/blob/master/rnn.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yjyzzSPFsoIy"
      },
      "source": [
        "# Predicting Movie Reviews\n",
        "\n",
        "In this exercise we are given a set of IMDB movie reviews and we train our model to predict other reviews. The output of the review is either positive or negative so it is a binary classification problem like the ship/not ship problem we dealt with before"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AxlaPRhjYIOj"
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "from tensorflow.keras.datasets import imdb\n",
        "import numpy as np\n",
        "#import cupy as np\n",
        "from keras.utils import to_categorical\n",
        "\n"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dUvFJR-vkYNh"
      },
      "source": [
        "## The data\n",
        "\n",
        "THe movie review dataset is a set of 50000 reviews of movies (half training, half test). Each review contains a set of words and is labeled positive (1) or negative (0). For convenience each word index refer to its frequency of occurence in the dataset. For example a word with index 5 is the fifth most frequently used data set. The indices 0,1 and 2 are reserved so 5 really means the third most frequent.\n",
        "\n",
        "Details about the dataset can be found here [Keras IMDB](https://keras.io/api/datasets/imdb/)\n",
        "\n",
        "In this exercise we choose only the first 10000 most frequent words to be included. Any word that is not among them is given the index 2.\n",
        "\n",
        "First we load data set without omitting any words"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XX_34kZwx7bp"
      },
      "source": [
        "### Data details\n",
        "\n",
        "We would like to have an idea about the number of reviews, the average length of a review. Also we compute how many entries with values 0,1,2 and 3. The number 0 is used for padding and 1 to denote the beginning of each sequence. The number 2 is used for missing words. Finally, the number 3 is never used since as you will see later we will shift the indices by 3."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l2VxN9cJyNez",
        "outputId": "9464371a-3719-4761-fb92-cd07425e36db",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "(x_train,y_train),(x_test,y_test)=tf.keras.datasets.imdb.load_data()\n",
        "\n",
        "print(\"The number of reviews in the x_train data set = {}\\n\".format(x_train.shape[0]))\n",
        "print(\"The average length of reviews = {}\".format(np.mean([len(x) for x in x_train])))\n",
        "print(\"With standard deviation = {}\".format(np.std([len(x) for x in x_train])))\n",
        "print(\"The number of 0's in the x_train data set = {}\\n\".format(sum([1 for x in np.hstack(x_train) if x==0])))\n",
        "print(\"The number of 1's in the x_train data set = {}\\n\".format(sum([1 for x in np.hstack(x_train) if x==1])))\n",
        "print(\"The number of 2's in the x_train data set = {}\\n\".format(sum([1 for x in np.hstack(x_train) if x==2])))\n",
        "print(\"The number of 3's in the x_train data set = {}\\n\".format(sum([1 for x in np.hstack(x_train) if x==3])))"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
            "17465344/17464789 [==============================] - 0s 0us/step\n",
            "The number of reviews in the x_train data set = 25000\n",
            "\n",
            "The average length of reviews = 238.71364\n",
            "With standard deviation = 176.49367364852034\n",
            "The number of 0's in the x_train data set = 0\n",
            "\n",
            "The number of 1's in the x_train data set = 25000\n",
            "\n",
            "The number of 2's in the x_train data set = 1\n",
            "\n",
            "The number of 3's in the x_train data set = 0\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ql1rL9j80jgO"
      },
      "source": [
        "Now when we choose only the first _max_words_ most frequent words and compute the number of 2's in the data set. As you can see the number of 2's is now very large since all the \"ignored\" words were given the code 2."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fBaHMaLXYSQT",
        "outputId": "bc496265-958b-44e5-d3c1-cb0a1080719d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "max_words=5000\n",
        "(x_train,y_train),(x_test,y_test)=imdb.load_data(num_words=max_words)\n",
        "print(\"The number of 2's in the x_train data set = {}\\n\".format(sum([1 for x in np.hstack(x_train) if x==2])))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The number of 2's in the x_train data set = 592372\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n9Sm_zbolngj"
      },
      "source": [
        "### Word index\n",
        "\n",
        "Keras provides also a dictionary of word to index. We don't need it for training but it helps us get an idea what the reviews are saying in plain English. From that we build a dictionary of index to words. We use the index_to_word to display the first review in the data set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5sKLW9l0YoYE",
        "outputId": "ab93bf16-c1eb-46f8-9d2f-7ca9538c2e3d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "\n",
        "word_to_index=imdb.get_word_index()\n",
        "index_to_word=dict([(key,val) for (val,key) in word_to_index.items()])\n",
        "review = \" \".join( [index_to_word.get(i - 3, \"***\") for i in x_train[0]] )\n",
        "print(review)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "*** this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert *** is an amazing actor and now the same being director *** father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for *** and would recommend it to everyone to watch and the fly *** was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also *** to the two little *** that played the *** of norman and paul they were just brilliant children are often left out of the *** list i think because the stars that play them all grown up are such a big *** for the whole film but these children are amazing and should be *** for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was *** with us all\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bqB0XxTDtS8G"
      },
      "source": [
        "## One hot Encoding\n",
        "\n",
        "Even though this problem is similar to the image classification problem that we saw we cannot feed the word indices to our model directly. First, not all the reviews are the same length. Second, the same word in different reviews can occur at different position. For exam the two reviews \" that was a good movie\" and \"That movie was good\" will be interpreted differently because the **same** words occur at different positions. So we need to do the following\n",
        "\n",
        "1. Truncate or pad all reviews to contain the same number of words\n",
        "1. Make sure the same word occurs at the same position in every review. This we do by using one hot encoding.\n",
        "\n",
        "Suppose that our vocabulary contains only 3 words labeled 1,2 and 3. Further, suppose that two reviews have the values [2,1]  and [1,2] respectively then the one hot representation of both is the same:\n",
        "[1,1,0] i.e. words 1 and 2 are present but 3 is missing\n",
        "\n",
        "**NOTE** our representation is not very efficient. A better approach is to use **word embedding** which we don't use here.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "owyL7pdcDqzq"
      },
      "source": [
        "## Implementation Details\n",
        "\n",
        "The indices of a numpy tensor can be arrays. For example suppose that we are given an array A of size _n_ filled with zeros and we need to set the values at positions 1,13,27 with 1. We can perform the operation in one statement as A[[1,13,27]]=1. Knowing that the function one_hot below goes through every review which is a array of indices and sets the corresponding positions to 1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kYprPNE5S-LD",
        "outputId": "0ef35f3e-a2b2-4cbb-8513-95ce83ff174c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "\n",
        "\n",
        "def one_hot(reviews, nb_words = max_words):\n",
        " res = np.zeros((len(reviews), nb_words))\n",
        " for i, review in enumerate(reviews):\n",
        "  res[i, review] = 1\n",
        " return res\n",
        "\n",
        "x_train_one_hot=one_hot(x_train)\n",
        "x_test_one_hot=one_hot(x_test)\n",
        "print(x_train_one_hot.shape)\n"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(25000, 5000)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nwic_ig4Fdt3",
        "outputId": "5b6c4086-1d37-4f15-a31f-a84397df6e51",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "model=tf.keras.models.Sequential()\n",
        "model.add(tf.keras.layers.Dense(1,input_shape=(x_train_one_hot.shape[1],),activation=\"sigmoid\"))\n",
        "model.summary()\n",
        "model.compile(optimizer=\"Adam\",loss=\"binary_crossentropy\",\n",
        "    metrics=[\"accuracy\"])\n",
        "model.fit(\n",
        "    x_train_one_hot,\n",
        "    y_train,\n",
        "    validation_data=(x_test_one_hot,y_test),\n",
        "    batch_size=500,\n",
        "    epochs=10)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "dense (Dense)                (None, 1)                 5001      \n",
            "=================================================================\n",
            "Total params: 5,001\n",
            "Trainable params: 5,001\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/10\n",
            "50/50 [==============================] - 1s 16ms/step - loss: 0.6095 - accuracy: 0.7236 - val_loss: 0.5459 - val_accuracy: 0.8066\n",
            "Epoch 2/10\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 0.4908 - accuracy: 0.8396 - val_loss: 0.4712 - val_accuracy: 0.8408\n",
            "Epoch 3/10\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 0.4276 - accuracy: 0.8637 - val_loss: 0.4274 - val_accuracy: 0.8549\n",
            "Epoch 4/10\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 0.3872 - accuracy: 0.8763 - val_loss: 0.3965 - val_accuracy: 0.8641\n",
            "Epoch 5/10\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 0.3577 - accuracy: 0.8849 - val_loss: 0.3755 - val_accuracy: 0.8692\n",
            "Epoch 6/10\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 0.3360 - accuracy: 0.8900 - val_loss: 0.3597 - val_accuracy: 0.8726\n",
            "Epoch 7/10\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 0.3186 - accuracy: 0.8955 - val_loss: 0.3466 - val_accuracy: 0.8756\n",
            "Epoch 8/10\n",
            "50/50 [==============================] - 1s 11ms/step - loss: 0.3044 - accuracy: 0.8990 - val_loss: 0.3379 - val_accuracy: 0.8779\n",
            "Epoch 9/10\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 0.2927 - accuracy: 0.9033 - val_loss: 0.3288 - val_accuracy: 0.8790\n",
            "Epoch 10/10\n",
            "50/50 [==============================] - 1s 10ms/step - loss: 0.2826 - accuracy: 0.9064 - val_loss: 0.3224 - val_accuracy: 0.8812\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f314a2a8780>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wc3Zyg21gP6r",
        "outputId": "235e2a11-0c83-4388-a6d0-81391746b43e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "model.evaluate(x_test_one_hot,y_test)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "782/782 [==============================] - 2s 2ms/step - loss: 0.3229 - accuracy: 0.8802\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.3229261636734009, 0.8802000284194946]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ItUdg_xKEvfA"
      },
      "source": [
        "## Word Embedding\n",
        "\n",
        "A better approach is to use word embedding. First we truncate/pad all reviews to the same length using the pad_sequences function from Keras. Then we add an Embedding layer as the first layer in our model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vhj6BKTLI6UZ"
      },
      "source": [
        "### EXAMPLE"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6inT2mdzI5Rl",
        "outputId": "40b9dc3a-94a0-49d8-b3c1-cc65ea109a88",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "model = tf.keras.Sequential()\n",
        "model.add(tf.keras.layers.Embedding(6, 1, input_length=4))\n",
        "#model.add(tf.keras.layers.Flatten())\n",
        "#input_array = np.random.randint(10, size=(3, 4))\n",
        "input_array=np.array([[1,2,3,4],[4,3,2,1],[3,3,4,5]])\n",
        "model.compile('rmsprop', 'mse')\n",
        "output_array = model.predict(input_array)\n",
        "print(output_array.shape)\n",
        "print(output_array)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(3, 4, 1)\n",
            "[[[ 0.02138605]\n",
            "  [ 0.02708221]\n",
            "  [ 0.00789357]\n",
            "  [ 0.01462214]]\n",
            "\n",
            " [[ 0.01462214]\n",
            "  [ 0.00789357]\n",
            "  [ 0.02708221]\n",
            "  [ 0.02138605]]\n",
            "\n",
            " [[ 0.00789357]\n",
            "  [ 0.00789357]\n",
            "  [ 0.01462214]\n",
            "  [-0.04709755]]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "81dCUlWSI8v_"
      },
      "source": [
        "## The Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6Q-utH9Rj3ll",
        "outputId": "6501f511-f21a-4382-f586-fe8f7825b588",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "x_train=pad_sequences(x_train,maxlen=200)\n",
        "x_test=pad_sequences(x_test,maxlen=200)\n",
        "emb=tf.keras.layers.Embedding(max_words,32,input_length=200,mask_zero=True)\n",
        "x=emb(x_train)\n",
        "x.shape\n"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([25000, 200, 32])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2VfoSbrbkPao",
        "outputId": "0b987f53-7e3f-47ce-dad8-2d537c622c68",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(x_train.shape)\n",
        "print(x_train_one_hot.shape)\n",
        "x_train_one_hot[0,100:120]\n"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(25000, 200)\n",
            "(25000, 5000)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1., 0., 0., 1., 1., 0., 1., 1., 0., 0., 0., 0., 1., 1., 0., 0., 0.,\n",
              "       1., 0., 0.])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OYBCzczk8afa",
        "outputId": "070c8d2e-644d-496b-a8c2-4056c81b5936",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "x_train_pad=pad_sequences(x_train,maxlen=200)\n",
        "x_test_pad=pad_sequences(x_test,maxlen=200)\n",
        "\n",
        "model=tf.keras.models.Sequential()\n",
        "model.add(tf.keras.layers.Embedding(max_words,32,input_length=200,mask_zero=True))\n",
        "model.add(tf.keras.layers.Flatten())\n",
        "model.add(tf.keras.layers.Dense(1,activation=\"sigmoid\"))\n",
        "model.summary()\n",
        "model.compile(optimizer=tf.keras.optimizers.Adam(),loss=tf.keras.losses.BinaryCrossentropy(),\n",
        "    metrics=[\"accuracy\"])\n",
        "history = model.fit(\n",
        "    x_train_pad,\n",
        "    y_train,\n",
        "    validation_data=(x_test_pad,y_test),\n",
        "    batch_size=256,\n",
        "    epochs=20)\n"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential_3\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding_3 (Embedding)      (None, 200, 32)           160000    \n",
            "_________________________________________________________________\n",
            "flatten_2 (Flatten)          (None, 6400)              0         \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 1)                 6401      \n",
            "=================================================================\n",
            "Total params: 166,401\n",
            "Trainable params: 166,401\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n",
            "Epoch 1/20\n",
            "98/98 [==============================] - 1s 11ms/step - loss: 0.6494 - accuracy: 0.6439 - val_loss: 0.5277 - val_accuracy: 0.7820\n",
            "Epoch 2/20\n",
            "98/98 [==============================] - 1s 12ms/step - loss: 0.3826 - accuracy: 0.8529 - val_loss: 0.3264 - val_accuracy: 0.8652\n",
            "Epoch 3/20\n",
            "98/98 [==============================] - 1s 11ms/step - loss: 0.2644 - accuracy: 0.8987 - val_loss: 0.2929 - val_accuracy: 0.8765\n",
            "Epoch 4/20\n",
            "98/98 [==============================] - 1s 11ms/step - loss: 0.2168 - accuracy: 0.9180 - val_loss: 0.2910 - val_accuracy: 0.8758\n",
            "Epoch 5/20\n",
            "98/98 [==============================] - 1s 11ms/step - loss: 0.1831 - accuracy: 0.9360 - val_loss: 0.2864 - val_accuracy: 0.8787\n",
            "Epoch 6/20\n",
            "98/98 [==============================] - 1s 11ms/step - loss: 0.1541 - accuracy: 0.9504 - val_loss: 0.2964 - val_accuracy: 0.8755\n",
            "Epoch 7/20\n",
            "98/98 [==============================] - 1s 11ms/step - loss: 0.1289 - accuracy: 0.9633 - val_loss: 0.3010 - val_accuracy: 0.8750\n",
            "Epoch 8/20\n",
            "98/98 [==============================] - 1s 11ms/step - loss: 0.1041 - accuracy: 0.9751 - val_loss: 0.3089 - val_accuracy: 0.8737\n",
            "Epoch 9/20\n",
            "98/98 [==============================] - 1s 11ms/step - loss: 0.0840 - accuracy: 0.9839 - val_loss: 0.3182 - val_accuracy: 0.8722\n",
            "Epoch 10/20\n",
            "98/98 [==============================] - 1s 11ms/step - loss: 0.0671 - accuracy: 0.9899 - val_loss: 0.3278 - val_accuracy: 0.8712\n",
            "Epoch 11/20\n",
            "98/98 [==============================] - 1s 10ms/step - loss: 0.0519 - accuracy: 0.9942 - val_loss: 0.3386 - val_accuracy: 0.8693\n",
            "Epoch 12/20\n",
            "98/98 [==============================] - 1s 10ms/step - loss: 0.0408 - accuracy: 0.9968 - val_loss: 0.3500 - val_accuracy: 0.8682\n",
            "Epoch 13/20\n",
            "98/98 [==============================] - 1s 10ms/step - loss: 0.0326 - accuracy: 0.9980 - val_loss: 0.3596 - val_accuracy: 0.8683\n",
            "Epoch 14/20\n",
            "98/98 [==============================] - 1s 10ms/step - loss: 0.0261 - accuracy: 0.9990 - val_loss: 0.3691 - val_accuracy: 0.8682\n",
            "Epoch 15/20\n",
            "98/98 [==============================] - 1s 10ms/step - loss: 0.0212 - accuracy: 0.9995 - val_loss: 0.3788 - val_accuracy: 0.8675\n",
            "Epoch 16/20\n",
            "98/98 [==============================] - 1s 10ms/step - loss: 0.0177 - accuracy: 0.9996 - val_loss: 0.3885 - val_accuracy: 0.8675\n",
            "Epoch 17/20\n",
            "98/98 [==============================] - 1s 10ms/step - loss: 0.0146 - accuracy: 0.9999 - val_loss: 0.3972 - val_accuracy: 0.8678\n",
            "Epoch 18/20\n",
            "98/98 [==============================] - 1s 10ms/step - loss: 0.0123 - accuracy: 0.9999 - val_loss: 0.4067 - val_accuracy: 0.8665\n",
            "Epoch 19/20\n",
            "98/98 [==============================] - 1s 10ms/step - loss: 0.0105 - accuracy: 0.9999 - val_loss: 0.4152 - val_accuracy: 0.8668\n",
            "Epoch 20/20\n",
            "98/98 [==============================] - 1s 11ms/step - loss: 0.0090 - accuracy: 1.0000 - val_loss: 0.4228 - val_accuracy: 0.8668\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}