tutorial-notebooks/2_embeddings_and_classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "2_embeddings_and_classification.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c0PLMrnTDnIk",
        "colab_type": "text"
      },
      "source": [
        "## Contents taken from Udacity tensorflow-2 course's Lesson 9."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "9uBA1i1BbiJn"
      },
      "source": [
        "# Word Embeddings and Sentiment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1iLe4E0dB7tj"
      },
      "source": [
        "In this colab, you'll work with word embeddings and train a basic recurrent neural network to predict text sentiment. At the end, you'll be able to visualize how the network sees the related sentiment of each word in the dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wqvz1jVgbwIN"
      },
      "source": [
        "## Import TensorFlow and related functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "XIG52aKPdpux",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pazU5OmxehIA"
      },
      "source": [
        "## Get the dataset\n",
        "\n",
        "We're going to use a dataset containing Amazon and Yelp reviews, with their related sentiment (1 for positive, 0 for negative). This dataset was originally extracted from [here](https://www.kaggle.com/marklvl/sentiment-labelled-sentences-data-set)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "qpwQT2E9ez5B",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 326
        },
        "outputId": "c9abcf80-68dd-4b25-fca6-8f4eee724555"
      },
      "source": [
        "!wget --no-check-certificate \\\n",
        "    -O /tmp/sentiment.csv https://drive.google.com/uc?id=13ySLC_ue6Umt9RJYSeM2t-V0kCv-4C-P"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2020-08-07 11:27:25--  https://drive.google.com/uc?id=13ySLC_ue6Umt9RJYSeM2t-V0kCv-4C-P\n",
            "Resolving drive.google.com (drive.google.com)... 172.217.218.101, 172.217.218.113, 172.217.218.102, ...\n",
            "Connecting to drive.google.com (drive.google.com)|172.217.218.101|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Moved Temporarily\n",
            "Location: https://doc-08-ak-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/i2t5jtrhemee57u8n48k94i8ld14phmb/1596799575000/11118900490791463723/*/13ySLC_ue6Umt9RJYSeM2t-V0kCv-4C-P [following]\n",
            "Warning: wildcards not supported in HTTP.\n",
            "--2020-08-07 11:27:26--  https://doc-08-ak-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/i2t5jtrhemee57u8n48k94i8ld14phmb/1596799575000/11118900490791463723/*/13ySLC_ue6Umt9RJYSeM2t-V0kCv-4C-P\n",
            "Resolving doc-08-ak-docs.googleusercontent.com (doc-08-ak-docs.googleusercontent.com)... 108.177.126.132, 2a00:1450:4013:c01::84\n",
            "Connecting to doc-08-ak-docs.googleusercontent.com (doc-08-ak-docs.googleusercontent.com)|108.177.126.132|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 127831 (125K) [text/csv]\n",
            "Saving to: ‘/tmp/sentiment.csv’\n",
            "\n",
            "\r/tmp/sentiment.csv    0%[                    ]       0  --.-KB/s               \r/tmp/sentiment.csv  100%[===================>] 124.83K  --.-KB/s    in 0.001s  \n",
            "\n",
            "2020-08-07 11:27:26 (105 MB/s) - ‘/tmp/sentiment.csv’ saved [127831/127831]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "6Zvp9NScfMnw",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "73e59be9-0326-4fc2-f921-b602ce5609c3"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "dataset = pd.read_csv('/tmp/sentiment.csv')\n",
        "\n",
        "sentences = dataset['text'].tolist()\n",
        "labels = dataset['sentiment'].tolist()\n",
        "\n",
        "# Separate out the sentences and labels into training and test sets\n",
        "training_size = int(len(sentences) * 0.8)\n",
        "\n",
        "training_sentences = sentences[0:training_size]\n",
        "testing_sentences = sentences[training_size:]\n",
        "training_labels = labels[0:training_size]\n",
        "testing_labels = labels[training_size:]\n",
        "\n",
        "# Make labels into numpy arrays for use with the network later\n",
        "training_labels_final = np.array(training_labels)\n",
        "testing_labels_final = np.array(testing_labels)\n",
        "\n",
        "print(\"Example training inputs and outputs\")\n",
        "for i in range(5):\n",
        "  print(training_sentences[i], \"---->\" , training_labels[i])\n"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Example training inputs and outputs\n",
            "So there is no way for me to plug it in here in the US unless I go by a converter. ----> 0\n",
            "Good case Excellent value. ----> 1\n",
            "Great for the jawbone. ----> 1\n",
            "Tied to charger for conversations lasting more than 45 minutes.MAJOR PROBLEMS!! ----> 0\n",
            "The mic is great. ----> 1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "NHpvqaSigcST"
      },
      "source": [
        "## Tokenize the dataset\n",
        "\n",
        "Tokenize the dataset, including padding and OOV, we'll follow the same process as we did in tutorial-1 on text_processing."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "78icewYRgfxh",
        "colab": {}
      },
      "source": [
        "vocab_size = 1000 ## Maximum vocabulary size, will consider only 1000 most \n",
        "                  ## occuring words and assign <OOV> id to everything else\n",
        "\n",
        "embedding_dim = 16\n",
        "max_length = 100  ## Maximum number of input tokens our architecture will read\n",
        "trunc_type='post'\n",
        "padding_type='post'\n",
        "oov_tok = \"<OOV>\"\n",
        "BATCH_SIZE=128\n",
        "\n",
        "\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "tokenizer = Tokenizer(num_words = vocab_size, oov_token=oov_tok)\n",
        "tokenizer.fit_on_texts(training_sentences)\n",
        "word_index = tokenizer.word_index\n",
        "sequences = tokenizer.texts_to_sequences(training_sentences)\n",
        "padded = pad_sequences(sequences,maxlen=max_length, padding=padding_type, \n",
        "                       truncating=trunc_type)\n",
        "\n",
        "testing_sequences = tokenizer.texts_to_sequences(testing_sentences)\n",
        "testing_padded = pad_sequences(testing_sequences,maxlen=max_length, \n",
        "                               padding=padding_type, truncating=trunc_type)\n",
        "\n"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "q4yIEk_8kszh"
      },
      "source": [
        "## Prepare tf.dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "JTU3FmVGk100",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "03abc8d6-019b-4d93-b57d-61abbec6a3a8"
      },
      "source": [
        "dataset = tf.data.Dataset.from_tensor_slices((padded, training_labels))\n",
        "dataset = dataset.batch(BATCH_SIZE)\n",
        "\n",
        "example_input_batch, example_target_batch = next(iter(dataset))\n",
        "example_input_batch.shape, example_target_batch.shape"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorShape([128, 100]), TensorShape([128]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RI91liJnlA92"
      },
      "source": [
        "## Train a Basic Sentiment Model with Embeddings\n",
        "\n",
        "The basic model looks like below figure:\n",
        "\n",
        "<p align=\"center\">\n",
        "  <img src=\"https://cdn.analyticsvidhya.com/wp-content/uploads/2019/01/Screenshot-from-2019-01-18-15-51-14.png\" />\n",
        "</p>\n",
        "\n",
        "*E* = represents embedding layer (converts token_id to a vector of length embedding_dim)  \n",
        "*a <sup>\\<i\\></sup>* = Hidden state propogating in RNN network  \n",
        "*y_hat* = classfication output"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "bBMgzp-_lMTp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 255
        },
        "outputId": "93ec7339-127c-4933-9879-3f747af31dfe"
      },
      "source": [
        "# Build a basic sentiment network\n",
        "# Note the embedding layer is first, \n",
        "# and the output is only 1 node as it is either 0 or 1 (negative or positive)\n",
        "\n",
        "\n",
        "## tf.keras.Sequential groups a linear stack of layers into a tf.keras.Model.\n",
        "\n",
        "\n",
        "\n",
        "model = tf.keras.Sequential([\n",
        "    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),\n",
        "    tf.keras.layers.SimpleRNN(embedding_dim),\n",
        "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "# Show model summary\n",
        "model.summary()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding (Embedding)        (None, 100, 16)           16000     \n",
            "_________________________________________________________________\n",
            "simple_rnn (SimpleRNN)       (None, 16)                528       \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 1)                 17        \n",
            "=================================================================\n",
            "Total params: 16,545\n",
            "Trainable params: 16,545\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uK5WWuHNT8YN",
        "colab_type": "text"
      },
      "source": [
        "### We can see that the model summary follows our architecture's image. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Pfl1W-zVldpn",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        },
        "outputId": "a59c7d9e-64b2-4f98-fbdc-37f835cb2c5d"
      },
      "source": [
        "num_epochs = 10\n",
        "model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])\n",
        "model.fit(padded, training_labels_final, epochs=num_epochs, validation_data=(testing_padded, testing_labels_final))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "50/50 [==============================] - 4s 71ms/step - loss: 0.6925 - accuracy: 0.5229 - val_loss: 0.6955 - val_accuracy: 0.4987\n",
            "Epoch 2/10\n",
            "50/50 [==============================] - 3s 67ms/step - loss: 0.6369 - accuracy: 0.6842 - val_loss: 0.6980 - val_accuracy: 0.5664\n",
            "Epoch 3/10\n",
            "50/50 [==============================] - 3s 67ms/step - loss: 0.5244 - accuracy: 0.7815 - val_loss: 0.6954 - val_accuracy: 0.6015\n",
            "Epoch 4/10\n",
            "50/50 [==============================] - 3s 66ms/step - loss: 0.4414 - accuracy: 0.8267 - val_loss: 0.7419 - val_accuracy: 0.6040\n",
            "Epoch 5/10\n",
            "50/50 [==============================] - 3s 68ms/step - loss: 0.3956 - accuracy: 0.8493 - val_loss: 0.8135 - val_accuracy: 0.5865\n",
            "Epoch 6/10\n",
            "50/50 [==============================] - 3s 69ms/step - loss: 0.3329 - accuracy: 0.8839 - val_loss: 0.8785 - val_accuracy: 0.5815\n",
            "Epoch 7/10\n",
            "50/50 [==============================] - 3s 67ms/step - loss: 0.3050 - accuracy: 0.8939 - val_loss: 0.8585 - val_accuracy: 0.5940\n",
            "Epoch 8/10\n",
            "50/50 [==============================] - 3s 67ms/step - loss: 0.2592 - accuracy: 0.9115 - val_loss: 0.8582 - val_accuracy: 0.5940\n",
            "Epoch 9/10\n",
            "50/50 [==============================] - 3s 66ms/step - loss: 0.2438 - accuracy: 0.9234 - val_loss: 0.8879 - val_accuracy: 0.6190\n",
            "Epoch 10/10\n",
            "50/50 [==============================] - 3s 67ms/step - loss: 0.2085 - accuracy: 0.9335 - val_loss: 0.9631 - val_accuracy: 0.5965\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7efc800bac50>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u43fMxzNWHBw",
        "colab_type": "text"
      },
      "source": [
        "The model didn't do very well on this data and it's understandable as the architecture is fairly small. But the idea was to show how to use *tf.keras.Sequential() API* to build a failry *simple many-to-one RNN acrhitecture*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GjMZ4ZFQl_48"
      },
      "source": [
        "## Get files for visualizing the network\n",
        "\n",
        "The code below will download two files for visualizing how your network \"sees\" the sentiment related to each word. Head to http://projector.tensorflow.org/ and load these files, then click the \"Sphereize\" checkbox."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "S2lB46FirAVx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "ed384f93-15e9-499d-c3ee-f6bf56a1c9f2"
      },
      "source": [
        "# First get the weights of the embedding layer\n",
        "e = model.layers[0]\n",
        "weights = e.get_weights()[0]\n",
        "print(weights.shape) # shape: (vocab_size, embedding_dim)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(1000, 16)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Xcha0oGemHX2",
        "colab": {}
      },
      "source": [
        "import io\n",
        "\n",
        "# Write out the embedding vectors and metadata\n",
        "out_v = io.open('vecs.tsv', 'w', encoding='utf-8')\n",
        "out_m = io.open('meta.tsv', 'w', encoding='utf-8')\n",
        "for word_num in range(1, vocab_size):\n",
        "  word = reverse_word_index[word_num]\n",
        "  embeddings = weights[word_num]\n",
        "  out_m.write(word + \"\\n\")\n",
        "  out_v.write('\\t'.join([str(x) for x in embeddings]) + \"\\n\")\n",
        "out_v.close()\n",
        "out_m.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "g-Q6ALywmWVz",
        "colab": {}
      },
      "source": [
        "# Download the files\n",
        "try:\n",
        "  from google.colab import files\n",
        "except ImportError:\n",
        "  pass\n",
        "else:\n",
        "  files.download('vecs.tsv')\n",
        "  files.download('meta.tsv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GNoxfY-i3Ir1"
      },
      "source": [
        "## Predicting Sentiment in New Reviews\n",
        "\n",
        "Now that you've trained and visualized your network, take a look below at how we can predict sentiment in new reviews the network has never seen before."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "QXtfw-OY3WoZ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 785
        },
        "outputId": "ea138e1b-fcef-4214-9cb3-16a6e4fcf3c2"
      },
      "source": [
        "# Use the model to predict a review   \n",
        "fake_reviews = ['I love this phone', 'I hate spaghetti', \n",
        "                'Everything was cold',\n",
        "                'Everything was hot exactly as I wanted', \n",
        "                'Everything was green', \n",
        "                'the host seated us immediately',\n",
        "                'they gave us free chocolate cake', \n",
        "                'not sure about the wilted flowers on the table',\n",
        "                'only works when I stand on tippy toes', \n",
        "                'does not work when I stand on my head']\n",
        "\n",
        "print(fake_reviews) \n",
        "\n",
        "# Create the sequences\n",
        "padding_type='post'\n",
        "sample_sequences = tokenizer.texts_to_sequences(fake_reviews)\n",
        "fakes_padded = pad_sequences(sample_sequences, padding=padding_type, maxlen=max_length)           \n",
        "\n",
        "print('\\nHOT OFF THE PRESS! HERE ARE SOME NEWLY MINTED, ABSOLUTELY GENUINE REVIEWS!\\n')              \n",
        "\n",
        "classes = model.predict(fakes_padded)\n",
        "\n",
        "# The closer the class is to 1, the more positive the review is deemed to be\n",
        "for x in range(len(fake_reviews)):\n",
        "  print(fake_reviews[x])\n",
        "  print(classes[x])\n",
        "  print('\\n')\n",
        "\n",
        "# Try adding reviews of your own\n",
        "# Add some negative words (such as \"not\") to the good reviews and see what happens\n",
        "# For example:\n",
        "# they gave us free chocolate cake and did not charge us"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['I love this phone', 'I hate spaghetti', 'Everything was cold', 'Everything was hot exactly as I wanted', 'Everything was green', 'the host seated us immediately', 'they gave us free chocolate cake', 'not sure about the wilted flowers on the table', 'only works when I stand on tippy toes', 'does not work when I stand on my head']\n",
            "\n",
            "HOT OFF THE PRESS! HERE ARE SOME NEWLY MINTED, ABSOLUTELY GENUINE REVIEWS!\n",
            "\n",
            "I love this phone\n",
            "[0.96117866]\n",
            "\n",
            "\n",
            "I hate spaghetti\n",
            "[0.02996874]\n",
            "\n",
            "\n",
            "Everything was cold\n",
            "[0.8984161]\n",
            "\n",
            "\n",
            "Everything was hot exactly as I wanted\n",
            "[0.94505495]\n",
            "\n",
            "\n",
            "Everything was green\n",
            "[0.514217]\n",
            "\n",
            "\n",
            "the host seated us immediately\n",
            "[0.9449767]\n",
            "\n",
            "\n",
            "they gave us free chocolate cake\n",
            "[0.92621785]\n",
            "\n",
            "\n",
            "not sure about the wilted flowers on the table\n",
            "[0.12957855]\n",
            "\n",
            "\n",
            "only works when I stand on tippy toes\n",
            "[0.76650923]\n",
            "\n",
            "\n",
            "does not work when I stand on my head\n",
            "[0.02863884]\n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Xag0gKhV7ge",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}