seq2seq_integration_with_android_app.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MaILF2Y66mjv"
   },
   "source": [
    "#Abstractive Summarization of Privacy Policies with Seq2Seq\n",
    "**FYP**\n",
    "BY:\n",
    "SHAHZEB AND MUNEEB\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AOgbGhKS6mjx"
   },
   "source": [
    "The objective of this project is to build a model that can create relevant abstractive summaries for Privacy Policies preserving the original context. \n",
    "To build our model we will use a two-layered bidirectional RNN with LSTMs on the input data and two layers, each with an LSTM using attention on the target data.\n",
    "The sections of this project are:\n",
    "\n",
    "- Preparing the Data\n",
    "- Building the Model\n",
    "- Training the Model\n",
    "- Making Our Own Abstractive Summaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 64
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3064,
     "status": "ok",
     "timestamp": 1583157206606,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "Tw0keeCK6mjy",
    "outputId": "2c0dbf3a-5640-47bd-b62c-1c4617afbd68"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "import time\n",
    "from tensorflow.python.layers.core import Dense\n",
    "#import data_utils\n",
    "from tensorflow.python.ops.rnn_cell_impl import _zero_state_tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3048,
     "status": "ok",
     "timestamp": 1583157206617,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "djqUeGR4CLdh",
    "outputId": "1c57222e-409f-4dee-c304-bd0f2892f0c5"
   },
   "outputs": [],
   "source": [
    "print('TensorFlow Version: {}'.format(tf.__version__))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 4456,
     "status": "ok",
     "timestamp": 1583157208060,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "V-9OEfMioyUa",
    "outputId": "d6d66fa9-0552-4420-fc5d-572e3b879d8a"
   },
   "outputs": [],
   "source": [
    "#device_name = tf.test.gpu_device_name()\n",
    "#if device_name != '/device:GPU:0':\n",
    "#  raise SystemError('GPU device not found')\n",
    "#print('Found GPU at: {}'.format(device_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "H0o8PPgD63ZP"
   },
   "source": [
    "### MOunt Drive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 141
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 33874,
     "status": "ok",
     "timestamp": 1583157237506,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "frzAZi2D67tn",
    "outputId": "c29a42cb-76d2-47e7-a917-30fd9cb9a2a5"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "currentDirectory = os.getcwd()\n",
    "print (currentDirectory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c7s5j-U46mj5"
   },
   "source": [
    "## Exploring The Data set\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "o24fDOcG6mj6"
   },
   "outputs": [],
   "source": [
    "headings = pd.read_csv('./data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 173
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6193,
     "status": "ok",
     "timestamp": 1583157246190,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "pv2B397Z6mj8",
    "outputId": "0f1df434-33d1-42d5-b223-9f0eac0e4e4f"
   },
   "outputs": [],
   "source": [
    "print(headings.shape)\n",
    "print(headings.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 87
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 5666,
     "status": "ok",
     "timestamp": 1583157246198,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "NbtLoD046mkD",
    "outputId": "6872b215-8d40-4b2c-cce4-9a19c046284c"
   },
   "outputs": [],
   "source": [
    "# Check for any null values in the dataset\n",
    "headings.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oaJHve336uPq"
   },
   "outputs": [],
   "source": [
    "headings = headings.drop(['Unnamed: 0'], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "m55yoyyc6mkG"
   },
   "outputs": [],
   "source": [
    "# Remove null values and unneeded features\n",
    "headings = headings.dropna()\n",
    "\n",
    "headings = headings.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 202
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 4710,
     "status": "ok",
     "timestamp": 1583157246203,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "p8tL54i16mkJ",
    "outputId": "0e708555-d8c7-4509-a5d9-013b56ff0e49"
   },
   "outputs": [],
   "source": [
    "headings.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 471
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 4161,
     "status": "ok",
     "timestamp": 1583157246205,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "Z054OKaL6mkM",
    "outputId": "233d6167-206d-4cb1-b157-a3dd013f2cef"
   },
   "outputs": [],
   "source": [
    "# Inspecting some of the items from the dataset\n",
    "for i in range(5):\n",
    "    print(\"Summary#\",i+1)\n",
    "    print(headings.title[i])\n",
    "    print(\"Original scenerio#\",i+1)\n",
    "    print(headings.content[i])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3836,
     "status": "ok",
     "timestamp": 1583157246206,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "zNP519m94Dl1",
    "outputId": "93fbc172-786c-41e8-8250-e8efa915496d"
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_4iCevEI6mkP"
   },
   "source": [
    "## Preparing the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Bgki8Rny6mkP"
   },
   "outputs": [],
   "source": [
    "# A list of contractions from http://stackoverflow.com/questions/19790188/expanding-english-language-contractions-in-python\n",
    "contractions = { \n",
    "\"ain't\": \"am not\",\n",
    "\"aren't\": \"are not\",\n",
    "\"can't\": \"cannot\",\n",
    "\"can't've\": \"cannot have\",\n",
    "\"'cause\": \"because\",\n",
    "\"could've\": \"could have\",\n",
    "\"couldn't\": \"could not\",\n",
    "\"couldn't've\": \"could not have\",\n",
    "\"didn't\": \"did not\",\n",
    "\"doesn't\": \"does not\",\n",
    "\"don't\": \"do not\",\n",
    "\"hadn't\": \"had not\",\n",
    "\"hadn't've\": \"had not have\",\n",
    "\"hasn't\": \"has not\",\n",
    "\"haven't\": \"have not\",\n",
    "\"he'd\": \"he would\",\n",
    "\"he'd've\": \"he would have\",\n",
    "\"he'll\": \"he will\",\n",
    "\"he's\": \"he is\",\n",
    "\"how'd\": \"how did\",\n",
    "\"how'll\": \"how will\",\n",
    "\"how's\": \"how is\",\n",
    "\"i'd\": \"i would\",\n",
    "\"i'll\": \"i will\",\n",
    "\"i'm\": \"i am\",\n",
    "\"i've\": \"i have\",\n",
    "\"isn't\": \"is not\",\n",
    "\"it'd\": \"it would\",\n",
    "\"it'll\": \"it will\",\n",
    "\"it's\": \"it is\",\n",
    "\"let's\": \"let us\",\n",
    "\"ma'am\": \"madam\",\n",
    "\"mayn't\": \"may not\",\n",
    "\"might've\": \"might have\",\n",
    "\"mightn't\": \"might not\",\n",
    "\"must've\": \"must have\",\n",
    "\"mustn't\": \"must not\",\n",
    "\"needn't\": \"need not\",\n",
    "\"oughtn't\": \"ought not\",\n",
    "\"shan't\": \"shall not\",\n",
    "\"sha'n't\": \"shall not\",\n",
    "\"she'd\": \"she would\",\n",
    "\"she'll\": \"she will\",\n",
    "\"she's\": \"she is\",\n",
    "\"should've\": \"should have\",\n",
    "\"shouldn't\": \"should not\",\n",
    "\"that'd\": \"that would\",\n",
    "\"that's\": \"that is\",\n",
    "\"there'd\": \"there had\",\n",
    "\"there's\": \"there is\",\n",
    "\"they'd\": \"they would\",\n",
    "\"they'll\": \"they will\",\n",
    "\"they're\": \"they are\",\n",
    "\"they've\": \"they have\",\n",
    "\"wasn't\": \"was not\",\n",
    "\"we'd\": \"we would\",\n",
    "\"we'll\": \"we will\",\n",
    "\"we're\": \"we are\",\n",
    "\"we've\": \"we have\",\n",
    "\"weren't\": \"were not\",\n",
    "\"what'll\": \"what will\",\n",
    "\"what're\": \"what are\",\n",
    "\"what's\": \"what is\",\n",
    "\"what've\": \"what have\",\n",
    "\"where'd\": \"where did\",\n",
    "\"where's\": \"where is\",\n",
    "\"who'll\": \"who will\",\n",
    "\"who's\": \"who is\",\n",
    "\"won't\": \"will not\",\n",
    "\"wouldn't\": \"would not\",\n",
    "\"you'd\": \"you would\",\n",
    "\"you'll\": \"you will\",\n",
    "\"you're\": \"you are\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kXEYjxGE6mkT"
   },
   "outputs": [],
   "source": [
    "def clean_text(text, remove_stopwords = True):\n",
    "    '''Remove unwanted characters, stopwords, and format the text to create fewer nulls word embeddings'''\n",
    "    \n",
    "    # Convert words to lower case\n",
    "    text = text.lower()\n",
    "    \n",
    "    # Replace shorter contractions with their longer forms \n",
    "    if True:\n",
    "        text = text.split()\n",
    "        new_text = []\n",
    "        for word in text:\n",
    "            if word in contractions:\n",
    "                new_text.append(contractions[word])\n",
    "            else:\n",
    "                new_text.append(word)\n",
    "        text = \" \".join(new_text)\n",
    "    \n",
    "    # Format words and remove unwanted characters if left although we have removed them already while scraping\n",
    "    text = re.sub(r'https?:\\/\\/.*[\\r\\n]*', '', text, flags=re.MULTILINE)\n",
    "    text = re.sub(r'\\<a href', ' ', text)\n",
    "    text = re.sub(r'&amp;', '', text) \n",
    "    text = re.sub(r'[_\"\\-;%()|+&=*%.,!?:#$@\\[\\]/]', ' ', text)\n",
    "    text = re.sub(r'<br />', ' ', text)\n",
    "    text = re.sub(r'\\'', ' ', text)\n",
    "    text = re.sub(r'\\n', ' ', text)\n",
    "    \n",
    "    # Optionally, remove stop words\n",
    "    if remove_stopwords:\n",
    "        text = text.split()\n",
    "        stops = set(stopwords.words(\"english\"))\n",
    "        text = [w for w in text if not w in stops]\n",
    "        text = \" \".join(text)\n",
    "\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FbYOStM-6mkV"
   },
   "source": [
    "We will remove the stopwords from the texts because they do not provide much use for training our model.\n",
    "However, we will keep them for our summaries so that they sound more like natural phrases. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 87
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3040,
     "status": "ok",
     "timestamp": 1583157248229,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "da1ofCGn6mkW",
    "outputId": "3d828598-a854-4d07-ae80-e1fbf4b121a7"
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean the summaries and texts\n",
    "clean_summaries = []\n",
    "for summary in headings.title:\n",
    "    clean_summaries.append(clean_text(summary, remove_stopwords=False))\n",
    "print(\"Summaries are complete.\")\n",
    "\n",
    "clean_texts = []\n",
    "for text in headings.content:\n",
    "    clean_texts.append(clean_text(text))\n",
    "print(\"Texts are complete.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ov6gNQzH6mkd"
   },
   "outputs": [],
   "source": [
    "def count_words(count_dict, text):\n",
    "    '''Count the number of occurrences of each word in a set of text'''\n",
    "    for sentence in text:\n",
    "        for word in sentence.split():\n",
    "            if word not in count_dict:\n",
    "                count_dict[word] = 1\n",
    "            else:\n",
    "                count_dict[word] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2431,
     "status": "ok",
     "timestamp": 1583157248233,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "EL2TwE406mkh",
    "outputId": "5ad9f7af-f7c4-4a46-a70e-5a084902ae33"
   },
   "outputs": [],
   "source": [
    "# Find the number of times each word was used and the size of the vocabulary\n",
    "word_counts = {}\n",
    "\n",
    "count_words(word_counts, clean_summaries)\n",
    "count_words(word_counts, clean_texts)\n",
    "            \n",
    "print(\"Size of Vocabulary:\", len(word_counts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a8CnYUDXaDs0"
   },
   "source": [
    "#Word Embeddings\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 39864,
     "status": "ok",
     "timestamp": 1583157287186,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "paiPI3pR6mko",
    "outputId": "b8e8e125-65d2-498d-ebe5-ed434221baac"
   },
   "outputs": [],
   "source": [
    "\n",
    "# Load Conceptnet Numberbatch's (CN) embeddings, probably better than Glove \n",
    "#(https://github.com/commonsense/conceptnet-numberbatch)\n",
    "\n",
    "embeddings_index = {}\n",
    "with open('./numberbatch-en.txt', encoding='utf-8') as f:\n",
    "    for line in f:\n",
    "        #print (line)\n",
    "        values = line.split(' ')\n",
    "        #print (values[0])\n",
    "        word = values[0]\n",
    "        #print (values[1:])\n",
    "        embedding = np.asarray(values[1:], dtype='float32')\n",
    "        embeddings_index[word] = embedding\n",
    "\n",
    "        \n",
    "print('Word embeddings:', len(embeddings_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 89
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 29030,
     "status": "ok",
     "timestamp": 1583157287188,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "kQgzpE_q6mkr",
    "outputId": "389111e0-1949-457e-8139-8f5aaea20944"
   },
   "outputs": [],
   "source": [
    "# Find the number of words that are missing from CN, and are used more than our threshold.\n",
    "missing_words = 0\n",
    "threshold = 20    #20threshold #Thrshold is here if any word appear more than 20 therefore\n",
    "#I use a threshold of 20, so that words not in CN can be added to our word_embedding_matrix, but they need to be common enough in the paragraphs so that the model can understand their meaning.\n",
    "rey=[]\n",
    "for word, count in word_counts.items():\n",
    "    if count > threshold:\n",
    "        if word not in embeddings_index:\n",
    "            missing_words += 1\n",
    "            rey.append(word)\n",
    "            \n",
    "missing_ratio = round(missing_words/len(word_counts),4)*100\n",
    "            \n",
    "print(\"Number of words missing from CN:\", missing_words)\n",
    "print(\"Percent of words that are missing from vocabulary: {}%\".format(missing_ratio))\n",
    "print(rey)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 23124,
     "status": "ok",
     "timestamp": 1583157287195,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "3wR-Uv4F6mky",
    "outputId": "8dd42163-c33a-4648-d900-6ac6394035e7"
   },
   "outputs": [],
   "source": [
    "# Limit the vocab that we will use to words that appear more tha threshold or are in GloVe\n",
    "\n",
    "#dictionary to convert words to integers\n",
    "vocab_to_int = {} \n",
    "\n",
    "value = 0\n",
    "for word, count in word_counts.items():\n",
    "    if count >= threshold or word in embeddings_index:\n",
    "        vocab_to_int[word] = value\n",
    "        value += 1\n",
    "\n",
    "# Special tokens that will be added to our vocab\n",
    "codes = [\"<UNK>\",\"<PAD>\",\"<EOS>\",\"<GO>\"]   \n",
    "\n",
    "# Add codes to vocab\n",
    "for code in codes:\n",
    "    vocab_to_int[code] = len(vocab_to_int)\n",
    "\n",
    "# Dictionary to convert integers to words\n",
    "int_to_vocab = {}\n",
    "for word, value in vocab_to_int.items():\n",
    "    int_to_vocab[value] = word\n",
    "\n",
    "usage_ratio = round(len(vocab_to_int) / len(word_counts),4)*100\n",
    "\n",
    "print(\"Total number of unique words:\", len(word_counts))\n",
    "print(\"Number of words we will use:\", len(vocab_to_int))\n",
    "print(\"Percent of words we will use: {}%\".format(usage_ratio))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 18779,
     "status": "ok",
     "timestamp": 1583157288711,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "7sPlq0P06mk4",
    "outputId": "eec00dc8-ed39-41b5-9218-cf58fea4bdf8"
   },
   "outputs": [],
   "source": [
    "# Need to use 300 for embedding dimensions to match CN's vectors.\n",
    "embedding_dim = 300\n",
    "nb_words = len(vocab_to_int)\n",
    "\n",
    "# Create matrix with default values of zero\n",
    "word_embedding_matrix = np.zeros((nb_words, embedding_dim), dtype=np.float32)\n",
    "for word, i in vocab_to_int.items():\n",
    "    if word in embeddings_index:\n",
    "        word_embedding_matrix[i] = embeddings_index[word]\n",
    "    else:\n",
    "        # If word not in CN, create a random embedding for it\n",
    "        new_embedding = np.array(np.random.uniform(-1.0, 1.0, embedding_dim))\n",
    "        #print (new_embedding)\n",
    "        embeddings_index[word] = new_embedding\n",
    "        word_embedding_matrix[i] = new_embedding \n",
    "\n",
    "# Check if value matches len(vocab_to_int)\n",
    "print(len(word_embedding_matrix))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lCQ36Smn6mk-"
   },
   "outputs": [],
   "source": [
    "def convert_to_ints(text, word_count, unk_count, eos=False):\n",
    "    '''Convert words in text to an integer.\n",
    "       If word is not in vocab_to_int, use UNK's integer.\n",
    "       Total the number of words and UNKs.\n",
    "       Add EOS token to the end of texts'''\n",
    "    ints = []\n",
    "    for sentence in text:\n",
    "        sentence_ints = []\n",
    "        for word in sentence.split():\n",
    "            word_count += 1\n",
    "            if word in vocab_to_int:\n",
    "                sentence_ints.append(vocab_to_int[word])\n",
    "            else:\n",
    "                sentence_ints.append(vocab_to_int[\"<UNK>\"])\n",
    "                unk_count += 1\n",
    "        if eos:\n",
    "            sentence_ints.append(vocab_to_int[\"<EOS>\"])\n",
    "        ints.append(sentence_ints)\n",
    "    return ints, word_count, unk_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 15663,
     "status": "ok",
     "timestamp": 1583157288715,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "547FFl3u6mlA",
    "outputId": "6af7c6bb-2092-43a1-be99-4c5538d3d4b4"
   },
   "outputs": [],
   "source": [
    "# Apply convert_to_ints to clean_summaries and clean_texts\n",
    "word_count = 0\n",
    "unk_count = 0\n",
    "\n",
    "int_summaries, word_count, unk_count = convert_to_ints(clean_summaries, word_count, unk_count)\n",
    "int_texts, word_count, unk_count = convert_to_ints(clean_texts, word_count, unk_count, eos=True)\n",
    "\n",
    "unk_percent = round(unk_count/word_count,4)*100\n",
    "\n",
    "print(\"Total number of words in Vocab:\", word_count)\n",
    "print(\"Total number of UNKs in Vocab:\", unk_count)\n",
    "print(\"Percent of words that are UNK: {}%\".format(unk_percent))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 107
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 14636,
     "status": "ok",
     "timestamp": 1583157288717,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "E8js9XjmKSEK",
    "outputId": "1f615feb-942e-4aba-c22e-4027a3c3fd95"
   },
   "outputs": [],
   "source": [
    "#print (int_to_vocab[value] for value in int_summaries[:5])\n",
    "for value in int_summaries[2:]:\n",
    "  for value2 in value:\n",
    "    print (int_to_vocab[value2],end=\" \")\n",
    "  break\n",
    "print ()\n",
    "print (len(int_summaries))\n",
    "print (int_texts[:5])\n",
    "print (len(int_texts))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dnRzSx6M6mlD"
   },
   "outputs": [],
   "source": [
    "def create_lengths(text):\n",
    "    '''Create a data frame of the sentence lengths from a text'''\n",
    "    lengths = []\n",
    "    for sentence in text:\n",
    "        lengths.append(len(sentence))\n",
    "    return pd.DataFrame(lengths, columns=['counts'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 381
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12951,
     "status": "ok",
     "timestamp": 1583157288719,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "5_i-9rYV6mlF",
    "outputId": "e87b1c9d-7cc0-41a5-d1eb-0d7d66f08ac0"
   },
   "outputs": [],
   "source": [
    "lengths_summaries = create_lengths(int_summaries)\n",
    "lengths_texts = create_lengths(int_texts)\n",
    "\n",
    "print(\"Extracted Summaries:\")\n",
    "print(lengths_summaries.describe())\n",
    "print()\n",
    "print(\"Paragraphs:\")\n",
    "print(lengths_texts.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 121
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 12070,
     "status": "ok",
     "timestamp": 1583157288721,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "gD0oMIbeLyJr",
    "outputId": "3542246a-ed07-4890-bb5a-352bbaf9c151"
   },
   "outputs": [],
   "source": [
    "print (lengths_summaries[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 11397,
     "status": "ok",
     "timestamp": 1583157288723,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "wVMlHmV46mlI",
    "outputId": "5813e474-9415-41ec-e619-d93dccada94b"
   },
   "outputs": [],
   "source": [
    "# Inspect the length of texts\n",
    "#print(lengths_texts,count)\n",
    "\n",
    "print(np.percentile(lengths_texts.counts, 90))\n",
    "print(np.percentile(lengths_texts.counts, 95))\n",
    "print(np.percentile(lengths_texts.counts, 99))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 10442,
     "status": "ok",
     "timestamp": 1583157288724,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "Ic8buaf16mlL",
    "outputId": "18fe3a52-6f04-4c2a-de06-23e90e4465d5"
   },
   "outputs": [],
   "source": [
    "# Inspect the length of summaries\n",
    "print(np.percentile(lengths_summaries.counts, 90))\n",
    "print(np.percentile(lengths_summaries.counts, 95))\n",
    "print(np.percentile(lengths_summaries.counts, 99))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IPW-kF136mlQ"
   },
   "outputs": [],
   "source": [
    "def unk_counter(sentence):\n",
    "    '''Counts the number of time UNK appears in a sentence.'''\n",
    "    unk_count = 0\n",
    "    for word in sentence:\n",
    "        if word == vocab_to_int[\"<UNK>\"]:\n",
    "            unk_count += 1\n",
    "    return unk_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 8292,
     "status": "ok",
     "timestamp": 1583157290503,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "EbFt7RTA6mlU",
    "outputId": "3ea92f8e-caad-44e8-c06c-213ba056ce17"
   },
   "outputs": [],
   "source": [
    "# takes a long time  , this is normal\n",
    "\n",
    "# Sort the summaries and texts by the length of the texts, shortest to longest\n",
    "# Limit the length of summaries and texts based on the min and max ranges.\n",
    "# Remove summaries that include too many UNKs\n",
    "\n",
    "sorted_summaries = []\n",
    "sorted_texts = []\n",
    "max_text_length = 60\n",
    "max_summary_length = 20\n",
    "min_length = 2\n",
    "unk_text_limit = 3\n",
    "unk_summary_limit = 1\n",
    "\n",
    "for length in range(min(lengths_texts.counts), max_text_length): \n",
    "    for count, words in enumerate(int_summaries):\n",
    "        if (len(int_summaries[count]) >= min_length and\n",
    "            len(int_summaries[count]) <= max_summary_length and\n",
    "            len(int_texts[count]) >= min_length and\n",
    "            unk_counter(int_summaries[count]) <= unk_summary_limit and\n",
    "            unk_counter(int_texts[count]) <= unk_text_limit and\n",
    "            length == len(int_texts[count])\n",
    "           ):\n",
    "            sorted_summaries.append(int_summaries[count])\n",
    "            sorted_texts.append(int_texts[count])\n",
    "        \n",
    "# Compare lengths to ensure they match\n",
    "print(len(sorted_summaries))\n",
    "print(len(sorted_texts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 55
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 191205,
     "status": "ok",
     "timestamp": 1581586460873,
     "user": {
      "displayName": "i160246 Muhammad Shahzeb Ali",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mDNXixi8uT005c0frFjGaAf4wiyWf4rXJj_nITRSg=s64",
      "userId": "18251415013789702811"
     },
     "user_tz": -300
    },
    "id": "ZRReKznRSPLo",
    "outputId": "ec4b1591-07b2-4b74-bdf2-0fec525b6fca"
   },
   "outputs": [],
   "source": [
    "print (sorted_summaries[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jYnQFLER6mlY"
   },
   "source": [
    "## Build Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "U1MIZPcl6mlZ"
   },
   "outputs": [],
   "source": [
    "def model_inputs():\n",
    "    '''Create palceholders for inputs to the model'''\n",
    "    \n",
    "    input_data = tf.placeholder(tf.int32, [None, None], name='input')\n",
    "    targets = tf.placeholder(tf.int32, [None, None], name='targets')\n",
    "    lr = tf.placeholder(tf.float32, name='learning_rate')\n",
    "    keep_prob = tf.placeholder(tf.float32, name='keep_prob')\n",
    "    summary_length = tf.placeholder(tf.int32, (None,), name='summary_length')\n",
    "    max_summary_length = tf.reduce_max(summary_length, name='max_dec_len')\n",
    "    text_length = tf.placeholder(tf.int32, (None,), name='text_length')\n",
    "\n",
    "    return input_data, targets, lr, keep_prob, summary_length, max_summary_length, text_length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dGoxdoHz6mlc"
   },
   "outputs": [],
   "source": [
    "def process_encoding_input(target_data, vocab_to_int, batch_size):\n",
    "    '''Remove the last word id from each batch and concat the <GO> to the begining of each batch'''\n",
    "    \n",
    "    ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])\n",
    "    dec_input = tf.concat([tf.fill([batch_size, 1], vocab_to_int['<GO>']), ending], 1)\n",
    "\n",
    "    return dec_input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aJ7p3phH6mld"
   },
   "outputs": [],
   "source": [
    "def encoding_layer(rnn_size, sequence_length, num_layers, rnn_inputs, keep_prob):\n",
    "    '''Create the encoding layer'''\n",
    "    \n",
    "    for layer in range(num_layers):\n",
    "        with tf.variable_scope('encoder_{}'.format(layer)):\n",
    "            cell_fw = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            cell_fw = tf.contrib.rnn.DropoutWrapper(cell_fw, \n",
    "                                                    input_keep_prob = keep_prob)\n",
    "\n",
    "            cell_bw = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            cell_bw = tf.contrib.rnn.DropoutWrapper(cell_bw, \n",
    "                                                    input_keep_prob = keep_prob)\n",
    "\n",
    "            enc_output, enc_state = tf.nn.bidirectional_dynamic_rnn(cell_fw, \n",
    "                                                                    cell_bw, \n",
    "                                                                    rnn_inputs,\n",
    "                                                                    sequence_length,\n",
    "                                                                    dtype=tf.float32)\n",
    "    # Join outputs since we are using a bidirectional RNN\n",
    "    enc_output = tf.concat(enc_output,2)\n",
    "    \n",
    "    return enc_output, enc_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ejCY-IG56mlg"
   },
   "outputs": [],
   "source": [
    "\n",
    "def training_decoding_layer(dec_embed_input, summary_length, dec_cell, initial_state, output_layer, \n",
    "                            vocab_size, max_summary_length):\n",
    "    '''Create the training logits'''\n",
    "    \n",
    "    training_helper = tf.contrib.seq2seq.TrainingHelper(inputs=dec_embed_input,\n",
    "                                                        sequence_length=summary_length,\n",
    "                                                        time_major=False)\n",
    "\n",
    "    training_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                       training_helper,\n",
    "                                                       initial_state,\n",
    "                                                       output_layer) \n",
    "\n",
    "    training_logits, _ , _ = tf.contrib.seq2seq.dynamic_decode(training_decoder,\n",
    "                                                           output_time_major=False,\n",
    "                                                           impute_finished=True,\n",
    "                                                           maximum_iterations=max_summary_length)\n",
    "    return training_decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "p5RpDsV56mll"
   },
   "outputs": [],
   "source": [
    "def inference_decoding_layer(embeddings, start_token, end_token, dec_cell, initial_state, output_layer,\n",
    "                             max_summary_length, batch_size):\n",
    "    '''Create the inference logits'''\n",
    "    \n",
    "    start_tokens = tf.tile(tf.constant([start_token], dtype=tf.int32), [batch_size], name='start_tokens')\n",
    "    \n",
    "    inference_helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(embeddings,\n",
    "                                                                start_tokens,\n",
    "                                                                end_token)\n",
    "                \n",
    "    inference_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                        inference_helper,\n",
    "                                                        initial_state,\n",
    "                                                        output_layer)\n",
    "                \n",
    "    inference_logits, _ , _ = tf.contrib.seq2seq.dynamic_decode(inference_decoder,\n",
    "                                                            output_time_major=False,\n",
    "                                                            impute_finished=True,\n",
    "                                                            maximum_iterations=max_summary_length)\n",
    "    \n",
    "    return inference_decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MBcDmTXx6mln"
   },
   "outputs": [],
   "source": [
    "\n",
    "def decoding_layer(dec_embed_input, embeddings, enc_output, enc_state, vocab_size, text_length, summary_length, \n",
    "                   max_summary_length, rnn_size, vocab_to_int, keep_prob, batch_size, num_layers):\n",
    "    '''Create the decoding cell and attention for the training and inference decoding layers'''\n",
    "    \n",
    "    for layer in range(num_layers):\n",
    "        with tf.variable_scope('decoder_{}'.format(layer)):\n",
    "            lstm = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                           initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            dec_cell = tf.contrib.rnn.DropoutWrapper(lstm, \n",
    "                                                     input_keep_prob = keep_prob)\n",
    "    \n",
    "    output_layer = Dense(vocab_size,\n",
    "                         kernel_initializer = tf.truncated_normal_initializer(mean = 0.0, stddev=0.1))\n",
    "    \n",
    "    attn_mech = tf.contrib.seq2seq.BahdanauAttention(rnn_size,\n",
    "                                                  enc_output,\n",
    "                                                  text_length,\n",
    "                                                  normalize=False,\n",
    "                                                  name='BahdanauAttention')\n",
    "\n",
    "    dec_cell = tf.contrib.seq2seq.AttentionWrapper(dec_cell,\n",
    "                                                          attn_mech,\n",
    "                                                          rnn_size)\n",
    "            \n",
    "    #initial_state = tf.contrib.seq2seq.AttentionWrapperState(enc_state[0],\n",
    "    #                                                                _zero_state_tensors(rnn_size, \n",
    "    #                                                                                    batch_size, \n",
    "    #                                                                                    tf.float32)) \n",
    "    initial_state = dec_cell.zero_state(batch_size=batch_size,dtype=tf.float32).clone(cell_state=enc_state[0])\n",
    "\n",
    "    with tf.variable_scope(\"decode\"):\n",
    "        training_decoder = training_decoding_layer(dec_embed_input, \n",
    "                                                  summary_length, \n",
    "                                                  dec_cell, \n",
    "                                                  initial_state,\n",
    "                                                  output_layer,\n",
    "                                                  vocab_size, \n",
    "                                                  max_summary_length)\n",
    "        \n",
    "        training_logits,_ ,_ = tf.contrib.seq2seq.dynamic_decode(training_decoder,\n",
    "                                  output_time_major=False,\n",
    "                                  impute_finished=True,\n",
    "                                  maximum_iterations=max_summary_length)\n",
    "    with tf.variable_scope(\"decode\", reuse=True):\n",
    "        inference_decoder = inference_decoding_layer(embeddings,  \n",
    "                                                    vocab_to_int['<GO>'], \n",
    "                                                    vocab_to_int['<EOS>'],\n",
    "                                                    dec_cell, \n",
    "                                                    initial_state, \n",
    "                                                    output_layer,\n",
    "                                                    max_summary_length,\n",
    "                                                    batch_size)\n",
    "        \n",
    "        inference_logits,_ ,_ = tf.contrib.seq2seq.dynamic_decode(inference_decoder,\n",
    "                                  output_time_major=False,\n",
    "                                  impute_finished=True,\n",
    "                                  maximum_iterations=max_summary_length)\n",
    "\n",
    "    return training_logits, inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JuVESpih6mlo"
   },
   "outputs": [],
   "source": [
    "def seq2seq_model(input_data, target_data, keep_prob, text_length, summary_length, max_summary_length, \n",
    "                  vocab_size, rnn_size, num_layers, vocab_to_int, batch_size):\n",
    "    '''Use the previous functions to create the training and inference logits'''\n",
    "    \n",
    "    # Use Numberbatch's embeddings and the newly created ones as our embeddings\n",
    "    embeddings = word_embedding_matrix\n",
    "    \n",
    "    enc_embed_input = tf.nn.embedding_lookup(embeddings, input_data)\n",
    "    enc_output, enc_state = encoding_layer(rnn_size, text_length, num_layers, enc_embed_input, keep_prob)\n",
    "    \n",
    "    dec_input = process_encoding_input(target_data, vocab_to_int, batch_size)\n",
    "    dec_embed_input = tf.nn.embedding_lookup(embeddings, dec_input)\n",
    "    \n",
    "    training_logits, inference_logits  = decoding_layer(dec_embed_input, \n",
    "                                                        embeddings,\n",
    "                                                        enc_output,\n",
    "                                                        enc_state, \n",
    "                                                        vocab_size, \n",
    "                                                        text_length, \n",
    "                                                        summary_length, \n",
    "                                                        max_summary_length,\n",
    "                                                        rnn_size, \n",
    "                                                        vocab_to_int, \n",
    "                                                        keep_prob, \n",
    "                                                        batch_size,\n",
    "                                                        num_layers)\n",
    "    \n",
    "    return training_logits, inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "C6w_YGts6mlq"
   },
   "outputs": [],
   "source": [
    "\n",
    "def pad_sentence_batch(sentence_batch):\n",
    "    \"\"\"Pad sentences with <PAD> so that each sentence of a batch has the same length\"\"\"\n",
    "    max_sentence = max([len(sentence) for sentence in sentence_batch])\n",
    "    return [sentence + [vocab_to_int['<PAD>']] * (max_sentence - len(sentence)) for sentence in sentence_batch]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kW6wc5VA6mls"
   },
   "outputs": [],
   "source": [
    "def get_batches(summaries, texts, batch_size):\n",
    "    \"\"\"Batch summaries, texts, and the lengths of their sentences together\"\"\"\n",
    "    for batch_i in range(0, len(texts)//batch_size):\n",
    "        start_i = batch_i * batch_size\n",
    "        summaries_batch = summaries[start_i:start_i + batch_size]\n",
    "        texts_batch = texts[start_i:start_i + batch_size]\n",
    "        pad_summaries_batch = np.array(pad_sentence_batch(summaries_batch))\n",
    "        pad_texts_batch = np.array(pad_sentence_batch(texts_batch))\n",
    "        \n",
    "        # Need the lengths for the _lengths parameters\n",
    "        pad_summaries_lengths = []\n",
    "        for summary in pad_summaries_batch:\n",
    "            pad_summaries_lengths.append(len(summary))\n",
    "        \n",
    "        pad_texts_lengths = []\n",
    "        for text in pad_texts_batch:\n",
    "            pad_texts_lengths.append(len(text))\n",
    "        \n",
    "        yield pad_summaries_batch, pad_texts_batch, pad_summaries_lengths, pad_texts_lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gUhPBryV6mlv"
   },
   "outputs": [],
   "source": [
    "# Set the Hyperparameters\n",
    "epochs = 50\n",
    "batch_size = 64\n",
    "rnn_size = 256\n",
    "num_layers = 6\n",
    "learning_rate = 0.005\n",
    "keep_probability = 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 505
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 5917,
     "status": "ok",
     "timestamp": 1583157328214,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "3I3bJqTD6mlx",
    "outputId": "5354efcc-53ad-43e5-b72c-bcfbf754e7c7"
   },
   "outputs": [],
   "source": [
    "# Build the graph\n",
    "train_graph = tf.Graph()\n",
    "# Set the graph to default to ensure that it is ready for training\n",
    "with train_graph.as_default():\n",
    "    \n",
    "    # Load the model inputs    \n",
    "    input_data, targets, lr, keep_prob, summary_length, max_summary_length, text_length = model_inputs()\n",
    "\n",
    "    # Create the training and inference logits\n",
    "    training_logits, inference_logits = seq2seq_model(tf.reverse(input_data, [-1]),\n",
    "                                                      targets, \n",
    "                                                      keep_prob,   \n",
    "                                                      text_length,\n",
    "                                                      summary_length,\n",
    "                                                      max_summary_length,\n",
    "                                                      len(vocab_to_int)+1,\n",
    "                                                      rnn_size, \n",
    "                                                      num_layers, \n",
    "                                                      vocab_to_int,\n",
    "                                                      batch_size)\n",
    "    \n",
    "    # Create tensors for the training logits and inference logits\n",
    "    training_logits = tf.identity(training_logits.rnn_output, 'logits')\n",
    "    inference_logits = tf.identity(inference_logits.sample_id, name='predictions')\n",
    "    \n",
    "    # Create the weights for sequence_loss\n",
    "    masks = tf.sequence_mask(summary_length, max_summary_length, dtype=tf.float32, name='masks')\n",
    "\n",
    "    with tf.name_scope(\"optimization\"):\n",
    "        # Loss function\n",
    "        cost = tf.contrib.seq2seq.sequence_loss(\n",
    "            training_logits,\n",
    "            targets,\n",
    "            masks)\n",
    "\n",
    "        # Optimizer\n",
    "        optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "\n",
    "        # Gradient Clipping\n",
    "        gradients = optimizer.compute_gradients(cost)\n",
    "        capped_gradients = [(tf.clip_by_value(grad, -5., 5.), var) for grad, var in gradients if grad is not None]\n",
    "        train_op = optimizer.apply_gradients(capped_gradients)\n",
    "print(\"Graph is built.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fmFOlquE6ml0"
   },
   "source": [
    "## Training the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2092,
     "status": "ok",
     "timestamp": 1583157328216,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "nlNcRq4j6ml3",
    "outputId": "d3606abd-a950-4b4b-d57e-a45f039e3229"
   },
   "outputs": [],
   "source": [
    "sorted_summaries_short = sorted_summaries\n",
    "sorted_texts_short = sorted_texts\n",
    "#print(sorted_summaries)\n",
    "print(\"The shortest text length:\", len(sorted_texts_short[0]))\n",
    "print(\"The longest text length:\",len(sorted_texts_short[-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1671751,
     "status": "ok",
     "timestamp": 1583093938956,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "XASfketR6ml6",
    "outputId": "8943bf9f-eb5c-4134-916a-2646da5225bd"
   },
   "outputs": [],
   "source": [
    "# Train the Model\n",
    "learning_rate_decay = 0.90\n",
    "min_learning_rate = 0.0005\n",
    "display_step = 50 # Check training loss after every 20 batches\n",
    "stop_early = 0 \n",
    "stop = 6000000000000000 #3 # If the update loss does not decrease in 3 consecutive update checks, stop training\n",
    "per_epoch = 1 # Make 3 update checks per epoch\n",
    "update_check = (len(sorted_texts_short)//batch_size//per_epoch)-1\n",
    "\n",
    "update_loss = 0 \n",
    "batch_loss = 0\n",
    "summary_update_loss = [] # Record the update losses for saving improvements in the model\n",
    "\n",
    "  \n",
    "tf.reset_default_graph()\n",
    "checkpoint = \"drive/My Drive/Colab Notebooks/fyp/models/model#1.ckpt\"  #300k sentence\n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # If we want to continue training a previous session\n",
    "    # loader = tf.train.import_meta_graph(checkpoint + '.meta')\n",
    "    # loader.restore(sess, checkpoint)\n",
    "    #sess.run(tf.local_variables_initializer())\n",
    "\n",
    "    for epoch_i in range(1, epochs+1):\n",
    "        update_loss = 0\n",
    "        batch_loss = 0\n",
    "        for batch_i, (summaries_batch, texts_batch, summaries_lengths, texts_lengths) in enumerate(\n",
    "                get_batches(sorted_summaries_short, sorted_texts_short, batch_size)):\n",
    "            start_time = time.time()\n",
    "            _, loss = sess.run(\n",
    "                [train_op, cost],\n",
    "                {input_data: texts_batch,\n",
    "                 targets: summaries_batch,\n",
    "                 lr: learning_rate,\n",
    "                 summary_length: summaries_lengths,\n",
    "                 text_length: texts_lengths,\n",
    "                 keep_prob: keep_probability})\n",
    "\n",
    "            batch_loss += loss\n",
    "            update_loss += loss\n",
    "            end_time = time.time()\n",
    "            batch_time = end_time - start_time\n",
    "\n",
    "            if batch_i % display_step == 0 and batch_i > 0:\n",
    "                print('Epoch {:>3}/{} Batch {:>4}/{} - Loss: {:>6.3f}, Seconds: {:>4.2f}'\n",
    "                      .format(epoch_i,\n",
    "                              epochs, \n",
    "                              batch_i, \n",
    "                              len(sorted_texts_short) // batch_size, \n",
    "                              batch_loss / display_step, \n",
    "                              batch_time*display_step))\n",
    "                batch_loss = 0\n",
    "                \n",
    "                saver = tf.train.Saver() \n",
    "                saver.save(sess, checkpoint)\n",
    "                \n",
    "            if batch_i % update_check == 0 and batch_i > 0:\n",
    "                print(\"Average loss for this update:\", round(update_loss/update_check,3))\n",
    "                summary_update_loss.append(update_loss)\n",
    "                \n",
    "              \n",
    "                  \n",
    "                # If the update loss is at a new minimum, save the model\n",
    "                if update_loss <= min(summary_update_loss):\n",
    "                    print('New Record!') \n",
    "                    stop_early = 0\n",
    "                    saver = tf.train.Saver() \n",
    "                    saver.save(sess, checkpoint)\n",
    "\n",
    "                else:\n",
    "                    print(\"No Improvement.\")\n",
    "                    #stop_early += 1\n",
    "                    #if stop_early == stop:\n",
    "                    #    break\n",
    "                update_loss = 0\n",
    "            \n",
    "                    \n",
    "        # Reduce learning rate, but not below its minimum value\n",
    "        learning_rate *= learning_rate_decay\n",
    "        if learning_rate < min_learning_rate:\n",
    "            learning_rate = min_learning_rate\n",
    "        \n",
    "        #if stop_early == stop:\n",
    "        #    print(\"Stopping Training.\")\n",
    "            #break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "S_4rooXoU3lP"
   },
   "outputs": [],
   "source": [
    "checkpoint = \"drive/My Drive/Colab Notebooks/best_model.ckpt\" \n",
    "\n",
    "loaded_graph = tf.Graph()\n",
    "with tf.Session(graph=loaded_graph) as sess:\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(checkpoint + '.meta')\n",
    "    loader.restore(sess, checkpoint)\n",
    "    names = []\n",
    "    [names.append(n.name) for n in loaded_graph.as_graph_def().node]\n",
    "names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qLTgl3MT6ml8"
   },
   "source": [
    "## Making Our Own Summaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gJsdh6eG6ml-"
   },
   "outputs": [],
   "source": [
    "def text_to_seq(text):\n",
    "    '''Prepare the text for the model'''\n",
    "    \n",
    "    text = clean_text(text)\n",
    "    return [vocab_to_int.get(word, vocab_to_int['<UNK>']) for word in text.split()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5L1jkGaSsGEP"
   },
   "outputs": [],
   "source": [
    "text=''' India on Thursday hit out at Turkish President Recep Tayyip Erdogan and the UK's junior foreign minister Nigel Adams for their \"irresponsible\" comments over violence in the national capital when it is going through a \"sensitive time\". '''\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "993IfWW3sPLC"
   },
   "outputs": [],
   "source": [
    "paragraphs = text.split('\\n')\n",
    "rm_paras = list()\n",
    "for para in paragraphs:\n",
    "  if len(para)<200:\n",
    "    continue\n",
    "  rm_paras.append(para)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 8207,
     "status": "ok",
     "timestamp": 1583094142458,
     "user": {
      "displayName": "Muneeb Sial",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GgrHZwwlxmJvXeNRBPRQ3bDeojuUaBRlm9pm-gwIg=s64",
      "userId": "02013851263585652343"
     },
     "user_tz": -300
    },
    "id": "m2Pbp2Cu6mmE",
    "outputId": "5ec3fb78-3d01-4c2a-eb57-b719d1110ff3"
   },
   "outputs": [],
   "source": [
    "# Create summaries\n",
    "\n",
    "def generate_summary(input_sentence):\n",
    "    paragraphs = input_sentence.split('\\n')\n",
    "    rm_paras = list()\n",
    "    for para in paragraphs:\n",
    "        if len(para)<200:\n",
    "            continue\n",
    "        rm_paras.append(para)\n",
    "\n",
    "    checkpoint = \"./model#1.ckpt\"\n",
    "    \n",
    "    i=1\n",
    "    for para in rm_paras:\n",
    "\n",
    "        text = text_to_seq(para)\n",
    "        loaded_graph = tf.Graph()\n",
    "        with tf.Session(graph=loaded_graph) as sess:\n",
    "            # Load saved model\n",
    "            loader = tf.train.import_meta_graph(checkpoint + '.meta')\n",
    "            loader.restore(sess, checkpoint)\n",
    "\n",
    "            input_data = loaded_graph.get_tensor_by_name('input:0')\n",
    "            logits = loaded_graph.get_tensor_by_name('predictions:0')\n",
    "            text_length = loaded_graph.get_tensor_by_name('text_length:0')\n",
    "            summary_length = loaded_graph.get_tensor_by_name('summary_length:0')\n",
    "            keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')\n",
    "\n",
    "            #Multiply by batch_size to match the model's input parameters\n",
    "            \n",
    "\n",
    "\n",
    "            answer_logits = sess.run(logits, {input_data: [text]*batch_size, \n",
    "                                              summary_length: [np.random.randint(5,8)], \n",
    "                                              text_length: [len(text)]*batch_size,\n",
    "                                              keep_prob: 1.0})[0] \n",
    "\n",
    "            # Remove the padding from the tweet\n",
    "        pad = vocab_to_int[\"<PAD>\"] \n",
    "        summary = \" \".join([int_to_vocab[i] for i in answer_logits if i != pad])\n",
    "        full_summary += (str(i)+'=>'+summary+'\\n')\n",
    "        i+=1\n",
    "    return full_summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import socket\n",
    "from _thread import *\n",
    "import threading\n",
    "import json\n",
    "from io import StringIO\n",
    "\n",
    "\n",
    "def threaded(c):\n",
    "    data = c.recv(1024)\n",
    "\n",
    "    print(data)\n",
    "    data = json.dumps( data.decode(\"utf-8\") )\n",
    "    data = json.loads( data )\n",
    "    input_sentence=data[13:-2]\n",
    "\n",
    "    smry = _generate_summary(input_sentence)\n",
    "    dt = json.dumps({\"label\": smry }).encode('utf-8')\n",
    "    dt += b'\\n'\n",
    "    c.send( dt )\n",
    "    \n",
    "    c.close()\n",
    "port = 5001\n",
    "s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)\n",
    "s.bind(( \"0.0.0.0\" , port ))\n",
    "print(\"socket binded to port\", port )\n",
    "s.listen(5)\n",
    "print(\"socket is listening\")\n",
    "all_conn = []\n",
    "while True:\n",
    "    conn, addr = s.accept()\n",
    "    print('Connected to :', addr[0], ':', addr[1])\n",
    "    all_conn.append( conn )\n",
    "    start_new_thread(threaded, (conn,))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s.close()"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "FYP1Code.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}