tutorial-notebooks/5_enc_dec_with_LuongAttention.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "5_enc_dec_with_LuongAttention.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mejJonsu2o7r",
        "colab_type": "text"
      },
      "source": [
        "#### Taken from [tensorflow-tutorial ](https://www.tensorflow.org/tutorials/text/nmt_with_attention ) and added a *translate_batch()* function to translate a batch and dump outputs into a file"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "J0Qjg6vuaHNt"
      },
      "source": [
        "# Neural machine translation with attention"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "tnxXKDjq3jEL",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.ticker as ticker\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "import unicodedata\n",
        "import re\n",
        "import numpy as np\n",
        "import os\n",
        "import io\n",
        "import time\n",
        "from utils.dataset import NMTDataset\n",
        "from tensorflow.python.ops import math_ops"
      ],
      "execution_count": 73,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YsrDDMlKA2nd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "d31e50ca-bb4b-4c79-e920-7d30dac76748"
      },
      "source": [
        "!ls utils\n"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "dataset.py\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GOi42V79Ydlr"
      },
      "source": [
        "We'll use the same dataset we worked on notebook-1 (text-processing). For our convenience we've created a utils/dataset.py file which returns train and validation tf.data.Dataset objects."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sy9mcFmxBP4J",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "a5f3a663-71c5-4d97-acd2-cc9249042d77"
      },
      "source": [
        "BUFFER_SIZE = 32000\n",
        "BATCH_SIZE = 64\n",
        "num_examples = 30000\n",
        "\n",
        "dataset_creator = NMTDataset('en-spa')\n",
        "train_dataset, val_dataset, inp_lang, targ_lang = dataset_creator.call(num_examples, BUFFER_SIZE, BATCH_SIZE)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip\n",
            "2646016/2638744 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wVtSz89FBk0D",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "48d1ab71-9d23-4369-f081-12c8927814c2"
      },
      "source": [
        "\n",
        "print(\"Inpute Vocabulary Size: {}\".format(len(inp_lang.word_index)))\n",
        "print(\"Target Vocabulary Size: {}\".format(len(targ_lang.word_index)))"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Inpute Vocabulary Size: 9414\n",
            "Target Vocabulary Size: 4935\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "qc6-NK1GtWQt",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "f35ff927-5a3c-480c-95ae-2b0d63844250"
      },
      "source": [
        "example_input_batch, example_target_batch = next(iter(train_dataset))\n",
        "example_input_batch.shape, example_target_batch.shape"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorShape([64, 16]), TensorShape([64, 11]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "TNfHIF71ulLu"
      },
      "source": [
        "## Write the encoder and decoder model\n",
        "\n",
        "Implement an encoder-decoder model with attention which you can read about in the TensorFlow [Neural Machine Translation (seq2seq) tutorial](https://github.com/tensorflow/nmt). This example uses a more recent set of APIs. This notebook implements the [attention equations](https://github.com/tensorflow/nmt#background-on-the-attention-mechanism) from the seq2seq tutorial. The following diagram shows that each input words is assigned a weight by the attention mechanism which is then used by the decoder to predict the next word in the sentence. The below picture and formulas are an example of attention mechanism from [Luong's paper](https://arxiv.org/abs/1508.04025v5). \n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_mechanism.jpg\" width=\"500\" alt=\"attention mechanism\">\n",
        "\n",
        "The input is put through an encoder model which gives us the encoder output of shape *(batch_size, max_length, hidden_size)* and the encoder hidden state of shape *(batch_size, hidden_size)*.\n",
        "\n",
        "Here are the equations that are implemented:\n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_equation_0.jpg\" alt=\"attention equation 0\" width=\"800\">\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_equation_1.jpg\" alt=\"attention equation 1\" width=\"800\">\n",
        "\n",
        "This tutorial uses [Luong attention](https://arxiv.org/abs/1508.04025) for the encoder. Let's decide on notation before writing the simplified form:\n",
        "\n",
        "* FC = Fully connected (dense) layer\n",
        "* EO = Encoder output\n",
        "* H = hidden state\n",
        "* X = input to the decoder\n",
        "\n",
        "And the pseudo-code:\n",
        "\n",
        "* `score = MatMul(Transpose(H)xE0)`\n",
        "* `attention weights = softmax(score, axis = 1)`. Softmax by default is applied on the last axis but here we want to apply it on the *1st axis*, since the shape of score is *(batch_size, max_length, hidden_size)*. `Max_length` is the length of our input. Since we are trying to assign a weight to each input, softmax should be applied on that axis.\n",
        "* `context vector = sum(attention weights * EO, axis = 1)`. Same reason as above for choosing axis as 1.\n",
        "* `embedding output` = The input to the decoder X is passed through an embedding layer.\n",
        "* `merged vector = concat(embedding output, context vector)`\n",
        "* This merged vector is then given to the GRU\n",
        "\n",
        "The shapes of all the vectors at each step have been specified in the comments in the code:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dgVsBjbVDBwX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Define some useful parameters for further use\n",
        "\n",
        "vocab_inp_size = len(inp_lang.word_index)+1\n",
        "vocab_tar_size = len(targ_lang.word_index)+1\n",
        "max_length_input = example_input_batch.shape[1]\n",
        "max_length_output = example_target_batch.shape[1]\n",
        "\n",
        "embedding_dim = 128\n",
        "units = 1024\n",
        "steps_per_epoch = num_examples//BATCH_SIZE"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "nZ2rI24i3jFg",
        "colab": {}
      },
      "source": [
        "# Encoder is composed of embedding layer and then one GRU layer. It produces outputs and last hidden states. \n",
        "# Encoder Outputs shape = (BATCH_SIZE, max_length_input, units)\n",
        "# Last Hidden State Shape = (BATCH_SIZE, units)\n",
        "\n",
        "class Encoder(tf.keras.Model):\n",
        "  def __init__(self, vocab_size, embedding_dim, enc_units, batch_sz):\n",
        "    super(Encoder, self).__init__()\n",
        "    self.batch_sz = batch_sz\n",
        "    self.enc_units = enc_units\n",
        "    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "    self.gru = tf.keras.layers.GRU(self.enc_units,\n",
        "                                   return_sequences=True,\n",
        "                                   return_state=True,\n",
        "                                   recurrent_initializer='glorot_uniform')\n",
        "\n",
        "  def call(self, x, hidden):\n",
        "    x = self.embedding(x)\n",
        "    output, state = self.gru(x, initial_state = hidden)\n",
        "    return output, state\n",
        "\n",
        "  def initialize_hidden_state(self):\n",
        "    return tf.zeros((self.batch_sz, self.enc_units))"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "60gSVh05Jl6l",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "d5ad818c-8ef1-4f16-e9fb-b04503a5ca8a"
      },
      "source": [
        "encoder = Encoder(vocab_inp_size, embedding_dim, units, BATCH_SIZE)\n",
        "\n",
        "# sample input\n",
        "sample_hidden = encoder.initialize_hidden_state()\n",
        "sample_output, sample_hidden = encoder(example_input_batch, sample_hidden)\n",
        "print ('Encoder output shape: (batch size, sequence length, units) {}'.format(sample_output.shape))\n",
        "print ('Encoder Hidden state shape: (batch size, units) {}'.format(sample_hidden.shape))"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Encoder output shape: (batch size, sequence length, units) (64, 16, 1024)\n",
            "Encoder Hidden state shape: (batch size, units) (64, 1024)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "umohpBN2OM94",
        "colab": {}
      },
      "source": [
        "def LuongAttention(query, values):\n",
        "  # query hidden state shape == (batch_size, hidden size)\n",
        "  # query_with_time_axis shape == (batch_size, 1, hidden size)\n",
        "  # values shape == (batch_size, max_len, hidden size)\n",
        "  # we are doing this to broadcast addition along the time axis to calculate the score\n",
        "  \n",
        "  query_with_time_axis = tf.expand_dims(query, 1)\n",
        "  # score shape == (batch_size, max_length, 1)\n",
        "  # we get 1 at the last axis because we are applying score to self.V\n",
        "  # the shape of the tensor before applying self.V is (batch_size, max_length, units)\n",
        "  \n",
        "  #-------- COMPUTING EQUATION (4: Luong's Attention)   ---------#  \n",
        "  score = tf.matmul(query_with_time_axis, values, transpose_b=True)\n",
        "\n",
        "  #-------- COMPUTING EQUATION (1) ------------#\n",
        "  # attention_weights shape == (batch_size, max_length, 1)\n",
        "  attention_weights = tf.nn.softmax(score, axis=1)\n",
        "\n",
        "  # Reshape attention_weights to shape = (batch_size, max_length, 1)\n",
        "  attention_weights = tf.reshape(attention_weights, shape=(-1, attention_weights.shape[2], 1))\n",
        "\n",
        "  #---------- COMPUTING EQUATION (2) -----------#\n",
        "  # context_vector shape after sum == (batch_size, hidden_size)\n",
        "  context_vector = attention_weights * values\n",
        "  # Context vector is passed on to the curren time step's decoder cell\n",
        "  context_vector = tf.reduce_sum(context_vector, axis=1)\n",
        "\n",
        "  return context_vector, attention_weights"
      ],
      "execution_count": 108,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "k534zTHiDjQU",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "423d6cff-fc52-47fb-ef7c-c6cf00496169"
      },
      "source": [
        "attention_result, attention_weights = LuongAttention(sample_hidden, sample_output)\n",
        "\n",
        "print(\"context vector shape: (batch size, units) {}\".format(attention_result.shape))\n",
        "print(\"Attention weights shape: (batch_size, sequence_length, 1) {}\".format(attention_weights.shape))"
      ],
      "execution_count": 109,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "context vector shape: (batch size, units) (64, 1024)\n",
            "Attention weights shape: (batch_size, sequence_length, 1) (64, 16, 1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LOaVVOaRvkVx",
        "colab_type": "text"
      },
      "source": [
        "In Luong Attention paper, the attention and context vector is computed on hidden state *h<sub>t</sub>* at the decoder time step t itself, unlike Bahdanau Attention which uses *h<sub>t-1</sub>* hidden state to compute attention weights and context vector. \n",
        "\n",
        "If explained in simple terms, Bahdanau Attention first passes *h<sub>t-1</sub>* to the attention layer which produces context vector *c<sub>t</sub>* at time step *t* which finally calculates *y<sub>t</sub> <sup>hat</sup>*.  \n",
        "\n",
        "Luong Attention, on the other hand, first passes *h<sub>t-1</sub>* to decoder's recurrent layer to calculate *h<sub>t</sub>*. This *h<sub>t</sub>* is passed to attention layer to calculate attention weights and context vector. A new *h<sub>t</sub><sup>~</sup>* is computed by concatenatin of *h<sub>t</sub>* and context vector *c<sub>t</sub>* which finally calculates *y<sub>t</sub> <sup>hat</sup>*.\n",
        "\n",
        "However, for simplicity we'll use Bahdanau's method of using *h<sub>t-1</sub>* to calculate attention weights and context vector. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "yJ_B3mhW3jFk",
        "colab": {}
      },
      "source": [
        "class Decoder(tf.keras.Model):\n",
        "  def __init__(self, vocab_size, embedding_dim, dec_units, batch_sz):\n",
        "    super(Decoder, self).__init__()\n",
        "    self.batch_sz = batch_sz\n",
        "    self.dec_units = dec_units\n",
        "    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "    self.gru = tf.keras.layers.GRU(self.dec_units,\n",
        "                                   return_sequences=True,\n",
        "                                   return_state=True,\n",
        "                                   recurrent_initializer='glorot_uniform')\n",
        "    self.fc = tf.keras.layers.Dense(vocab_size)\n",
        "\n",
        "  def call(self, x, hidden, enc_output):\n",
        "    # enc_output shape == (batch_size, max_length, hidden_size)\n",
        "    context_vector, attention_weights = LuongAttention(hidden, enc_output)\n",
        "\n",
        "    # x shape after passing through embedding == (batch_size, 1, embedding_dim)\n",
        "    x = self.embedding(x)\n",
        "\n",
        "    #------------ COMPUTING EQUATION (3) ----------------------#\n",
        "    # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)\n",
        "    x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)\n",
        "\n",
        "    # passing the concatenated vector to the GRU\n",
        "    output, state = self.gru(x)\n",
        "\n",
        "    # output shape == (batch_size * 1, hidden_size)\n",
        "    output = tf.reshape(output, (-1, output.shape[2]))\n",
        "\n",
        "    # output shape == (batch_size, vocab)\n",
        "    x = self.fc(output)\n",
        "\n",
        "    return x, state, attention_weights"
      ],
      "execution_count": 110,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "P5UY8wko3jFp",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "68f67959-2aac-401f-b38c-54d64c28fbb4"
      },
      "source": [
        "decoder = Decoder(vocab_tar_size, embedding_dim, units, BATCH_SIZE)\n",
        "\n",
        "sample_decoder_output, _, _ = decoder(tf.random.uniform((BATCH_SIZE, 1)),\n",
        "                                      sample_hidden, sample_output)\n",
        "\n",
        "print ('Decoder output shape: (batch_size, vocab size) {}'.format(sample_decoder_output.shape))"
      ],
      "execution_count": 111,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Decoder output shape: (batch_size, vocab size) (64, 4936)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_ch_71VbIRfK"
      },
      "source": [
        "## Define the optimizer and the loss function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "WmTHr5iV3jFr",
        "colab": {}
      },
      "source": [
        "optimizer = tf.keras.optimizers.Adam()\n",
        "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
        "    from_logits=True, reduction='none')\n",
        "\n",
        "def loss_function(real, pred):\n",
        "  mask = tf.math.logical_not(tf.math.equal(real, 0))\n",
        "  loss_ = loss_object(real, pred)\n",
        "\n",
        "  mask = tf.cast(mask, dtype=loss_.dtype)\n",
        "  loss_ *= mask\n",
        "\n",
        "  return tf.reduce_mean(loss_)"
      ],
      "execution_count": 112,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "DMVWzzsfNl4e"
      },
      "source": [
        "## Checkpoints (Object-based saving)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "Zj8bXQTgNwrF",
        "colab": {}
      },
      "source": [
        "checkpoint_dir = './training_checkpoints'\n",
        "checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt\")\n",
        "checkpoint = tf.train.Checkpoint(optimizer=optimizer,\n",
        "                                 encoder=encoder,\n",
        "                                 decoder=decoder)"
      ],
      "execution_count": 113,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hpObfY22IddU"
      },
      "source": [
        "## Training\n",
        "\n",
        "1. Pass the *input* through the *encoder* which return *encoder output* and the *encoder hidden state*.\n",
        "2. The encoder output, encoder hidden state and the decoder input (which is the *start token*) is passed to the decoder.\n",
        "3. The decoder returns the *predictions* and the *decoder hidden state*.\n",
        "4. The decoder hidden state is then passed back into the model and the predictions are used to calculate the loss.\n",
        "5. Use *teacher forcing* to decide the next input to the decoder.\n",
        "6. *Teacher forcing* is the technique where the *target word* is passed as the *next input* to the decoder.\n",
        "7. The final step is to calculate the gradients and apply it to the optimizer and backpropagate."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "sC9ArXSsVfqn",
        "colab": {}
      },
      "source": [
        "@tf.function\n",
        "def train_step(inp, targ, enc_hidden):\n",
        "  loss = 0\n",
        "\n",
        "  with tf.GradientTape() as tape:\n",
        "    enc_output, enc_hidden = encoder(inp, enc_hidden)\n",
        "\n",
        "    dec_hidden = enc_hidden\n",
        "\n",
        "    dec_input = tf.expand_dims([targ_lang.word_index['<start>']] * BATCH_SIZE, 1)\n",
        "\n",
        "    # Teacher forcing - feeding the target as the next input\n",
        "    for t in range(1, targ.shape[1]):\n",
        "      # passing enc_output to the decoder\n",
        "      predictions, dec_hidden, _ = decoder(dec_input, dec_hidden, enc_output)\n",
        "\n",
        "      loss += loss_function(targ[:, t], predictions)\n",
        "\n",
        "      # using teacher forcing\n",
        "      dec_input = tf.expand_dims(targ[:, t], 1)\n",
        "\n",
        "  batch_loss = (loss / int(targ.shape[1]))\n",
        "\n",
        "  variables = encoder.trainable_variables + decoder.trainable_variables\n",
        "\n",
        "  gradients = tape.gradient(loss, variables)\n",
        "\n",
        "  optimizer.apply_gradients(zip(gradients, variables))\n",
        "\n",
        "  return batch_loss"
      ],
      "execution_count": 114,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ddefjBMa3jF0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "6fda27f1-2f3c-40ea-b3bb-3e52da82c6c4"
      },
      "source": [
        "EPOCHS = 10\n",
        "\n",
        "for epoch in range(EPOCHS):\n",
        "  start = time.time()\n",
        "\n",
        "  enc_hidden = encoder.initialize_hidden_state()\n",
        "  total_loss = 0\n",
        "\n",
        "  for (batch, (inp, targ)) in enumerate(train_dataset.take(steps_per_epoch)):\n",
        "    batch_loss = train_step(inp, targ, enc_hidden)\n",
        "    total_loss += batch_loss\n",
        "\n",
        "    if batch % 100 == 0:\n",
        "      print('Epoch {} Batch {} Loss {:.4f}'.format(epoch + 1,\n",
        "                                                   batch,\n",
        "                                                   batch_loss.numpy()))\n",
        "  # saving (checkpoint) the model every 2 epochs\n",
        "  if (epoch + 1) % 2 == 0:\n",
        "    checkpoint.save(file_prefix = checkpoint_prefix)\n",
        "\n",
        "  print('Epoch {} Loss {:.4f}'.format(epoch + 1,\n",
        "                                      total_loss / steps_per_epoch))\n",
        "  print('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))"
      ],
      "execution_count": 115,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1 Batch 0 Loss 4.5059\n",
            "Epoch 1 Batch 100 Loss 1.6796\n",
            "Epoch 1 Batch 200 Loss 1.3949\n",
            "Epoch 1 Batch 300 Loss 0.9678\n",
            "Epoch 1 Loss 1.1854\n",
            "Time taken for 1 epoch 32.85179924964905 sec\n",
            "\n",
            "Epoch 2 Batch 0 Loss 0.7490\n",
            "Epoch 2 Batch 100 Loss 0.7875\n",
            "Epoch 2 Batch 200 Loss 0.6171\n",
            "Epoch 2 Batch 300 Loss 0.6758\n",
            "Epoch 2 Loss 0.5563\n",
            "Time taken for 1 epoch 22.12682318687439 sec\n",
            "\n",
            "Epoch 3 Batch 0 Loss 0.4191\n",
            "Epoch 3 Batch 100 Loss 0.3951\n",
            "Epoch 3 Batch 200 Loss 0.4340\n",
            "Epoch 3 Batch 300 Loss 0.4464\n",
            "Epoch 3 Loss 0.3493\n",
            "Time taken for 1 epoch 21.796282529830933 sec\n",
            "\n",
            "Epoch 4 Batch 0 Loss 0.2658\n",
            "Epoch 4 Batch 100 Loss 0.2748\n",
            "Epoch 4 Batch 200 Loss 0.2455\n",
            "Epoch 4 Batch 300 Loss 0.2689\n",
            "Epoch 4 Loss 0.2475\n",
            "Time taken for 1 epoch 22.35062575340271 sec\n",
            "\n",
            "Epoch 5 Batch 0 Loss 0.2523\n",
            "Epoch 5 Batch 100 Loss 0.2465\n",
            "Epoch 5 Batch 200 Loss 0.2239\n",
            "Epoch 5 Batch 300 Loss 0.2614\n",
            "Epoch 5 Loss 0.1937\n",
            "Time taken for 1 epoch 21.80947232246399 sec\n",
            "\n",
            "Epoch 6 Batch 0 Loss 0.1714\n",
            "Epoch 6 Batch 100 Loss 0.2163\n",
            "Epoch 6 Batch 200 Loss 0.2070\n",
            "Epoch 6 Batch 300 Loss 0.1617\n",
            "Epoch 6 Loss 0.1566\n",
            "Time taken for 1 epoch 22.31173324584961 sec\n",
            "\n",
            "Epoch 7 Batch 0 Loss 0.1840\n",
            "Epoch 7 Batch 100 Loss 0.1516\n",
            "Epoch 7 Batch 200 Loss 0.1474\n",
            "Epoch 7 Batch 300 Loss 0.1448\n",
            "Epoch 7 Loss 0.1320\n",
            "Time taken for 1 epoch 21.937301635742188 sec\n",
            "\n",
            "Epoch 8 Batch 0 Loss 0.0897\n",
            "Epoch 8 Batch 100 Loss 0.1685\n",
            "Epoch 8 Batch 200 Loss 0.1937\n",
            "Epoch 8 Batch 300 Loss 0.1470\n",
            "Epoch 8 Loss 0.1142\n",
            "Time taken for 1 epoch 22.343220949172974 sec\n",
            "\n",
            "Epoch 9 Batch 0 Loss 0.1179\n",
            "Epoch 9 Batch 100 Loss 0.1566\n",
            "Epoch 9 Batch 200 Loss 0.1017\n",
            "Epoch 9 Batch 300 Loss 0.1572\n",
            "Epoch 9 Loss 0.1003\n",
            "Time taken for 1 epoch 21.950446844100952 sec\n",
            "\n",
            "Epoch 10 Batch 0 Loss 0.1155\n",
            "Epoch 10 Batch 100 Loss 0.0810\n",
            "Epoch 10 Batch 200 Loss 0.1014\n",
            "Epoch 10 Batch 300 Loss 0.1623\n",
            "Epoch 10 Loss 0.0927\n",
            "Time taken for 1 epoch 22.422590970993042 sec\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mU3Ce8M6I3rz"
      },
      "source": [
        "## Translate\n",
        "\n",
        "* The evaluate function is similar to the training loop, except we don't use *teacher forcing* here. The input to the decoder at each time step is its previous predictions along with the hidden state and the encoder output.\n",
        "* Stop predicting when the model predicts the *end token*.\n",
        "* And store the *attention weights for every time step*.\n",
        "\n",
        "Note: The encoder output is calculated only once for one input."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "EbQpyYs13jF_",
        "colab": {}
      },
      "source": [
        "def evaluate(sentence):\n",
        "\n",
        "  sentence = dataset_creator.preprocess_sentence(sentence)\n",
        "\n",
        "  inputs = [inp_lang.word_index[i] for i in sentence.split(' ')]\n",
        "  inputs = tf.keras.preprocessing.sequence.pad_sequences([inputs],\n",
        "                                                         maxlen=max_length_input,\n",
        "                                                         padding='post')\n",
        "  inputs = tf.convert_to_tensor(inputs)\n",
        "\n",
        "  result = ''\n",
        "\n",
        "  hidden = [tf.zeros((1, units))]\n",
        "  enc_out, enc_hidden = encoder(inputs, hidden)\n",
        "\n",
        "  dec_hidden = enc_hidden\n",
        "  dec_input = tf.expand_dims([targ_lang.word_index['<start>']], 0)\n",
        "\n",
        "  for t in range(max_length_output):\n",
        "    predictions, dec_hidden, attention_weights = decoder(dec_input,\n",
        "                                                         dec_hidden,\n",
        "                                                         enc_out)\n",
        "\n",
        "    # storing the attention weights to plot later on\n",
        "\n",
        "    predicted_id = tf.argmax(predictions[0]).numpy()\n",
        "\n",
        "    result += targ_lang.index_word[predicted_id] + ' '\n",
        "\n",
        "    if targ_lang.index_word[predicted_id] == '<end>':\n",
        "      return result, sentence\n",
        "\n",
        "    # the predicted ID is fed back into the model\n",
        "    dec_input = tf.expand_dims([predicted_id], 0)\n",
        "\n",
        "  return result, sentence"
      ],
      "execution_count": 116,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "sl9zUHzg3jGI",
        "colab": {}
      },
      "source": [
        "def translate(sentence):\n",
        "  result, sentence = evaluate(sentence)\n",
        "\n",
        "  print('Input: %s' % (sentence))\n",
        "  print('Predicted translation: {}'.format(result))"
      ],
      "execution_count": 117,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "n250XbnjOaqP"
      },
      "source": [
        "## Restore the latest checkpoint and test"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "UJpT9D5_OgP6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "5094c130-ab05-4b88-ae20-4a8813c7fb44"
      },
      "source": [
        "# restoring the latest checkpoint in checkpoint_dir\n",
        "checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))"
      ],
      "execution_count": 118,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x7f288ddc1550>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 118
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "WrAM0FDomq3E",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "e697bb11-9c7c-42b8-d1ec-0054d03ca0bb"
      },
      "source": [
        "translate(u'hace mucho frio aqui.')"
      ],
      "execution_count": 119,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input: <start> hace mucho frio aqui . <end>\n",
            "Predicted translation: it s very cold here . <end> \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "zSx2iM36EZQZ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "10967ebc-1c5f-4680-83bd-aede4165758a"
      },
      "source": [
        "translate(u'esta es mi vida.')"
      ],
      "execution_count": 120,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input: <start> esta es mi vida . <end>\n",
            "Predicted translation: this is my life . <end> \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "A3LLCx3ZE0Ls",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "62edd9f7-0744-4b90-dc1b-60f1aca11912"
      },
      "source": [
        "translate(u'¿todavia estan en casa?')"
      ],
      "execution_count": 121,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input: <start> ¿ todavia estan en casa ? <end>\n",
            "Predicted translation: are you still home ? <end> \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "DUQVLVqUE1YW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "878f0a15-4627-4166-b8bf-cefc60f57265"
      },
      "source": [
        "# wrong translation\n",
        "translate(u'trata de averiguarlo.')"
      ],
      "execution_count": 122,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input: <start> trata de averiguarlo . <end>\n",
            "Predicted translation: try to find to find to find to find to find \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "92aXUSuvwzOt",
        "colab": {}
      },
      "source": [
        "def translate_batch(test_dataset):\n",
        "  with open('output_text.txt', 'w') as f:\n",
        "    for (inputs, targets) in test_dataset:\n",
        "      outputs = np.zeros((BATCH_SIZE, max_length_output),dtype=np.int16)\n",
        "      hidden_state = tf.zeros((BATCH_SIZE, units))\n",
        "      enc_output, dec_h = encoder(inputs, hidden_state)\n",
        "      dec_input = tf.expand_dims([targ_lang.word_index['<start>']] * BATCH_SIZE, 1)\n",
        "      for t in range(max_length_output):\n",
        "        preds, dec_h, _ = decoder(dec_input, dec_h, enc_output)\n",
        "        predicted_id = tf.argmax(preds, axis=1).numpy()\n",
        "        outputs[:, t] = predicted_id\n",
        "        dec_input = tf.expand_dims(predicted_id, 1)\n",
        "      outputs = targ_lang.sequences_to_texts(outputs)\n",
        "      for t, item in enumerate(outputs):\n",
        "        try:\n",
        "          i = item.index('<end>')\n",
        "          f.write(\"%s\\n\" %item[:i])\n",
        "        except: \n",
        "          f.write(\"%s \\n\" % item) # For those translated sequences which didn't correctly translated and have <end> token.\n",
        "\n",
        "outputs = translate_batch(val_dataset)"
      ],
      "execution_count": 123,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "rbHSSD1dT1D0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "a84b23b2-0182-4743-af54-ad8e37b0f494"
      },
      "source": [
        "!head output_text.txt\n",
        "! wc -l output_text.txt"
      ],
      "execution_count": 124,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "speak slowly . \n",
            "i ll keep my revenge . \n",
            "come back . \n",
            "i love reading . \n",
            "don t bother me ! \n",
            "hold the rope . \n",
            "do ! \n",
            "tom betrayed you . \n",
            "get on . \n",
            "i was very nervous . \n",
            "5952 output_text.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t2Ci8zsFbL8q",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "bfb8101c-89d4-4734-8bca-7af173357a81"
      },
      "source": [
        "val_targets = list(val_dataset.take(1))\n",
        "val_targets = np.asarray(val_targets[0][1])\n",
        "print(type(val_targets))\n",
        "targ_lang.sequences_to_texts(val_targets)[:10]"
      ],
      "execution_count": 125,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'numpy.ndarray'>\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['<start> walk more slowly . <end> <OOV> <OOV> <OOV> <OOV> <OOV>',\n",
              " '<start> i ll have my revenge . <end> <OOV> <OOV> <OOV>',\n",
              " '<start> welcome back . <end> <OOV> <OOV> <OOV> <OOV> <OOV> <OOV>',\n",
              " '<start> i love reading . <end> <OOV> <OOV> <OOV> <OOV> <OOV>',\n",
              " '<start> don t push me ! <end> <OOV> <OOV> <OOV> <OOV>',\n",
              " '<start> he let go of the rope . <end> <OOV> <OOV>',\n",
              " '<start> run ! <end> <OOV> <OOV> <OOV> <OOV> <OOV> <OOV> <OOV>',\n",
              " '<start> tom betrayed you . <end> <OOV> <OOV> <OOV> <OOV> <OOV>',\n",
              " '<start> hold on . <end> <OOV> <OOV> <OOV> <OOV> <OOV> <OOV>',\n",
              " '<start> he was very nervous . <end> <OOV> <OOV> <OOV> <OOV>']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 125
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kk7sbSx9c1a9",
        "colab_type": "text"
      },
      "source": [
        "We can see that the model worked well. Despite being not very accurate the translations, however do make some sense. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "okmWWHmMc8T6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}