language_translation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Language Translation\n",
    "In this project, I’ll be training a sequence to sequence model on a dataset of English and French sentences that can translate new sentences from English to French."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the Data\n",
    "Since translating the whole language of English to French will take lots of time to train, henve only a small portion of the English corpus is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import helper\n",
    "import problem_unittests as tests\n",
    "\n",
    "source_path = 'data/small_vocab_en'\n",
    "target_path = 'data/small_vocab_fr'\n",
    "source_text = helper.load_data(source_path)\n",
    "target_text = helper.load_data(target_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the Data\n",
    "Play around with view_sentence_range to view different parts of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset Stats\n",
      "Roughly the number of unique words: 227\n",
      "Number of sentences: 137861\n",
      "Average number of words in a sentence: 13.225277634719028\n",
      "\n",
      "English sentences 0 to 10:\n",
      "new jersey is sometimes quiet during autumn , and it is snowy in april .\n",
      "the united states is usually chilly during july , and it is usually freezing in november .\n",
      "california is usually quiet during march , and it is usually hot in june .\n",
      "the united states is sometimes mild during june , and it is cold in september .\n",
      "your least liked fruit is the grape , but my least liked is the apple .\n",
      "his favorite fruit is the orange , but my favorite is the grape .\n",
      "paris is relaxing during december , but it is usually chilly in july .\n",
      "new jersey is busy during spring , and it is never hot in march .\n",
      "our least liked fruit is the lemon , but my least liked is the grape .\n",
      "the united states is sometimes busy during january , and it is sometimes warm in november .\n",
      "\n",
      "French sentences 0 to 10:\n",
      "new jersey est parfois calme pendant l' automne , et il est neigeux en avril .\n",
      "les états-unis est généralement froid en juillet , et il gèle habituellement en novembre .\n",
      "california est généralement calme en mars , et il est généralement chaud en juin .\n",
      "les états-unis est parfois légère en juin , et il fait froid en septembre .\n",
      "votre moins aimé fruit est le raisin , mais mon moins aimé est la pomme .\n",
      "son fruit préféré est l'orange , mais mon préféré est le raisin .\n",
      "paris est relaxant en décembre , mais il est généralement froid en juillet .\n",
      "new jersey est occupé au printemps , et il est jamais chaude en mars .\n",
      "notre fruit est moins aimé le citron , mais mon moins aimé est le raisin .\n",
      "les états-unis est parfois occupé en janvier , et il est parfois chaud en novembre .\n"
     ]
    }
   ],
   "source": [
    "view_sentence_range = (0, 10)\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "print('Dataset Stats')\n",
    "print('Roughly the number of unique words: {}'.format(len({word: None for word in source_text.split()})))\n",
    "\n",
    "sentences = source_text.split('\\n')\n",
    "word_counts = [len(sentence.split()) for sentence in sentences]\n",
    "print('Number of sentences: {}'.format(len(sentences)))\n",
    "print('Average number of words in a sentence: {}'.format(np.average(word_counts)))\n",
    "\n",
    "print()\n",
    "print('English sentences {} to {}:'.format(*view_sentence_range))\n",
    "print('\\n'.join(source_text.split('\\n')[view_sentence_range[0]:view_sentence_range[1]]))\n",
    "print()\n",
    "print('French sentences {} to {}:'.format(*view_sentence_range))\n",
    "print('\\n'.join(target_text.split('\\n')[view_sentence_range[0]:view_sentence_range[1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implement Preprocessing Function\n",
    "### Text to Word Ids\n",
    "As you did with other RNNs, you must turn the text into a number so the computer can understand it. In the function text_to_ids(), you'll turn source_text and target_text from words to ids. However, you need to add the <EOS> word id at the end of target_text. This will help the neural network predict when the sentence should end.\n",
    "\n",
    "You can get the <EOS> word id by doing:\n",
    "\n",
    "    target_vocab_to_int['<EOS>']\n",
    "You can get other word ids using source_vocab_to_int and target_vocab_to_int."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def text_to_ids(source_text, target_text, source_vocab_to_int, target_vocab_to_int):\n",
    "    source_id_text, target_id_text = [], []\n",
    "    for sentence in source_text.split('\\n'):\n",
    "        arr = [source_vocab_to_int[word] for word in sentence.split(' ') if word != '']\n",
    "        source_id_text.append(arr)\n",
    "    \n",
    "    for sentence in target_text.split('\\n'):\n",
    "        arr = [target_vocab_to_int[word] for word in sentence.split(' ') if word != '']\n",
    "        arr.append(target_vocab_to_int['<EOS>'])\n",
    "        target_id_text.append(arr)\n",
    "    \n",
    "    return (source_id_text, target_id_text)\n",
    "\n",
    "tests.test_text_to_ids(text_to_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess all the data and save it\n",
    "Renning the code cell below will preprocess all the data and save it to file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "helper.preprocess_and_save_data(source_path, target_path, text_to_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check Point\n",
    "This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import helper\n",
    "import problem_unittests as tests\n",
    "\n",
    "(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the Version of TensorFlow and Access to GPU\n",
    "This will check to make sure you have the correct version of TensorFlow and access to a GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow Version: 1.1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/krypten/miniconda3/lib/python3.6/site-packages/ipykernel/__main__.py:12: UserWarning: No GPU found. Please use a GPU to train your neural network.\n"
     ]
    }
   ],
   "source": [
    "from distutils.version import LooseVersion\n",
    "import warnings\n",
    "import tensorflow as tf\n",
    "from tensorflow.python.layers.core import Dense\n",
    "\n",
    "# Check TensorFlow Version\n",
    "assert LooseVersion(tf.__version__) >= LooseVersion('1.1'), 'Please use TensorFlow version 1.1 or newer'\n",
    "print('TensorFlow Version: {}'.format(tf.__version__))\n",
    "\n",
    "# Check for a GPU\n",
    "if not tf.test.gpu_device_name():\n",
    "    warnings.warn('No GPU found. Please use a GPU to train your neural network.')\n",
    "else:\n",
    "    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the Neural Network\n",
    "You'll build the components necessary to build a Sequence-to-Sequence model by implementing the following functions below:\n",
    "* model_inputs\n",
    "* process_decoder_input\n",
    "* encoding_layer\n",
    "* decoding_layer_train\n",
    "* decoding_layer_infer\n",
    "* decoding_layer\n",
    "* seq2seq_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input\n",
    "Implement the model_inputs() function to create TF Placeholders for the Neural Network. It should create the following placeholders:\n",
    "\n",
    "* Input text placeholder named \"input\" using the TF Placeholder name parameter with rank 2.\n",
    "* Targets placeholder with rank 2.\n",
    "* Learning rate placeholder with rank 0.\n",
    "* Keep probability placeholder named \"keep_prob\" using the TF Placeholder name parameter with rank 0.\n",
    "* Target sequence length placeholder named \"target_sequence_length\" with rank 1\n",
    "* Max target sequence length tensor named \"max_target_len\" getting its value from applying tf.reduce_max on the target_sequence_length placeholder. Rank 0.\n",
    "* Source sequence length placeholder named \"source_sequence_length\" with rank 1\n",
    "\n",
    "Return the placeholders in the following the tuple (input, targets, learning rate, keep probability, target sequence length, max target sequence length, source sequence length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def model_inputs():\n",
    "    input = tf.placeholder(tf.int32, [None, None], name='input')\n",
    "    target = tf.placeholder(tf.int32, [None, None], name='target')\n",
    "    learning_rate = tf.placeholder(tf.float32, name='learning_rate')\n",
    "    keep_prob = tf.placeholder(tf.float32, name='keep_prob')\n",
    "    source_sequence_length = tf.placeholder(tf.int32, [None], name='source_sequence_length')\n",
    "    target_sequence_length = tf.placeholder(tf.int32, [None], name='target_sequence_length')\n",
    "    max_target_len = tf.reduce_max(target_sequence_length)\n",
    "    \n",
    "    return input, target, learning_rate, keep_prob, target_sequence_length, max_target_len, source_sequence_length\n",
    "\n",
    "tests.test_model_inputs(model_inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Process Decoder Input\n",
    "Implement process_decoder_input by removing the last word id from each batch in target_data and concat the GO ID to the begining of each batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def process_decoder_input(target_data, target_vocab_to_int, batch_size):\n",
    "    ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])\n",
    "    dec_input = tf.concat([tf.fill([batch_size, 1], target_vocab_to_int['<GO>']), ending], 1)\n",
    "    return dec_input\n",
    "\n",
    "tests.test_process_encoding_input(process_decoder_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding\n",
    "Implement encoding_layer() to create a Encoder RNN layer:\n",
    "* Embed the encoder input using tf.contrib.layers.embed_sequence\n",
    "* Construct a stacked tf.contrib.rnn.LSTMCell wrapped in a tf.contrib.rnn.DropoutWrapper\n",
    "* Pass cell and embedded input to tf.nn.dynamic_rnn()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "from imp import reload\n",
    "reload(tests)\n",
    "\n",
    "def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob, \n",
    "                   source_sequence_length, source_vocab_size, \n",
    "                   encoding_embedding_size):\n",
    "    # Encoding\n",
    "    embed = tf.contrib.layers.embed_sequence(rnn_inputs, \n",
    "                                            vocab_size=source_vocab_size, \n",
    "                                            embed_dim=encoding_embedding_size)\n",
    "    \n",
    "    # LSTM Cell\n",
    "    def build_cell(lstm_size):\n",
    "        return tf.contrib.rnn.BasicLSTMCell(lstm_size)\n",
    "    cell = tf.contrib.rnn.MultiRNNCell([build_cell(rnn_size) for _ in range(num_layers)])\n",
    "    cell = tf.contrib.rnn.DropoutWrapper(cell, output_keep_prob=keep_prob)\n",
    "    \n",
    "    output, final_state = tf.nn.dynamic_rnn(cell, embed, dtype=tf.float32)\n",
    "    return (output, final_state)\n",
    "\n",
    "tests.test_encoding_layer(encoding_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoding - Training\n",
    "Create a training decoding layer:\n",
    "\n",
    "* Create a tf.contrib.seq2seq.TrainingHelper\n",
    "* Create a tf.contrib.seq2seq.BasicDecoder\n",
    "* Obtain the decoder outputs from tf.contrib.seq2seq.dynamic_decode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def decoding_layer_train(encoder_state, dec_cell, dec_embed_input, \n",
    "                         target_sequence_length, max_summary_length, \n",
    "                         output_layer, keep_prob):\n",
    "    helper = tf.contrib.seq2seq.TrainingHelper(dec_embed_input, target_sequence_length)\n",
    "    decoder = tf.contrib.seq2seq.BasicDecoder(\n",
    "        cell=dec_cell,\n",
    "        helper=helper,\n",
    "        initial_state=encoder_state,\n",
    "        output_layer=output_layer)\n",
    "    final_outputs, final_state = tf.contrib.seq2seq.dynamic_decode(decoder,\n",
    "                                                                   impute_finished=True,\n",
    "                                                                   maximum_iterations=max_summary_length)\n",
    "    \n",
    "    return final_outputs\n",
    "\n",
    "tests.test_decoding_layer_train(decoding_layer_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoding - Inference\n",
    "Create inference decoder:\n",
    "* Create a tf.contrib.seq2seq.GreedyEmbeddingHelper\n",
    "* Create a tf.contrib.seq2seq.BasicDecoder\n",
    "* Obtain the decoder outputs from tf.contrib.seq2seq.dynamic_decode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id,\n",
    "                         end_of_sequence_id, max_target_sequence_length,\n",
    "                         vocab_size, output_layer, batch_size, keep_prob):\n",
    "    \n",
    "    start_tokens = tf.tile(tf.constant([start_of_sequence_id], dtype=tf.int32), [batch_size], name='start_tokens')\n",
    "    helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(dec_embeddings, start_tokens, end_of_sequence_id)\n",
    "    \n",
    "    decoder = tf.contrib.seq2seq.BasicDecoder(\n",
    "        cell=dec_cell,\n",
    "        helper=helper,\n",
    "        initial_state=encoder_state,\n",
    "        output_layer=output_layer)\n",
    "    final_outputs, final_state = tf.contrib.seq2seq.dynamic_decode(decoder,\n",
    "                                                                   impute_finished=True,\n",
    "                                                                   maximum_iterations=max_target_sequence_length)\n",
    "    return final_outputs\n",
    "\n",
    "tests.test_decoding_layer_infer(decoding_layer_infer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the Decoding Layer\n",
    "Implement decoding_layer() to create a Decoder RNN layer.\n",
    "* Embed the target sequences\n",
    "* Construct the decoder LSTM cell (just like you constructed the encoder cell above)\n",
    "* Create an output layer to map the outputs of the decoder to the elements of our vocabulary\n",
    "* Use the your decoding_layer_train(encoder_state, dec_cell, dec_embed_input, target_sequence_length, max_target_sequence_length, output_layer, keep_prob) function to get the training logits.\n",
    "* Use your decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id, max_target_sequence_length, vocab_size, output_layer, batch_size, keep_prob) function to get the inference logits.\n",
    "\n",
    "Note: You'll need to use tf.variable_scope to share variables between training and inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def decoding_layer(dec_input, encoder_state,\n",
    "                   target_sequence_length, max_target_sequence_length,\n",
    "                   rnn_size,\n",
    "                   num_layers, target_vocab_to_int, target_vocab_size,\n",
    "                   batch_size, keep_prob, decoding_embedding_size):\n",
    "    # Embedding\n",
    "    dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, decoding_embedding_size]))\n",
    "    dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)\n",
    "    \n",
    "    # LSTM Cell\n",
    "    def build_cell(lstm_size):\n",
    "        return tf.contrib.rnn.BasicLSTMCell(lstm_size)\n",
    "    cell = tf.contrib.rnn.MultiRNNCell([build_cell(rnn_size) for _ in range(num_layers)])\n",
    "    dec_cell = tf.contrib.rnn.DropoutWrapper(cell, output_keep_prob=keep_prob)\n",
    "    \n",
    "    output_layer = Dense(target_vocab_size,\n",
    "                         kernel_initializer = tf.truncated_normal_initializer(mean = 0.0, stddev=0.1))\n",
    "    \n",
    "    with tf.variable_scope(\"decode\"):\n",
    "        output_training = decoding_layer_train(\n",
    "                                    encoder_state,\n",
    "                                    dec_cell,\n",
    "                                    dec_embed_input,\n",
    "                                    target_sequence_length,\n",
    "                                    max_target_sequence_length,\n",
    "                                    output_layer,\n",
    "                                    keep_prob)\n",
    "    \n",
    "    with tf.variable_scope(\"decode\", reuse=True):\n",
    "        output_inference = decoding_layer_infer(\n",
    "                                    encoder_state,\n",
    "                                    dec_cell,\n",
    "                                    dec_embeddings,\n",
    "                                    target_vocab_to_int['<GO>'],\n",
    "                                    target_vocab_to_int['<EOS>'],\n",
    "                                    max_target_sequence_length,\n",
    "                                    target_vocab_size,\n",
    "                                    output_layer,\n",
    "                                    batch_size,\n",
    "                                    keep_prob)\n",
    "    return (output_training, output_inference)\n",
    "\n",
    "tests.test_decoding_layer(decoding_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the Neural Network\n",
    "Apply the functions you implemented above to:\n",
    "* Encode the input using your encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob,  source_sequence_length, source_vocab_size, encoding_embedding_size).\n",
    "* Process target data using your process_decoder_input(target_data, target_vocab_to_int, batch_size) function.\n",
    "* Decode the encoded input using your decoding_layer(dec_input, enc_state, target_sequence_length, max_target_sentence_length, rnn_size, num_layers, target_vocab_to_int, target_vocab_size, batch_size, keep_prob, dec_embedding_size) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def seq2seq_model(input_data, target_data, keep_prob, batch_size,\n",
    "                  source_sequence_length, target_sequence_length,\n",
    "                  max_target_sentence_length,\n",
    "                  source_vocab_size, target_vocab_size,\n",
    "                  enc_embedding_size, dec_embedding_size,\n",
    "                  rnn_size, num_layers, target_vocab_to_int):\n",
    "    (output, enc_state) = encoding_layer(input_data,\n",
    "                   rnn_size,\n",
    "                   num_layers,\n",
    "                   keep_prob,\n",
    "                   source_sequence_length,\n",
    "                   source_vocab_size,\n",
    "                   enc_embedding_size)\n",
    "    dec_input = process_decoder_input(target_data, target_vocab_to_int, batch_size)\n",
    "    (output_training, output_inference) = decoding_layer(dec_input,\n",
    "                   enc_state,\n",
    "                   target_sequence_length,\n",
    "                   max_target_sentence_length,\n",
    "                   rnn_size,\n",
    "                   num_layers,\n",
    "                   target_vocab_to_int,\n",
    "                   target_vocab_size,\n",
    "                   batch_size,\n",
    "                   keep_prob,\n",
    "                   dec_embedding_size) \n",
    "    \n",
    "    return output_training, output_inference\n",
    "\n",
    "tests.test_seq2seq_model(seq2seq_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Network Training\n",
    "## Hyperparameters\n",
    "Tune the following parameters:\n",
    "* Set epochs to the number of epochs.\n",
    "* Set batch_size to the batch size.\n",
    "* Set rnn_size to the size of the RNNs.\n",
    "* Set num_layers to the number of layers.\n",
    "* Set encoding_embedding_size to the size of the embedding for the encoder.\n",
    "* Set decoding_embedding_size to the size of the embedding for the decoder.\n",
    "* Set learning_rate to the learning rate.\n",
    "* Set keep_probability to the Dropout keep probability\n",
    "* Set display_step to state how many steps between each debug output statement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Number of Epochs\n",
    "epochs = 5\n",
    "# Batch Size\n",
    "batch_size = 128\n",
    "# RNN Size\n",
    "rnn_size = 256\n",
    "# Number of Layers\n",
    "num_layers = 2\n",
    "# Embedding Size\n",
    "encoding_embedding_size = 30\n",
    "decoding_embedding_size = 30\n",
    "# Learning Rate\n",
    "learning_rate = 0.01\n",
    "# Dropout Keep Probability\n",
    "keep_probability = 0.8\n",
    "display_step = 500"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the Graph\n",
    "Build the graph using the neural network you implemented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "save_path = 'checkpoints/dev'\n",
    "(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()\n",
    "max_target_sentence_length = max([len(sentence) for sentence in source_int_text])\n",
    "\n",
    "train_graph = tf.Graph()\n",
    "with train_graph.as_default():\n",
    "    input_data, targets, lr, keep_prob, target_sequence_length, max_target_sequence_length, source_sequence_length = model_inputs()\n",
    "    \n",
    "    input_shape = tf.shape(input_data)\n",
    "    train_logits, inference_logits = seq2seq_model(tf.reverse(input_data, [-1]),\n",
    "                                                   targets,\n",
    "                                                   keep_prob,\n",
    "                                                   batch_size,\n",
    "                                                   source_sequence_length,\n",
    "                                                   target_sequence_length,\n",
    "                                                   max_target_sequence_length,\n",
    "                                                   len(source_vocab_to_int),\n",
    "                                                   len(target_vocab_to_int),\n",
    "                                                   encoding_embedding_size,\n",
    "                                                   decoding_embedding_size,\n",
    "                                                   rnn_size,\n",
    "                                                   num_layers,\n",
    "                                                   target_vocab_to_int)\n",
    "    \n",
    "    training_logits = tf.identity(train_logits.rnn_output, name='logits')\n",
    "    inference_logits = tf.identity(inference_logits.sample_id, name='predictions')\n",
    "    \n",
    "    masks = tf.sequence_mask(target_sequence_length, max_target_sequence_length, dtype=tf.float32, name='masks')\n",
    "    \n",
    "    with tf.name_scope(\"optimization\"):\n",
    "        # Loss function\n",
    "        cost = tf.contrib.seq2seq.sequence_loss(\n",
    "            training_logits,\n",
    "            targets,\n",
    "            masks)\n",
    "        \n",
    "        # Optimizer\n",
    "        optimizer = tf.train.AdamOptimizer(lr)\n",
    "        \n",
    "        # Gradient Clipping\n",
    "        gradients = optimizer.compute_gradients(cost)\n",
    "        capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]\n",
    "        train_op = optimizer.apply_gradients(capped_gradients)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Batch and pad the source and target sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pad_sentence_batch(sentence_batch, pad_int):\n",
    "    max_sentence = max([len(sentence) for sentence in sentence_batch])\n",
    "    return [sentence + [pad_int] * (max_sentence - len(sentence)) for sentence in sentence_batch]\n",
    "\n",
    "def get_batches(sources, targets, batch_size, source_pad_int, target_pad_int):\n",
    "    for batch_i in range(0, len(sources)//batch_size):\n",
    "        start_i = batch_i * batch_size\n",
    "        \n",
    "        # Slice the right amount for the batch\n",
    "        sources_batch = sources[start_i:start_i + batch_size]\n",
    "        targets_batch = targets[start_i:start_i + batch_size]\n",
    "        \n",
    "        # Pad\n",
    "        pad_sources_batch = np.array(pad_sentence_batch(sources_batch, source_pad_int))\n",
    "        pad_targets_batch = np.array(pad_sentence_batch(targets_batch, target_pad_int))\n",
    "        \n",
    "        # Need the lengths for the _lengths parameters\n",
    "        pad_sources_lengths = []\n",
    "        for source in pad_sources_batch:\n",
    "            pad_sources_lengths.append(len(source))\n",
    "            \n",
    "        pad_targets_lengths = []\n",
    "        for target in pad_targets_batch:\n",
    "            pad_targets_lengths.append(len(target))\n",
    "        \n",
    "        yield pad_sources_batch, pad_targets_batch, pad_sources_lengths, pad_targets_lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train\n",
    "Train the neural network on the preprocessed data. If you have a hard time getting a good loss, check the forms to see if anyone is having the same problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   0 Batch  500/1077 - Train Accuracy: 0.7316, Validation Accuracy: 0.7372, Loss: 0.3766\n",
      "Epoch   0 Batch 1000/1077 - Train Accuracy: 0.9003, Validation Accuracy: 0.8619, Loss: 0.1119\n",
      "Epoch   1 Batch  500/1077 - Train Accuracy: 0.9430, Validation Accuracy: 0.9219, Loss: 0.0398\n",
      "Epoch   1 Batch 1000/1077 - Train Accuracy: 0.9457, Validation Accuracy: 0.9357, Loss: 0.0392\n",
      "Epoch   2 Batch  500/1077 - Train Accuracy: 0.9668, Validation Accuracy: 0.9400, Loss: 0.0250\n",
      "Epoch   2 Batch 1000/1077 - Train Accuracy: 0.9449, Validation Accuracy: 0.9563, Loss: 0.0314\n",
      "Epoch   3 Batch  500/1077 - Train Accuracy: 0.9691, Validation Accuracy: 0.9382, Loss: 0.0249\n",
      "Epoch   3 Batch 1000/1077 - Train Accuracy: 0.9494, Validation Accuracy: 0.9521, Loss: 0.0300\n",
      "Epoch   4 Batch  500/1077 - Train Accuracy: 0.9688, Validation Accuracy: 0.9368, Loss: 0.0222\n",
      "Epoch   4 Batch 1000/1077 - Train Accuracy: 0.9628, Validation Accuracy: 0.9315, Loss: 0.0333\n",
      "Model Trained and Saved\n"
     ]
    }
   ],
   "source": [
    "def get_accuracy(target, logits):\n",
    "    \"\"\"\n",
    "    Calculate accuracy\n",
    "    \"\"\"\n",
    "    max_seq = max(target.shape[1], logits.shape[1])\n",
    "    if max_seq - target.shape[1]:\n",
    "        target = np.pad(\n",
    "            target,\n",
    "            [(0,0),(0,max_seq - target.shape[1])],\n",
    "            'constant')\n",
    "    if max_seq - logits.shape[1]:\n",
    "        logits = np.pad(\n",
    "            logits,\n",
    "            [(0,0),(0,max_seq - logits.shape[1])],\n",
    "            'constant')\n",
    "\n",
    "    return np.mean(np.equal(target, logits))\n",
    "\n",
    "# Split data to training and validation sets\n",
    "train_source = source_int_text[batch_size:]\n",
    "train_target = target_int_text[batch_size:]\n",
    "valid_source = source_int_text[:batch_size]\n",
    "valid_target = target_int_text[:batch_size]\n",
    "(valid_sources_batch, valid_targets_batch, valid_sources_lengths, valid_targets_lengths ) = next(get_batches(valid_source, valid_target, batch_size, source_vocab_to_int['<PAD>'], target_vocab_to_int['<PAD>']))\n",
    "\n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    for epoch_i in range(epochs):\n",
    "        for batch_i, (source_batch, target_batch, sources_lengths, targets_lengths) in enumerate(\n",
    "                get_batches(train_source, train_target, batch_size,\n",
    "                            source_vocab_to_int['<PAD>'],\n",
    "                            target_vocab_to_int['<PAD>'])):\n",
    "            \n",
    "            _, loss = sess.run([train_op, cost],\n",
    "                               {input_data: source_batch,\n",
    "                                targets: target_batch,\n",
    "                                lr: learning_rate,\n",
    "                                target_sequence_length: targets_lengths,\n",
    "                                source_sequence_length: sources_lengths,\n",
    "                                keep_prob: keep_probability})\n",
    "            if batch_i % display_step == 0 and batch_i > 0:\n",
    "                batch_train_logits = sess.run(\n",
    "                    inference_logits,\n",
    "                    {input_data: source_batch,\n",
    "                     source_sequence_length: sources_lengths,\n",
    "                     target_sequence_length: targets_lengths,\n",
    "                     keep_prob: 1.0})\n",
    "                \n",
    "                batch_valid_logits = sess.run(\n",
    "                    inference_logits,\n",
    "                    {input_data: valid_sources_batch,\n",
    "                     source_sequence_length: valid_sources_lengths,\n",
    "                     target_sequence_length: valid_targets_lengths,\n",
    "                     keep_prob: 1.0})\n",
    "                \n",
    "                train_acc = get_accuracy(target_batch, batch_train_logits)\n",
    "                valid_acc = get_accuracy(valid_targets_batch, batch_valid_logits)\n",
    "                \n",
    "                print('Epoch {:>3} Batch {:>4}/{} - Train Accuracy: {:>6.4f}, Validation Accuracy: {:>6.4f}, Loss: {:>6.4f}'\n",
    "                      .format(epoch_i, batch_i, len(source_int_text) // batch_size, train_acc, valid_acc, loss))\n",
    "                \n",
    "    # Save Model\n",
    "    saver = tf.train.Saver()\n",
    "    saver.save(sess, save_path)\n",
    "    print('Model Trained and Saved')                "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Parameters\n",
    "Save the batch_size and save_path parameters for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Save parameters for checkpoint\n",
    "helper.save_params(save_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Checkpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import helper\n",
    "import problem_unittests as tests\n",
    "\n",
    "_, (source_vocab_to_int, target_vocab_to_int), (source_int_to_vocab, target_int_to_vocab) = helper.load_preprocess()\n",
    "load_path = helper.load_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentence to Sequence\n",
    "To feed a sentence into the model for translation, you first need to preprocess it. Implement the function sentence_to_seq() to preprocess new sentences.\n",
    "* Convert the sentence to lowercase\n",
    "* Convert words into ids using vocab_to_int\n",
    " * Convert words not in the vocabulary, to the <UNK> word id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def sentence_to_seq(sentence, vocab_to_int):\n",
    "    sentence = sentence.lower()\n",
    "    words = sentence.split()\n",
    "    word_id_list = [vocab_to_int.get(word, vocab_to_int['<UNK>']) for word in words]\n",
    "    return word_id_list\n",
    "\n",
    "tests.test_sentence_to_seq(sentence_to_seq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Translate\n",
    "This will translate translate_sentence from English to French."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from checkpoints/dev\n",
      "Input\n",
      "  Word Ids:      [71, 19, 202, 157, 119, 97, 100]\n",
      "  English Words: ['he', 'saw', 'a', 'old', 'yellow', 'truck', '.']\n",
      "\n",
      "Prediction\n",
      "  Word Ids:      [262, 339, 115, 154, 142, 246, 21, 297, 1]\n",
      "  French Words: il a vu un vieux camion jaune . <EOS>\n"
     ]
    }
   ],
   "source": [
    "translate_sentence = 'he saw a old yellow truck .'\n",
    "translate_sentence = sentence_to_seq(translate_sentence, source_vocab_to_int)\n",
    "\n",
    "loaded_graph = tf.Graph()\n",
    "with tf.Session(graph=loaded_graph) as sess:\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(load_path + '.meta')\n",
    "    loader.restore(sess, load_path)\n",
    "    \n",
    "    input_data = loaded_graph.get_tensor_by_name('input:0')\n",
    "    logits = loaded_graph.get_tensor_by_name('predictions:0')\n",
    "    target_sequence_length = loaded_graph.get_tensor_by_name('target_sequence_length:0')\n",
    "    source_sequence_length = loaded_graph.get_tensor_by_name('source_sequence_length:0')\n",
    "    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')\n",
    "    \n",
    "    translate_logits = sess.run(logits, {input_data: [translate_sentence]*batch_size,\n",
    "                                         target_sequence_length: [len(translate_sentence)*2]*batch_size,\n",
    "                                         source_sequence_length: [len(translate_sentence)]*batch_size,\n",
    "                                         keep_prob: 1.0})[0]\n",
    "\n",
    "print('Input')\n",
    "print('  Word Ids:      {}'.format([i for i in translate_sentence]))\n",
    "print('  English Words: {}'.format([source_int_to_vocab[i] for i in translate_sentence]))\n",
    "\n",
    "print('\\nPrediction')\n",
    "print('  Word Ids:      {}'.format([i for i in translate_logits]))\n",
    "print('  French Words: {}'.format(\" \".join([target_int_to_vocab[i] for i in translate_logits])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imperfect Translation\n",
    "You might notice that some sentences translate better than others. Since the dataset you're using only has a vocabulary of 227 English words of the thousands that you use, you're only going to see good results using these words. For this project, you don't need a perfect translation. However, if you want to create a better translation model, you'll need better data.\n",
    "\n",
    "You can train on the WMT10 French-English corpus. This dataset has more vocabulary and richer in topics discussed. However, this will take you days to train, so make sure you've a GPU and the neural network is performing well on dataset we provided. Just make sure you play with the WMT10 corpus after you've submitted this project.\n",
    "## Submitting This Project\n",
    "When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as \"dlnd_language_translation.ipynb\" and save it as a HTML file under \"File\" -> \"Download as\". Include the \"helper.py\" and \"problem_unittests.py\" files in your submission."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}