t81_558_class_11_03_embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 11: Natural Language Processing and Speech Recognition**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 11 Material\n",
    "\n",
    "* Part 11.1: Getting Started with Spacy in Python [[Video]](https://www.youtube.com/watch?v=A5BtU9vXzu8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_01_spacy.ipynb)\n",
    "* Part 11.2: Word2Vec and Text Classification [[Video]](https://www.youtube.com/watch?v=nWxtRlpObIs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_02_word2vec.ipynb)\n",
    "* **Part 11.3: What are Embedding Layers in Keras** [[Video]](https://www.youtube.com/watch?v=OuNH5kT-aD0&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_03_embedding.ipynb)\n",
    "* Part 11.4: Natural Language Processing with Spacy and Keras [[Video]](https://www.youtube.com/watch?v=BKgwjhao5DU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_11_04_text_nlp.ipynb)\n",
    "* Part 11.5: Learning English from Scratch with Keras and TensorFlow [[Video]](https://www.youtube.com/watch?v=Y1khuuSjZzc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=58) [[Notebook]](t81_558_class_11_05_english_scratch.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 11.3: What are Embedding Layers in Keras\n",
    "\n",
    "[Embedding Layers](https://keras.io/layers/embeddings/) are a powerful feature of Keras that allow additional information to be automatically inserted into your neural network.  In the previous section you saw that Word2Vec can expand words to a 300 dimension vector.  An embedding layer would allow you to automatically insert these 300-dimension vectors in the place of word-indexes.  \n",
    "\n",
    "Embedding layers are often used with Natural Language Processing (NLP); however, they can be used in any instance where you wish to insert a larger vector in the place of an index value.  In some ways you can think of an embedding layer as dimension expansion. However, the hope is that these additional dimensions will provide more information to the model and provide a better score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple Embedding Layer Example\n",
    "\n",
    "* **input_dim** = How large is the vocabulary?  How many categories are you encoding. This is the number of items in your \"lookup table\".\n",
    "* **output_dim** = How many numbers in the vector that you wish to return. \n",
    "* **input_length** = How many items are in the input feature vector that you need to transform?\n",
    "\n",
    "Now we create one that has a vocabulary size of 10, will reduce those values between 0-9 to 4 number vectors.  Each feature vector coming in will have 2 such features.  This neural network does nothing more than pass the embedding on to the output.  But it does let us see what the embedding is doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Embedding\n",
    "import numpy as np\n",
    "\n",
    "model = Sequential()\n",
    "embedding_layer = Embedding(input_dim=10, output_dim=4, input_length=2)\n",
    "model.add(embedding_layer)\n",
    "model.compile('adam', 'mse')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets query the neural network with 2 rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 2)\n",
      "[[[-0.04465513 -0.02065047 -0.01658659 -0.03898759]\n",
      "  [ 0.01235443 -0.00267823 -0.03883423 -0.012385  ]]]\n"
     ]
    }
   ],
   "source": [
    "input_data = np.array([\n",
    "    [1,2]\n",
    "])\n",
    "\n",
    "pred = model.predict(input_data)\n",
    "\n",
    "print(input_data.shape)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([[ 0.03261134,  0.0120466 , -0.04478213, -0.04514688],\n",
       "        [-0.04465513, -0.02065047, -0.01658659, -0.03898759],\n",
       "        [ 0.01235443, -0.00267823, -0.03883423, -0.012385  ],\n",
       "        [-0.01451837, -0.0023061 ,  0.00900034, -0.03496481],\n",
       "        [ 0.03273893,  0.01941523, -0.04344552,  0.03974229],\n",
       "        [-0.02380162,  0.0377434 , -0.01789006, -0.011664  ],\n",
       "        [-0.0165769 , -0.03008759,  0.00087344, -0.02701756],\n",
       "        [ 0.0103844 ,  0.01787665, -0.00957631, -0.00504515],\n",
       "        [ 0.01307121,  0.02575408,  0.00197864, -0.02592713],\n",
       "        [ 0.04115922, -0.03220158,  0.04251255,  0.03303945]],\n",
       "       dtype=float32)]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_layer.get_weights()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transferring An Embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Embedding\n",
    "import numpy as np\n",
    "\n",
    "embedding_lookup = np.array([\n",
    "    [1,0,0],\n",
    "    [0,1,0],\n",
    "    [0,0,1]\n",
    "])\n",
    "\n",
    "model = Sequential()\n",
    "embedding_layer = Embedding(input_dim=3, output_dim=3, input_length=2)\n",
    "model.add(embedding_layer)\n",
    "model.compile('adam', 'mse')\n",
    "\n",
    "embedding_layer.set_weights([embedding_lookup])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 2)\n",
      "[[[1. 0. 0.]\n",
      "  [0. 1. 0.]]]\n"
     ]
    }
   ],
   "source": [
    "input_data = np.array([\n",
    "    [0,1]\n",
    "])\n",
    "\n",
    "pred = model.predict(input_data)\n",
    "\n",
    "print(input_data.shape)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training an Embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy import array\n",
    "from tensorflow.keras.preprocessing.text import one_hot\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Flatten, Embedding, Dense"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define 10 resturant reviews.\n",
    "reviews = [\n",
    "    'Never coming back!',\n",
    "    'Horrible service',\n",
    "    'Rude waitress',\n",
    "    'Cold food.',\n",
    "    'Horrible food!',\n",
    "    'Awesome',\n",
    "    'Awesome service!',\n",
    "    'Rocks!',\n",
    "    'poor work',\n",
    "    'Couldn\\'t have done better']\n",
    "\n",
    "# Define labels (1=negative, 0=positive)\n",
    "labels = array([1,1,1,1,1,0,0,0,0,0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoded reviews: [[41, 20, 38], [31, 38], [34, 14], [17, 1], [31, 1], [36], [36, 38], [10], [17, 38], [9, 24, 30, 26]]\n"
     ]
    }
   ],
   "source": [
    "VOCAB_SIZE = 50\n",
    "encoded_reviews = [one_hot(d, VOCAB_SIZE) for d in reviews]\n",
    "print(f\"Encoded reviews: {encoded_reviews}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[41 20 38  0]\n",
      " [31 38  0  0]\n",
      " [34 14  0  0]\n",
      " [17  1  0  0]\n",
      " [31  1  0  0]\n",
      " [36  0  0  0]\n",
      " [36 38  0  0]\n",
      " [10  0  0  0]\n",
      " [17 38  0  0]\n",
      " [ 9 24 30 26]]\n"
     ]
    }
   ],
   "source": [
    "MAX_LENGTH = 4\n",
    "\n",
    "padded_reviews = pad_sequences(encoded_reviews, maxlen=MAX_LENGTH, padding='post')\n",
    "print(padded_reviews)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, 4, 8)              400       \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 32)                0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 433\n",
      "Trainable params: 433\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "embedding_layer = Embedding(VOCAB_SIZE, 8, input_length=MAX_LENGTH)\n",
    "model.add(embedding_layer)\n",
    "model.add(Flatten())\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])\n",
    "\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W0820 09:30:57.185041 140736216040320 deprecation.py:323] From /Users/jheaton/miniconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x637738780>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit the model\n",
    "model.fit(padded_reviews, labels, epochs=100, verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50, 8)\n",
      "[array([[-1.27638459e-01, -7.32154772e-02, -8.23497996e-02,\n",
      "         1.55757844e-01,  9.42496359e-02, -8.72198120e-02,\n",
      "         8.76339972e-02, -7.52256885e-02],\n",
      "       [ 1.57531679e-01, -1.83092132e-01,  1.09754540e-01,\n",
      "        -5.59859611e-02, -1.42092615e-01,  1.27459422e-01,\n",
      "        -8.70256722e-02,  1.14011139e-01],\n",
      "       [-1.01960078e-02,  2.40092352e-03, -8.82671028e-03,\n",
      "        -2.32898351e-02, -1.27082579e-02, -3.45976949e-02,\n",
      "        -4.36593667e-02,  3.87248509e-02],\n",
      "       [ 2.55999826e-02,  2.62987129e-02, -4.63925712e-02,\n",
      "         3.27251665e-02,  1.01461262e-03, -1.76734850e-03,\n",
      "         1.70630924e-02, -3.78757827e-02],\n",
      "       [ 1.76978819e-02, -3.00942902e-02, -2.69936211e-02,\n",
      "         4.15484644e-02,  2.93739177e-02,  3.24756764e-02,\n",
      "        -4.73561659e-02, -2.77424101e-02],\n",
      "       [-2.61650570e-02, -4.27419059e-02,  2.94128768e-02,\n",
      "        -1.84099898e-02, -2.91952491e-02, -1.45032890e-02,\n",
      "        -4.00351174e-02, -1.71758048e-02],\n",
      "       [ 2.21025981e-02,  4.90714945e-02, -4.98351455e-02,\n",
      "        -3.42762619e-02,  6.00985438e-03,  4.58834507e-02,\n",
      "         6.29239157e-03, -7.09176064e-03],\n",
      "       [ 1.20469220e-02,  2.80299895e-02, -1.56085566e-03,\n",
      "        -4.35840152e-02, -2.99129020e-02,  4.92892973e-02,\n",
      "         1.20616183e-02, -4.93034385e-02],\n",
      "       [ 3.39503326e-02,  2.32699178e-02, -3.47792022e-02,\n",
      "         3.85733508e-02, -8.75700638e-03,  3.85055281e-02,\n",
      "        -3.33492644e-02,  4.12801839e-02],\n",
      "       [-6.53073043e-02,  6.43995628e-02,  1.41115680e-01,\n",
      "         1.37434706e-01,  1.44881666e-01,  5.91219440e-02,\n",
      "        -1.11271143e-01,  1.40764132e-01],\n",
      "       [-1.17150545e-01,  1.17336661e-01,  1.60248488e-01,\n",
      "         1.36416152e-01,  9.63227600e-02,  6.36191666e-02,\n",
      "        -1.21699564e-01,  9.33679193e-02],\n",
      "       [-2.41705664e-02,  4.84076254e-02, -4.28567193e-02,\n",
      "        -4.62335236e-02,  6.78142160e-03, -2.56966427e-03,\n",
      "        -9.06240940e-03, -2.71082651e-02],\n",
      "       [ 2.51625217e-02, -4.79624644e-02,  7.86251947e-03,\n",
      "         4.71709482e-02, -4.46544550e-02,  4.79450487e-02,\n",
      "         2.57652439e-02, -2.59684678e-02],\n",
      "       [ 1.05608217e-02, -4.60422300e-02, -7.94041902e-04,\n",
      "         2.83906944e-02, -1.57152489e-03, -3.91790643e-02,\n",
      "         4.63046916e-02, -1.16083734e-02],\n",
      "       [ 1.33147553e-01, -1.77805409e-01,  5.86671084e-02,\n",
      "        -9.35554877e-02, -7.03387260e-02,  1.40483305e-01,\n",
      "        -7.25190490e-02,  6.09031245e-02],\n",
      "       [ 3.41262259e-02,  2.06382014e-02, -1.08463764e-02,\n",
      "         2.90428437e-02,  3.04693840e-02, -4.75016832e-02,\n",
      "        -4.54979017e-03, -3.73430140e-02],\n",
      "       [-3.16906944e-02,  2.65635364e-02, -4.06373851e-02,\n",
      "         4.35189046e-02, -1.13512874e-02, -4.59094755e-02,\n",
      "         3.24223749e-02,  3.14243920e-02],\n",
      "       [-3.96284536e-02,  5.71823530e-02,  4.92128506e-02,\n",
      "         1.03677876e-01,  3.96134146e-02,  5.59797026e-02,\n",
      "        -2.97813024e-02,  5.13190478e-02],\n",
      "       [-2.48364937e-02, -4.48569171e-02, -4.55492139e-02,\n",
      "        -3.69077101e-02, -1.06841698e-02, -8.06143135e-03,\n",
      "        -2.45194919e-02, -3.60227004e-02],\n",
      "       [-2.02316884e-02, -1.80474892e-02, -2.13288553e-02,\n",
      "        -2.99344063e-02, -4.20380943e-02,  1.18177310e-02,\n",
      "        -1.46966092e-02, -1.01048127e-02],\n",
      "       [ 6.07240684e-02, -1.48598492e-01,  1.42832220e-01,\n",
      "        -8.70573670e-02, -1.12305626e-01,  8.60677660e-02,\n",
      "        -9.77530107e-02,  9.56647396e-02],\n",
      "       [ 2.69803666e-02, -4.91398685e-02,  2.36564316e-02,\n",
      "         2.21328177e-02,  1.51170827e-02, -1.12605691e-02,\n",
      "         2.65229382e-02, -1.49989836e-02],\n",
      "       [-1.91685446e-02,  2.53888108e-02, -3.73181477e-02,\n",
      "         1.69311650e-02,  7.80122355e-03,  3.13251875e-02,\n",
      "        -2.38785278e-02, -2.78156046e-02],\n",
      "       [ 4.21969779e-02, -4.63155657e-03,  1.06233954e-02,\n",
      "         4.93466146e-02, -1.36091225e-02, -3.42379212e-02,\n",
      "        -4.18865681e-02,  7.48466700e-04],\n",
      "       [-1.16421141e-01,  1.18699647e-01, -1.44919947e-01,\n",
      "         1.42912403e-01,  7.32873976e-02, -9.24467593e-02,\n",
      "         1.05575934e-01, -4.83868383e-02],\n",
      "       [ 2.01657750e-02, -1.83456168e-02,  4.46293838e-02,\n",
      "        -3.28654423e-02, -7.43005425e-03, -2.03108322e-02,\n",
      "        -6.00732490e-03,  1.02709159e-02],\n",
      "       [ 1.27216980e-01,  1.33295357e-01,  1.16956219e-01,\n",
      "         1.04745403e-01,  4.60819639e-02,  1.47473231e-01,\n",
      "         1.46169677e-01,  1.36659043e-02],\n",
      "       [-1.96083635e-03, -4.99341153e-02, -3.08993347e-02,\n",
      "        -1.64446719e-02,  1.56081803e-02,  2.11512335e-02,\n",
      "         2.99200900e-02, -1.26866810e-02],\n",
      "       [-3.45591903e-02,  4.72246893e-02,  4.96268272e-04,\n",
      "         1.83574595e-02,  1.44737475e-02, -1.83882341e-02,\n",
      "        -1.56709924e-02,  2.85472758e-02],\n",
      "       [-2.24688295e-02,  1.96416304e-03,  3.01393308e-02,\n",
      "        -2.52007246e-02, -2.79577263e-02,  9.71044227e-03,\n",
      "        -4.52019945e-02,  3.48014720e-02],\n",
      "       [-9.06442851e-02, -5.04268445e-02,  1.24671265e-01,\n",
      "        -1.15815401e-01,  1.16480939e-01, -1.38771787e-01,\n",
      "        -1.66879356e-01,  8.52634460e-02],\n",
      "       [ 1.34694874e-01, -1.59155518e-01, -1.47928432e-01,\n",
      "        -9.95706022e-02, -9.48849916e-02, -1.30862430e-01,\n",
      "         1.32454976e-01, -1.24046236e-01],\n",
      "       [ 4.29037847e-02, -1.18198022e-02, -3.23562026e-02,\n",
      "         2.77888030e-04,  3.57273109e-02, -4.55552340e-02,\n",
      "        -1.71355009e-02, -3.58140841e-02],\n",
      "       [ 1.47119872e-02, -8.39581341e-03, -4.97337095e-02,\n",
      "         4.14907932e-04,  4.49181721e-03, -3.88239622e-02,\n",
      "        -4.76641431e-02, -3.72699872e-02],\n",
      "       [ 1.21359736e-01, -1.41499981e-01, -1.73865706e-01,\n",
      "        -1.02452829e-01, -8.52961168e-02, -8.68981779e-02,\n",
      "         1.04418553e-01, -1.16582833e-01],\n",
      "       [ 1.05563886e-02,  4.88700904e-02,  4.19358872e-02,\n",
      "         2.77933218e-02,  4.00756933e-02,  4.18898575e-02,\n",
      "        -1.10643990e-02, -4.68294397e-02],\n",
      "       [-9.51950252e-02,  1.49518698e-01,  8.94053802e-02,\n",
      "         1.29261225e-01,  1.54982850e-01,  5.19184060e-02,\n",
      "        -8.58952329e-02,  8.13751444e-02],\n",
      "       [ 5.69326803e-03,  3.60976718e-02, -2.84970757e-02,\n",
      "        -4.30928133e-02,  4.85866107e-02,  4.27313931e-02,\n",
      "        -3.79558317e-02, -4.41237092e-02],\n",
      "       [ 9.48985964e-02,  1.33492753e-01, -1.33382708e-01,\n",
      "         1.14643596e-01,  1.09594285e-01,  5.01741432e-02,\n",
      "         8.93134698e-02, -1.19802400e-01],\n",
      "       [ 9.17690992e-03,  4.67830189e-02,  1.17662549e-02,\n",
      "         4.45526876e-02, -3.63004431e-02,  4.46372293e-02,\n",
      "        -1.81698203e-02, -3.30165848e-02],\n",
      "       [ 1.14683062e-03, -2.60279328e-03, -4.54955362e-02,\n",
      "         2.40893401e-02,  1.16487369e-02, -9.96934250e-03,\n",
      "         5.20550087e-03,  4.65192311e-02],\n",
      "       [ 1.15355372e-01, -1.22557804e-01, -1.59467891e-01,\n",
      "        -1.33583352e-01, -5.86559512e-02, -1.12420522e-01,\n",
      "         9.34256315e-02, -9.48332548e-02],\n",
      "       [ 1.26518644e-02, -3.46521512e-02, -4.85382974e-04,\n",
      "        -4.32965532e-02,  5.85318729e-03,  4.25877012e-02,\n",
      "        -1.05450749e-02, -1.25950575e-03],\n",
      "       [ 1.41703226e-02,  4.64644283e-03, -3.20282951e-02,\n",
      "         1.28390454e-02, -2.88494118e-02, -2.85348650e-02,\n",
      "        -2.35750526e-03,  4.94522713e-02],\n",
      "       [-4.55510989e-02, -3.92590053e-02, -9.10770148e-04,\n",
      "         4.16951440e-02,  3.82498391e-02,  3.29226516e-02,\n",
      "        -1.85016282e-02, -3.19965705e-02],\n",
      "       [-1.97278745e-02,  2.49309801e-02,  1.07770078e-02,\n",
      "        -4.14409637e-02, -9.98783112e-03, -2.36777067e-02,\n",
      "         4.83242907e-02, -1.39058605e-02],\n",
      "       [-2.59440020e-03, -2.25135218e-02,  4.73927893e-02,\n",
      "         1.39506906e-03, -1.27900764e-03,  9.10118967e-03,\n",
      "        -2.74577141e-02,  6.77590445e-03],\n",
      "       [ 4.02823202e-02,  3.52226384e-02,  3.61128338e-02,\n",
      "        -3.43546271e-05, -2.78259870e-02,  3.51261534e-02,\n",
      "        -3.73568051e-02,  2.22474448e-02],\n",
      "       [ 5.29993698e-03,  4.95826714e-02,  3.75631116e-02,\n",
      "         1.11380927e-02, -3.87401804e-02,  3.56944539e-02,\n",
      "        -2.46392619e-02, -3.50308307e-02],\n",
      "       [ 5.34796715e-03,  1.39355399e-02,  2.06021219e-03,\n",
      "        -2.76877880e-02,  9.81122255e-03, -5.76760620e-03,\n",
      "         4.36641090e-02,  4.68137600e-02]], dtype=float32)]\n"
     ]
    }
   ],
   "source": [
    "print(embedding_layer.get_weights()[0].shape)\n",
    "print(embedding_layer.get_weights())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "loss, accuracy = model.evaluate(padded_reviews, labels, verbose=0)\n",
    "print(f'Accuracy: {accuracy}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}