sentiment_analysis.py.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercice 2 : Sentiment Analysis on IMDB Reviews using LSTM and Keras\n",
    "created by Hans Michael\n",
    "<hr>\n",
    "\n",
    "### Steps\n",
    "<ol type=\"1\">\n",
    "    <li>Load the dataset (50K IMDB Movie Review)</li>\n",
    "    <li>Clean Dataset</li>\n",
    "    <li>Encode Sentiments</li>\n",
    "    <li>Split Dataset</li>\n",
    "    <li>Tokenize and Pad/Truncate Reviews</li>\n",
    "    <li>Build Architecture/Model</li>\n",
    "    <li>Train and Test</li>\n",
    "</ol>\n",
    "\n",
    "<hr>\n",
    "<i>Import all the libraries needed</i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\thome\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd    # to load dataset\n",
    "import numpy as np     # for mathematic equation\n",
    "import nltk\n",
    "nltk.download('stopwords')  # to download stopwords\n",
    "from nltk.corpus import stopwords   # to get collection of stopwords\n",
    "from sklearn.model_selection import train_test_split       # for splitting dataset\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer  # to encode text to int\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences   # to do padding or truncating\n",
    "from tensorflow.keras.models import Sequential     # the model\n",
    "from tensorflow.keras.layers import Embedding, LSTM, Dense # layers of the architecture\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint   # save model\n",
    "from tensorflow.keras.models import load_model   # load saved model\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "<i>Preview dataset</i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                  review sentiment\n",
      "0      One of the other reviewers has mentioned that ...  positive\n",
      "1      A wonderful little production. <br /><br />The...  positive\n",
      "2      I thought this was a wonderful way to spend ti...  positive\n",
      "3      Basically there's a family where a little boy ...  negative\n",
      "4      Petter Mattei's \"Love in the Time of Money\" is...  positive\n",
      "...                                                  ...       ...\n",
      "49995  I thought this movie did a down right good job...  positive\n",
      "49996  Bad plot, bad dialogue, bad acting, idiotic di...  negative\n",
      "49997  I am a Catholic taught in parochial elementary...  negative\n",
      "49998  I'm going to have to disagree with the previou...  negative\n",
      "49999  No one expects the Star Trek movies to be high...  negative\n",
      "\n",
      "[50000 rows x 2 columns]\n"
     ]
    }
   ],
   "source": [
    "data = pd.read_csv('IMDB Dataset.csv')\n",
    "\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "<b>Stop Word</b> is a commonly used words in a sentence, usually a search engine is programmed to ignore this words (i.e. \"the\", \"a\", \"an\", \"of\", etc.)\n",
    "\n",
    "<i>Declaring the english stop words</i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "english_stops = set(stopwords.words('english'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Load and Clean Dataset\n",
    "\n",
    "In the original dataset, the reviews are still dirty. There are still html tags, numbers, uppercase, and punctuations. This will not be good for training, so in <b>load_dataset()</b> function, beside loading the dataset using <b>pandas</b>, I also pre-process the reviews by removing html tags, non alphabet (punctuations and numbers), stop words, and lower case all of the reviews.\n",
    "\n",
    "### Encode Sentiments\n",
    "In the same function, I also encode the sentiments into integers (0 and 1). Where 0 is for negative sentiments and 1 is for positive sentiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reviews\n",
      "0        [one, reviewers, mentioned, watching, oz, epis...\n",
      "1        [a, wonderful, little, production, the, filmin...\n",
      "2        [i, thought, wonderful, way, spend, time, hot,...\n",
      "3        [basically, family, little, boy, jake, thinks,...\n",
      "4        [petter, mattei, love, time, money, visually, ...\n",
      "                               ...                        \n",
      "49995    [i, thought, movie, right, good, job, it, crea...\n",
      "49996    [bad, plot, bad, dialogue, bad, acting, idioti...\n",
      "49997    [i, catholic, taught, parochial, elementary, s...\n",
      "49998    [i, going, disagree, previous, comment, side, ...\n",
      "49999    [no, one, expects, star, trek, movies, high, a...\n",
      "Name: review, Length: 50000, dtype: object \n",
      "\n",
      "Sentiment\n",
      "0        1\n",
      "1        1\n",
      "2        1\n",
      "3        0\n",
      "4        1\n",
      "        ..\n",
      "49995    1\n",
      "49996    0\n",
      "49997    0\n",
      "49998    0\n",
      "49999    0\n",
      "Name: sentiment, Length: 50000, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "def load_dataset():\n",
    "    df = pd.read_csv('IMDB Dataset.csv')\n",
    "    x_data = df['review']       # Reviews/Input\n",
    "    y_data = df['sentiment']    # Sentiment/Output\n",
    "\n",
    "    # PRE-PROCESS REVIEW\n",
    "    x_data = x_data.replace({'<.*?>': ''}, regex = True)          # remove html tag\n",
    "    x_data = x_data.replace({'[^A-Za-z]': ' '}, regex = True)     # remove non alphabet\n",
    "    x_data = x_data.apply(lambda review: [w for w in review.split() if w not in english_stops])  # remove stop words\n",
    "    x_data = x_data.apply(lambda review: [w.lower() for w in review])   # lower case\n",
    "    \n",
    "    # ENCODE SENTIMENT -> 0 & 1\n",
    "    y_data = y_data.replace('positive', 1)\n",
    "    y_data = y_data.replace('negative', 0)\n",
    "\n",
    "    return x_data, y_data\n",
    "\n",
    "x_data, y_data = load_dataset()\n",
    "\n",
    "print('Reviews')\n",
    "print(x_data, '\\n')\n",
    "print('Sentiment')\n",
    "print(y_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Split Dataset\n",
    "In this work, I decided to split the data into 80% of Training and 20% of Testing set using <b>train_test_split</b> method from Scikit-Learn. By using this method, it automatically shuffles the dataset. We need to shuffle the data because in the original dataset, the reviews and sentiments are in order, where they list positive reviews first and then negative reviews. By shuffling the data, it will be distributed equally in the model, so it will be more accurate for predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Set\n",
      "32564    [before, last, call, w, carson, daly, local, n...\n",
      "19142    [as, i, peruse, hundreds, comments, loyal, rea...\n",
      "23163    [course, i, wrong, now, i, never, expected, li...\n",
      "22749    [this, fabulous, film, the, plot, good, yarn, ...\n",
      "46651    [make, a, you, sacks, of, german, staples, wel...\n",
      "                               ...                        \n",
      "38762    [remember, two, stories, fondly, first, set, d...\n",
      "7478     [do, not, watch, this, sad, excuse, for, a, fi...\n",
      "34967    [matthew, aldrich, this, name, worthy, remembr...\n",
      "40832    [i, first, came, across, my, tutor, friend, ac...\n",
      "1168     [i, love, movie, the, cast, terrific, portraya...\n",
      "Name: review, Length: 40000, dtype: object \n",
      "\n",
      "22041    [this, could, well, worst, film, i, ever, seen...\n",
      "33422    [normally, would, say, loved, movie, acting, a...\n",
      "46360    [steven, vasquez, directed, co, wrote, james, ...\n",
      "3680     [you, sit, half, hour, watch, story, believing...\n",
      "49776    [first, things, first, i, means, picky, movie,...\n",
      "                               ...                        \n",
      "46790    [for, wonderful, rollercoaster, film, bears, r...\n",
      "41890    [i, seen, movies, released, the, informer, wor...\n",
      "28388    [human, traffic, without, doubt, original, com...\n",
      "27247    [i, think, french, could, make, bad, movie, i,...\n",
      "12247    [an, exquisite, film, they, make, like, we, ea...\n",
      "Name: review, Length: 10000, dtype: object \n",
      "\n",
      "Test Set\n",
      "32564    0\n",
      "19142    1\n",
      "23163    0\n",
      "22749    1\n",
      "46651    0\n",
      "        ..\n",
      "38762    1\n",
      "7478     0\n",
      "34967    0\n",
      "40832    1\n",
      "1168     1\n",
      "Name: sentiment, Length: 40000, dtype: int64 \n",
      "\n",
      "22041    0\n",
      "33422    0\n",
      "46360    0\n",
      "3680     1\n",
      "49776    0\n",
      "        ..\n",
      "46790    1\n",
      "41890    0\n",
      "28388    1\n",
      "27247    0\n",
      "12247    1\n",
      "Name: sentiment, Length: 10000, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "x_train, x_test, y_train, y_test = train_test_split(x_data, y_data, test_size = 0.2)\n",
    "\n",
    "print('Train Set')\n",
    "print(x_train, '\\n')\n",
    "print(x_test, '\\n')\n",
    "print('Test Set')\n",
    "print(y_train, '\\n')\n",
    "print(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "<i>Function for getting the maximum review length, by calculating the mean of all the reviews length (using <b>numpy.mean</b>)</i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_max_length():\n",
    "    review_length = []\n",
    "    for review in x_train:\n",
    "        review_length.append(len(review))\n",
    "\n",
    "    return int(np.ceil(np.mean(review_length)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Tokenize and Pad/Truncate Reviews\n",
    "A Neural Network only accepts numeric data, so we need to encode the reviews. I use <b>tensorflow.keras.preprocessing.text.Tokenizer</b> to encode the reviews into integers, where each unique word is automatically indexed (using <b>fit_on_texts</b> method) based on <b>x_train</b>. <br>\n",
    "<b>x_train</b> and <b>x_test</b> is converted into integers using <b>texts_to_sequences</b> method.\n",
    "\n",
    "Each reviews has a different length, so we need to add padding (by adding 0) or truncating the words to the same length (in this case, it is the mean of all reviews length) using <b>tensorflow.keras.preprocessing.sequence.pad_sequences</b>.\n",
    "\n",
    "\n",
    "<b>post</b>, pad or truncate the words in the back of a sentence<br>\n",
    "<b>pre</b>, pad or truncate the words in front of a sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoded X Train\n",
      " [[ 1606   142   557 ...     0     0     0]\n",
      " [  108     1 39956 ...   884     3   512]\n",
      " [  169     1   264 ...   581  9627   489]\n",
      " ...\n",
      " [ 3676 31442     8 ...     0     0     0]\n",
      " [    1    23   287 ... 13117 18170  1715]\n",
      " [    1    41     3 ...     0     0     0]] \n",
      "\n",
      "Encoded X Test\n",
      " [[    8    27    16 ... 79978   238 11685]\n",
      " [ 1800    12    58 ...     0     0     0]\n",
      " [ 2035 73395   427 ... 17006 10962  7311]\n",
      " ...\n",
      " [  303  5003   116 ...     0     0     0]\n",
      " [    1    31   611 ...   488  1912     7]\n",
      " [  700  5421     4 ...     0     0     0]] \n",
      "\n",
      "Maximum review length:  130\n"
     ]
    }
   ],
   "source": [
    "# ENCODE REVIEW\n",
    "token = Tokenizer(lower=False)    # no need lower, because already lowered the data in load_data()\n",
    "token.fit_on_texts(x_train)\n",
    "x_train = token.texts_to_sequences(x_train)\n",
    "x_test = token.texts_to_sequences(x_test)\n",
    "\n",
    "max_length = get_max_length()\n",
    "\n",
    "x_train = pad_sequences(x_train, maxlen=max_length, padding='post', truncating='post')\n",
    "x_test = pad_sequences(x_test, maxlen=max_length, padding='post', truncating='post')\n",
    "\n",
    "total_words = len(token.word_index) + 1   # add 1 because of 0 padding\n",
    "\n",
    "print('Encoded X Train\\n', x_train, '\\n')\n",
    "print('Encoded X Test\\n', x_test, '\\n')\n",
    "print('Maximum review length: ', max_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Build Architecture/Model\n",
    "<b>Embedding Layer</b>: in simple terms, it creates word vectors of each word in the <i>word_index</i> and group words that are related or have similar meaning by analyzing other words around them.\n",
    "\n",
    "<b>LSTM Layer</b>: to make a decision to keep or throw away data by considering the current input, previous output, and previous memory. There are some important components in LSTM.\n",
    "<ul>\n",
    "    <li><b>Forget Gate</b>, decides information is to be kept or thrown away</li>\n",
    "    <li><b>Input Gate</b>, updates cell state by passing previous output and current input into sigmoid activation function</li>\n",
    "    <li><b>Cell State</b>, calculate new cell state, it is multiplied by forget vector (drop value if multiplied by a near 0), add it with the output from input gate to update the cell state value.</li>\n",
    "    <li><b>Ouput Gate</b>, decides the next hidden state and used for predictions</li>\n",
    "</ul>\n",
    "\n",
    "<b>Dense Layer</b>: compute the input with the weight matrix and bias (optional), and using an activation function. I use <b>Sigmoid</b> activation function for this work because the output is only 0 or 1.\n",
    "\n",
    "The optimizer is <b>Adam</b> and the loss function is <b>Binary Crossentropy</b> because again the output is only 0 and 1, which is a binary number.\n",
    "\n",
    "### Edit Model\n",
    "\n",
    "I tried to reduce the overfitting of the training model. It was 0.97 accuracy for training and 0.86 accuracy for testing. I reduced the number of layer EMBED_DIM and LSTM_OUT from 64, 32 to 2, 2 so the accuracy for training is now 0.90 and for testing is 0.86. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_5\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_5 (Embedding)     (None, 130, 2)            184720    \n",
      "                                                                 \n",
      " lstm_5 (LSTM)               (None, 2)                 40        \n",
      "                                                                 \n",
      " dense_5 (Dense)             (None, 1)                 3         \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 184763 (721.73 KB)\n",
      "Trainable params: 184763 (721.73 KB)\n",
      "Non-trainable params: 0 (0.00 Byte)\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# ARCHITECTURE\n",
    "EMBED_DIM = 2\n",
    "LSTM_OUT = 2\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(total_words, EMBED_DIM, input_length = max_length))\n",
    "model.add(LSTM(LSTM_OUT))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])\n",
    "\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Training\n",
    "For training, it is simple. We only need to fit our <b>x_train</b> (input) and <b>y_train</b> (output/label) data. For this training, I use a mini-batch learning method with a <b>batch_size</b> of <i>128</i> and <i>5</i> <b>epochs</b>.\n",
    "\n",
    "Also, I added a callback called **checkpoint** to save the model locally for every epoch if its accuracy improved from the previous epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint = ModelCheckpoint(\n",
    "    'models/LSTM.h5',\n",
    "    monitor='accuracy',\n",
    "    save_best_only=True,\n",
    "    verbose=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - ETA: 0s - loss: 0.6929 - accuracy: 0.5061\n",
      "Epoch 1: accuracy improved from -inf to 0.50612, saving model to models\\LSTM.h5\n",
      "313/313 [==============================] - 25s 58ms/step - loss: 0.6929 - accuracy: 0.5061\n",
      "Epoch 2/5\n",
      "  2/313 [..............................] - ETA: 20s - loss: 0.6918 - accuracy: 0.5273"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Python311\\Lib\\site-packages\\keras\\src\\engine\\training.py:3079: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.\n",
      "  saving_api.save_model(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313/313 [==============================] - ETA: 0s - loss: 0.5815 - accuracy: 0.7037\n",
      "Epoch 2: accuracy improved from 0.50612 to 0.70372, saving model to models\\LSTM.h5\n",
      "313/313 [==============================] - 19s 61ms/step - loss: 0.5815 - accuracy: 0.7037\n",
      "Epoch 3/5\n",
      "313/313 [==============================] - ETA: 0s - loss: 0.4248 - accuracy: 0.8455\n",
      "Epoch 3: accuracy improved from 0.70372 to 0.84548, saving model to models\\LSTM.h5\n",
      "313/313 [==============================] - 20s 63ms/step - loss: 0.4248 - accuracy: 0.8455\n",
      "Epoch 4/5\n",
      "313/313 [==============================] - ETA: 0s - loss: 0.3508 - accuracy: 0.8835\n",
      "Epoch 4: accuracy improved from 0.84548 to 0.88355, saving model to models\\LSTM.h5\n",
      "313/313 [==============================] - 19s 61ms/step - loss: 0.3508 - accuracy: 0.8835\n",
      "Epoch 5/5\n",
      "313/313 [==============================] - ETA: 0s - loss: 0.2966 - accuracy: 0.9073\n",
      "Epoch 5: accuracy improved from 0.88355 to 0.90728, saving model to models\\LSTM.h5\n",
      "313/313 [==============================] - 49s 156ms/step - loss: 0.2966 - accuracy: 0.9073\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x2adc1f2a350>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train, batch_size = 128, epochs = 5, callbacks=[checkpoint])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "### Testing\n",
    "To evaluate the model, we need to predict the sentiment using our <b>x_test</b> data and comparing the predictions with <b>y_test</b> (expected output) data. Then, we calculate the accuracy of the model by dividing numbers of correct prediction with the total data. Resulted an accuracy of <b>86.63%</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "79/79 [==============================] - 6s 27ms/step\n",
      "Correct Prediction: 8588\n",
      "Wrong Prediction: 1412\n",
      "Accuracy: 85.88\n"
     ]
    }
   ],
   "source": [
    "y_pred = model.predict(x_test, batch_size = 128)\n",
    "y_pred = (y_pred > 0.5).astype('int') \n",
    "\n",
    "true = 0\n",
    "for i, y in enumerate(y_test):\n",
    "    if y == y_pred[i]:\n",
    "        true += 1\n",
    "\n",
    "print('Correct Prediction: {}'.format(true))\n",
    "print('Wrong Prediction: {}'.format(len(y_pred) - true))\n",
    "print('Accuracy: {}'.format(true/len(y_pred)*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### Load Saved Model\n",
    "\n",
    "Load saved model and use it to predict a movie review statement's sentiment (positive or negative)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "loaded_model = load_model('models/LSTM.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Receives a review as an input to be predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "review = str(input('Movie Review: '))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input must be pre processed before it is passed to the model to be predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cleaned:  In summary writing a movie review involves a careful balance of plot summary thesis development opinion sharing and thorough analysis of various aspects of the film including characters cinematography directorial choices music and sound and technical elements\n",
      "Filtered:  ['in summary writing movie review involves careful balance plot summary thesis development opinion sharing thorough analysis various aspects film including characters cinematography directorial choices music sound technical elements']\n"
     ]
    }
   ],
   "source": [
    "# Pre-process input\n",
    "regex = re.compile(r'[^a-zA-Z\\s]')\n",
    "review = regex.sub('', review)\n",
    "print('Cleaned: ', review)\n",
    "\n",
    "words = review.split(' ')\n",
    "filtered = [w for w in words if w not in english_stops]\n",
    "filtered = ' '.join(filtered)\n",
    "filtered = [filtered.lower()]\n",
    "\n",
    "print('Filtered: ', filtered)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, we need to tokenize and encode the words. I use the tokenizer which was previously declared because we want to encode the words based on words that are known by the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[   50  2515   397     3   610  2165  4193  2859    40  2515 13366   856\n",
      "    560  5725 12288  4959   887  1320     4   485    30   523  3721  2567\n",
      "    113   355  1773   715     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0     0     0     0     0     0     0\n",
      "      0     0     0     0     0     0     0     0     0     0]]\n"
     ]
    }
   ],
   "source": [
    "tokenize_words = token.texts_to_sequences(filtered)\n",
    "tokenize_words = pad_sequences(tokenize_words, maxlen=max_length, padding='post', truncating='post')\n",
    "print(tokenize_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the result of the prediction which shows the **confidence score** of the review statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 1s 1s/step\n",
      "[[0.8879976]]\n"
     ]
    }
   ],
   "source": [
    "result = loaded_model.predict(tokenize_words)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the confidence score is close to 0, then the statement is **negative**. On the other hand, if the confidence score is close to 1, then the statement is **positive**. I use a threshold of **0.7** to determine which confidence score is positive and negative, so if it is equal or greater than 0.7, it is **positive** and if it is less than 0.7, it is **negative**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "positive\n"
     ]
    }
   ],
   "source": [
    "if result >= 0.7:\n",
    "    print('positive')\n",
    "else:\n",
    "    print('negative')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}