TP3_4_Project/Exercices/Exercice1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This notebook is inspired by the [Keras Example](https://github.com/keras-team/keras-io/blob/master/examples/nlp/text_classification_with_transformer.py)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1: Introduction\n",
    "\n",
    "#### Exploring Transformers with Keras\n",
    "\n",
    "Welcome to this Jupyter Notebook, where we will embark on an exciting journey to explore and understand the Transformer architecture using Keras. This notebook is tailored for the Artificial Intelligence course at ESIR – Université Rennes, focusing on practical aspects of implementing and utilizing Transformers in natural language processing (NLP).\n",
    "\n",
    "#### Background\n",
    "\n",
    "Transformers, since their introduction in the paper “Attention Is All You Need” by Vaswani et al., have revolutionized the field of NLP. Known for their efficiency in handling sequential data and their ability to capture long-range dependencies, Transformers have set new benchmarks in a wide range of NLP tasks.\n",
    "\n",
    "In this notebook, we will:\n",
    "\n",
    "- Implement a Transformer block as a Keras layer.\n",
    "- Use this Transformer for text classification.\n",
    "- Compare its performance with a traditional LSTM-based approach.\n",
    "- Experiment with different hyperparameters to understand their impact.\n",
    "\n",
    "#### Dataset\n",
    "\n",
    "We will be working with the Ohsumed dataset, a collection of medical abstracts, which presents a challenging yet insightful task for text classification. This dataset differs from the usual IMDB dataset typically used in transformer examples, providing a more domain-specific challenge.\n",
    "\n",
    "**Update**: Instead we will be using the IMDB dataset, as it requires less preprocessing and is easier to work with.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "### Part 2: Setup and Imports\n",
    "\n",
    "First, let's set up our environment by importing the necessary libraries. We'll need TensorFlow and its high-level API, Keras, for building and training our models. The layers submodule from Keras provides us with the necessary layers to construct our Transformer model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From c:\\Users\\codya\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\keras\\src\\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Importing necessary libraries\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "TensorFlow will serve as our backend for matrix operations and neural network functionalities, while Keras will offer a user-friendly interface to build and train our models.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Part 3: Transformer Block Implementation\n",
    "\n",
    "#### Implementing the Transformer Block\n",
    "\n",
    "The core of the Transformer model is its unique architecture, which we will implement as a custom layer in Keras. The `TransformerBlock` class encapsulates the key components of the Transformer's architecture.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransformerBlock(layers.Layer):\n",
    "    def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1):\n",
    "        super().__init__()\n",
    "        self.att = layers.MultiHeadAttention(num_heads=num_heads, key_dim=embed_dim)\n",
    "        self.ffn = keras.Sequential(\n",
    "            [\n",
    "                layers.Dense(ff_dim, activation=\"relu\"),\n",
    "                layers.Dense(embed_dim),\n",
    "            ]\n",
    "        )\n",
    "        self.layernorm1 = layers.LayerNormalization(epsilon=1e-6)\n",
    "        self.layernorm2 = layers.LayerNormalization(epsilon=1e-6)\n",
    "        self.dropout1 = layers.Dropout(rate)\n",
    "        self.dropout2 = layers.Dropout(rate)\n",
    "\n",
    "    def call(self, inputs, training):\n",
    "        attn_output = self.att(inputs, inputs)\n",
    "        attn_output = self.dropout1(attn_output, training=training)\n",
    "        out1 = self.layernorm1(inputs + attn_output)\n",
    "        ffn_output = self.ffn(out1)\n",
    "        ffn_output = self.dropout2(ffn_output, training=training)\n",
    "        return self.layernorm2(out1 + ffn_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Explanation\n",
    "\n",
    "This class is a representation of a single Transformer block, which consists of:\n",
    "\n",
    "1. **Multi-Head Attention Mechanism**: This allows the model to jointly attend to information from different representation subspaces at different positions.\n",
    "\n",
    "2. **Feed Forward Network**: A simple fully connected neural network applied to each position separately and identically.\n",
    "\n",
    "3. **Layer Normalization and Dropout**: Used for regularization and to stabilize the training process.\n",
    "\n",
    "4. **Residual Connections**: Encourage gradient flow through the network.\n",
    "\n",
    "This block can process a batch of sequences in parallel, making it highly efficient for NLP tasks.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 4: Data Preparation\n",
    "\n",
    "#### Loading the Dataset\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size = 20000  # Only consider the top 20k words\n",
    "maxlen = 200  # Only consider the first 200 words of each movie review\n",
    "\n",
    "(x_train, y_train), (x_val, y_val) = keras.datasets.imdb.load_data(num_words=vocab_size)\n",
    "x_train = keras.utils.pad_sequences(x_train, maxlen=maxlen)\n",
    "x_val = keras.utils.pad_sequences(x_val, maxlen=maxlen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "### Part 5: Model Building\n",
    "\n",
    "#### Creating the Classifier Model\n",
    "\n",
    "Now, we will use the previously defined Transformer block to build our classifier model. The model architecture is designed to process the input text, apply the Transformer block, and then classify the text into appropriate categories.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TokenAndPositionEmbedding(layers.Layer):\n",
    "    def __init__(self, maxlen, vocab_size, embed_dim):\n",
    "        super().__init__()\n",
    "        self.token_emb = layers.Embedding(input_dim=vocab_size, output_dim=embed_dim)\n",
    "        self.pos_emb = layers.Embedding(input_dim=maxlen, output_dim=embed_dim)\n",
    "\n",
    "    def call(self, x):\n",
    "        maxlen = tf.shape(x)[-1]\n",
    "        positions = tf.range(start=0, limit=maxlen, delta=1)\n",
    "        positions = self.pos_emb(positions)\n",
    "        x = self.token_emb(x)\n",
    "        return x + positions\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "embed_dim = 32  # Embedding size for each token\n",
    "num_heads = 2  # Number of attention heads\n",
    "ff_dim = 32  # Hidden layer size in feed forward network inside transformer\n",
    "\n",
    "inputs = layers.Input(shape=(maxlen,))\n",
    "embedding_layer = TokenAndPositionEmbedding(maxlen, vocab_size, embed_dim)\n",
    "x = embedding_layer(inputs)\n",
    "transformer_block = TransformerBlock(embed_dim, num_heads, ff_dim)\n",
    "x = transformer_block(x)\n",
    "x = layers.GlobalAveragePooling1D()(x)\n",
    "x = layers.Dropout(0.1)(x)\n",
    "x = layers.Dense(20, activation=\"relu\")(x)\n",
    "x = layers.Dropout(0.1)(x)\n",
    "outputs = layers.Dense(2, activation=\"softmax\")(x)\n",
    "\n",
    "model = keras.Model(inputs=inputs, outputs=outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "#### Explanation\n",
    "\n",
    "In this architecture:\n",
    "\n",
    "- The `TokenAndPositionEmbedding` layer first processes the input sequences.\n",
    "- The `TransformerBlock` applies the attention mechanism and feedforward network.\n",
    "- The `GlobalAveragePooling1D` layer condenses the output from the transformer block.\n",
    "- The `Dense` layers at the end serve as a classifier on top of the transformer outputs.\n",
    "\n",
    "### Part 6: Model Compilation\n",
    "\n",
    "#### Compiling the Model\n",
    "\n",
    "After building the model, it needs to be compiled with the appropriate loss function and optimizer. Since this is a classification task, we'll use the 'sparse_categorical_crossentropy' loss.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=\"adam\", loss=\"sparse_categorical_crossentropy\", metrics=[\"accuracy\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "391/391 [==============================] - 22s 56ms/step - loss: 0.1071 - accuracy: 0.9640 - val_loss: 0.4235 - val_accuracy: 0.8605\n",
      "Epoch 2/5\n",
      "391/391 [==============================] - 22s 57ms/step - loss: 0.0716 - accuracy: 0.9784 - val_loss: 0.4915 - val_accuracy: 0.8576\n",
      "Epoch 3/5\n",
      "391/391 [==============================] - 23s 59ms/step - loss: 0.0479 - accuracy: 0.9866 - val_loss: 0.6163 - val_accuracy: 0.8458\n",
      "Epoch 4/5\n",
      "391/391 [==============================] - 23s 59ms/step - loss: 0.0401 - accuracy: 0.9882 - val_loss: 0.6446 - val_accuracy: 0.8424\n",
      "Epoch 5/5\n",
      "391/391 [==============================] - 23s 60ms/step - loss: 0.0316 - accuracy: 0.9916 - val_loss: 0.7246 - val_accuracy: 0.8472\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train, y_train, batch_size=64, epochs=5, validation_data=(x_val, y_val))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ploting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAG2CAYAAACDLKdOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA99UlEQVR4nO3deVyU5f7/8fcMywyg4IKCC+5mWu4LYZ6TqcXJ8pvWKTVPkmVlaWmcTmmZtpyk5Wh60rI6LmWllqX5O5ZllHkySxMpy6VSSy1AzWRTWWbu3x/AyLDJIDDM7ev5eNwP7rnnuuf+XNzSvLvuzWIYhiEAAACTsHq7AAAAgOpEuAEAAKZCuAEAAKZCuAEAAKZCuAEAAKZCuAEAAKZCuAEAAKZCuAEAAKZCuAEAAKZCuAEAAKbi1XCzadMmDRs2TM2bN5fFYtGaNWvOus7GjRvVq1cv2Ww2dejQQUuXLq3xOgEAgO/warjJzs5W9+7dtWDBgkq1P3DggK6++mpdfvnlSk5O1pQpUzR+/Hh9+OGHNVwpAADwFZa68uBMi8Wi1atXa/jw4eW2efDBB7Vu3Tp99913rmWjRo3SiRMntH79+lqoEgAA1HX+3i7AE1u2bNGQIUPclsXGxmrKlCnlrpOTk6OcnBzXa6fTqePHj6tx48ayWCw1VSoAAKhGhmEoMzNTzZs3l9Va8YEnnwo3qampioiIcFsWERGhjIwMnTp1SkFBQaXWSUhI0GOPPVZbJQIAgBp06NAhtWzZssI2PhVuqmLatGmKj493vU5PT1erVq106NAhhYaGerEyAABQWRkZGYqKilL9+vXP2tanwk1kZKTS0tLclqWlpSk0NLTMURtJstlsstlspZaHhoYSbgAA8DGVOaXEp+5zExMTo8TERLdlGzZsUExMjJcqAgAAdY1Xw01WVpaSk5OVnJwsqeBS7+TkZB08eFBSwSGlsWPHutpPmDBB+/fv1wMPPKA9e/bohRde0FtvvaX77rvPG+UDAIA6yKuHpb7++mtdfvnlrtdF58bExcVp6dKlSklJcQUdSWrbtq3WrVun++67T/PmzVPLli31n//8R7GxsbVeOwAAdZVhGHI4DeU5DOU6nMp3OJXnMJTncBZO7vP5Dqdyy5nPcziVW2y+vM/JL9xWnsOpdk3q6cG/XOi1/teZ+9zUloyMDIWFhSk9PZ1zbgAAZ2UYRoXBoOR8fuHP3HLmy1+vIEQUzZcMJuWFlPLW86aerRpo9d2XVutnevL97VMnFAMA6qaikQKHYcgwdGbeKTkK3zMMo9h8sTaGIYez4LXTMJTvPDN6cLZAUNHoQb4rEJzbevlOc4wB+FstCvCzKsCv6KdVAf4WBVgL5v0LlwcWmy/Vvoz5stZrWr/0hTy12levbh0AqpFR+MVY9CVZ8FNyFn6JOg1DzsIvW2fJNsaZ9VxtCtuV2abYF/JZv7Rd88XaFP/CL9mmeEBw+3ydqdvVh5L9UbH6yumzoRL9L+f35NaPsmo/M3++CSz6cve3yt9qVaBrvvCLvth8+aGgcL0y5s8WKIrm/QtDRYC/pbCO8kKK5by6cS3hBoDXOZyGsnLyC6bT+crKyVNWjsM1n3k6X9k5jsLl+co8XdA2u9h80es8x/n3Resr/KwWWS2S1WIpnC947Zq3WuRX+J5bOPC3KqCMeX8/S2HIKPaFXl7AqGC9wML3y1rPNe9vLRzhKKjvfAoKvohwA6BKDMPQqTxHsUBS7GfhVDyEZJ3OV2bhz+xc99en8hy1Vrdf4Reo1Vr4JVv4per2JVv05VtGm6L3ir7g/ArXs1jOfDFbLGe2Y7FY5Gct47OLf7EXbd+1nbK//F3hoKgPrkCgEuuVCBCu0KCz1FlxG9c2iv/+yuxL2b9joLYQboDzTJ7DWSqEFAWNUiGkKKTk5CvrdF5hUHEos3C+uo9GBPhZVN8eoHo2f4XY/FXf5q96dv8zrwvnXZO9xE+bv4ID/dxGACyFoYH/2wbOH4QbwAc4nYZO5jkqOGRTPISUf8gm83S+cvKr9yoKi0WqF1h20Cj1uoz36tsCFGLzUz27v2z+ftVaG4DzE+EGqEE5+Q5X2HA7RFPeeSOFh2wySxzmyc7NV3XftMEeYC0jdASoXmHQqGcLUH27v0IC/VSvcDSlvr1gBKVesVGUoAA/DjkAqFMIN0AJRSe3lhVCih+ycY2gFDtkk1V00mvhOtV9cquf1VLuYZn6ttLBI6TYe/Xs/goJPBNQAvx86ukrAFBphBucF/IcTv2elaujmTk6mnVaRzJyCucLfxbOH8vMUXZu9Z/cGhzoV2YIcQsebueYFByqqW8LcDuUYw+wct4IAJwF4QY+yzAMpZ/KcwsnRzJKBJbC5cezcz3+/KKTW0NsfgWHaKpwcmvRaIkfh20AoNYQblDnnM5z6Ghmjo5klhxdOV0qtHhy2MffalF4PZua1C+cis8XTuH1bAq1+3NyKwD4MMINaoXDaej37NIjKsVDzLHCn5k5+R59dlhQgJrUt6lpBaGlST2bGgYHcuIrAJwHCDeoMsMwlJmT7x5YSpzHUhRcjmfneHRPFJu/1RVMzoQWexkjLYGMsAAA3BBuUEpOvkPHik6+dQstp0sFmNN5lb9nisUiNQ5xH01pGlr2SEt9mz8nzgIAqoRwc55wOg2dOJWnIyXPWynjiqETJ/M8+uz6Nv+CUZSzhJZGwYHy5/JjAEANI9z4uJO5+aUOAZUVWo5l5Sjfg+NCAX6W0iMqrtd21+Gi8Ho2BQVyWAgAUHcQbuqgfIdTv2fnFgstpQ8HFU2e3pOlUUhgBaHlzPktYUEBHBYCAPgkwk0tMQxDGafyC24gV3KEpURoOX4y16Nb7QcF+JU+DFTOZc7clRYAYHaEm2pyPDtX234+XuYN5Youcc51VP7kWz+rReH1AksHlXpnDgsVTfVs7EYAAIrwrVhNfkjL1J3Ltp+1Xajdv/Dwj73cQ0NN6hfck4W72gIA4DnCTTVpHhakHlEN3O/LUiK0hNezyR7AybcAANQkwk01adU4WGsmXurtMgAAOO9xdikAADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AADAVr4ebBQsWqE2bNrLb7YqOjtbWrVvLbZuXl6fHH39c7du3l91uV/fu3bV+/fparBYAANR1Xg03K1euVHx8vGbOnKmkpCR1795dsbGxOnLkSJntp0+frpdeeknPP/+8du3apQkTJmjEiBHasWNHLVcOAADqKothGIa3Nh4dHa2+fftq/vz5kiSn06moqCjdc889mjp1aqn2zZs318MPP6yJEye6ll1//fUKCgrS66+/XqltZmRkKCwsTOnp6QoNDa2ejgAAgBrlyfe310ZucnNztX37dg0ZMuRMMVarhgwZoi1btpS5Tk5Ojux2u9uyoKAgff755+VuJycnRxkZGW4TAAAwL6+Fm2PHjsnhcCgiIsJteUREhFJTU8tcJzY2VnPmzNGPP/4op9OpDRs26N1331VKSkq520lISFBYWJhrioqKqtZ+AACAusXrJxR7Yt68eerYsaMuvPBCBQYGatKkSRo3bpys1vK7MW3aNKWnp7umQ4cO1WLFAACgtnkt3ISHh8vPz09paWluy9PS0hQZGVnmOk2aNNGaNWuUnZ2tX375RXv27FG9evXUrl27crdjs9kUGhrqNgEAAPPyWrgJDAxU7969lZiY6FrmdDqVmJiomJiYCte12+1q0aKF8vPz9c477+jaa6+t6XIBAICP8PfmxuPj4xUXF6c+ffqoX79+mjt3rrKzszVu3DhJ0tixY9WiRQslJCRIkr766iv9+uuv6tGjh3799Vc9+uijcjqdeuCBB7zZDQAAUId4NdyMHDlSR48e1YwZM5SamqoePXpo/fr1rpOMDx486HY+zenTpzV9+nTt379f9erV09ChQ7Vs2TI1aNDASz0AAAB1jVfvc+MN3OcGAADf4xP3uQEAAKgJhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAqhBsAAGAq/t4uwDTSvpdW3Sb5+UvWAMkvoPBn8df+5SyvqN1Z1rP6VfEz/bz9GwMAoEYQbqpLTpZ0dLe3q/CApQqhyL/g9VlDlwfhrFLr+Z/lMwMki8Xbv1AAQB1BuKkuTTpJY9+THPmSM09y5BX8dDrOzDvyJGd+sdf55SyvpnbF3yvFkBy5BVNZb/saS3kjWB4GK79Ayd5ACm4sBTcqnBqfmYIaSf6B3u4tAKAChJvqEtRAajfQ21WUzTAKQpbHAamqwapkqKvsZ1ayneEso48OKd8h6XTN/z4D6xcLPcXDT6OC8FM8DBUtIxABQK0h3JwPLJaCUQo/fykgyNvVnDuns+IQdC5hLT9HOn1COvm7dPK4+89TxwuCVW5mwXTil8rXbAstJ/w0LD0yVBSK/AJq7FcIAGZGuIHvsVolq03yt9Xudp3OguBz6o/C0PN7iRBUOH+q2OtTfxQEopyMgumPnyu/PVtYOeGnxGiR672GBCIAEOEGqDyr9cx5OI3bV26dokBUPAAVDz8nf5dOlghLp/6QZEg56QWTx4GorPBTwSEzP/4zAMBc+K8aUJOKByJ1qNw6Tod0Or2ckaGicFRytKhkIDpQ+RrtYSUOiVVwMrVrhIj/dACou/gvFFDXWP2KBaKOlVvH6ZBOnShjZKiM84ZcIemEJKMgSJ1Ol7S/8jXaG5QRfhqWHhkq/h73VgJQSwg3gBlY/aSQxgVTZRUPRGUeMis5QvR7wSE2qeDn6RPS8coGIsuZEaLyRodKHjYLakAgAnxR0RW6XhzhJdwA56uqBCJHfrGryco6ZFbGydanT6hghOhEYSDaV8mNWQoCTllXkpW6DL/wfQIRUMDpkPJPF1wBmn/afT6vxOtSP0u+X6JN3qkK1i382SpGuvUDr3WfcAOg8vz8pZDwgqmyHPlnQk+pk6lLjA4VvX86XZJRsN6pP6Tff6rkxiwFl937BxbckNHqX/DTL7DwJo0BZ+atxeZLvm8NKHt5qc8s8fnWstYpZ1tWf+6sbWZG4Y1Sy/riLxUuSrY5S3g4a7g4VXBrC29y5Hh184QbADXLz1+q16RgqixHXmEgqsTJ1EU/cwoDUU665N3/rlZeqSBVIjBVOpyVCE9VCWeV2ZbVz7cCWXmjF5UJB2WOZng48lFXWP0lf3vB7TP8gwp/2kv/DLCXvdz1s+T7JdoEFPtsL99TjXADoO7xC5DqNS2YKqsoEJ1OP/NoEUf+mXlnsXlH4Y0bi+adxeZLvu8s3raqn1nOY1B87hEoliqOcnkYwAznuY1aFL329uhFcRWGhHICQqUCRnkBpXCZn+28vLrx/OsxAHOqSiCqTYZRTmAqEZ7cAlOJ8FQyMBUPZOV9psfhrNh6pcKBUXC4wcuHHKqksqMXpQJGWT/PEkBKjoD4BfrWiJcJEG4AoDZYLAXnAvnSc8ZcjzqpamCqYjiTpfQIRIUBo7yAcn6PXpzP2NsAgLJ561EnwDmyersAAACA6kS4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApuL1cLNgwQK1adNGdrtd0dHR2rp1a4Xt586dq06dOikoKEhRUVG67777dPr06VqqFgAA1HVeDTcrV65UfHy8Zs6cqaSkJHXv3l2xsbE6cuRIme3ffPNNTZ06VTNnztTu3bu1aNEirVy5Ug899FAtVw4AAOoqr4abOXPm6Pbbb9e4cePUpUsXLVy4UMHBwVq8eHGZ7b/44gtdeumluummm9SmTRtdeeWVGj169FlHewAAwPnDa+EmNzdX27dv15AhQ84UY7VqyJAh2rJlS5nr9O/fX9u3b3eFmf379+v999/X0KFDy91OTk6OMjIy3CYAAGBe/t7a8LFjx+RwOBQREeG2PCIiQnv27ClznZtuuknHjh3TgAEDZBiG8vPzNWHChAoPSyUkJOixxx6r1toBAEDd5fUTij2xceNGzZo1Sy+88IKSkpL07rvvat26dXriiSfKXWfatGlKT093TYcOHarFigEAQG3z2shNeHi4/Pz8lJaW5rY8LS1NkZGRZa7zyCOP6Oabb9b48eMlSV27dlV2drbuuOMOPfzww7JaS2c1m80mm81W/R0AAAB1ktdGbgIDA9W7d28lJia6ljmdTiUmJiomJqbMdU6ePFkqwPj5+UmSDMOouWIBAIDP8NrIjSTFx8crLi5Offr0Ub9+/TR37lxlZ2dr3LhxkqSxY8eqRYsWSkhIkCQNGzZMc+bMUc+ePRUdHa2ffvpJjzzyiIYNG+YKOQAA4Pzm1XAzcuRIHT16VDNmzFBqaqp69Oih9evXu04yPnjwoNtIzfTp02WxWDR9+nT9+uuvatKkiYYNG6Ynn3zSW10AAAB1jMU4z47nZGRkKCwsTOnp6QoNDfV2OQAAoBI8+f72qaulAAAAzsbjcNOmTRs9/vjjOnjwYE3UAwAAcE48DjdTpkzRu+++q3bt2umKK67QihUrlJOTUxO1AQAAeKxK4SY5OVlbt25V586ddc8996hZs2aaNGmSkpKSaqJGAACASjvnE4rz8vL0wgsv6MEHH1ReXp66du2qe++9V+PGjZPFYqmuOqsNJxQDAOB7PPn+rvKl4Hl5eVq9erWWLFmiDRs26JJLLtFtt92mw4cP66GHHtLHH3+sN998s6ofDwAAUCUeh5ukpCQtWbJEy5cvl9Vq1dixY/Xcc8/pwgsvdLUZMWKE+vbtW62FAgAAVIbH4aZv37664oor9OKLL2r48OEKCAgo1aZt27YaNWpUtRQIAADgCY/Dzf79+9W6desK24SEhGjJkiVVLgoAAKCqPL5a6siRI/rqq69KLf/qq6/09ddfV0tRAAAAVeVxuJk4caIOHTpUavmvv/6qiRMnVktRAAAAVeVxuNm1a5d69epVannPnj21a9euaikKAACgqjwONzabTWlpaaWWp6SkyN/fqw8ZBwAA8DzcXHnllZo2bZrS09Ndy06cOKGHHnpIV1xxRbUWBwAA4CmPh1r+9a9/6c9//rNat26tnj17SpKSk5MVERGhZcuWVXuBAAAAnvA43LRo0ULffvut3njjDX3zzTcKCgrSuHHjNHr06DLveQMAAFCbqnSSTEhIiO64447qrgUAAOCcVfkM4F27dungwYPKzc11W/5///d/51wUAABAVVXpDsUjRozQzp07ZbFYVPRQ8aIngDscjuqtEAAAwAMeXy01efJktW3bVkeOHFFwcLC+//57bdq0SX369NHGjRtroEQAAIDK83jkZsuWLfrkk08UHh4uq9Uqq9WqAQMGKCEhQffee6927NhRE3UCAABUiscjNw6HQ/Xr15ckhYeH67fffpMktW7dWnv37q3e6gAAADzk8cjNxRdfrG+++UZt27ZVdHS0nnnmGQUGBurll19Wu3btaqJGAACASvM43EyfPl3Z2dmSpMcff1zXXHON/vSnP6lx48ZauXJltRcIAADgCYtRdLnTOTh+/LgaNmzoumKqLsvIyFBYWJjS09MVGhrq7XIAAEAlePL97dE5N3l5efL399d3333ntrxRo0Y+EWwAAID5eRRuAgIC1KpVK+5lAwAA6iyPr5Z6+OGH9dBDD+n48eM1UQ8AAMA58fiE4vnz5+unn35S8+bN1bp1a4WEhLi9n5SUVG3FAQAAeMrjcDN8+PAaKAMAAKB6VMvVUr6Eq6UAAPA9NXa1FAAAQF3n8WEpq9Va4WXfXEkFAAC8yeNws3r1arfXeXl52rFjh1599VU99thj1VYYAABAVVTbOTdvvvmmVq5cqffee686Pq7GcM4NAAC+xyvn3FxyySVKTEysro8DAACokmoJN6dOndK///1vtWjRojo+DgAAoMo8Puem5AMyDcNQZmamgoOD9frrr1drcQAAAJ7yONw899xzbuHGarWqSZMmio6OVsOGDau1OAAAAE95HG5uueWWGigDAACgenh8zs2SJUv09ttvl1r+9ttv69VXX62WogAAAKrK43CTkJCg8PDwUsubNm2qWbNmVUtRAAAAVeVxuDl48KDatm1bannr1q118ODBaikKAACgqjwON02bNtW3335bavk333yjxo0bV0tRAAAAVeVxuBk9erTuvfdeffrpp3I4HHI4HPrkk080efJkjRo1qiZqBAAAqDSPr5Z64okn9PPPP2vw4MHy9y9Y3el0auzYsZxzAwAAvK7Kz5b68ccflZycrKCgIHXt2lWtW7eu7tpqBM+WAgDA93jy/e3xyE2Rjh07qmPHjlVdHQAAoEZ4fM7N9ddfr6effrrU8meeeUY33HBDtRQFAABQVR6Hm02bNmno0KGlll911VXatGlTtRQFAABQVR6Hm6ysLAUGBpZaHhAQoIyMjGopCgAAoKo8Djddu3bVypUrSy1fsWKFunTpUi1FAQAAVJXHJxQ/8sgjuu6667Rv3z4NGjRIkpSYmKg333xTq1atqvYCAQAAPOFxuBk2bJjWrFmjWbNmadWqVQoKClL37t31ySefqFGjRjVRIwAAQKVV+T43RTIyMrR8+XItWrRI27dvl8PhqK7aagT3uQEAwPd48v3t8Tk3RTZt2qS4uDg1b95cs2fP1qBBg/Tll19W9eMAAACqhUeHpVJTU7V06VItWrRIGRkZuvHGG5WTk6M1a9ZwMjEAAKgTKj1yM2zYMHXq1Enffvut5s6dq99++03PP/98TdYGAADgsUqP3HzwwQe69957ddddd/HYBQAAUGdVeuTm888/V2Zmpnr37q3o6GjNnz9fx44dq8naAAAAPFbpcHPJJZfolVdeUUpKiu68806tWLFCzZs3l9Pp1IYNG5SZmVmTdQIAAFTKOV0KvnfvXi1atEjLli3TiRMndMUVV2jt2rXVWV+141JwAAB8T61cCi5JnTp10jPPPKPDhw9r+fLl5/JRAAAA1eKcwk0RPz8/DR8+vMqjNgsWLFCbNm1kt9sVHR2trVu3ltt24MCBslgspaarr766quUDAAATqZZwcy5Wrlyp+Ph4zZw5U0lJSerevbtiY2N15MiRMtu/++67SklJcU3fffed/Pz8dMMNN9Ry5QAAoC7yeriZM2eObr/9do0bN05dunTRwoULFRwcrMWLF5fZvlGjRoqMjHRNGzZsUHBwMOEGAABI8nK4yc3N1fbt2zVkyBDXMqvVqiFDhmjLli2V+oxFixZp1KhRCgkJKfP9nJwcZWRkuE0AAMC8vBpujh07JofDoYiICLflERERSk1NPev6W7du1Xfffafx48eX2yYhIUFhYWGuKSoq6pzrBgAAdZfXD0udi0WLFqlr167q169fuW2mTZum9PR013To0KFarBAAANQ2jx6cWd3Cw8Pl5+entLQ0t+VpaWmKjIyscN3s7GytWLFCjz/+eIXtbDabbDbbOdcKAAB8g1dHbgIDA9W7d28lJia6ljmdTiUmJiomJqbCdd9++23l5OTob3/7W02XCQAAfIhXR24kKT4+XnFxcerTp4/69eunuXPnKjs7W+PGjZMkjR07Vi1atFBCQoLbeosWLdLw4cPVuHFjb5QNAADqKK+Hm5EjR+ro0aOaMWOGUlNT1aNHD61fv951kvHBgwdltboPMO3du1eff/65PvroI2+UDAAA6rBzeraUL+LZUgAA+J5ae7YUAABAXUO4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApkK4AQAApuL1cLNgwQK1adNGdrtd0dHR2rp1a4XtT5w4oYkTJ6pZs2ay2Wy64IIL9P7779dStQAAoK7z9+bGV65cqfj4eC1cuFDR0dGaO3euYmNjtXfvXjVt2rRU+9zcXF1xxRVq2rSpVq1apRYtWuiXX35RgwYNar94AABQJ1kMwzC8tfHo6Gj17dtX8+fPlyQ5nU5FRUXpnnvu0dSpU0u1X7hwoZ599lnt2bNHAQEBVdpmRkaGwsLClJ6ertDQ0HOqHwAA1A5Pvr+9dlgqNzdX27dv15AhQ84UY7VqyJAh2rJlS5nrrF27VjExMZo4caIiIiJ08cUXa9asWXI4HOVuJycnRxkZGW4TAAAwL6+Fm2PHjsnhcCgiIsJteUREhFJTU8tcZ//+/Vq1apUcDofef/99PfLII5o9e7b++c9/lrudhIQEhYWFuaaoqKhq7QcAAKhbvH5CsSecTqeaNm2ql19+Wb1799bIkSP18MMPa+HCheWuM23aNKWnp7umQ4cO1WLFAACgtnnthOLw8HD5+fkpLS3NbXlaWpoiIyPLXKdZs2YKCAiQn5+fa1nnzp2Vmpqq3NxcBQYGllrHZrPJZrNVb/EAAKDO8trITWBgoHr37q3ExETXMqfTqcTERMXExJS5zqWXXqqffvpJTqfTteyHH35Qs2bNygw2AADg/OPVw1Lx8fF65ZVX9Oqrr2r37t266667lJ2drXHjxkmSxo4dq2nTprna33XXXTp+/LgmT56sH374QevWrdOsWbM0ceJEb3UBAADUMV69z83IkSN19OhRzZgxQ6mpqerRo4fWr1/vOsn44MGDslrP5K+oqCh9+OGHuu+++9StWze1aNFCkydP1oMPPuitLgAAgDrGq/e58QbucwMAgO/xifvcAAAA1ATCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBXCDQAAMBV/bxcAADA/h8OhvLw8b5eBOi4gIEB+fn7n/DmEGwBAjcrKytLhw4dlGIa3S0EdZ7FY1LJlS9WrV++cPodwAwCoMQ6HQ4cPH1ZwcLCaNGkii8Xi7ZJQRxmGoaNHj+rw4cPq2LHjOY3gEG4AADUmLy9PhmGoSZMmCgoK8nY5qOOaNGmin3/+WXl5eecUbjihGABQ4xixQWVU178Twg0AADAVwg0AADAVwg0AADAVwg0AADAVwg0AAD6AmyBWHuEGAFBrDMPQydx8r0ye3kRw/fr1GjBggBo0aKDGjRvrmmuu0b59+1zvHz58WKNHj1ajRo0UEhKiPn366KuvvnK9///+3/9T3759ZbfbFR4erhEjRrjes1gsWrNmjdv2GjRooKVLl0qSfv75Z1ksFq1cuVKXXXaZ7Ha73njjDf3+++8aPXq0WrRooeDgYHXt2lXLly93+xyn06lnnnlGHTp0kM1mU6tWrfTkk09KkgYNGqRJkya5tT969KgCAwOVmJjo0e+nLuM+NwCAWnMqz6EuMz70yrZ3PR6r4MDKf+1lZ2crPj5e3bp1U1ZWlmbMmKERI0YoOTlZJ0+e1GWXXaYWLVpo7dq1ioyMVFJSkpxOpyRp3bp1GjFihB5++GG99tprys3N1fvvv+9xzVOnTtXs2bPVs2dP2e12nT59Wr1799aDDz6o0NBQrVu3TjfffLPat2+vfv36SZKmTZumV155Rc8995wGDBiglJQU7dmzR5I0fvx4TZo0SbNnz5bNZpMkvf7662rRooUGDRrkcX11FeEGAIAyXH/99W6vFy9erCZNmmjXrl364osvdPToUW3btk2NGjWSJHXo0MHV9sknn9SoUaP02GOPuZZ1797d4xqmTJmi6667zm3Z/fff75q/55579OGHH+qtt95Sv379lJmZqXnz5mn+/PmKi4uTJLVv314DBgyQJF133XWaNGmS3nvvPd14442SpKVLl+qWW24x1b2ICDcAgFoTFOCnXY/Hem3bnvjxxx81Y8YMffXVVzp27JhrVObgwYNKTk5Wz549XcGmpOTkZN1+++3nXHOfPn3cXjscDs2aNUtvvfWWfv31V+Xm5ionJ0fBwcGSpN27dysnJ0eDBw8u8/PsdrtuvvlmLV68WDfeeKOSkpL03Xffae3atedca11CuAEA1BqLxeLRoSFvGjZsmFq3bq1XXnlFzZs3l9Pp1MUXX6zc3NyzPkribO9bLJZS5wCVdcJwSEiI2+tnn31W8+bN09y5c9W1a1eFhIRoypQpys3NrdR2pYJDUz169NDhw4e1ZMkSDRo0SK1btz7rer6EE4oBACjh999/1969ezV9+nQNHjxYnTt31h9//OF6v1u3bkpOTtbx48fLXL9bt24VnqDbpEkTpaSkuF7/+OOPOnny5Fnr2rx5s6699lr97W9/U/fu3dWuXTv98MMPrvc7duyooKCgCrfdtWtX9enTR6+88orefPNN3XrrrWfdrq8h3AAAUELDhg3VuHFjvfzyy/rpp5/0ySefKD4+3vX+6NGjFRkZqeHDh2vz5s3av3+/3nnnHW3ZskWSNHPmTC1fvlwzZ87U7t27tXPnTj399NOu9QcNGqT58+drx44d+vrrrzVhwgQFBAScta6OHTtqw4YN+uKLL7R7927deeedSktLc71vt9v14IMP6oEHHtBrr72mffv26csvv9SiRYvcPmf8+PF66qmnZBiG21VcZkG4AQCgBKvVqhUrVmj79u26+OKLdd999+nZZ591vR8YGKiPPvpITZs21dChQ9W1a1c99dRTridZDxw4UG+//bbWrl2rHj16aNCgQdq6datr/dmzZysqKkp/+tOfdNNNN+n+++93nTdTkenTp6tXr16KjY3VwIEDXQGruEceeUR///vfNWPGDHXu3FkjR47UkSNH3NqMHj1a/v7+Gj16tOx2+zn8puomi+Hphf8+LiMjQ2FhYUpPT1doaKi3ywEAUzt9+rQOHDigtm3bmvJL1Ff9/PPPat++vbZt26ZevXp5uxyXiv69ePL97RtndQEAgHOWl5en33//XdOnT9cll1xSp4JNdeKwFAAA54nNmzerWbNm2rZtmxYuXOjtcmoMIzcAAJwnBg4c6PFjKHwRIzcAAMBUCDcAAMBUCDcAAMBUCDcAAMBUCDcAAMBUCDcAAMBUCDcAANSANm3aaO7cud4u47xEuAEAAKZCuAEAAG4cDoecTqe3y6gywg0AoPYYhpSb7Z3Jgzvzvvzyy2revHmpL/hrr71Wt956q/bt26drr71WERERqlevnvr27auPP/64yr+WOXPmqGvXrgoJCVFUVJTuvvtuZWVlubXZvHmzBg4cqODgYDVs2FCxsbH6448/JElOp1PPPPOMOnToIJvNplatWunJJ5+UJG3cuFEWi0UnTpxwfVZycrIsFot+/vlnSdLSpUvVoEEDrV27Vl26dJHNZtPBgwe1bds2XXHFFQoPD1dYWJguu+wyJSUludV14sQJ3XnnnYqIiJDdbtfFF1+s//73v8rOzlZoaKhWrVrl1n7NmjUKCQlRZmZmlX9fZ8PjFwAAtSfvpDSruXe2/dBvUmBIpZrecMMNuueee/Tpp59q8ODBkqTjx49r/fr1ev/995WVlaWhQ4fqySeflM1m02uvvaZhw4Zp7969atWqlcelWa1W/fvf/1bbtm21f/9+3X333XrggQf0wgsvSCoII4MHD9att96qefPmyd/fX59++qkcDockadq0aXrllVf03HPPacCAAUpJSdGePXs8quHkyZN6+umn9Z///EeNGzdW06ZNtX//fsXFxen555+XYRiaPXu2hg4dqh9//FH169eX0+nUVVddpczMTL3++utq3769du3aJT8/P4WEhGjUqFFasmSJ/vrXv7q2U/S6fv36Hv+eKotwAwBACQ0bNtRVV12lN9980xVuVq1apfDwcF1++eWyWq3q3r27q/0TTzyh1atXa+3atZo0aZLH25syZYprvk2bNvrnP/+pCRMmuMLNM888oz59+rheS9JFF10kScrMzNS8efM0f/58xcXFSZLat2+vAQMGeFRDXl6eXnjhBbd+DRo0yK3Nyy+/rAYNGuizzz7TNddco48//lhbt27V7t27dcEFF0iS2rVr52o/fvx49e/fXykpKWrWrJmOHDmi999//5xGuSqDcAMAqD0BwQUjKN7atgfGjBmj22+/XS+88IJsNpveeOMNjRo1SlarVVlZWXr00Ue1bt06paSkKD8/X6dOndLBgwerVNrHH3+shIQE7dmzRxkZGcrPz9fp06d18uRJBQcHKzk5WTfccEOZ6+7evVs5OTmuEFZVgYGB6tatm9uytLQ0TZ8+XRs3btSRI0fkcDh08uRJVz+Tk5PVsmVLV7ApqV+/frrooov06quvaurUqXr99dfVunVr/fnPfz6nWs+Gc24AALXHYik4NOSNyWLxqNRhw4bJMAytW7dOhw4d0v/+9z+NGTNGknT//fdr9erVmjVrlv73v/8pOTlZXbt2VW5urse/kp9//lnXXHONunXrpnfeeUfbt2/XggULJMn1eUFBQeWuX9F7UsEhL0luTwPPy8sr83MsJX5HcXFxSk5O1rx58/TFF18oOTlZjRs3rlRdRcaPH6+lS5dKKjgkNW7cuFLbqW6EGwAAymC323XdddfpjTfe0PLly9WpUyf16tVLUsHJvbfccotGjBihrl27KjIy0nVyrqe2b98up9Op2bNn65JLLtEFF1yg335zH93q1q2bEhMTy1y/Y8eOCgoKKvf9Jk2aSJJSUlJcy5KTkytV2+bNm3Xvvfdq6NChuuiii2Sz2XTs2DG3ug4fPqwffvih3M/429/+pl9++UX//ve/tWvXLtehs5pEuAEAoBxjxozRunXrtHjxYteojVQQKN59910lJyfrm2++0U033VTlS6c7dOigvLw8Pf/889q/f7+WLVumhQsXurWZNm2atm3bprvvvlvffvut9uzZoxdffFHHjh2T3W7Xgw8+qAceeECvvfaa9u3bpy+//FKLFi1yfX5UVJQeffRR/fjjj1q3bp1mz55dqdo6duyoZcuWaffu3frqq680ZswYt9Gayy67TH/+8591/fXXa8OGDTpw4IA++OADrV+/3tWmYcOGuu666/SPf/xDV155pVq2bFml35MnCDcAAJRj0KBBatSokfbu3aubbrrJtXzOnDlq2LCh+vfvr2HDhik2NtY1quOp7t27a86cOXr66ad18cUX64033lBCQoJbmwsuuEAfffSRvvnmG/Xr108xMTF677335O9fcOrsI488or///e+aMWOGOnfurJEjR+rIkSOSpICAAC1fvlx79uxRt27d9PTTT+uf//xnpWpbtGiR/vjjD/Xq1Us333yz7r33XjVt2tStzTvvvKO+fftq9OjR6tKlix544AHXVVxFbrvtNuXm5urWW2+t0u/IUxbD8ODCfxPIyMhQWFiY0tPTFRoa6u1yAMDUTp8+rQMHDqht27ay2+3eLgdesmzZMt1333367bffFBgYWG67iv69ePL9zdVSAACgRpw8eVIpKSl66qmndOedd1YYbKoTh6UAAKhBb7zxhurVq1fmVHSvGrN65plndOGFFyoyMlLTpk2rte1yWAoAUGM4LFVwk720tLQy3wsICFDr1q1ruaK6i8NSAAD4gPr169foowZQGoelAAA17jw7SIAqqq5/J4QbAECN8fPzk6Qq3bkX55+ifydF/26qisNSAIAa4+/vr+DgYB09elQBAQGuRwEAJTmdTh09elTBwcGu+/dUFeEGAFBjLBaLmjVrpgMHDuiXX37xdjmo46xWq1q1anXOz54i3AAAalRgYKA6duzIoSmcVWBgYLWM7hFuAAA1zmq1nreXgqP21YmDnwsWLFCbNm1kt9sVHR2trVu3ltt26dKlslgsbhN/MAAAoIjXw83KlSsVHx+vmTNnKikpSd27d1dsbKzrgV9lCQ0NVUpKimviOC4AACji9XAzZ84c3X777Ro3bpy6dOmihQsXKjg4WIsXLy53HYvFosjISNcUERFRixUDAIC6zKvn3OTm5mr79u1uz5uwWq0aMmSItmzZUu56WVlZat26tZxOp3r16qVZs2aV+3yOnJwc5eTkuF6np6dLKriNMwAA8A1F39uVudGfV8PNsWPH5HA4So28REREaM+ePWWu06lTJy1evFjdunVTenq6/vWvf6l///76/vvv1bJly1LtExIS9Nhjj5VaHhUVVT2dAAAAtSYzM1NhYWEVtvG5q6ViYmIUExPjet2/f3917txZL730kp544olS7adNm6b4+HjXa6fTqePHj6tx48bnfB19SRkZGYqKitKhQ4dM+VBOs/dPMn8f6Z/vM3sf6Z/vq6k+GoahzMxMNW/e/KxtvRpuwsPD5efnV+ppqWlpaYqMjKzUZwQEBKhnz5766aefynzfZrPJZrO5LWvQoEGV6q2s0NBQ0/6jlczfP8n8faR/vs/sfaR/vq8m+ni2EZsiXj2hODAwUL1791ZiYqJrmdPpVGJiotvoTEUcDod27typZs2a1VSZAADAh3j9sFR8fLzi4uLUp08f9evXT3PnzlV2drbGjRsnSRo7dqxatGihhIQESdLjjz+uSy65RB06dNCJEyf07LPP6pdfftH48eO92Q0AAFBHeD3cjBw5UkePHtWMGTOUmpqqHj16aP369a6TjA8ePOh2K+Y//vhDt99+u1JTU9WwYUP17t1bX3zxhbp06eKtLrjYbDbNnDmz1GEwszB7/yTz95H++T6z95H++b660EeLUZlrqgAAAHyE12/iBwAAUJ0INwAAwFQINwAAwFQINwAAwFQINx5asGCB2rRpI7vdrujoaG3durXC9m+//bYuvPBC2e12de3aVe+//34tVVo1nvRv6dKlslgsbpPdbq/Faj2zadMmDRs2TM2bN5fFYtGaNWvOus7GjRvVq1cv2Ww2dejQQUuXLq3xOqvK0/5t3Lix1P6zWCxKTU2tnYI9lJCQoL59+6p+/fpq2rSphg8frr179551PV/6G6xKH33p7/DFF19Ut27dXDd3i4mJ0QcffFDhOr60/zztny/tu7I89dRTslgsmjJlSoXtvLEPCTceWLlypeLj4zVz5kwlJSWpe/fuio2N1ZEjR8ps/8UXX2j06NG67bbbtGPHDg0fPlzDhw/Xd999V8uVV46n/ZMK7kCZkpLimn755ZdarNgz2dnZ6t69uxYsWFCp9gcOHNDVV1+tyy+/XMnJyZoyZYrGjx+vDz/8sIYrrRpP+1dk7969bvuwadOmNVThufnss880ceJEffnll9qwYYPy8vJ05ZVXKjs7u9x1fO1vsCp9lHzn77Bly5Z66qmntH37dn399dcaNGiQrr32Wn3//fdltve1/edp/yTf2Xclbdu2TS+99JK6detWYTuv7UMDldavXz9j4sSJrtcOh8No3ry5kZCQUGb7G2+80bj66qvdlkVHRxt33nlnjdZZVZ72b8mSJUZYWFgtVVe9JBmrV6+usM0DDzxgXHTRRW7LRo4cacTGxtZgZdWjMv379NNPDUnGH3/8USs1VbcjR44YkozPPvus3Da+9jdYUmX66Mt/h4ZhGA0bNjT+85//lPmer+8/w6i4f7667zIzM42OHTsaGzZsMC677DJj8uTJ5bb11j5k5KaScnNztX37dg0ZMsS1zGq1asiQIdqyZUuZ62zZssWtvSTFxsaW296bqtI/ScrKylLr1q0VFRV11v9D8TW+tP/ORY8ePdSsWTNdccUV2rx5s7fLqbT09HRJUqNGjcpt4+v7sDJ9lHzz79DhcGjFihXKzs4u93E7vrz/KtM/yTf33cSJE3X11VeX2jdl8dY+JNxU0rFjx+RwOFx3Ti4SERFR7jkKqampHrX3pqr0r1OnTlq8eLHee+89vf7663I6nerfv78OHz5cGyXXuPL2X0ZGhk6dOuWlqqpPs2bNtHDhQr3zzjt65513FBUVpYEDByopKcnbpZ2V0+nUlClTdOmll+riiy8ut50v/Q2WVNk++trf4c6dO1WvXj3ZbDZNmDBBq1evLvcO8764/zzpn6/tO0lasWKFkpKSXI9EOhtv7UOvP34BvismJsbt/0j69++vzp0766WXXtITTzzhxcpQGZ06dVKnTp1cr/v37699+/bpueee07Jly7xY2dlNnDhR3333nT7//HNvl1JjKttHX/s77NSpk5KTk5Wenq5Vq1YpLi5On332WZ14hE518KR/vrbvDh06pMmTJ2vDhg11/sRnwk0lhYeHy8/PT2lpaW7L09LSFBkZWeY6kZGRHrX3pqr0r6SAgAD17NlTP/30U02UWOvK23+hoaEKCgryUlU1q1+/fnU+MEyaNEn//e9/tWnTJrVs2bLCtr70N1icJ30sqa7/HQYGBqpDhw6SpN69e2vbtm2aN2+eXnrppVJtfXH/edK/kur6vtu+fbuOHDmiXr16uZY5HA5t2rRJ8+fPV05Ojvz8/NzW8dY+5LBUJQUGBqp3795KTEx0LXM6nUpMTCz3eGpMTIxbe0nasGFDhcdfvaUq/SvJ4XBo586datasWU2VWat8af9Vl+Tk5Dq7/wzD0KRJk7R69Wp98sknatu27VnX8bV9WJU+luRrf4dOp1M5OTllvudr+68sFfWvpLq+7wYPHqydO3cqOTnZNfXp00djxoxRcnJyqWAjeXEf1ujpyiazYsUKw2azGUuXLjV27dpl3HHHHUaDBg2M1NRUwzAM4+abbzamTp3qar9582bD39/f+Ne//mXs3r3bmDlzphEQEGDs3LnTW12okKf9e+yxx4wPP/zQ2Ldvn7F9+3Zj1KhRht1uN77//ntvdaFCmZmZxo4dO4wdO3YYkow5c+YYO3bsMH755RfDMAxj6tSpxs033+xqv3//fiM4ONj4xz/+YezevdtYsGCB4efnZ6xfv95bXaiQp/177rnnjDVr1hg//vijsXPnTmPy5MmG1Wo1Pv74Y291oUJ33XWXERYWZmzcuNFISUlxTSdPnnS18fW/war00Zf+DqdOnWp89tlnxoEDB4xvv/3WmDp1qmGxWIyPPvrIMAzf33+e9s+X9l15Sl4tVVf2IeHGQ88//7zRqlUrIzAw0OjXr5/x5Zdfut677LLLjLi4OLf2b731lnHBBRcYgYGBxkUXXWSsW7euliv2jCf9mzJliqttRESEMXToUCMpKckLVVdO0aXPJaeiPsXFxRmXXXZZqXV69OhhBAYGGu3atTOWLFlS63VXlqf9e/rpp4327dsbdrvdaNSokTFw4EDjk08+8U7xlVBW3yS57RNf/xusSh996e/w1ltvNVq3bm0EBgYaTZo0MQYPHuz64jcM399/nvbPl/ZdeUqGm7qyDy2GYRg1OzYEAABQezjnBgAAmArhBgAAmArhBgAAmArhBgAAmArhBgAAmArhBgAAmArhBgAAmArhBsB5z2KxaM2aNd4uA0A1IdwA8KpbbrlFFoul1PSXv/zF26UB8FE8FRyA1/3lL3/RkiVL3JbZbDYvVQPA1zFyA8DrbDabIiMj3aaGDRtKKjhk9OKLL+qqq65SUFCQ2rVrp1WrVrmtv3PnTg0aNEhBQUFq3Lix7rjjDmVlZbm1Wbx4sS666CLZbDY1a9ZMkyZNcnv/2LFjGjFihIKDg9WxY0etXbu2ZjsNoMYQbgDUeY888oiuv/56ffPNNxozZoxGjRql3bt3S5Kys7MVGxurhg0batu2bXr77bf18ccfu4WXF198URMnTtQdd9yhnTt3au3aterQoYPbNh577DHdeOON+vbbbzV06FCNGTNGx48fr9V+AqgmNf5oTgCoQFxcnOHn52eEhIS4TU8++aRhGAVPyp4wYYLbOtHR0cZdd91lGIZhvPzyy0bDhg2NrKws1/vr1q0zrFarkZqaahiGYTRv3tx4+OGHy61BkjF9+nTX66ysLEOS8cEHH1RbPwHUHs65AeB1l19+uV588UW3ZY0aNXLNx8TEuL0XExOj5ORkSdLu3bvVvXt3hYSEuN6/9NJL5XQ6tXfvXlksFv32228aPHhwhTV069bNNR8SEqLQ0FAdOXKkql0C4EWEGwBeFxISUuowUXUJCgqqVLuAgAC31xaLRU6nsyZKAlDDOOcGQJ335ZdflnrduXNnSVLnzp31zTffKDs72/X+5s2bZbVa1alTJ9WvX19t2rRRYmJirdYMwHsYuQHgdTk5OUpNTXVb5u/vr/DwcEnS22+/rT59+mjAgAF64403tHXrVi1atEiSNGbMGM2cOVNxcXF69NFHdfToUd1zzz26+eabFRERIUl69NFHNWHCBDVt2lRXXXWVMjMztXnzZt1zzz2121EAtYJwA8Dr1q9fr2bNmrkt69Spk/bs2SOp4EqmFStW6O6771azZs20fPlydenSRZIUHBysDz/8UJMnT1bfvn0VHBys66+/XnPmzHF9VlxcnE6fPq3nnntO999/v8LDw/XXv/619joIoFZZDMMwvF0EAJTHYrFo9erVGj58uLdLAeAjOOcGAACYCuEGAACYCufcAKjTOHIOwFOM3AAAAFMh3AAAAFMh3AAAAFMh3AAAAFMh3AAAAFMh3AAAAFMh3AAAAFMh3AAAAFMh3AAAAFP5/2mkUVBSKu2ZAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Plotting \n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.plot(history.history[\"accuracy\"], label=\"accuracy\")\n",
    "plt.plot(history.history[\"val_accuracy\"], label=\"val_accuracy\")\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.ylim([0.5, 1])\n",
    "plt.legend(loc=\"lower right\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Compare with the LSTM baseline\n",
    "\n",
    "![image.png](../../TP1_2/Exercices/exo1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion and Further Steps\n",
    "\n",
    "\n",
    "### Model Comparison: Transformer vs. LSTM\n",
    "\n",
    "#### LSTM Baseline Performance\n",
    "- **Convergence Speed**: The LSTM baseline exhibits a slower convergence rate towards achieving optimal accuracy. This can be particularly noticeable in complex tasks or with larger datasets, where the LSTM's recurrent nature might contribute to longer training times.\n",
    "- **Handling Long Sequences**: Another aspect where LSTM struggles is in processing very long sequences due to its sequential data processing nature, which might lead to issues like vanishing or exploding gradients.\n",
    "\n",
    "#### Transformer Model Advantages\n",
    "- **Faster Convergence**: In contrast to the LSTM, the Transformer model demonstrates a faster convergence to high accuracy. This efficiency is primarily due to its parallel processing capabilities and the self-attention mechanism, which allows it to learn dependencies between words irrespective of their position in the sequence.\n",
    "- **Scalability with Sequence Length**: Unlike LSTMs, Transformers handle long sequences more effectively due to their attention mechanism, which provides a global understanding of the input sequence, leading to better performance in tasks involving longer texts.\n",
    "- **Better Contextual Understanding**: The self-attention mechanism in Transformers offers a more nuanced understanding of context. It dynamically weighs the significance of different parts of the input data, which can lead to improved performance in tasks like sentiment analysis, where context plays a critical role.\n",
    "- **Flexibility and Adaptability**: Transformers have shown remarkable adaptability to a variety of tasks beyond text classification, including language translation, summarization, and even non-language tasks. This versatility makes them a valuable tool in the broader field of natural language processing and beyond.\n",
    "\n",
    "#### Conclusion\n",
    "While the LSTM provides a robust baseline, the Transformer model clearly demonstrates superior performance in terms of convergence speed, scalability, and contextual understanding, making it a more efficient and versatile choice for advanced text processing tasks.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}