transformers_summarization_wandb.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine Tuning Transformer for Summary Generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Introduction\n",
    "\n",
    "In this tutorial we will be fine tuning a transformer model for **Summarization Task**. \n",
    "In this task a summary of a given article/document is generated when passed through a network. There are 2 types of summary generation mechanisms:\n",
    "\n",
    "1. ***Extractive Summary:*** the network calculates the most important sentences from the article and gets them together to provide the most meaningful information from the article.\n",
    "2. ***Abstractive Summary***: The network creates new sentences to encapsulate maximum gist of the article and generates that as output. The sentences in the summary may or may not be contained in the article. \n",
    "\n",
    "In this tutorial we will be generating ***Abstractive Summary***. \n",
    "\n",
    "#### Flow of the notebook\n",
    "\n",
    "* As with all the tutorials previously, this notebook also follows a easy to follow steps. Making the process of fine tuning and training a Transformers model a straight forward task.\n",
    "* However, unlike the other notebooks, in the tutorial, most of the sections have been created into functions, and they are called from the `main()` in the end of the notebook. \n",
    "* This is done to leverage the [Weights and Biases Service](https://www.wandb.com/) WandB in short.\n",
    "* It is a experiment tracking, parameter optimization and artifact management service. That can be very easily integrated to any of the Deep learning or Machine learning frameworks. \n",
    "\n",
    "The notebook will be divided into separate sections to provide a organized walk through for the process used. This process can be modified for individual use cases. The sections are:\n",
    "\n",
    "1. [Preparing Environment and Importing Libraries](#section01)\n",
    "2. [Preparing the Dataset for data processing: Class](#section02)\n",
    "3. [Fine Tuning the Model: Function](#section03)\n",
    "4. [Validating the Model Performance: Function](#section04)\n",
    "5. [Main Function](#section05)\n",
    "    * [Initializing WandB](#section501)\n",
    "    * [Importing and Pre-Processing the domain data](#section502)\n",
    "    * [Creation of Dataset and Dataloader](#section503)\n",
    "    * [Neural Network and Optimizer](#section504)\n",
    "    * [Training Model and Logging to WandB](#section505)\n",
    "    * [Validation and generation of Summary](#section506)\n",
    "6. [Examples of the Summary Generated from the model](#section06)\n",
    "\n",
    "\n",
    "#### Technical Details\n",
    "\n",
    "This script leverages on multiple tools designed by other teams. Details of the tools used below. Please ensure that these elements are present in your setup to successfully implement this script.\n",
    "\n",
    "- **Data**:\n",
    "\t- We are using the News Summary dataset available at [Kaggle](https://www.kaggle.com/sunnysai12345/news-summary)\n",
    "\t- This dataset is the collection created from Newspapers published in India, extracting, details that are listed below.  We are referring only to the first csv file from the data dump: `news_summary.csv`\n",
    "\t- There are`4514` rows of data.  Where each row has the following data-point:\n",
    "\t\t- **author** : Author of the article\n",
    "\t\t- **date** : Date the article was published\n",
    "\t\t- **headline**: Headline for the published article\n",
    "\t\t- **read_more** : URL for the article to follow online\n",
    "\t\t- **text**: This is the summary of the article\n",
    "\t\t- **ctext**: This is the complete article\n",
    "\n",
    "\n",
    "- **Language Model Used**: \n",
    "    - This notebook uses one of the most recent and novel transformers model ***T5***. [Research Paper](https://arxiv.org/abs/1910.10683)    \n",
    "    - ***T5*** in many ways is one of its kind transformers architecture that not only gives state of the art results in many NLP tasks, but also has a very radical approach to NLP tasks.\n",
    "    - **Text-2-Text** - According to the graphic taken from the T5 paper. All NLP tasks are converted to a **text-to-text** problem. Tasks such as translation, classification, summarization and question answering, all of them are treated as a text-to-text conversion problem, rather than seen as separate unique problem statements.\n",
    "    - **Unified approach for NLP Deep Learning** - Since the task is reflected purely in the text input and output, you can use the same model, objective, training procedure, and decoding process to ANY task. Above framework can be used for any task - show Q&A, summarization, etc. \n",
    "   - We will be taking inputs from the T5 paper to prepare our dataset prior to fine tuning and training.    \n",
    "   - [Documentation for python](https://huggingface.co/transformers/model_doc/t5.html)\n",
    "\n",
    "![**Each NLP problem as a “text-to-text” problem** - input: text, output: text](https://miro.medium.com/max/4006/1*D0J1gNQf8vrrUpKeyD8wPA.png) \n",
    "\t \n",
    "\n",
    "\n",
    "- Hardware Requirements: \n",
    "\t- Python 3.6 and above\n",
    "\t- Pytorch, Transformers and\n",
    "\t- All the stock Python ML Library\n",
    "\t- GPU enabled setup \n",
    "   \n",
    "\n",
    "- **Script Objective**:\n",
    "\t- The objective of this script is to fine tune ***T5 *** to be able to generate summary, that a close to or better than the actual summary  while ensuring the important information from the article is not lost.\n",
    "\n",
    "---\n",
    "NOTE: \n",
    "We are using the Weights and Biases Tool-set in  this tutorial. The different components will be explained as we go through the article."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section01'></a>\n",
    "### Preparing Environment and Importing Libraries\n",
    "\n",
    "At this step we will be installing the necessary libraries followed by importing the libraries and modules needed to run our script. \n",
    "We will be installing:\n",
    "* transformers\n",
    "* wandb\n",
    "\n",
    "Libraries imported are:\n",
    "* Pandas\n",
    "* Pytorch\n",
    "* Pytorch Utils for Dataset and Dataloader\n",
    "* Transformers\n",
    "* T5 Model and Tokenizer\n",
    "* wandb\n",
    "\n",
    "Followed by that we will preapre the device for CUDA execeution. This configuration is needed if you want to leverage on onboard GPU. First we will check the GPU avaiable to us, using the nvidia command followed by defining our device.\n",
    "\n",
    "Finally, we will be logging into the [wandb](https://www.wandb.com/) serice using the login command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 316
    },
    "colab_type": "code",
    "id": "WD_vnyLXZQzD",
    "outputId": "b2ff57b8-a147-4893-80bd-e40d18042f98"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[K     |████████████████████████████████| 665kB 3.5MB/s \n",
      "\u001b[K     |████████████████████████████████| 890kB 9.6MB/s \n",
      "\u001b[K     |████████████████████████████████| 1.1MB 22.0MB/s \n",
      "\u001b[K     |████████████████████████████████| 3.8MB 30.8MB/s \n",
      "\u001b[?25h  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "\u001b[K     |████████████████████████████████| 1.4MB 3.4MB/s \n",
      "\u001b[K     |████████████████████████████████| 102kB 9.8MB/s \n",
      "\u001b[K     |████████████████████████████████| 460kB 18.8MB/s \n",
      "\u001b[K     |████████████████████████████████| 102kB 9.9MB/s \n",
      "\u001b[K     |████████████████████████████████| 112kB 23.9MB/s \n",
      "\u001b[K     |████████████████████████████████| 71kB 8.8MB/s \n",
      "\u001b[K     |████████████████████████████████| 71kB 7.5MB/s \n",
      "\u001b[?25h  Building wheel for subprocess32 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for watchdog (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for gql (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for pathtools (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for graphql-core (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
     ]
    }
   ],
   "source": [
    "!pip install transformers -q\n",
    "!pip install wandb -q\n",
    "\n",
    "# Code for TPU packages install\n",
    "# !curl -q https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py\n",
    "# !python pytorch-xla-env-setup.py --apt-packages libomp5 libopenblas-dev"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "pzM1_ykHaFur",
    "outputId": "58fa0ba8-b486-4b26-aaea-c0331b343b70"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m W&B installed but not logged in.  Run `wandb login` or set the WANDB_API_KEY env variable.\n"
     ]
    }
   ],
   "source": [
    "# Importing stock libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import Dataset, DataLoader, RandomSampler, SequentialSampler\n",
    "\n",
    "# Importing the T5 modules from huggingface/transformers\n",
    "from transformers import T5Tokenizer, T5ForConditionalGeneration\n",
    "\n",
    "# WandB – Import the wandb library\n",
    "import wandb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 316
    },
    "colab_type": "code",
    "id": "KvPxXdKJguYB",
    "outputId": "6c523635-a25a-429b-cbd8-7b8bf9636972"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mon Jun  1 03:23:27 2020       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 440.82       Driver Version: 418.67       CUDA Version: 10.1     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n",
      "| N/A   39C    P0    25W / 250W |      0MiB / 16280MiB |      0%      Default |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                       GPU Memory |\n",
      "|  GPU       PID   Type   Process name                             Usage      |\n",
      "|=============================================================================|\n",
      "|  No running processes found                                                 |\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "# Checking out the GPU we have access to. This is output is from the google colab version. \n",
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NLxxwd1scQNv"
   },
   "outputs": [],
   "source": [
    "# # Setting up the device for GPU usage\n",
    "from torch import cuda\n",
    "device = 'cuda' if cuda.is_available() else 'cpu'\n",
    "\n",
    "# Preparing for TPU usage\n",
    "# import torch_xla\n",
    "# import torch_xla.core.xla_model as xm\n",
    "# device = xm.xla_device()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 87
    },
    "colab_type": "code",
    "id": "L-ePh9dEKXMw",
    "outputId": "a35fd305-1c09-48ff-978c-fa1d0762c5e2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: You can find your API key in your browser here: https://app.wandb.ai/authorize\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Paste an API key from your profile and hit enter: \n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Appending key for api.wandb.ai to your netrc file: /root/.netrc\n",
      "\u001b[32mSuccessfully logged in to Weights & Biases!\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Login to wandb to log the model run and all the parameters\n",
    "!wandb login"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section02'></a>\n",
    "### Preparing the Dataset for data processing: Class\n",
    "\n",
    "We will start with creation of Dataset class - This defines how the text is pre-processed before sending it to the neural network. This dataset will be used the the Dataloader method that will feed  the data in batches to the neural network for suitable training and processing. \n",
    "The Dataloader and Dataset will be used inside the `main()`.\n",
    "Dataset and Dataloader are constructs of the PyTorch library for defining and controlling the data pre-processing and its passage to neural network. For further reading into Dataset and Dataloader read the [docs at PyTorch](https://pytorch.org/docs/stable/data.html)\n",
    "\n",
    "#### *CustomDataset* Dataset Class\n",
    "- This class is defined to accept the Dataframe as input and generate tokenized output that is used by the **T5** model for training. \n",
    "- We are using the **T5** tokenizer to tokenize the data in the `text` and `ctext` column of the dataframe. \n",
    "- The tokenizer uses the ` batch_encode_plus` method to perform tokenization and generate the necessary outputs, namely: `source_id`, `source_mask` from the actual text and `target_id` and `target_mask` from the summary text.\n",
    "- To read further into the tokenizer, [refer to this document](https://huggingface.co/transformers/model_doc/t5.html#t5tokenizer)\n",
    "- The *CustomDataset* class is used to create 2 datasets, for training and for validation.\n",
    "- *Training Dataset* is used to fine tune the model: **80% of the original data**\n",
    "- *Validation Dataset* is used to evaluate the performance of the model. The model has not seen this data during training. \n",
    "\n",
    "#### Dataloader: Called inside the `main()`\n",
    "- Dataloader is used to for creating training and validation dataloader that load data to the neural network in a defined manner. This is needed because all the data from the dataset cannot be loaded to the memory at once, hence the amount of data loaded to the memory and then passed to the neural network needs to be controlled.\n",
    "- This control is achieved using the parameters such as `batch_size` and `max_len`.\n",
    "- Training and Validation dataloaders are used in the training and validation part of the flow respectively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "932p8NhxeNw4"
   },
   "outputs": [],
   "source": [
    "# Creating a custom dataset for reading the dataframe and loading it into the dataloader to pass it to the neural network at a later stage for finetuning the model and to prepare it for predictions\n",
    "\n",
    "class CustomDataset(Dataset):\n",
    "\n",
    "    def __init__(self, dataframe, tokenizer, source_len, summ_len):\n",
    "        self.tokenizer = tokenizer\n",
    "        self.data = dataframe\n",
    "        self.source_len = source_len\n",
    "        self.summ_len = summ_len\n",
    "        self.text = self.data.text\n",
    "        self.ctext = self.data.ctext\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.text)\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        ctext = str(self.ctext[index])\n",
    "        ctext = ' '.join(ctext.split())\n",
    "\n",
    "        text = str(self.text[index])\n",
    "        text = ' '.join(text.split())\n",
    "\n",
    "        source = self.tokenizer.batch_encode_plus([ctext], max_length= self.source_len, pad_to_max_length=True,return_tensors='pt')\n",
    "        target = self.tokenizer.batch_encode_plus([text], max_length= self.summ_len, pad_to_max_length=True,return_tensors='pt')\n",
    "\n",
    "        source_ids = source['input_ids'].squeeze()\n",
    "        source_mask = source['attention_mask'].squeeze()\n",
    "        target_ids = target['input_ids'].squeeze()\n",
    "        target_mask = target['attention_mask'].squeeze()\n",
    "\n",
    "        return {\n",
    "            'source_ids': source_ids.to(dtype=torch.long), \n",
    "            'source_mask': source_mask.to(dtype=torch.long), \n",
    "            'target_ids': target_ids.to(dtype=torch.long),\n",
    "            'target_ids_y': target_ids.to(dtype=torch.long)\n",
    "        }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section03'></a>\n",
    "### Fine Tuning the Model: Function\n",
    "\n",
    "Here we define a training function that trains the model on the training dataset created above, specified number of times (EPOCH), An epoch defines how many times the complete data will be passed through the network. \n",
    "\n",
    "This function is called in the `main()`\n",
    "\n",
    "Following events happen in this function to fine tune the neural network:\n",
    "- The epoch, tokenizer, model, device details, testing_ dataloader and optimizer are passed to the `train ()` when its called from the `main()`\n",
    "- The dataloader passes data to the model based on the batch size.\n",
    "- `language_model_labels` are calculated from the `target_ids` also, `source_id` and `attention_mask` are extracted.\n",
    "- The model outputs first element gives the loss for the forward pass. \n",
    "- Loss value is used to optimize the weights of the neurons in the network.\n",
    "- After every 10 steps the loss value is logged in the wandb service. This log is then used to generate graphs for analysis. Such as [these](https://app.wandb.ai/abhimishra-91/transformers_tutorials_summarization?workspace=user-abhimishra-91)\n",
    "- After every 500 steps the loss value is printed in the console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SaPAR7TWmxoM"
   },
   "outputs": [],
   "source": [
    "# Creating the training function. This will be called in the main function. It is run depending on the epoch value.\n",
    "# The model is put into train mode and then we wnumerate over the training loader and passed to the defined network \n",
    "\n",
    "def train(epoch, tokenizer, model, device, loader, optimizer):\n",
    "    model.train()\n",
    "    for _,data in enumerate(loader, 0):\n",
    "        y = data['target_ids'].to(device, dtype = torch.long)\n",
    "        y_ids = y[:, :-1].contiguous()\n",
    "        lm_labels = y[:, 1:].clone().detach()\n",
    "        lm_labels[y[:, 1:] == tokenizer.pad_token_id] = -100\n",
    "        ids = data['source_ids'].to(device, dtype = torch.long)\n",
    "        mask = data['source_mask'].to(device, dtype = torch.long)\n",
    "\n",
    "        outputs = model(input_ids = ids, attention_mask = mask, decoder_input_ids=y_ids, lm_labels=lm_labels)\n",
    "        loss = outputs[0]\n",
    "        \n",
    "        if _%10 == 0:\n",
    "            wandb.log({\"Training Loss\": loss.item()})\n",
    "\n",
    "        if _%500==0:\n",
    "            print(f'Epoch: {epoch}, Loss:  {loss.item()}')\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        # xm.optimizer_step(optimizer)\n",
    "        # xm.mark_step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section04'></a>\n",
    "### Validating the Model Performance: Function\n",
    "\n",
    "During the validation stage we pass the unseen data(Testing Dataset), trained model, tokenizer and device details to the function to perform the validation run. This step generates new summary for dataset that it has not seen during the training session. \n",
    "\n",
    "This function is called in the `main()`\n",
    "\n",
    "This unseen data is the 20% of `news_summary.csv` which was seperated during the Dataset creation stage. \n",
    "During the validation stage the weights of the model are not updated. We use the generate method for generating new text for the summary. \n",
    "\n",
    "It depends on the `Beam-Search coding` method developed for sequence generation for models with LM head. \n",
    "\n",
    "The generated text and originally summary are decoded from tokens to text and returned to the `main()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "j9TNdHlQ0CLz"
   },
   "outputs": [],
   "source": [
    "def validate(epoch, tokenizer, model, device, loader):\n",
    "    model.eval()\n",
    "    predictions = []\n",
    "    actuals = []\n",
    "    with torch.no_grad():\n",
    "        for _, data in enumerate(loader, 0):\n",
    "            y = data['target_ids'].to(device, dtype = torch.long)\n",
    "            ids = data['source_ids'].to(device, dtype = torch.long)\n",
    "            mask = data['source_mask'].to(device, dtype = torch.long)\n",
    "\n",
    "            generated_ids = model.generate(\n",
    "                input_ids = ids,\n",
    "                attention_mask = mask, \n",
    "                max_length=150, \n",
    "                num_beams=2,\n",
    "                repetition_penalty=2.5, \n",
    "                length_penalty=1.0, \n",
    "                early_stopping=True\n",
    "                )\n",
    "            preds = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True) for g in generated_ids]\n",
    "            target = [tokenizer.decode(t, skip_special_tokens=True, clean_up_tokenization_spaces=True)for t in y]\n",
    "            if _%100==0:\n",
    "                print(f'Completed {_}')\n",
    "\n",
    "            predictions.extend(preds)\n",
    "            actuals.extend(target)\n",
    "    return predictions, actuals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section05'></a>\n",
    "### Main Function\n",
    "\n",
    "The `main()` as the name suggests is the central location to execute all the functions/flows created above in the notebook. The following steps are executed in the `main()`:\n",
    "\n",
    "\n",
    "<a id='section501'></a>\n",
    "#### Initializing WandB \n",
    "\n",
    "* The `main()` begins with initializing WandB run under a specific project. This command initiates a new run for each execution of this command. \n",
    "\n",
    "* Before we proceed any further i will give a brief overview of the **[WandB Service](https://www.wandb.com/)**\n",
    "\n",
    "* This service has been created to track ML experiments, Optimize the experiments and save artifacts. It is designed to seamlessly integrate with all the Machine Learning and Deep Learning Frameworks. Each script can be organized into *Project* and each execution of the script will be registered as a *run* in the respective project.\n",
    "\n",
    "* The service can be configured to log several default metrics, such a network weights, hardware usage, gradients and weights of the network. \n",
    "\n",
    "* It can also be used to log user defined metrics, such a loss in the `train()`.\n",
    "\n",
    "* This particular tutorial is logged in the project: **[transformers_tutorials_summarization](https://app.wandb.ai/abhimishra-91/transformers_tutorials_summarization?workspace=user-abhimishra-91)**\n",
    "\n",
    "**One of the charts from the project**\n",
    "![](meta/wandb.png)\n",
    "\n",
    "* Visit the project page to see the details of different runs and what information is logged by the service. \n",
    "\n",
    "* Following the initialization of the WandB service we define configuration parameters that will be used across the tutorial such as `batch_size`, `epoch`, `learning_rate` etc.\n",
    "\n",
    "* These parameters are also passed to the WandB config. The config construct with all the parameters can be optimized using the Sweep service from WandB. Currently, that is outof scope of this tutorial. \n",
    "\n",
    "* Next we defining seed values so that the experiment and results can be reproduced.\n",
    "\n",
    "\n",
    "<a id='section502'></a>\n",
    "#### Importing and Pre-Processing the domain data\n",
    "\n",
    "We will be working with the data and preparing it for fine tuning purposes. \n",
    "*Assuming that the `news_summary.csv` is already downloaded in your `data` folder*\n",
    "\n",
    "* The file is imported as a dataframe and give it the headers as per the documentation.\n",
    "* Cleaning the file to remove the unwanted columns.\n",
    "* A new string is added to the main article column `summarize: ` prior to the actual article. This is done because **T5** had similar formatting for the summarization dataset. \n",
    "* The final Dataframe will be something like this:\n",
    "\n",
    "|text|ctext|\n",
    "|--|--|\n",
    "|summary-1|summarize: article 1|\n",
    "|summary-2|summarize: article 2|\n",
    "|summary-3|summarize: article 3|\n",
    "\n",
    "* Top 5 rows of the dataframe are printed on the console.\n",
    "\n",
    "<a id='section503'></a>\n",
    "#### Creation of Dataset and Dataloader\n",
    "\n",
    "* The updated dataframe is divided into 80-20 ratio for test and validation. \n",
    "* Both the data-frames are passed to the `CustomerDataset` class for tokenization of the new articles and their summaries.\n",
    "* The tokenization is done using the length parameters passed to the class.\n",
    "* Train and Validation parameters are defined and passed to the `pytorch Dataloader contstruct` to create `train` and `validation` data loaders.\n",
    "* These dataloaders will be passed to `train()` and `validate()` respectively for training and validation action.\n",
    "* The shape of datasets is printed in the console.\n",
    "\n",
    "\n",
    "<a id='section504'></a>\n",
    "#### Neural Network and Optimizer\n",
    "\n",
    "* In this stage we define the model and optimizer that will be used for training and to update the weights of the network. \n",
    "* We are using the `t5-base` transformer model for our project. You can read about the `T5 model` and its features above. \n",
    "* We use the `T5ForConditionalGeneration.from_pretrained(\"t5-base\")` commad to define our model. The `T5ForConditionalGeneration` adds a Language Model head to our `T5 model`. The Language Model head allows us to generate text based on the training of `T5 model`.\n",
    "* We are using the `Adam` optimizer for our project. This has been a standard for all our tutorials and is something that can be changed updated to see how different optimizer perform with different learning rates. \n",
    "* There is also a scope for doing more with Optimizer such a decay, momentum to dynamically update the Learning rate and other parameters. All those concepts have been kept out of scope for these tutorials. \n",
    "\n",
    "\n",
    "<a id='section505'></a>\n",
    "#### Training Model and Logging to WandB\n",
    "\n",
    "* Now we log all the metrics in WandB project that we have initialized above.\n",
    "* Followed by that we call the `train()` with all the necessary parameters.\n",
    "* Loss at every 500th step is printed on the console.\n",
    "* Loss at every 10th step is logged as Loss in the WandB service.\n",
    "\n",
    "\n",
    "<a id='section506'></a>\n",
    "#### Validation and generation of Summary\n",
    "\n",
    "* After the training is completed, the validation step is initiated.\n",
    "* As defined in the validation function, the model weights are not updated. We use the fine tuned model to generate new summaries based on the article text.\n",
    "* An output is printed on the console giving a count of how many steps are complete after every 100th step. \n",
    "* The original summary and generated summary are converted into a list and returned to the main function. \n",
    "* Both the lists are used to create the final dataframe with 2 columns **Generated Summary** and **Actual Summary**\n",
    "* The dataframe is saved as a csv file in the local drive.\n",
    "* A qualitative analysis can be done with the Dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 691,
     "referenced_widgets": [
      "694ec243104f470093820e5e0dbbfc8e",
      "730be2945e39401dac1ac1247cf2d5fb",
      "71a4e2121f554668b2fd6461de3b2dcb",
      "9218011c0e1544ffb8128e0add0439c6",
      "9754d7526eb249d6ba849cb30f415ffc",
      "6afe5bad5a2b4e8c94791949b0f08ead",
      "4fca290286b3402fae9ddd0b1ce54504",
      "ae78c7bf57934f03a0f553b8829e44d4",
      "5c5331a892a64b95a2a130d7cb953e27",
      "a84d01823be4413db834ac23cffd9c26",
      "7b6f595e9c6a45f9b1edc6bd5512d205",
      "662b4ba823df409d8169696c81dccb46",
      "94d2a8fa47a3440a8332cb36036ea68e",
      "98459f90e8e94b29b93f576b5fdebe58",
      "e46fe62d895043878985a1014c3d853a",
      "a7edcb37126443c298eca2251194bb66",
      "54502d44bc774e2cb8067b59faa3f1bf",
      "1e6cc5a0d54c4fdab326fcaa2660e781",
      "ec8ba72354fd4b1d92c3b7d061e3c464",
      "d9627b05dcda4043aa149c940d7d2f57",
      "caf8968694c64b4fbbf5a15ed4948c34",
      "bb96976987544a5f800e71f2124b97dd",
      "4ee0316176b245239434bd88fe8f8572",
      "a0002b369f23475ca440c59df649e450"
     ]
    },
    "colab_type": "code",
    "id": "ZtNs9ytpCow2",
    "outputId": "80545587-0a82-455a-a9ba-13eb3fcb1550"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                Logging results to <a href=\"https://wandb.com\" target=\"_blank\">Weights & Biases</a> <a href=\"https://docs.wandb.com/integrations/jupyter.html\" target=\"_blank\">(Documentation)</a>.<br/>\n",
       "                Project page: <a href=\"https://app.wandb.ai/abhimishra-91/transformers_tutorials_summarization\" target=\"_blank\">https://app.wandb.ai/abhimishra-91/transformers_tutorials_summarization</a><br/>\n",
       "                Run page: <a href=\"https://app.wandb.ai/abhimishra-91/transformers_tutorials_summarization/runs/2erhbv26\" target=\"_blank\">https://app.wandb.ai/abhimishra-91/transformers_tutorials_summarization/runs/2erhbv26</a><br/>\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "694ec243104f470093820e5e0dbbfc8e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=791656.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "                                                text                                              ctext\n",
      "0  The Administration of Union Territory Daman an...  summarize: The Daman and Diu administration on...\n",
      "1  Malaika Arora slammed an Instagram user who tr...  summarize: From her special numbers to TV?appe...\n",
      "2  The Indira Gandhi Institute of Medical Science...  summarize: The Indira Gandhi Institute of Medi...\n",
      "3  Lashkar-e-Taiba's Kashmir commander Abu Dujana...  summarize: Lashkar-e-Taiba's Kashmir commander...\n",
      "4  Hotels in Maharashtra will train their staff t...  summarize: Hotels in Mumbai and other Indian c...\n",
      "FULL Dataset: (4514, 2)\n",
      "TRAIN Dataset: (3611, 2)\n",
      "TEST Dataset: (903, 2)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5c5331a892a64b95a2a130d7cb953e27",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=1199.0, style=ProgressStyle(description…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "54502d44bc774e2cb8067b59faa3f1bf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=891691430.0, style=ProgressStyle(descri…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Initiating Fine-Tuning for the model on our dataset\n",
      "/n\n",
      "Epoch: 0, Loss:  5.861971378326416\n",
      "Epoch: 0, Loss:  1.923175573348999\n",
      "Epoch: 0, Loss:  1.489040493965149\n",
      "Epoch: 0, Loss:  1.9674493074417114\n",
      "Epoch: 1, Loss:  2.0224180221557617\n",
      "Epoch: 1, Loss:  1.2057034969329834\n",
      "Epoch: 1, Loss:  1.2185782194137573\n",
      "Epoch: 1, Loss:  1.6962525844573975\n",
      "Now generating summaries on our fine tuned model for the validation dataset and saving it in a dataframe\n",
      "/n\n",
      "Completed 0\n",
      "Completed 100\n",
      "Completed 200\n",
      "Completed 300\n",
      "Completed 400\n",
      "Output Files generated for review\n"
     ]
    }
   ],
   "source": [
    "def main():\n",
    "    # WandB – Initialize a new run\n",
    "    wandb.init(project=\"transformers_tutorials_summarization\")\n",
    "\n",
    "    # WandB – Config is a variable that holds and saves hyperparameters and inputs\n",
    "    # Defining some key variables that will be used later on in the training  \n",
    "    config = wandb.config          # Initialize config\n",
    "    config.TRAIN_BATCH_SIZE = 2    # input batch size for training (default: 64)\n",
    "    config.VALID_BATCH_SIZE = 2    # input batch size for testing (default: 1000)\n",
    "    config.TRAIN_EPOCHS = 2        # number of epochs to train (default: 10)\n",
    "    config.VAL_EPOCHS = 1 \n",
    "    config.LEARNING_RATE = 1e-4    # learning rate (default: 0.01)\n",
    "    config.SEED = 42               # random seed (default: 42)\n",
    "    config.MAX_LEN = 512\n",
    "    config.SUMMARY_LEN = 150 \n",
    "\n",
    "    # Set random seeds and deterministic pytorch for reproducibility\n",
    "    torch.manual_seed(config.SEED) # pytorch random seed\n",
    "    np.random.seed(config.SEED) # numpy random seed\n",
    "    torch.backends.cudnn.deterministic = True\n",
    "\n",
    "    # tokenzier for encoding the text\n",
    "    tokenizer = T5Tokenizer.from_pretrained(\"t5-base\")\n",
    "    \n",
    "\n",
    "    # Importing and Pre-Processing the domain data\n",
    "    # Selecting the needed columns only. \n",
    "    # Adding the summarzie text in front of the text. This is to format the dataset similar to how T5 model was trained for summarization task. \n",
    "    df = pd.read_csv('./data/news_summary.csv',encoding='latin-1')\n",
    "    df = df[['text','ctext']]\n",
    "    df.ctext = 'summarize: ' + df.ctext\n",
    "    print(df.head())\n",
    "\n",
    "    \n",
    "    # Creation of Dataset and Dataloader\n",
    "    # Defining the train size. So 80% of the data will be used for training and the rest will be used for validation. \n",
    "    train_size = 0.8\n",
    "    train_dataset=df.sample(frac=train_size, random_state = config.SEED).reset_index(drop=True)\n",
    "    val_dataset=df.drop(train_dataset.index).reset_index(drop=True)\n",
    "\n",
    "    print(\"FULL Dataset: {}\".format(df.shape))\n",
    "    print(\"TRAIN Dataset: {}\".format(train_dataset.shape))\n",
    "    print(\"TEST Dataset: {}\".format(val_dataset.shape))\n",
    "\n",
    "\n",
    "    # Creating the Training and Validation dataset for further creation of Dataloader\n",
    "    training_set = CustomDataset(train_dataset, tokenizer, config.MAX_LEN, config.SUMMARY_LEN)\n",
    "    val_set = CustomDataset(val_dataset, tokenizer, config.MAX_LEN, config.SUMMARY_LEN)\n",
    "\n",
    "    # Defining the parameters for creation of dataloaders\n",
    "    train_params = {\n",
    "        'batch_size': config.TRAIN_BATCH_SIZE,\n",
    "        'shuffle': True,\n",
    "        'num_workers': 0\n",
    "        }\n",
    "\n",
    "    val_params = {\n",
    "        'batch_size': config.VALID_BATCH_SIZE,\n",
    "        'shuffle': False,\n",
    "        'num_workers': 0\n",
    "        }\n",
    "\n",
    "    # Creation of Dataloaders for testing and validation. This will be used down for training and validation stage for the model.\n",
    "    training_loader = DataLoader(training_set, **train_params)\n",
    "    val_loader = DataLoader(val_set, **val_params)\n",
    "\n",
    "\n",
    "    \n",
    "    # Defining the model. We are using t5-base model and added a Language model layer on top for generation of Summary. \n",
    "    # Further this model is sent to device (GPU/TPU) for using the hardware.\n",
    "    model = T5ForConditionalGeneration.from_pretrained(\"t5-base\")\n",
    "    model = model.to(device)\n",
    "\n",
    "    # Defining the optimizer that will be used to tune the weights of the network in the training session. \n",
    "    optimizer = torch.optim.Adam(params =  model.parameters(), lr=config.LEARNING_RATE)\n",
    "\n",
    "    # Log metrics with wandb\n",
    "    wandb.watch(model, log=\"all\")\n",
    "    # Training loop\n",
    "    print('Initiating Fine-Tuning for the model on our dataset')\n",
    "\n",
    "    for epoch in range(config.TRAIN_EPOCHS):\n",
    "        train(epoch, tokenizer, model, device, training_loader, optimizer)\n",
    "\n",
    "\n",
    "    # Validation loop and saving the resulting file with predictions and acutals in a dataframe.\n",
    "    # Saving the dataframe as predictions.csv\n",
    "    print('Now generating summaries on our fine tuned model for the validation dataset and saving it in a dataframe')\n",
    "    for epoch in range(config.VAL_EPOCHS):\n",
    "        predictions, actuals = validate(epoch, tokenizer, model, device, val_loader)\n",
    "        final_df = pd.DataFrame({'Generated Text':predictions,'Actual Text':actuals})\n",
    "        final_df.to_csv('./models/predictions.csv')\n",
    "        print('Output Files generated for review')\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section06'></a>\n",
    "### Examples of the Summary Generated from the model\n",
    "\n",
    "##### Example 1\n",
    "\n",
    "**Original Text**\n",
    "New Delhi, Apr 25 (PTI) Union minister Vijay Goel today batted for the unification of the three municipal corporations in the national capital saying a discussion over the issue was pertinent. The BJP leader, who was confident of a good show by his party in the MCD polls, the results of which will be declared tomorrow, said the civic bodies needed to be \"revamped\" in order to deliver the services to the people more effectively. The first thing needed was a discussion on the unification of the three municipal corporations and there should also be an end to the practice of sending Delhi government officials to serve in the civic bodies, said the Union Minister of State (Independent Charge) for Youth Affairs and Sports. \"Barring one, the two other civic bodies have been incurring losses. It would be more fruitful and efficient if all the three were merged,\" he said, referring to the north, south and east Delhi municipal corporations. The erstwhile Municipal Corporation of Delhi (MCD) was trifurcated into NDMC, SDMC and EDMC by the then Sheila Dikshit-led Delhi government in 2012. Goel predicted a \"thumping\" victory for the BJP in the MCD polls. He said the newly-elected BJP councillors will be trained on the functioning of the civic bodies and dealing with the bureaucracy. \n",
    "\n",
    "\n",
    "**Original Summary**\n",
    "Union Minister Vijay Goel has favoured unification of three MCDs ? North, South and East ? in order to deliver the services more effectively. \"Barring one, the two other civic bodies have been incurring losses. It would be more fruitful and efficient if all the three were merged,\" he said. MCD was trifurcated into EDMC, NDMC and SDMC in 2012.\n",
    "\n",
    "**Generated Summary**\n",
    "BJP leader Vijay Goel on Saturday batted for the unification of three municipal corporations in the national capital saying a discussion over this was pertinent. \"Barring one, two other civic bodies have been incurring losses,\" said Goels. The erstwhile Municipal Corporations of Delhi (MCD) were trifurcated into NDMC and SDMC by the then Sheilha Dikshi-led government in 2012. Notably, the MCD poll results will be declared tomorrow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Example 2\n",
    "\n",
    "**Original Text**\n",
    "After much wait, the first UDAN flight took off from Shimla today after being flagged off by Prime Minister Narendra Modi.The flight will be operated by Alliance Air, the regional arm of Air India. PM Narendra Modi handed over boarding passes to some of passengers travelling via the first UDAN flight at the Shimla airport.Tomorrow PM @narendramodi will flag off the first UDAN flight under the Regional Connectivity Scheme, on Shimla-Delhi sector.Air India yesterday opened bookings for the first launch flight from Shimla to Delhi with all inclusive fares starting at Rs2,036.THE GREAT 'UDAN'The UDAN (Ude Desh ka Aam Naagrik) scheme seeks to make flying more affordable for the common people, holding a plan to connect over 45 unserved and under-served airports.Under UDAN, 50 per cent of the seats on each flight would have a cap of Rs 2,500 per seat/hour. The government has also extended subsidy in the form of viability gap funding to the operators flying on these routes.The scheme was launched to \"make air travel accessible to citizens in regionally important cities,\" and has been described as \"a first-of-its-kind scheme globally to stimulate regional connectivity through a market-based mechanism.\" Report have it the first flight today will not be flying at full capacity on its 70-seater ATR airplane because of payload restrictions related to the short Shimla airfield.|| Read more ||Udan scheme: Now you can fly to these 43 cities, see the full list hereUDAN scheme to fly hour-long flights capped at Rs 2,500 to smaller cities \n",
    "\n",
    "\n",
    "**Original Summary**\n",
    "PM Narendra Modi on Thursday launched Ude Desh ka Aam Nagrik (UDAN) scheme for regional flight connectivity by flagging off the inaugural flight from Shimla to Delhi. Under UDAN, government will connect small towns by air with 50% plane seats' fare capped at?2,500 for a one-hour journey of 500 kilometres. UDAN will connect over 45 unserved and under-served airports.\n",
    "\n",
    "**Generated Summary**\n",
    "UDAN (Ude Desh Ka Aam Naagrik) scheme, launched to make air travel accessible in regionally important cities under the Regional Connectivity Scheme, took off from Shimla on Tuesday. The first flight will be operated by Alliance Air, which is the regional arm of India's Air India. Under the scheme, 50% seats would have?2,500 per seat/hour and 50% of the seats would have capped at this rate. It was also extended subsidy in form-based funding for operators flying these routes as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Example 3\n",
    "\n",
    "**Original Text**\n",
    "New Delhi, Apr 25 (PTI) The Income Tax department has issued a Rs 24,646 crore tax demand notice to Sahara Groups Aamby Valley Limited (AVL) after conducting a special audit of the company. The department, as part of a special investigation and audit into the account books of AVL, found that an income of over Rs 48,000 crore for a particular assessment year was allegedly not reflected in the record books of the firm and hence it raised a fresh tax demand and penalty amount on it. A Sahara Group spokesperson confirmed the development to PTI. \"Yes, the Income Tax Department has raised Rs 48,085.79 crores to the income of the Aamby Valley Limited with a total demand of income tax of Rs 24,646.96 crores on the Aamby Valley Limited,\" the spokesperson said in a brief statement. Officials said the notice was issued by the taxman in January this year after the special audit of AVLs income for the Assessment Year 2012-13 found that the parent firm had allegedly floated a clutch of Special Purpose Vehicles whose incomes were later accounted on the account of AVL as they were merged with the former in due course of time. The AVL, in its income return filed for AY 2012-13, had reflected a loss of few crores but the special I-T audit brought up the added income, a senior official said. The Supreme Court, last week, had asked the Bombay High Courts official liquidator to sell the Rs 34,000 crore worth of properties of Aamby Valley owned by the Sahara Group and directed its chief Subrata Roy to personally appear before it on April 28.  \n",
    "\n",
    "\n",
    "**Original Summary**\n",
    "The Income Tax Department has issued a ?24,646 crore tax demand notice to Sahara Group's Aamby Valley Limited. The department's audit found that an income of over ?48,000 crore for the assessment year 2012-13 was not reflected in the record books of the firm. A week ago, the SC ordered Bombay HC to auction Sahara's Aamby Valley worth ?34,000 crore.\n",
    "\n",
    "**Generated Summary**\n",
    "the Income Tax department has issued a?24,646 crore tax demand notice to Sahara Groups Aamby Valley Limited (AVL) after conducting an audit of the company. The notice was issued in January this year after the special audit found that the parent firm had floated Special Purpose Vehicle income for the Assessment Year 2012-13 and later accounted on its account as they were merged with the former. \"Yes...the Income Tax Department raised Rs48,085.79 crores to the income,\" he added earlier said at the notice."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "transformers_summarization_wandb.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "1e6cc5a0d54c4fdab326fcaa2660e781": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4ee0316176b245239434bd88fe8f8572": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4fca290286b3402fae9ddd0b1ce54504": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "54502d44bc774e2cb8067b59faa3f1bf": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_ec8ba72354fd4b1d92c3b7d061e3c464",
       "IPY_MODEL_d9627b05dcda4043aa149c940d7d2f57"
      ],
      "layout": "IPY_MODEL_1e6cc5a0d54c4fdab326fcaa2660e781"
     }
    },
    "5c5331a892a64b95a2a130d7cb953e27": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_7b6f595e9c6a45f9b1edc6bd5512d205",
       "IPY_MODEL_662b4ba823df409d8169696c81dccb46"
      ],
      "layout": "IPY_MODEL_a84d01823be4413db834ac23cffd9c26"
     }
    },
    "662b4ba823df409d8169696c81dccb46": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a7edcb37126443c298eca2251194bb66",
      "placeholder": "​",
      "style": "IPY_MODEL_e46fe62d895043878985a1014c3d853a",
      "value": " 1.20k/1.20k [00:16&lt;00:00, 73.7B/s]"
     }
    },
    "694ec243104f470093820e5e0dbbfc8e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_71a4e2121f554668b2fd6461de3b2dcb",
       "IPY_MODEL_9218011c0e1544ffb8128e0add0439c6"
      ],
      "layout": "IPY_MODEL_730be2945e39401dac1ac1247cf2d5fb"
     }
    },
    "6afe5bad5a2b4e8c94791949b0f08ead": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "71a4e2121f554668b2fd6461de3b2dcb": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6afe5bad5a2b4e8c94791949b0f08ead",
      "max": 791656,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_9754d7526eb249d6ba849cb30f415ffc",
      "value": 791656
     }
    },
    "730be2945e39401dac1ac1247cf2d5fb": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7b6f595e9c6a45f9b1edc6bd5512d205": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_98459f90e8e94b29b93f576b5fdebe58",
      "max": 1199,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_94d2a8fa47a3440a8332cb36036ea68e",
      "value": 1199
     }
    },
    "9218011c0e1544ffb8128e0add0439c6": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ae78c7bf57934f03a0f553b8829e44d4",
      "placeholder": "​",
      "style": "IPY_MODEL_4fca290286b3402fae9ddd0b1ce54504",
      "value": " 792k/792k [00:00&lt;00:00, 2.85MB/s]"
     }
    },
    "94d2a8fa47a3440a8332cb36036ea68e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "9754d7526eb249d6ba849cb30f415ffc": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "98459f90e8e94b29b93f576b5fdebe58": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a0002b369f23475ca440c59df649e450": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a7edcb37126443c298eca2251194bb66": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a84d01823be4413db834ac23cffd9c26": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ae78c7bf57934f03a0f553b8829e44d4": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bb96976987544a5f800e71f2124b97dd": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "caf8968694c64b4fbbf5a15ed4948c34": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "d9627b05dcda4043aa149c940d7d2f57": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a0002b369f23475ca440c59df649e450",
      "placeholder": "​",
      "style": "IPY_MODEL_4ee0316176b245239434bd88fe8f8572",
      "value": " 892M/892M [00:15&lt;00:00, 57.8MB/s]"
     }
    },
    "e46fe62d895043878985a1014c3d853a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ec8ba72354fd4b1d92c3b7d061e3c464": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_bb96976987544a5f800e71f2124b97dd",
      "max": 891691430,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_caf8968694c64b4fbbf5a15ed4948c34",
      "value": 891691430
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}