GPT-J-6B-8-bit-Prompt-Tuning-Headline-Generation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Prompt Tuning GPT-J with 8-bit weights for Headline Generation"
      ],
      "metadata": {
        "id": "x5i-iMKTRbXW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This notebook contains the code for prompt tuning a GPT-J Model using OpenPrompt on the task of Headline Generation from News Articles.\n",
        "\n",
        "*   GPT-J-6B: A 6 billion parameter, autoregressive text generation model by EleutherAI trained on [The Pile](https://pile.eleuther.ai/) using [Mesh Transformer JAX](https://github.com/kingoflolz/mesh-transformer-jax/) (Ben Wang and Aran Komatsuzaki).\n",
        "*   GPT-J-6B-8bit: A quantized GPT-J-6B with 8-bit weights for scalable and cost-efficient fine-tuning by Hivemind with [LoRA](https://arxiv.org/pdf/2106.09685.pdf) and [8-bit Adam](https://arxiv.org/abs/2110.02861).\n",
        "*   Prompt Tuning: Latest paradigm to adapt pre-trained language models (PLMs) to downstream NLP tasks by learning and adding \"soft prompts\" to the model input (Brian Lester, Rami Al-Rfou, Noah Constant), [Paper](https://arxiv.org/abs/2104.08691).\n",
        "*   OpenPrompt: An Open-Source Framework for Prompt-learning, [Code](https://arxiv.org/abs/2104.08691) [Docs](https://thunlp.github.io/OpenPrompt/) [Paper](https://arxiv.org/abs/2111.01998)\n",
        "\n",
        "*Note: Running this notebook excedes the ressources of free google colab. \\\n",
        "For reproduction, we trained with Colab Pro Extended RAM and Premium GPU Settings. Extended RAM is necessary. Premium GPU is not, but it speeds up the training process bx factor x3.*"
      ],
      "metadata": {
        "id": "shHA1dDosB2C"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Installs & Imports\n",
        "\n",
        "Freeze library versions to make it reproducible."
      ],
      "metadata": {
        "id": "BmkK6bqPcRC9"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "lFibai4vnZtG",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "0b1f9624-4550-4014-acf9-a25144ec1ef2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting transformers==4.21.0\n",
            "  Downloading transformers-4.21.0-py3-none-any.whl (4.7 MB)\n",
            "\u001b[K     |████████████████████████████████| 4.7 MB 14.5 MB/s \n",
            "\u001b[?25hCollecting regex!=2019.12.17\n",
            "  Downloading regex-2022.10.31-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (772 kB)\n",
            "\u001b[K     |████████████████████████████████| 772 kB 73.5 MB/s \n",
            "\u001b[?25hCollecting filelock\n",
            "  Downloading filelock-3.8.2-py3-none-any.whl (10 kB)\n",
            "Collecting huggingface-hub<1.0,>=0.1.0\n",
            "  Downloading huggingface_hub-0.11.1-py3-none-any.whl (182 kB)\n",
            "\u001b[K     |████████████████████████████████| 182 kB 94.6 MB/s \n",
            "\u001b[?25hCollecting tqdm>=4.27\n",
            "  Downloading tqdm-4.64.1-py2.py3-none-any.whl (78 kB)\n",
            "\u001b[K     |████████████████████████████████| 78 kB 8.9 MB/s \n",
            "\u001b[?25hCollecting packaging>=20.0\n",
            "  Downloading packaging-22.0-py3-none-any.whl (42 kB)\n",
            "\u001b[K     |████████████████████████████████| 42 kB 1.7 MB/s \n",
            "\u001b[?25hCollecting requests\n",
            "  Downloading requests-2.28.1-py3-none-any.whl (62 kB)\n",
            "\u001b[K     |████████████████████████████████| 62 kB 1.8 MB/s \n",
            "\u001b[?25hCollecting pyyaml>=5.1\n",
            "  Downloading PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (701 kB)\n",
            "\u001b[K     |████████████████████████████████| 701 kB 86.8 MB/s \n",
            "\u001b[?25hCollecting tokenizers!=0.11.3,<0.13,>=0.11.1\n",
            "  Downloading tokenizers-0.12.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (6.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 6.6 MB 76.3 MB/s \n",
            "\u001b[?25hCollecting numpy>=1.17\n",
            "  Downloading numpy-1.24.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 17.3 MB 85.9 MB/s \n",
            "\u001b[?25hCollecting typing-extensions>=3.7.4.3\n",
            "  Downloading typing_extensions-4.4.0-py3-none-any.whl (26 kB)\n",
            "Collecting urllib3<1.27,>=1.21.1\n",
            "  Downloading urllib3-1.26.13-py2.py3-none-any.whl (140 kB)\n",
            "\u001b[K     |████████████████████████████████| 140 kB 86.6 MB/s \n",
            "\u001b[?25hCollecting charset-normalizer<3,>=2\n",
            "  Downloading charset_normalizer-2.1.1-py3-none-any.whl (39 kB)\n",
            "Collecting certifi>=2017.4.17\n",
            "  Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)\n",
            "\u001b[K     |████████████████████████████████| 155 kB 89.6 MB/s \n",
            "\u001b[?25hCollecting idna<4,>=2.5\n",
            "  Downloading idna-3.4-py3-none-any.whl (61 kB)\n",
            "\u001b[K     |████████████████████████████████| 61 kB 119 kB/s \n",
            "\u001b[?25hInstalling collected packages: urllib3, idna, charset-normalizer, certifi, typing-extensions, tqdm, requests, pyyaml, packaging, filelock, tokenizers, regex, numpy, huggingface-hub, transformers\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "scipy 1.7.3 requires numpy<1.23.0,>=1.16.5, but you have numpy 1.24.0 which is incompatible.\n",
            "numba 0.56.4 requires numpy<1.24,>=1.18, but you have numpy 1.24.0 which is incompatible.\n",
            "google-cloud-bigquery 3.3.6 requires packaging<22.0.0dev,>=14.3, but you have packaging 22.0 which is incompatible.\u001b[0m\n",
            "Successfully installed certifi-2022.12.7 charset-normalizer-2.1.1 filelock-3.8.2 huggingface-hub-0.11.1 idna-3.4 numpy-1.24.0 packaging-22.0 pyyaml-6.0 regex-2022.10.31 requests-2.28.1 tokenizers-0.12.1 tqdm-4.64.1 transformers-4.21.0 typing-extensions-4.4.0 urllib3-1.26.13\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "certifi",
                  "numpy"
                ]
              }
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting openprompt==1.0.1\n",
            "  Downloading openprompt-1.0.1-py3-none-any.whl (146 kB)\n",
            "\u001b[K     |████████████████████████████████| 146 kB 14.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: nltk in /usr/local/lib/python3.8/dist-packages (from openprompt==1.0.1) (3.7)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from openprompt==1.0.1) (1.7.3)\n",
            "Requirement already satisfied: pyarrow in /usr/local/lib/python3.8/dist-packages (from openprompt==1.0.1) (9.0.0)\n",
            "Requirement already satisfied: dill in /usr/local/lib/python3.8/dist-packages (from openprompt==1.0.1) (0.3.6)\n",
            "Requirement already satisfied: tqdm>=4.62.2 in /usr/local/lib/python3.8/dist-packages (from openprompt==1.0.1) (4.64.1)\n",
            "Requirement already satisfied: transformers>=4.10.0 in /usr/local/lib/python3.8/dist-packages (from openprompt==1.0.1) (4.21.0)\n",
            "Collecting rouge==1.0.0\n",
            "  Downloading rouge-1.0.0-py3-none-any.whl (14 kB)\n",
            "Collecting yacs\n",
            "  Downloading yacs-0.1.8-py3-none-any.whl (14 kB)\n",
            "Collecting sentencepiece==0.1.96\n",
            "  Downloading sentencepiece-0.1.96-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.2 MB 93.1 MB/s \n",
            "\u001b[?25hCollecting datasets\n",
            "  Downloading datasets-2.8.0-py3-none-any.whl (452 kB)\n",
            "\u001b[K     |████████████████████████████████| 452 kB 88.9 MB/s \n",
            "\u001b[?25hCollecting tensorboardX\n",
            "  Downloading tensorboardX-2.5.1-py2.py3-none-any.whl (125 kB)\n",
            "\u001b[K     |████████████████████████████████| 125 kB 102.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.8/dist-packages (from rouge==1.0.0->openprompt==1.0.1) (1.15.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from transformers>=4.10.0->openprompt==1.0.1) (6.0)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.1.0 in /usr/local/lib/python3.8/dist-packages (from transformers>=4.10.0->openprompt==1.0.1) (0.11.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers>=4.10.0->openprompt==1.0.1) (3.8.2)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from transformers>=4.10.0->openprompt==1.0.1) (2.28.1)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.8/dist-packages (from transformers>=4.10.0->openprompt==1.0.1) (1.24.0)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.13,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers>=4.10.0->openprompt==1.0.1) (0.12.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers>=4.10.0->openprompt==1.0.1) (2022.10.31)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from transformers>=4.10.0->openprompt==1.0.1) (22.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub<1.0,>=0.1.0->transformers>=4.10.0->openprompt==1.0.1) (4.4.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.8/dist-packages (from datasets->openprompt==1.0.1) (3.8.3)\n",
            "Collecting xxhash\n",
            "  Downloading xxhash-3.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)\n",
            "\u001b[K     |████████████████████████████████| 212 kB 98.9 MB/s \n",
            "\u001b[?25hCollecting multiprocess\n",
            "  Downloading multiprocess-0.70.14-py38-none-any.whl (132 kB)\n",
            "\u001b[K     |████████████████████████████████| 132 kB 95.7 MB/s \n",
            "\u001b[?25hCollecting responses<0.19\n",
            "  Downloading responses-0.18.0-py3-none-any.whl (38 kB)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from datasets->openprompt==1.0.1) (1.3.5)\n",
            "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.8/dist-packages (from datasets->openprompt==1.0.1) (2022.11.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets->openprompt==1.0.1) (1.3.3)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets->openprompt==1.0.1) (22.1.0)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets->openprompt==1.0.1) (1.8.2)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets->openprompt==1.0.1) (4.0.2)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets->openprompt==1.0.1) (1.3.1)\n",
            "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets->openprompt==1.0.1) (2.1.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets->openprompt==1.0.1) (6.0.3)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->transformers>=4.10.0->openprompt==1.0.1) (1.26.13)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->transformers>=4.10.0->openprompt==1.0.1) (2022.12.7)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->transformers>=4.10.0->openprompt==1.0.1) (3.4)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.8/dist-packages (from nltk->openprompt==1.0.1) (1.2.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.8/dist-packages (from nltk->openprompt==1.0.1) (7.1.2)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas->datasets->openprompt==1.0.1) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas->datasets->openprompt==1.0.1) (2022.6)\n",
            "Collecting numpy>=1.17\n",
            "  Downloading numpy-1.22.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.9 MB)\n",
            "\u001b[K     |████████████████████████████████| 16.9 MB 89.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: protobuf<=3.20.1,>=3.8.0 in /usr/local/lib/python3.8/dist-packages (from tensorboardX->openprompt==1.0.1) (3.19.6)\n",
            "Installing collected packages: numpy, xxhash, responses, multiprocess, yacs, tensorboardX, sentencepiece, rouge, datasets, openprompt\n",
            "  Attempting uninstall: numpy\n",
            "    Found existing installation: numpy 1.24.0\n",
            "    Uninstalling numpy-1.24.0:\n",
            "      Successfully uninstalled numpy-1.24.0\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "google-cloud-bigquery 3.3.6 requires packaging<22.0.0dev,>=14.3, but you have packaging 22.0 which is incompatible.\u001b[0m\n",
            "Successfully installed datasets-2.8.0 multiprocess-0.70.14 numpy-1.22.4 openprompt-1.0.1 responses-0.18.0 rouge-1.0.0 sentencepiece-0.1.96 tensorboardX-2.5.1 xxhash-3.1.0 yacs-0.1.8\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "numpy"
                ]
              }
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting accelerate==0.15.0\n",
            "  Downloading accelerate-0.15.0-py3-none-any.whl (191 kB)\n",
            "\u001b[K     |████████████████████████████████| 191 kB 14.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: pyyaml in /usr/local/lib/python3.8/dist-packages (from accelerate==0.15.0) (6.0)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.8/dist-packages (from accelerate==0.15.0) (5.4.8)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.8/dist-packages (from accelerate==0.15.0) (1.22.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from accelerate==0.15.0) (22.0)\n",
            "Requirement already satisfied: torch>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from accelerate==0.15.0) (1.13.0+cu116)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.8/dist-packages (from torch>=1.4.0->accelerate==0.15.0) (4.4.0)\n",
            "Installing collected packages: accelerate\n",
            "Successfully installed accelerate-0.15.0\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting bitsandbytes==0.35.4\n",
            "  Downloading bitsandbytes-0.35.4-py3-none-any.whl (62.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 62.5 MB 143.4 MB/s \n",
            "\u001b[?25hInstalling collected packages: bitsandbytes\n",
            "Successfully installed bitsandbytes-0.35.4\n"
          ]
        }
      ],
      "source": [
        "!pip install -I transformers==4.21.0 \n",
        "# Transformers is pinned to this specific version because it fixes bug \n",
        "# https://github.com/huggingface/transformers/issues/19290\n",
        "!pip install openprompt==1.0.1\n",
        "!pip install accelerate==0.15.0\n",
        "!pip install bitsandbytes==0.35.4"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from openprompt.pipeline_base import PromptForGeneration\n",
        "from openprompt.prompts.generation_verbalizer import GenerationVerbalizer\n",
        "from tqdm import tqdm\n",
        "from openprompt.data_utils import PROCESSORS\n",
        "import torch\n",
        "from openprompt.data_utils.utils import InputExample\n",
        "import argparse\n",
        "import numpy as np\n",
        "\n",
        "from openprompt import PromptDataLoader\n",
        "from openprompt.prompts import ManualVerbalizer\n",
        "from openprompt.prompts import SoftTemplate\n",
        "from openprompt import PromptForClassification\n",
        "import time\n",
        "import os\n",
        "import re\n",
        "from openprompt.utils.crossfit_metrics import evaluate as crossfit_evaluate"
      ],
      "metadata": {
        "id": "lZjW4Iso8Nwf"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Patches for Model conversion to 8 bits\n",
        "\n",
        "We use an 8-bit version of EleutherAI's GPT-J-6B model converted with facebook's bitsandbytes library. This reduces the model's size from 24GB down to 6GB.\n",
        "\n",
        "*    large weight tensors are quantized using dynamic 8-bit quantization and de-quantized just-in-time for multiplication \n",
        "*    using gradient checkpoints to store one only activation per layer: using dramatically less memory at the cost of 30% slower training"
      ],
      "metadata": {
        "id": "P4gz5vhxw3ZQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import transformers\n",
        "import pandas as pd\n",
        "import torch\n",
        "import torch.nn.functional as F\n",
        "from torch import nn\n",
        "from torch.cuda.amp import custom_fwd, custom_bwd\n",
        "\n",
        "import accelerate\n",
        "\n",
        "from bitsandbytes.functional import quantize_blockwise, dequantize_blockwise\n",
        "from torch.utils.data import DataLoader\n",
        "from bitsandbytes.optim import Adam8bit"
      ],
      "metadata": {
        "id": "4rYnS8aiw5fP"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class FrozenBNBLinear(nn.Module):\n",
        "    def __init__(self, weight, absmax, code, bias=None):\n",
        "        assert isinstance(bias, nn.Parameter) or bias is None\n",
        "        super().__init__()\n",
        "        self.out_features, self.in_features = weight.shape\n",
        "        self.register_buffer(\"weight\", weight.requires_grad_(False))\n",
        "        self.register_buffer(\"absmax\", absmax.requires_grad_(False))\n",
        "        self.register_buffer(\"code\", code.requires_grad_(False))\n",
        "        self.adapter = None\n",
        "        self.bias = bias\n",
        " \n",
        "    def forward(self, input):\n",
        "        output = DequantizeAndLinear.apply(input, self.weight, self.absmax, self.code, self.bias)\n",
        "        if self.adapter:\n",
        "            output += self.adapter(input)\n",
        "        return output\n",
        " \n",
        "    @classmethod\n",
        "    def from_linear(cls, linear: nn.Linear) -> \"FrozenBNBLinear\":\n",
        "        weights_int8, state = quantize_blockise_lowmemory(linear.weight)\n",
        "        return cls(weights_int8, *state, linear.bias)\n",
        " \n",
        "    def __repr__(self):\n",
        "        return f\"{self.__class__.__name__}({self.in_features}, {self.out_features})\"\n",
        " \n",
        " \n",
        "class DequantizeAndLinear(torch.autograd.Function): \n",
        "    @staticmethod\n",
        "    @custom_fwd\n",
        "    def forward(ctx, input: torch.Tensor, weights_quantized: torch.ByteTensor,\n",
        "                absmax: torch.FloatTensor, code: torch.FloatTensor, bias: torch.FloatTensor):\n",
        "        weights_deq = dequantize_blockwise(weights_quantized, absmax=absmax, code=code)\n",
        "        ctx.save_for_backward(input, weights_quantized, absmax, code)\n",
        "        ctx._has_bias = bias is not None\n",
        "        return F.linear(input, weights_deq, bias).clone()\n",
        " \n",
        "    @staticmethod\n",
        "    @custom_bwd\n",
        "    def backward(ctx, grad_output: torch.Tensor):\n",
        "        assert not ctx.needs_input_grad[1] and not ctx.needs_input_grad[2] and not ctx.needs_input_grad[3]\n",
        "        input, weights_quantized, absmax, code = ctx.saved_tensors\n",
        "        # grad_output: [*batch, out_features]\n",
        "        weights_deq = dequantize_blockwise(weights_quantized, absmax=absmax, code=code)\n",
        "        grad_input = grad_output @ weights_deq\n",
        "        grad_bias = grad_output.flatten(0, -2).sum(dim=0) if ctx._has_bias else None\n",
        "        return grad_input, None, None, None, grad_bias\n",
        " \n",
        " \n",
        "class FrozenBNBEmbedding(nn.Module):\n",
        "    def __init__(self, weight, absmax, code):\n",
        "        super().__init__()\n",
        "        self.num_embeddings, self.embedding_dim = weight.shape\n",
        "        self.register_buffer(\"weight\", weight.requires_grad_(False))\n",
        "        self.register_buffer(\"absmax\", absmax.requires_grad_(False))\n",
        "        self.register_buffer(\"code\", code.requires_grad_(False))\n",
        "        self.adapter = None\n",
        " \n",
        "    def forward(self, input, **kwargs):\n",
        "        with torch.no_grad():\n",
        "            # note: both quantized weights and input indices are not differentiable\n",
        "            weight_deq = dequantize_blockwise(self.weight, absmax=self.absmax, code=self.code)\n",
        "            output = F.embedding(input, weight_deq, **kwargs)\n",
        "        if self.adapter:\n",
        "            output += self.adapter(input)\n",
        "        return output \n",
        " \n",
        "    @classmethod\n",
        "    def from_embedding(cls, embedding: nn.Embedding) -> \"FrozenBNBEmbedding\":\n",
        "        weights_int8, state = quantize_blockise_lowmemory(embedding.weight)\n",
        "        return cls(weights_int8, *state)\n",
        " \n",
        "    def __repr__(self):\n",
        "        return f\"{self.__class__.__name__}({self.num_embeddings}, {self.embedding_dim})\"\n",
        " \n",
        " \n",
        "def quantize_blockise_lowmemory(matrix: torch.Tensor, chunk_size: int = 2 ** 20):\n",
        "    assert chunk_size % 4096 == 0\n",
        "    code = None\n",
        "    chunks = []\n",
        "    absmaxes = []\n",
        "    flat_tensor = matrix.view(-1)\n",
        "    for i in range((matrix.numel() - 1) // chunk_size + 1):\n",
        "        input_chunk = flat_tensor[i * chunk_size: (i + 1) * chunk_size].clone()\n",
        "        quantized_chunk, (absmax_chunk, code) = quantize_blockwise(input_chunk, code=code)\n",
        "        chunks.append(quantized_chunk)\n",
        "        absmaxes.append(absmax_chunk)\n",
        " \n",
        "    matrix_i8 = torch.cat(chunks).reshape_as(matrix)\n",
        "    absmax = torch.cat(absmaxes)\n",
        "    return matrix_i8, (absmax, code)\n",
        " \n",
        " \n",
        "def convert_to_int8(model):\n",
        "    \"\"\"Convert linear and embedding modules to 8-bit with optional adapters\"\"\"\n",
        "    for module in list(model.modules()):\n",
        "        for name, child in module.named_children():\n",
        "            if isinstance(child, nn.Linear):\n",
        "                print(name, child)\n",
        "                setattr(\n",
        "                    module,\n",
        "                    name,\n",
        "                    FrozenBNBLinear(\n",
        "                        weight=torch.zeros(child.out_features, child.in_features, dtype=torch.uint8),\n",
        "                        absmax=torch.zeros((child.weight.numel() - 1) // 4096 + 1),\n",
        "                        code=torch.zeros(256),\n",
        "                        bias=child.bias,\n",
        "                    ),\n",
        "                )\n",
        "            elif isinstance(child, nn.Embedding):\n",
        "                setattr(\n",
        "                    module,\n",
        "                    name,\n",
        "                    FrozenBNBEmbedding(\n",
        "                        weight=torch.zeros(child.num_embeddings, child.embedding_dim, dtype=torch.uint8),\n",
        "                        absmax=torch.zeros((child.weight.numel() - 1) // 4096 + 1),\n",
        "                        code=torch.zeros(256),\n",
        "                    )\n",
        "                )"
      ],
      "metadata": {
        "id": "lzw5FWEHT25v"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class GPTJBlock(transformers.models.gptj.modeling_gptj.GPTJBlock):\n",
        "    def __init__(self, config):\n",
        "        super().__init__(config)\n",
        "\n",
        "        convert_to_int8(self.attn)\n",
        "        convert_to_int8(self.mlp)\n",
        "\n",
        "\n",
        "class GPTJModel(transformers.models.gptj.modeling_gptj.GPTJModel):\n",
        "    def __init__(self, config):\n",
        "        super().__init__(config)\n",
        "        convert_to_int8(self)\n",
        "        \n",
        "\n",
        "class GPTJForCausalLM(transformers.models.gptj.modeling_gptj.GPTJForCausalLM):\n",
        "    def __init__(self, config):\n",
        "        super().__init__(config)\n",
        "        convert_to_int8(self)\n",
        "\n",
        "\n",
        "transformers.models.gptj.modeling_gptj.GPTJBlock = GPTJBlock"
      ],
      "metadata": {
        "id": "qGIxiqjQvoG9"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Prompt Tuning Setup"
      ],
      "metadata": {
        "id": "eZT8VNLQODtr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Step 1: Define a task\n",
        "\n",
        "We attempt to prompt tune for headline generation. This means we input a news articles fulltext and expect a headline like output. \\\n",
        "OpenPrompt provides the ```InputExample``` class for task/data definition. \n",
        "* ```text_a``` is the input text of the data, in our case a news articles fulltext\n",
        "* ```tgt_text``` is the ground truth, in our case the news articles headline\n",
        "* ```guid``` is a unique identifier for each input example\n"
      ],
      "metadata": {
        "id": "zP3t-4EM9NFD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Option 1: Create small dataset manually (for testing purposes only)"
      ],
      "metadata": {
        "id": "zUehi7K44ikf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create datasets manually\n",
        "from openprompt.data_utils import InputExample\n",
        "dataset = [ \n",
        "    # For simplicity, there's only four examples\n",
        "    # text_a is the input text of the data\n",
        "    # tgt_text is the headline target text\n",
        "    # guid is a unique identifier for each input example.\n",
        "    InputExample(\n",
        "        guid = 1,\n",
        "        text_a = \"Noch vor wenigen Wochen war FTX eine der größten Kryptobörsen der Welt. Das Vermögen ihres Gründers Sam Bankman-Fried (SBF) wurde von Forbes und Bloomberg zeitweise mit über 26 Milliarden Dollar bewertet.\\n\\nFTX-Pleite: SBF als CEO zurückgetreten\\n\\nAnfang November 2022 ging dann alles ganz schnell: Innerhalb weniger Tage rutschte FTX in die Pleite – Milliarden Dollar wurden abgezogen, Rettungsversuche scheiterten, Bankman-Fried trat als Firmenchef zurück und meldete Insolvenz an.\\n\\nJetzt hat sich SBF zum ersten Mal seit Mitte November öffentlich zu Wort gemeldet. Von seinem Wohnsitz auf den Bahamas aus war er bei dem von der New York Times veranstalteten Dealbook Summit zugeschaltet.\\n\\nKomplett versagt, aber nicht betrogen?\\n\\nDabei zeigte sich der FTX-Pleitier reumütig: „Wir haben komplett versagt“, gestand Bankman-Fried ein. Zudem habe er nie versucht, jemanden zu betrügen. Er sei schockiert darüber, was aus dem florierenden Geschäft im vergangenen Monat geworden sei.\\n\\nWarum er, wie ihm vorgeworfen wird, Kundengelder in Milliardenhöhe von FTX abgezogen habe, um sein anderes Unternehmen, Alameda Research, zu retten, darauf ging Bankman-Fried nicht genauer ein.\\n\\nScharfe Kritik von SBF-Nachfolger bei FTX\\n\\nEin wenig Licht ins Dunkel bringt hier wohl eher der FTX-Chef John Ray, der im Zuge des Konkursverfahrens die SBF-Nachfolge antrat. Er habe noch nie in seiner Karriere „solch ein komplettes Versagen an Unternehmenskontrolle und so einen Mangel an vertrauenswürdigen Finanzinformationen erlebt“, wird Ray von Spiegel Online zitiert.\\n\\nWas in der Führungsriege bei FTX angestellt worden sei, sei „inakzeptabel“. Abzuwarten bleibt freilich, was bei den in den USA laufenden Ermittlungen und Sammelklagen gegen Bankman-Fried herauskommt.\\n\\nGenug Geld für US-Kunden von FTX da\\n\\nFür die FTX-Kund:innen in den USA hat Bankman-Fried derweil gute Nachrichten – wenn man dem trauen kann. Die US-Plattform sei solvent. FTX verfüge über genügend Geld, um die Kund:innen dort auszuzahlen.\\n\\nIhm selbst sei im Vergleich zu den finanziell rosigen Zeiten der vergangenen Monate nur noch wenig von seinem einstigen Vermögen geblieben. Auf einem Bankkonto habe er noch 100.000 Dollar, so SBF.\",\n",
        "        tgt_text = \"Sam Bankman-Fried: Nach FTX-Pleite sind von 26 Milliarden nur noch 100.000 Dollar übrig\"\n",
        "    ),\n",
        "    InputExample(\n",
        "        guid = 2,\n",
        "        text_a = \"Am 8. Dezember 2022 haben Bundesverkehrsminister Volker Wissing und Deutsche-Bahn-Infrastrukturvorstand Berthold Huber eine Vereinbarung für eine Milliardenfinanzierung zur Modernisierung der Bahnstrecken unterzeichnet. Gefördert werden soll das Projekt „Digitale Schiene Deutschland“ mit etwa 2,7 Milliarden Euro durch das Bundesministerium für Digitales und Verkehr, kurz BMDV.\\n\\nEine Milliarde Euro mehr\\n\\nEs gab bereits einen bestehenden Finanzierungsvertrag in Höhe von etwa 1,7 Milliarden Euro – mit der jüngsten Vereinbarung wurde er um eine Milliarde erhöht. Zusätzlich gab es für die Deutsche Bahn (DB) noch Finanzierungszusagen für weitere Projekte.\\n\\nMit dem Projekt „Digitale Schiene Deutschland“ sollen Strecken modernisiert werden, ohne dass Gleise ausgetauscht werden müssen. Ein Ziel: Langfristig sollen Strecken und Schienenknoten mit dem europäischen Zugbeeinflussungssystem ETCS, das steht für „European Train Control System“, ausgerüstet werden. Zudem sollen digitale Stellwerke eingebaut werden.\\n\\nMehr Kapazität durch Modernisierung\\n\\nDas soll laut Website des DB-Programms helfen, ohne neue Gleise mehr Züge auf die Strecken zu bekommen. Positiv auswirken soll es sich außerdem auf die Zuverlässigkeit im Güter- und Personentransport. Konkret soll die Modernisierung für 35 Prozent mehr Kapazität sorgen.\\n\\nGelder werden auf verschiedenen Strecken eingesetzt\\n\\nDas Geld wird unter anderem auf dem Güterverkehrskorridor Rhein-Alpen eingesetzt – da soll das europäischen Eisenbahnverkehrsleitsystem ERTMS, kurz für „European Rail Traffic Management System“, eingesetzt werden. Gelder in Höhe von etwa 307 Millionen Euro sind für Modernisierungen des Korridors Skandinavien-Mittelmeer geplant.\\n\\nEtwa 83 Millionen Euro sollen für den „Digitalen Knoten Stuttgart“ genutzt werden. Elf Millionen Euro sind dann noch für die Schnellfahrstrecke Köln-Frankfurt/Main einkalkuliert.\",\n",
        "        tgt_text = \"Geld für moderne Schienen: Deutsche Bahn und Verkehrsministerium erhöhen Milliardenfinanzierung\"\n",
        "    ),\n",
        "    InputExample(\n",
        "        guid = 3,\n",
        "        text_a = \"Dass ein Asteroideneinschlag auf der Erde verheerende Folgen haben könnte, dürfte allen klar sein. Vorgeführt zu bekommen, was genau ein solches Ereignis auslösen könnte, ist aber noch einmal beeindruckender. Genau dafür sorgt eine neu entwickelte Website.\\n\\nPunktgenaue Simulation\\n\\nDas Projekt nennt sich „Asteroid Launcher“ und wurde von Neal Agarwal entwickelt, der ein Händchen für faszinierende Datenprojekte hat.\\n\\nMan kann damit nicht nur punktgenau simulieren, wo der Asteroid einschlägt, sondern auch zahlreiche weitere Parameter eingeben: Aus welchem Material soll der Asteroid bestehen? Eisen? Stein? Oder etwa Gold? Welchen Durchmesser soll er haben, mit welcher Geschwindigkeit und in welchem Winkel soll er auf die Erde prallen?\\n\\nErdbeben und Tornados könnten auf Einschlag folgen\\n\\nDann berechnet die Seite, welche enorme Schäden er anrichten würde. Die Ergebnisse, die dabei präsentiert werden, sind erschütternd und grotesk, denn es werden nicht nur die Zahlen der vom Einschlag hervorgerufenen Todesopfer präsentiert, sondern auch die Folgeauswirkungen des Einschlages mit einkalkuliert.\\n\\nEin solcher könnte nämlich auch Erdbeben und Tornados zur Folge haben, ganz zu schweigen von der Schockwelle und dem Feuerball, den er auslösen würde.\\n\\nNasa könnte über das Projekt erfreut sein\\n\\nAgarwal hat sein neuestes Projekt auf Twitter lapidar mit den Worten „Bauen Sie ihren eigenen Asteroiden und lassen sie ihn auf die Erde rasen, um zu sehen, welche Auswirkungen er hat“ angekündigt.\\n\\nSo gruselig die Seite anmutet, die Nasa könnte darüber erfreut sein. Laut der Technologie-Plattform Futurism wird den Menschen dadurch vermittelt, welch verheerende Auswirkungen solch ein Szenario haben könnte. Und das könnte der Nasa in der öffentliche Wahrnehmung helfen, ihre große Investitionen, die sie zur Prävention solcher potenziellen Einschläge tätigt, zu legitimieren.\",\n",
        "        tgt_text = \"Asteroideneinschlag in deiner Stadt: Hier erfährst du, was passieren würde\"\n",
        "    ),\n",
        "    InputExample(\n",
        "        guid = 4,\n",
        "        text_a = \"„Deutschland soll führender Startup-Standort in Europa werden“, so haben es die Ampel-Parteien im November 2021 in ihren Koalitionsvertrag für die kommenden vier Jahre geschrieben. Zudem sollte sich Deutschland laut SPD, Grünen und FDP in ein klimaneutrales Industrieland transformieren: „Die Klimaschutzziele von Paris zu erreichen, hat für uns oberste Priorität. Klimaschutz sichert Freiheit, Gerechtigkeit und nachhaltigen Wohlstand“, hieß es von der Ampel.\\n\\nWas ist seitdem passiert?\\n\\nEin Jahr Koalitionsvertrag – so steht es um die Versprechen für die Startup-Szene\\n\\nEin Jahr Koalitionsvertrag – wir haben auf Basis eines Papiers des Startup-Verbands, das t3n vorliegt, einige der wichtigsten Änderungen der letzten zwölf Monate zusammengefasst.\\n\\n1. Startup-Strategie\\n\\nNoch in den ersten 100 Tagen hat die Bundesregierung, die im Koalitionsvertrag angekündigte „umfassende Startup-Strategie“ in Angriff genommen. Im Juli hat das Bundeskabinett dann erstmals überhaupt eine Startup-Strategie einer Bundesregierung verabschiedet.\\n\\n„Wir bewerten die schnelle Verabschiedung der Startup-Strategie, in deren Erarbeitung wir als Startup-Verband intensiv eingebunden wurden, insgesamt positiv. Damit hat sich die Ampel eine eigene To-Do-List erstellt. Nun kommt es auf eine schnelle Umsetzung an“, kommentiert der Startup-Verband.\\n\\nIn der Strategie werden für zehn Themenfelder Handlungsvorhaben definiert, die innerhalb der Legislaturperiode umgesetzt werden. Dazu gehören unter anderem die Themen Startup-Finanzierung, Diversität und Mitarbeiterbeteiligung.\\n\\n2. ESOP/Mitarbeiterkapitalbeteiligungen\\n\\n„Wir werden die Mitarbeiterkapitalbeteiligung für Start-ups attraktiver gestalten“, heißt es unmissverständlich im Koalitionsvertrag. Bereits kurz nach Amtsantritt Ende 2021 bekräftigte der dafür federführende Bundesfinanzminister Christian Lindner das Versprechen und weckte damit Erwartungen auf eine schnelle Umsetzung. Doch bislang gibt es hierzu noch keinen Entschluss.\\n\\nIm Sommer wurden Eckpunkte zum sogenannten „Zukunftsfinanzierungsgesetz“ vorgestellt, die jedoch laut Startup-Verband wichtige Bereiche, wie zum Beispiel die Vermeidung der sogenannten Dry-Income-Besteuerung, außer Acht ließen.\\n\\nMit einem neuen Referent:innenentwurf wird noch in diesem Jahr gerechnet. Das Gesetzgebungsverfahren soll nach den aktuellen Planungen in der ersten Jahreshälfte 2023 abgeschlossen werden, damit das Gesetz zum Ende nächsten Jahres in Kraft treten kann.\\n\\nDie Verbesserung der Rahmenbedingungen von Mitarbeiterkapitalbeteiligungen seien für Deutschland von grundlegender Bedeutung, um die Wettbewerbsfähigkeit als Startup-Standort zu sichern, so der Startup-Verband.\\n\\n3. Universitäten als Startup-Schmieden\\n\\nLaut Koalitionsvertrag sollten Universitäten eine wichtigere Rolle übernehmen, wenn es darum geht, junge Menschen für Gründungen zu begeistern. Für einen „echten Innovationsschub“ sollten Ausgründungen vorangetrieben und Mittel des Bundes „zur Schaffung einer Gründungsinfrastruktur für technologisches wie soziales Unternehmertum“ bereitgestellt werden.\\n\\nDie Startup-Strategie knüpft daran an. Danach soll unter anderem „in Ergänzung zu EXIST-Potentialen“ ein Leuchtturmwettbewerb für Entrepreneurship-Center aufgelegt werden. Die Konzeptphase soll 2023 beginnen und abgeschlossen werden. Hierfür wurde bereits eine Kommission einberufen, die zeitnah Ergebnisse präsentieren soll.\\n\\n„Für den Erfolg des Projektes wird entscheidend sein, die dafür erforderlichen Haushaltsmittel ab 2024 zur Verfügung zu stellen“, schreibt der Startup-Verband in seiner Einschätzung vom 7. Dezember. „Neben der Förderung von Hochschul-Ausgründungen sehen wir auch beim Transfer von außeruniversitären Forschungsinstituten viel ungenutztes Potential, das dringend genutzt werden sollte.“\\n\\nDazu gehöre auch, den IP-Transfer zu vereinfachen. Um das zu erreichen, konnte mit dem Start der Initiative „IP for virtual shares“ von SprinD ein erster wichtiger Beitrag geleistet werden.\\n\\n4. Cannabislegalisierung\\n\\nVon vielen mit großer Begeisterung wurde das Kapitel des Koalitionsvertrags aufgenommen, in dem es um die Legalisierung von Cannabis ging. „Die kontrollierte Abgabe von Cannabis an Erwachsene zu Genusszwecken in lizenzierten Geschäften“ sollte laut Koalitionsvertrag möglich gemacht werden.\\n\\nUm das zu erreichen, hat Gesundheitsminister Karl Lauterbach Ende Oktober 2022 ein Eckpunktepapier vorgelegt. „Die Drogenpolitik muss erneuert werden. Wir wollen den Cannabis-Konsum unter Gesundheitsaspekten reformieren“, sagte Lauterbach dazu. Cannabis und THC sollen künftig nicht mehr als Betäubungsmittel (BtM) eingestuft werden.\\n\\nIm Jahr 2021 schätzte der Medizin-Professor und FDP-Politiker Andrew Ullmann die Steuereinnahmen durch Cannabis auf circa 1,3 Milliarden Euro pro Jahr.\\n\\nBis es zur Cannabislegalisierung in Deutschland kommt, gibt es allerdings noch einige Hürden zu überwinden.\\n\\n„Das größte Fragezeichen beim vorliegenden Eckpunktepapier ist und bleibt die Interpretation durch die Europäische Kommission“, sagt Cannabis-Unternehmer Niklas Kouparanis im Podcast mit t3n.\\n\\n5. Unternehmensgründung in 24 Stunden\\n\\n\",\n",
        "        tgt_text = \"Ein Jahr Koalitionsvertrag – so steht es um die Versprechen für die Startup-Szene\"\n",
        "    ),\n",
        "]\n",
        "\n",
        "# Split: train 50%, test 50%\n",
        "train_dataset = dataset[0:2]\n",
        "test_dataset = dataset[2:4]"
      ],
      "metadata": {
        "id": "pbpRS6z84Tio"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Option 2: Load dataset from drive"
      ],
      "metadata": {
        "id": "plOsGS9L4d5Y"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "First mount drive to access dataset files stored in google drive."
      ],
      "metadata": {
        "id": "_Ybb2xB060Rn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Mount drive to load datasets from drive\n",
        "\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "id": "7XepFSEv2eaB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4b54ce95-7002-443a-a2f0-d1bf18f7ae7f"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Then load and adapt the dataset from drive. Adaptation is neccesary to transform data into  OpenPromts ```InputExample``` format. \\\n",
        "We used a dataset with 1000 datapoints and a 90/10 train/test split. \n",
        "\n",
        "*Note: If you try to reproduce this, you will need to store your own dataset in drive. \\\n",
        "Most likely yo will need to write your own ```adapt_dataset``` function too.*"
      ],
      "metadata": {
        "id": "1-5iSfp67CEJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Load and adapt datasets\n",
        "# split: train 90%, test 10%\n",
        "\n",
        "def adapt_dataset(path_to_file):\n",
        "  dataset = []\n",
        "  with open(path_to_file, \"r\") as f:\n",
        "    data = f.read()\n",
        "    examples = data.split(\"[Text]:\")\n",
        "    examples = examples[1:]\n",
        "    for guid, example in enumerate(examples):\n",
        "      fulltext, title = example.split(\"[Titel]:\")\n",
        "\n",
        "      dataset.append(InputExample(\n",
        "            guid = guid,\n",
        "            text_a = fulltext,\n",
        "            tgt_text = title.strip(\"\\n\\t\\t\\t\\t\\t\").strip(\"\\n\\\"\")\n",
        "        ))\n",
        "  return dataset\n",
        "  \n",
        "train_dataset = adapt_dataset(\"/content/drive/MyDrive/train.csv\")\n",
        "test_dataset = adapt_dataset(\"/content/drive/MyDrive/test.csv\")"
      ],
      "metadata": {
        "id": "KeAQzx6ahSg6"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Optional: Have a look at the data"
      ],
      "metadata": {
        "id": "tavPc3OY7aN0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Have a look at the data (optional)\n",
        "print(f\"Train Examples: {train_dataset[:3]}\")\n",
        "print(f\"Data points in train dataset: {len(train_dataset)}\")\n",
        "print(f\"Test Examples: {test_dataset[:3]}\")\n",
        "print(f\"Data points in test dataset: {len(test_dataset)}\")"
      ],
      "metadata": {
        "id": "Hvkju5prjXuZ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ad980398-6279-45cf-eefd-2f07dd71fde8"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train Examples: [{\n",
            "  \"guid\": 0,\n",
            "  \"label\": null,\n",
            "  \"meta\": {},\n",
            "  \"text_a\": \"Berlin. Die Klagen der Fans waren laut und zahlreich. Zu soft, zu beliebig sei die neue Single, hie\\u00df es in den Sozialen Netzwerken. F\\u00fcr eine Band wie Linkin Park nichts Neues.\\nOft schon haben die Amerikaner mit Stilen gespielt und Erwartungen entt\\u00e4uscht. Wohin steuert die Band mit ihrem neuen Album \\\"\\\"One More Light\\\"\\\" (Warner)?\\nHarte Gitarrenriffs, schn\\u00f6rkelloser Rock? All das sucht man auf dem siebten Studioalbum vergeblich. Mit dem Vorg\\u00e4nger \\\"\\\"The Hunting Party\\\"\\\" waren Linkin Park noch zu ihren Wurzeln zur\\u00fcckgekehrt; \\\"\\\"Heavy\\\"\\\", die erste Single, hingegen ist astreiner Radiopop. Chester Bennington singt schwerm\\u00fctig \\u00fcber seine Gedanken. Im hymnischen Refrain fragt er, warum alles so schwer sei. Dann \\u00fcbernimmt die 21-j\\u00e4hrige S\\u00e4ngerin Kiara als weiblicher Konterpart. Schema F.\\nDie Reaktionen der Fans? \\\"\\\"Was ein Witz, Linkin Park. Hoffentlich klingt so nicht das ganze Album\\\"\\\", schreibt ein Facebook-Nutzer. Ein anderer notiert, er habe das Lied achtmal geh\\u00f6rt - und es werde immer schlechter. Andere bedanken sich: \\\"\\\"Fehlender Wandel ist langweilig.\\\"\\\"\\nWer kann es schon allen recht machen? Mit mehr als 61 Millionen Likes ist Linkin Park laut Label die Band mit den meisten Facebook-Fans. Die Kalifornier f\\u00fcllen Stadien und spielen Headlinershows vor Zehntausenden. Ihr Deb\\u00fctalbum \\\"\\\"Hybrid Theory\\\"\\\" ist das meistverkaufte Deb\\u00fct des Jahrhunderts. Und jetzt: \\\"\\\"Heavy\\\"\\\". So hei\\u00dft der erste Singlevorbote, der im Februar f\\u00fcr reichlich Unmut sorgte.\\nDie Band selbst sieht den Frust der Fans gelassen. Fr\\u00fcher h\\u00e4tte er sich \\u00fcber jemanden aufgeregt, der seine Musik nicht mag. \\\"\\\"Ich h\\u00e4tte es pers\\u00f6nlich genommen\\\"\\\", sagt Bennington im Interview der Deutschen Presse-Agentur in Berlin. Heute mache er sich keine Gedanken mehr. Sein Ziel sei nie gewesen, m\\u00f6glichst viele Platten zu verkaufen oder Stadien zu f\\u00fcllen. \\\"\\\"Unsere Absicht ist, Musik zu machen, die wir lieben.\\\"\\\"\\nAuf \\\"\\\"One More Light\\\"\\\" klingt diese Musik ziemlich glattgeb\\u00fcgelt. Die elektronischen Experimente vergangener Tage sind genauso Geschichte wie der Nu Metal aus den Anfangszeiten. Auch der charakteristische Rap von Mike Shinoda ist eine Seltenheit. Balladen (\\\"\\\"One More Light\\\"\\\") stehen neben eing\\u00e4ngigem Poprock (\\\"\\\"Talking To Myself\\\"\\\") oder gef\\u00fchliger Akustikgitarre (\\\"\\\"Sharp Edges\\\"\\\"). Maroon 5, Justin Bieber und andere Popsternchen lassen gar nicht mal so dezent gr\\u00fc\\u00dfen.\\nErstmals hat die Band die Aufnahmen nicht mit musikalischen Proben begonnen, sondern mit Gespr\\u00e4chen. \\\"\\\"Wor\\u00fcber mache ich mir Gedanken? Was ist f\\u00fcr mich gerade am wichtigsten?\\\"\\\", seien Fragen dieser Gespr\\u00e4che gewesen, aus denen sich die Ideen f\\u00fcr neue Lieder ergeben h\\u00e4tten: Depression und Tod, aber eben auch das manchmal banale Familienleben mit Kindern. Hast du die Z\\u00e4hne geputzt? Wo ist dein Rucksack? \\\"\\\"Das ist ein typischer Tag, wenn du Kinder hast\\\"\\\", sagt Bennington. Aber dies seien auch Themen, die vor 15 Jahren niemand in der Band verstanden h\\u00e4tte. \\\"\\\"Es kommt darauf an, wo wir im Leben stehen.\\\"\\\"\\nDie Linkin-Park-Mitglieder sind \\u00e4lter geworden - und mit ihnen ihre Fans. Einige haben sich innerhalb der vergangenen 20 Jahre abgewendet. Allerdings kommen mit jedem Album auch neue Fans hinzu. \\\"\\\"Und bei den Konzerten kommen alle zusammen\\\"\\\", sagt S\\u00e4nger und Produzent Shinoda.\\n\",\n",
            "  \"text_b\": \"\",\n",
            "  \"tgt_text\": \"Linkin Park:\\u00a0Radio-Pop statt harter Gitarrenrock\"\n",
            "}\n",
            ", {\n",
            "  \"guid\": 1,\n",
            "  \"label\": null,\n",
            "  \"meta\": {},\n",
            "  \"text_a\": \"Der dreij\\u00e4hrige Klaus, ein Blondschopf mit gro\\u00dfen traurigen Augen, kaute stundenlang, wenn er etwas essen sollte. Seiner 23j\\u00e4hrigen Stiefmutter gingen die Nerven durch. Sie sehlug und. mi\\u00dfhandelte das schwierige l\\u00f6nd. Ein Hainjwiser.Ajta&ge*. rieht sprich die Hausfrau Gisela N. gfrte\\u00f6S ? zwar schuldig, T^m\\u00c4ft steh *be* die Verurteilung zu einer Geldstrafe von 2400 Mark bei einer Bew\\u00e4hrungszeit von zwei Jahren vor.\\nMit blauen Flecken am ganzen K\\u00f6rper war Klaus im M\\u00e4rz dieses Jahres in ein Hamburger Krankenhaus eingeliefert worden. Die Angeklagte: \\\"\\\"Mir war'\\u00f6fter mal die Hand ausgerutscht, weil Klaus nicht essen wollte.\\\"\\\" Das Kind war bereits vorher im Krankenhaus gewesen und hatte nur Tabletten bekommen. Und eine 33j\\u00e4hrige Sozialp\\u00e4dagogin, die die Familie betreute, hatte keine Ratschl\\u00e4ge gegeben.\\nDas Motiv der Sozialp\\u00e4dagogin: Es ist keine qualifizierte Sozialarbeit, mit Ratschl\\u00e4gen zu helfen. Der angeklagten Stiefmutter wurde das Fehlen \\u00e4rztlichen und menschlichen Rates strafmildernd angerechnet, frk\\n\",\n",
            "  \"text_b\": \"\",\n",
            "  \"tgt_text\": \"Kind mi\\u00dfhandelt: Bew\\u00e4hrung\"\n",
            "}\n",
            ", {\n",
            "  \"guid\": 2,\n",
            "  \"label\": null,\n",
            "  \"meta\": {},\n",
            "  \"text_a\": \"19.08.78\\nDer Senat hilft wieder heizen\\nAuch in diesem Jahr gew\\u00e4hrt der Senat f\\u00fcr bed\\u00fcrftige alte Mitb\\u00fcrger wieder Heizungs- und Weihnachtsbeihilfe, vorausgesetzt, das Einkommen \\u00fcbersteigt den Bedarfssatz der Sozialhilfe nicht wesentlich. Damit die Beihilfe-Empf\\u00e4nger rechtzeitig Brennstoff einkaufen k\\u00f6nnen, wird die Heizungshilfe f\\u00fcr den n\\u00e4chsten Winter schon im September/Oktober in einer Summe ausbezahlt. Die Weihnachtsbeihilfe gibt es Anfang Dezember. Antr\\u00e4ge sollen zwischen dem 15. und 30. September montags, donnerstags und freitags von 8 bis 14 Uhr in ihrer Sozialdienststelle gestellt werden. Unterlagen \\u00fcber die Einkommensverh\\u00e4ltnisse aller Familienangeh\\u00f6rigen sowie \\u00fcber die H\\u00f6he der Miete m\\u00fcssen vorgelegt werden. Sozialhilfe-Empf\\u00e4nger bekommen die Heizungs- und Weihnachtsbeihilfe ohne Antrag.\\n\",\n",
            "  \"text_b\": \"\",\n",
            "  \"tgt_text\": \"Der Senat hilft wieder heizen\"\n",
            "}\n",
            "]\n",
            "Data points in train dataset: 990\n",
            "Test Examples: [{\n",
            "  \"guid\": 0,\n",
            "  \"label\": null,\n",
            "  \"meta\": {},\n",
            "  \"text_a\": \"Auf deutschen Stra\\u00dfen, Schienen und Wasserwegen liegt vieles im Argen. Die Infrastruktur des Landes ist in einem schlechtem Zustand. Schleswig-Holsteins Ministerpr\\u00e4sident Torsten Albig (SPD) hat die Diskussion dar\\u00fcber mit seiner Forderung nach einer Sonderabgabe zur Sanierung der Verkehrswege neu entfacht. Professor Thomas Straubhaar, 56, Direktor des Hamburgischen WeltWirtschaftsInstituts (HWWI), h\\u00e4lt diesen Vorsto\\u00df nicht f\\u00fcr umfassend genug, um Stra\\u00dfen, Schienen und Wasserstra\\u00dfen ausreichend zu erneuern. Im Abendblatt-Interview erl\\u00e4utert der renommierte \\u00d6konom seine Vorschl\\u00e4ge.\\nHamburger Abendblatt: Herr Professor Straubhaar, Schleswig-Holsteins Ministerpr\\u00e4sident Torsten Albig hat mit seinem Vorschlag, einen Sonderfonds f\\u00fcr die Sanierung der deutschen Verkehrswege einzurichten, eine heftige Debatte ausgel\\u00f6st und viel Kritik bekommen. Ist sein Vorschlag abwegig?\\nThomas Straubhaar: Im Gegenteil, Herr Albig hat recht. Der Zustand der Verkehrsinfrastruktur in Deutschland, die einst die beste der Welt war, wird immer schlechter. Leistungsf\\u00e4hige Verkehrswege \\u2013 Stra\\u00dfen, Schienen, Luft- und Wasserwege \\u2013 sind aber f\\u00fcr ein Industrieland wie Deutschland existenziell wichtig. Die Debatte um die Erneuerung der Infrastruktur geh\\u00f6rt deshalb in die Mitte der Gesellschaft.\\nDie Gro\\u00dfe Koalition will in dieser Legislaturperiode zu den j\\u00e4hrlichen Mitteln f\\u00fcnf Milliarden Euro zus\\u00e4tzlich f\\u00fcr die Sanierung der Verkehrswege zur Verf\\u00fcgung stellen. Ist das zu wenig?\\nStraubhaar: Zu wenig ist gar kein Ausdruck. An diese Summe m\\u00fcsste mindestens eine Null angeh\\u00e4ngt werden. Und selbst bei einem Volumen von 50 Milliarden Euro w\\u00e4re ich nicht sicher, ob das ausreicht. Seinen heutigen Wohlstand verdankt Deutschland auch der Tatsache, dass es mit dem Aufbau der Eisenbahn und mit der Erfindung des Automobils weltweite Standards setzte \\u2013 weil es die neuen Verkehrstr\\u00e4ger auch selbst im gro\\u00dfen Stil industriell herstellte und bis heute weltweit mit gro\\u00dfem Erfolg verkauft. Deutschland liegt als Transitland mitten in einem europ\\u00e4ischen Binnenmarkt. Logistik- und damit Transportwirtschaft genauso wie die Industrie laufen auf Hochtouren und sorgen f\\u00fcr Millionen von Jobs in Deutschland. Die Grundlage daf\\u00fcr sind unsere Verkehrswege, und die sind vielerorts in einem bedenklichen Zustand, man nehme als Beispiel nur die maroden Schleusenanlagen des Nord-Ostsee-Kanals in Brunsb\\u00fcttel.\\nWie soll der deutsche Staat einen zweistelligen Milliardenbetrag f\\u00fcr die Sanierung der Verkehrswege finanzieren?\\nStraubhaar: Die Diskussion um die Infrastruktur wird v\\u00f6llig \\u00fcberlagert von der ebenso notwendigen Debatte zur Sanierung der \\u00f6ffentlichen Haushalte. Diese beiden Themen m\\u00fcssen getrennt betrachtet werden. F\\u00fcr Stra\\u00dfen, Schienen, Wasserwege brauchen wir \\u2013 wie es Ministerpr\\u00e4sident Albig fordert \\u2013 einen Sonderfonds, der aus staatlichen Mitteln ebenso gespeist wird wie aus dem Kapital privater Anleger. Wichtig ist, dass die Basis f\\u00fcr die Finanzierung eines solchen Fonds so breit wie m\\u00f6glich ist. Nicht nur die unmittelbaren Nutzer der Infrastruktur, die Ministerpr\\u00e4sident Albig vorrangig im Blick hat, m\\u00fcssen einen Beitrag leisten, sondern alle Staatsb\\u00fcrger in Deutschland. Denn eine schlechte Infrastruktur schadet dem gesamten Land. Obendrein muss ein solcher Fonds, etwa durch Anleihen, offen sein f\\u00fcr die Akquisition von privatem Kapital in gro\\u00dfen Stil.\\nVerb\\u00e4nde wie der ADAC verweisen darauf, dass \\u00fcber Mineral\\u00f6l- oder Kfz-Steuern und Lkw-Maut j\\u00e4hrlich bereits gut 50 bis 60 Milliarden Euro aus dem Automobilverkehr an den Bund flie\\u00dfen, dass dieses Geld aber nur zum Teil f\\u00fcr die Sanierung verwendet wird.\\nStraubhaar: Das zeigt ja das Problem. Wer aus dem laufenden Haushalt mehr Geld f\\u00fcr die Verkehrswege haben will, muss zun\\u00e4chst mehr auf Effizienz achten und nicht so schlampig vorgehen wie beim Berliner Flughafen, und er darf das Geld nicht verschwenden wie etwa bei zu vielen zu kleinen Regionalflugh\\u00e4fen, die schlicht nicht wirtschaftlich betrieben werden k\\u00f6nnen. Zudem muss er an anderer Stelle k\\u00fcrzen, im Zweifel bei den Sozialabgaben. Das gilt als extrem unpopul\\u00e4r. Deshalb wird sich \\u2013 gerade in der jetzigen Gro\\u00dfen Koalition \\u2013 daf\\u00fcr keine Mehrheit finden. Die Infrastruktur muss es aus den Diskussionszyklen einer Legislaturperiode herausgel\\u00f6st werden.\\nBund und L\\u00e4nder d\\u00fcrfen k\\u00fcnftig keine zus\\u00e4tzlichen Schulden mehr aufnehmen. Woher soll das Geld kommen?\\nStraubhaar: Gro\\u00dfe gesellschaftliche Themen wie die Infrastruktur oder die Bildung sind nicht das Problem unserer Staatsverschuldung. Es sind vor allem die Sozialausgaben. Klar ist doch: Investitionen in die Infrastruktur erbringen f\\u00fcr das gesamte Land eine weit h\\u00f6here Rendite, als die Kapitalkosten etwa aus Anleihen ausmachen, die dagegen st\\u00fcnden. Wann, wenn nicht jetzt in einer Phase historisch niedriger Zinsen, sollte der Staat sich daf\\u00fcr am Kapitalmarkt bedienen? Wir m\\u00fcssen weg vom konsumptiven zur\\u00fcck zum investiven Staat. Das aber \\u2013 man muss es in aller Klarheit sagen \\u2013 bedeutet, dass man an vielen Stellen den Sozialstaat beschr\\u00e4nken muss und ihn nicht, wie jetzt mit der Rente mit 63, weiter ausbauen sollte.\\nSie glauben, dass Privatanleger ihr Geld in Stra\\u00dfen investieren?\\nStraubhaar: Selbstverst\\u00e4ndlich. Weltweit sucht Kapital nach attraktiven Anlagem\\u00f6glichkeiten, vom deutschen Sparer bis zum US-Pensionsfonds. Wer \\u00fcber einen Zeitraum von 20 oder 30 Jahren eine staatlich abgesicherte Verzinsung von, sagen wir, drei Prozent bekommen kann, wird diese Gelegenheit im heutigen Zinsumfeld nutzen.\\nDie Transportwirtschaft fordert gerade f\\u00fcr Hamburg h\\u00f6here \\u00f6ffentliche Investitionen f\\u00fcr Sanierung und Neubau.\\nStraubhaar: Hamburg ist in besonderer Weise vom Verschlei\\u00df der Verkehrswege betroffen. Die Stadt ist \\u2013 durch den Hafen und die Industrie \\u2013 eines der Zentren auch f\\u00fcr den Schwerverkehr in ganz Deutschland. Verschlissene Stra\\u00dfen und Br\\u00fccken belasten die Situation in der Stadt ebenso wie die Verz\\u00f6gerung bei Neubauprojekten, etwa der s\\u00fcdlichen oder der n\\u00f6rdlichen Elbquerung.\\nMacht der Hamburger Senat genug?\\nStraubhaar: Der Senat besch\\u00e4ftigt sich intensiv mit dem Thema, er kann aber nicht kurzfristig l\\u00f6sen, was langfristig vers\\u00e4umt wurde, zumal Verkehrswege wie die Autobahnen in der Hoheit des Bundes liegen. Hamburg braucht so etwas wie einen Masterplan Verkehr, der auf der Einbettung der Metropolregion in Nordeuropa basiert, der neben dem Hafen, neben Stra\\u00dfen und Schienen auch die Rolle der Luftverkehrsindustrie einbezieht.\\nIn der Hamburger City herrscht gef\\u00fchlt Dauerstau, weil sich immer mehr Lkw in den Pkw-Verkehr dr\\u00e4ngen.\\nStraubhaar: Ja, beim Baustellenmanagement besteht definitiv noch eine Menge Verbesserungspotenzial! Interessant finde ich, dass viele j\\u00fcngere Menschen auch in Hamburg kein Interesse mehr an einem eigenen Auto haben, sondern an Mobilit\\u00e4t, an der Verbindung zwischen den Verkehrstr\\u00e4gern. Der \\u00f6ffentliche Personennahverkehr in Hamburg ist zunehmend popul\\u00e4r. Deswegen greift es vielleicht zu kurz, mal \\u00fcber Busbeschleunigungsspuren zu diskutieren, mal \\u00fcber eine Stadtbahn oder \\u00fcber eine neue U-Bahn-Linie. Wichtiger w\\u00e4re doch eine Debatte dar\\u00fcber, wie der \\u00f6ffentliche Verkehrsverbund in und um Hamburg im Jahr 2025 als Gesamtsystem aussehen k\\u00f6nnte.\\nDie Elektromobilit\\u00e4t oder der Fahrradverkehr sind in Hamburg bislang nicht besonders fortschrittlich organisiert.\\nStraubhaar: Wenn Hamburg eine leistungsf\\u00e4hige Infrastruktur etwa f\\u00fcr das gro\\u00dfe Zukunftsthema Elektromobilit\\u00e4t schaffen w\\u00fcrde \\u2013 mit mehr und besseren Ladestationen, Verbund- und Leitsystemen \\u2013, wenn sie auch wegweisende Konzepte f\\u00fcr den \\u00f6ffentlichen Nahverkehr entwickelte, w\\u00fcrde das einen wirtschaftlichen und sozialen Nutzen f\\u00fcr Jahrzehnte und einen Wissensvorsprung f\\u00fcr eine moderne Verkehrsindustrie mit entsprechenden Besch\\u00e4ftigungseffekten schaffen. So wie fr\\u00fcher mit der Eisenbahn- und Autoindustrie.\\n\",\n",
            "  \"text_b\": \"\",\n",
            "  \"tgt_text\": \"Straubhaar: \\u201eWeniger Sozialstaat, daf\\u00fcr bessere Stra\\u00dfen\\u201c\"\n",
            "}\n",
            ", {\n",
            "  \"guid\": 1,\n",
            "  \"label\": null,\n",
            "  \"meta\": {},\n",
            "  \"text_a\": \"W\\\"\\\"tte\\\"\\\" da\\u00df . . .? (Sbd. II)\\n-\\nJetzt scheint er seine Show endlich in den Griff zu bekommen: Frank Elstner wirkte konzentrierter als sonst, begann mit den Wetten \\\"\\\"schon\\\"\\\" im ersten Drittel der Sendezeit und \\u00fcberzog auch nur ein paar Minuten. Und Mogelei durfte ihm diesmal auch niemand vorwerfen wollen, denn die meisten Wetten endeten mit einer \\u00dcberraschung. Problematisch scheint nach wie vor die Au\\u00dfen\\u00fcbertragung, die diesmal in Hameln optisch v\\u00f6llig danebenging. Das allerdings geh\\u00f6rt wohl zu den Pannen, die bei jeder Livesendung einkalkuliert werden m\\u00fcssen und mit denen Frank Elstner normalerweise sehr souver\\u00e4n fertig wird. Wollen wir ihm w\\u00fcnschen, da\\u00df seine Show noch nicht sobald am Mangel von originellen und vor allem fernsehwirksamen Wetten eingeht BRIGITTE EHRICH\\nIm Schlaraffenland (Stg. II)\\nDie Handschrift des entsetzlich fr\\u00fch verstorbenen Regisseurs Fritz Umgelter war auch hier unverkennbar: Die lockere Genauigkeit, mit der er Szenen dirigierte wie jene, in der die gro\\u00dfen Herren rauchend sich am Roulette der Amazonen-P\\u00fcppchen vergn\\u00fcgen. Oder die Sicherheit, mit der er Milieu schaffte. Oder das Unterhaltliche, das er so souver\\u00e4n beherrschte. Dem Vergn\\u00fcgen der Zuschauer war Umgelters Arbeit immer gewidmet, vom \\\"\\\"Gr\\u00fcnen Strand der Spree\\\"\\\" bis zum \\\"\\\"Trenck\\\"\\\" und dar\\u00fcber hinaus. Mit sicherem Griff f\\u00fcr das, was sich anst\\u00e4ndig verfilmen l\\u00e4\\u00dft, w\\u00e4hlte Umgelter aus Heinrich Manns Werk \\\"\\\"Im Schlaraffenland\\\"\\\" aus. Gewi\\u00df nicht den bedeutendsten Roman des gro\\u00dfen Erz\\u00e4hlers, aber gerade deshalb den am ehesten bildlich umsetzbaren. Es sei dahingestellt, ob die\\nscharfe Gesellschaftskritik des Autors von Umgelter in den Film eingebracht wurde und ob er alle Darsteller gen\\u00fcgend am Z\\u00fcgel hielt. In jedem Fall aber hat er den Roman in fl\\u00fcssiger Weise einem gro\\u00dfen Publikum empfohlen. PETER DREESSEN\\nNorton (Stg. II)\\n\\\"\\\"Die Steine Danzigs erz\\u00e4hlen die Geschichte, die man nicht wegwischen kann\\\"\\\", stellt am Schlu\\u00df der Reihe ein polnischer Danziger fest. Die Film-Trilogie schilderte jedoch vorwiegend das Martyrium polnischer B\\u00fcrger Danzigs. Diese Gewichtung historischer Tatsachen ist anfechtbar. Denn nach dem 1. Weltkrieg, als der Versailler Vertrag Danzig zur \\\"\\\"Freien Stadt\\\"\\\" degradierte, war Danzig rein deutsch und zu Beginn des 2. Weltkrieges immerhin noch zu 93 Prozent deutsch. Die Vertreibung von Hunderttausenden von Deutschen aus der verw\\u00fcsteten Stadt wurde jedoch nur beil\\u00e4ufig als \\\"\\\"furchtbares Schicksal\\\"\\\" erw\\u00e4hnt. Die nachfolgende Diskussion zwischen zwei Deutschen und zwei Polen best\\u00e4tigte die unvereinbaren Standpunkte. Fragen nach polnischen Ausschreitungen an Deutschen w\\u00e4hrend des \\\"\\\"Bromberger Blutsonntages\\\"\\\" blieben unbeantwortet. Auch die polnische Auffassung, eigentlich k\\u00f6nne man nach 1945 nicht von einer Vertreibung der Deutschen sprechen, ist geschichtlich unhaltbar. \\\"\\\"Die Wunden vernarben, aber wir f\\u00fchlen sie heute noch\\\"\\\", sagte einmal der polnische Danziger zu seinem heimatvertriebenen deutschen Freund. So ist es. An diesen \\\"\\\"Narben\\\"\\\" hat das Versalller. Diktat viel schuld. So bleibt die Frage, die der Film stellt, offen? K\\u00f6nnen Menschen \\u00fcberhaupt aus der Geschichte lernen?\\nIn der zweiten Folge l\\u00e4uft heute die ZDF-Fr\\u00fchabendserie \\\"\\\"Der Fuchs von \\u00d6velg\\u00f6nne\\\"\\\" (18.20 Uhr). Da sind nat\\u00fcrlich die Hamburger aufgerufen mitzuerleben, wie ein sturmerprobter Barkassenf\\u00fchrer bei \\u00fcberraschenden Untiefen man\\u00f6vriert. Herbert Fleischmann und Diana K\\u00f6rner (Foto) sind dabei.\\n\",\n",
            "  \"text_b\": \"\",\n",
            "  \"tgt_text\": \"GESEHEN AM WOCHENENDE\"\n",
            "}\n",
            ", {\n",
            "  \"guid\": 2,\n",
            "  \"label\": null,\n",
            "  \"meta\": {},\n",
            "  \"text_a\": \"Mit 190 km/h und mehr hatte Hurrikan \\\"\\\"Jeanne\\\"\\\" in der Nacht vom 18. auf den 19. September Teile Floridas gepr\\u00fcgelt. Er war das - hoffentlich - letzte von f\\u00fcnf solcher Monster, die Hunderte Menschenleben forderten und schwerste Verw\\u00fcstungen anrichteten. Auch die Reiseindustrie wurde durcheinandergewirbelt.\\n\\\"\\\"Da die Wetterbedingungen zahlreiche Versp\\u00e4tungen und Streichungen von Fl\\u00fcgen verursachen, rufen Sie bitte die Nr. xyz an, um den Flugstatus abzufragen.\\\"\\\" Dieser Hinweis, typisch f\\u00fcr die Lage, stand bis vor wenigen Tagen auf der Internet-Homepage der amerikanischen United Airlines. Inzwischen wird auch dort der Flugplan wieder eingehalten.\\nRiesengl\\u00fcck hatte die Lufthansa, die Miami t\\u00e4glich von Frankfurt/M. (im Winter auch von M\\u00fcnchen) aus anfliegt. \\\"\\\"Beim Hurrikan ,Frances', der vom 25. August bis 7. September tobte, ist unser Flug kurz vorher in Miami gelandet. Der n\\u00e4chste war zu einem Zeitpunkt angesetzt, da ,Frances' sich abgeschw\\u00e4cht hatte und Miami Airport wieder \\u00f6ffnete. Wir konnten also den Flugplan einhalten\\\"\\\", berichtet Lufthansa-Sprecher Thomas Jachnow.\\nDie Condor mu\\u00dfte zwei Fl\\u00fcge nach Orlando und Fort Myers verschieben. Steffen Milchsack, Sprecher der Thomas Cook AG: \\\"\\\"Viele unserer Passagiere haben Apartments oder H\\u00e4user in Florida, sie buchen bei uns ,Condor individuell' und einen Mietwagen. Sie sind es gewohnt, flexibel zu sein. Deshalb gab es keine Stornierungen.\\\"\\\"\\nRelativ entspannt blieb auch Pierre de la Motte bei der LTU: \\\"\\\"Alle Fl\\u00fcge nach Miami (zwei pro Woche), Fort Myers (drei pro Woche) und Orlando (einmal) konnten stattfinden.\\\"\\\" Anders die Delta, einer der f\\u00fchrenden Akteure der US-Airlinebranche mit 104 Millionen Passagieren weltweit im Jahr 2003: Sie mu\\u00dfte wegen aller Wirbelst\\u00fcrme der letzten Wochen rund 3000 Fl\\u00fcge streichen. Siebzig Prozent der Ausf\\u00e4lle betrafen Florida. Allein \\\"\\\"Jeanne\\\"\\\" verursachte 900 Flugannullierungen.\\nMarion Krimmer, Delta-Sprecherin in Deutschland: \\\"\\\"Im Delta-Hauptquartier in Atlanta habe ich in der meteorologischen Abteilung den Verlauf der Unwetter mit verfolgen k\\u00f6nnen. Da sich deren mutma\\u00dflicher Weg auf den Monitoren relativ lange vorher abzeichnet, konnte Delta entsprechend reagieren. Teuer wurden f\\u00fcr uns die \\u00dcbernachtungen der vielen Passagiere, die blockiert waren.\\\"\\\"\\nAuch bei den Reiseveranstaltern, die stark auf die USA, die Karibik und Kanada setzen, gl\\u00fchten die Dr\\u00e4hte. Canusa, f\\u00fchrender deutscher Spezialist f\\u00fcr Nordamerika, mu\\u00dfte etwa zwei Prozent seiner Urlauber nach Kalifornien oder South Carolina umbuchen. Betroffen war vor allem, wer auf den Inseln Sanibel und Captiva an Floridas Golfk\\u00fcste gebucht hatte. Dort haben die Unwetter so schwer gew\\u00fctet, da\\u00df etliche Hotels monatelang ausfallen. Canusa-Gesch\\u00e4ftsf\\u00fchrer Tilo Krause-D\\u00fcnow sagt: \\\"\\\"Im Sommer standen wir f\\u00fcr Florida gegen\\u00fcber 2003 bei einem Buchungsplus von 80 Prozent. Der gesamte s\\u00fcdliche Bereich, auch die Keys-Inseln, blieb fast ganz verschont. Der Florida-Panhandle wurde allerdings schwer verw\\u00fcstet. Dort oben ist die K\\u00fcste bis hin nach New Orleans extrem betroffen. Wer in diese Region wollte, dem haben wir Alternativen angeboten.\\\"\\\" Dagegen sind alle Themenparks um Orlando wie Disney und Sea World wieder ge\\u00f6ffnet.\\nAntje G\\u00fcnther vom Deutschen Reiseb\\u00fcro DER, das Reisen in die USA haupts\\u00e4chlich nach dem Baukastensystem verkauft, mu\\u00dfte einigen der rund 100 G\\u00e4ste, die w\\u00e4hrend \\\"\\\"Jeanne\\\"\\\" in Florida unterwegs waren, die Schlie\\u00dfung des \\\"\\\"South Seas Resort\\\"\\\" auf Captiva Island und des \\\"\\\"West Wind Inn\\\"\\\" auf Sanibel mitteilen. \\\"\\\"Zu Schaden gekommen ist niemand, wir haben aber einige Florida-Rundreisen umleiten m\\u00fcssen. Inzwischen l\\u00e4uft fast alles wieder nach Plan.\\\"\\\"\\nHaben wir Ihre Reiselust geweckt? Aktuelle Angebote finden Sie auf unserem Partnerportal www.globista.de!\\n\",\n",
            "  \"text_b\": \"\",\n",
            "  \"tgt_text\": \"Kr\\u00e4ftig durchgesch\\u00fcttelt\"\n",
            "}\n",
            "]\n",
            "Data points in test dataset: 10\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Step 2: Obtain a Pretrained Language Model (PML). \n",
        "\n",
        "The GPT-J model and tokenizer are not in the OpenPrompt default configuration. So we load them separately adapting from this [Tutorial](https://github.com/thunlp/OpenPrompt/blob/main/tutorial/6.1_chinese_dataset_uer_t5.py)."
      ],
      "metadata": {
        "id": "R6xx_Ezl4r6k"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Here the combination of tokenizer and model is not in the default configurations of openprompt.\n",
        "# So we load the tokenizer and model separately.\n",
        "from transformers import AutoTokenizer\n",
        "from openprompt.plms.lm import LMTokenizerWrapper\n",
        "\n",
        "# We use the model configuration and the tokenizer of GPT-J-6B and set pad_token as eos_token.\n",
        "config = transformers.GPTJConfig.from_pretrained(\"EleutherAI/gpt-j-6B\")\n",
        "config.pad_token_id = config.eos_token_id\n",
        "# Parameter setting use_fast=False is neccessary for usage with OpenPromt Templates later\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"EleutherAI/gpt-j-6B\", add_special_tokens=False, use_fast=False)\n",
        "tokenizer.pad_token = config.pad_token_id\n",
        "\n",
        "# We load the pre-trained gpt-j-6b with 8-bit weights from huggingface. \n",
        "# To reduce the peak RAM usage, we add the argument, low_cpu_mem_usage=True to from_pretrained.\n",
        "model = GPTJForCausalLM.from_pretrained(\"hivemind/gpt-j-6B-8bit\", low_cpu_mem_usage=True)\n",
        "\n",
        "# Set tokenizer wrapper\n",
        "tokenizer_wrapper = LMTokenizerWrapper(max_seq_length=2048, tokenizer=tokenizer)"
      ],
      "metadata": {
        "id": "KGcHNiP8-FVP",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000,
          "referenced_widgets": [
            "da60012bab304a4699db5dfc0b9b065e",
            "0ff758dabc7b4dd7b82e1af62cba6ed7",
            "30746c3f34354794a6f4c4185e0ef17f",
            "4e8e715c770f456d89e5034e039f739d",
            "df74ae246b35418db7156a32293bbc21",
            "5df02a3f9bba4f7c902b5f0dccf59daa",
            "90b85c8bfa7a40a582c2e0ded1324242",
            "364746915c8a40e883f5bf914cfd80d5",
            "160152c938b1456daa6a78b0cb2a775b",
            "e5eb6257ffd64aca9eafbc56bcba8a99",
            "b799e82c985642f2ab13b61b43e6a080",
            "36137d130dc94499ac9d13cd10912b92",
            "1b89d549a1e2407cae34b4174c29db82",
            "c8b5aba865bb4caaaa1d5d6fcd8b3177",
            "3e7370d47fdd4da5acb84e535b314467",
            "2ea8015398fa4e79b7f0f43028372dc5",
            "655315af305e4295a42bc2953f95db16",
            "79f3a1db0d84435a9c60297cbd370191",
            "e7eade04e7e94608bf9cd25307971bb1",
            "6aa16a6f06f241f79f3cdab00d774323",
            "071ea6d869ea46cb8d5a1faa34eb8c7e",
            "bbcf8a3bd33c45d990d8398204ac3352",
            "f2c6f43858884863a08743d4ad788af4",
            "662cc28c71f0474a964119c25df1bf5d",
            "99841b8f20ec41f1aa96854582157d69",
            "12648a4714aa494995059c62eaf2e02b",
            "8e7823408cc64d8ebc8dff23d62cea0b",
            "1164884164e04fe5b7d9e17848ada002",
            "41fbc1338aa344cfa41769b4672e33c6",
            "1599ca43c46b4f8bb6bd1f2b15ef8b45",
            "04c2f3d20d2649a0873c49350cbbc82a",
            "a7be14ab72ca467f944974aff7ae4531",
            "4f98325a51f7424d821e107be831e5ec",
            "e967e1cf29654a37a1af7f1155c9e568",
            "5aca21caea704b49b88116c7c9617fef",
            "15756c8474434fc9b03394b073133e71",
            "b910a4eeeb3549c397b505e555f8d726",
            "7d4eab9e6cce4827a84afcd392a71e64",
            "a15c0d794480484f86b55fd64ab3a2ed",
            "a01c1232f66f4ea59e855476c6921d7f",
            "a1bfec5be1784818bde1a35c070c1bcc",
            "8ed6bdc22f7441d3ad60ac17ca70424c",
            "196bb2754187484fb6895366d921cc54",
            "7a77cbc4fd1d44be8d6134ec3c183406",
            "615167f544294d67a666195d2e23e6a4",
            "b6d9514e7c7549c199a65faebd4caaa2",
            "aeed5f319a65432a896fb2202e37476b",
            "f8e4e3a5b6434d6d96351ebba7bd0b52",
            "218a68e2ae1043b1804f160b7da529ef",
            "fc37a085c1564d88b620a57862e08028",
            "1fc6f05a9bcc45f9987be24fa1113418",
            "4b7385439e1945d28e02e4c588f37308",
            "e8cea6239dae47a4afbf2a05724570f1",
            "6b54452715a9458ba807cdd32b986b67",
            "e875db33e3a34501aece161f81bf90e8",
            "f6e9c393cb5a4b2fa741cf7e6aa9de32",
            "1f14dfd47eba4289bde65744334a7a4d",
            "a38c17251e7b4fda84868d54c9359845",
            "46358fe9395e44bc8194af50a78670fa",
            "92014f8298ad40c5a31425017c662a63",
            "28fccfae13f64450b9915750443c428e",
            "94c7a26b5bf44dc1b84e08a6db67c9f3",
            "57407b9fcd144020ae1cc1ed4e48486d",
            "ace1532307cd4c4285a0b7bc01f67e39",
            "e56069e3e16f47c6913d546f55e40a17",
            "5aa4668fa00d4558ab16a885cfa5c599",
            "019def6362534cfc83a3410be1a26f35",
            "b5edcb8ec72b4d0c9eed1e4421a5393a",
            "d0051821eccb43aeb7d03630f3b6852d",
            "88a2b318ef6443a09b83ddaf08f308f6",
            "1096085eac0f457e946b62b6262304b9",
            "508c3db3c83049998020875c2e18a1d1",
            "294fff82831d44219caf5987a54a5836",
            "3a2075a8a5e841629b0f82736668dcc3",
            "6ca2237a97b34e2d875b6dc730d64b08",
            "f9931d7c143b4f7da5e08c427c6ac4c0",
            "764131af5dbe40c9adb111e5c05aca71",
            "44e78c01090b41c3a8711b51fe28dd31",
            "795d59c55e6547369d12442261f870f9",
            "1cb7a1664da346fe9a8f7ba3ecb25a8d",
            "ec4e2f5341284349bfb823b84b6e99e7",
            "98dc7c8e822e4ca797edebc24ade05f3",
            "d0fc0f742acf4163aa18efea70e20216",
            "e2518a40a1694fc9bab37fa85d881ed6",
            "344950151f6749c0abbf32ba64fbecfc",
            "9ca2aee5bfb14a9097194443386f12ef",
            "01e1f611141846aaa429d0c47f7b7166",
            "82460c68e84c48ac8a667ca3bf8b1f6c"
          ]
        },
        "outputId": "9bfe0f63-9b92-42d0-b956-5de853e61d32"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading config.json:   0%|          | 0.00/930 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "da60012bab304a4699db5dfc0b9b065e"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading tokenizer_config.json:   0%|          | 0.00/619 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "36137d130dc94499ac9d13cd10912b92"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading vocab.json:   0%|          | 0.00/779k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f2c6f43858884863a08743d4ad788af4"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading merges.txt:   0%|          | 0.00/446k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "e967e1cf29654a37a1af7f1155c9e568"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading added_tokens.json:   0%|          | 0.00/3.94k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "615167f544294d67a666195d2e23e6a4"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading special_tokens_map.json:   0%|          | 0.00/357 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f6e9c393cb5a4b2fa741cf7e6aa9de32"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading config.json:   0%|          | 0.00/1.00k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "019def6362534cfc83a3410be1a26f35"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading pytorch_model.bin:   0%|          | 0.00/5.75G [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "44e78c01090b41c3a8711b51fe28dd31"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "k_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "v_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "q_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "out_proj Linear(in_features=4096, out_features=4096, bias=False)\n",
            "fc_in Linear(in_features=4096, out_features=16384, bias=True)\n",
            "fc_out Linear(in_features=16384, out_features=4096, bias=True)\n",
            "lm_head Linear(in_features=4096, out_features=50400, bias=True)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Optional: Quick test to check that model generation works"
      ],
      "metadata": {
        "id": "769ccmKu3QMu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Just a quick test, to check whether generation works (optional)\n",
        "from transformers import Text2TextGenerationPipeline\n",
        "\n",
        "# Check if gpu ia available\n",
        "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "print(device)\n",
        "\n",
        "# Move model accordingly\n",
        "model = model.to(device)\n",
        "\n",
        "# Initialize text generation pipeline and generate a test text\n",
        "text2text_generator = Text2TextGenerationPipeline(model, tokenizer, device=0)\n",
        "generated_text = text2text_generator(\"Hallo, mein Name ist\", max_length=50, do_sample=False)\n",
        "print(generated_text)"
      ],
      "metadata": {
        "id": "wycXMu1_z6zE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Step 3: Define a Template\n",
        "\n",
        "A Template defines how to modify the original text for input. It is one of the most important modules in prompt-learning."
      ],
      "metadata": {
        "id": "J8incYByvuKL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from openprompt.prompts import ManualTemplate, PtuningTemplate\n",
        "\n",
        "promptTemplate = ManualTemplate(\n",
        "    text = '{\"soft\": None} {\"soft\": \"Titel\"} {\"soft\": None} {\"soft\": \"folgendem\"} {\"soft\": \"Zeitungsartikel:\"} {\"placeholder\":\"text_a\"} {\"mask\"}',\n",
        "    tokenizer = tokenizer,\n",
        ")"
      ],
      "metadata": {
        "id": "cF6PQzPp6bux"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Step 4: Construct a Prompt Model\n",
        "\n",
        "Given the task, now we have a PLM and a Template, so we combine them into a ```PromptModel```."
      ],
      "metadata": {
        "id": "qc2pEvSUFleM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "use_cuda = True\n",
        "prompt_model = PromptForGeneration(plm=model,template=promptTemplate, freeze_plm=False, plm_eval_mode=False)\n",
        "if use_cuda:\n",
        "    prompt_model=  prompt_model.cuda()"
      ],
      "metadata": {
        "id": "bkFzEBfXFlEu"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Step 5: Define a DataLoader\n",
        "\n",
        "A ```PromptDataLoader``` is basically OpenPrompts version of pytorch Dataloader, which also includes a Tokenizer and a Template."
      ],
      "metadata": {
        "id": "VCnYtwcfvzaX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define DataLoader\n",
        "\n",
        "from openprompt import PromptDataLoader\n",
        "\n",
        "train_dataloader = PromptDataLoader(\n",
        "    dataset = train_dataset,\n",
        "    tokenizer = tokenizer,\n",
        "    template = promptTemplate,\n",
        "    tokenizer_wrapper_class=LMTokenizerWrapper,\n",
        ")\n",
        "validation_dataloader = PromptDataLoader(\n",
        "    dataset = test_dataset,\n",
        "    tokenizer = tokenizer,\n",
        "    template = promptTemplate,\n",
        "    tokenizer_wrapper_class=LMTokenizerWrapper,\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UmgJblwB7iwY",
        "outputId": "fd1048c1-efdb-4582-862a-8aad5dd41366"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "tokenizing: 0it [00:00, ?it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (5531 > 2048). Running this sequence through the model will result in indexing errors\n",
            "tokenizing: 990it [00:06, 156.79it/s]\n",
            "tokenizing: 10it [00:00, 135.92it/s]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Optional: Visualize some encoded data"
      ],
      "metadata": {
        "id": "SVDh6-FNEZmo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# visualize some encoded data (optional)\n",
        "print(next(iter(train_dataloader)))\n",
        "print(tokenizer.decode(next(iter(train_dataloader))['input_ids'][0]))\n",
        "#print(tokenizer.decode(next(iter(train_dataloader))['decoder_input_ids'][0]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HnubVYFgqnvr",
        "outputId": "25d986fe-04eb-4608-d50e-a6e6c09887e3"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{\"input_ids\": [[11307, 13, 6733, 14770, 11286, 4587, 24964, 266, 5757, 300, 2306, 3318, 1976, 15668, 260, 488, 13, 1168, 84, 2705, 11, 1976, 84, 1250, 14261, 384, 72, 4656, 497, 518, 14206, 11, 289, 494, 39683, 1658, 287, 2853, 1406, 89, 498, 268, 3433, 89, 15448, 3464, 13, 376, 25151, 304, 500, 10243, 266, 494, 7502, 259, 3250, 299, 488, 912, 3169, 947, 13, 198, 46, 701, 5513, 261, 387, 11722, 4656, 33391, 49894, 263, 10255, 520, 346, 268, 308, 9774, 72, 2120, 3318, 5256, 24657, 2150, 268, 920, 83, 11033, 385, 21474, 13, 370, 1219, 259, 2876, 84, 861, 4656, 10243, 10255, 1312, 71, 2787, 497, 84, 268, 23999, 13538, 3198, 3125, 4401, 15931, 357, 13195, 1008, 19427, 198, 39, 32074, 402, 7940, 918, 81, 10203, 11, 264, 1349, 30570, 17164, 13416, 4631, 30, 1439, 288, 292, 884, 83, 582, 257, 3046, 1357, 264, 494, 65, 1452, 11733, 40916, 27373, 2436, 488, 13, 11707, 1357, 569, 2398, 11033, 782, 263, 13538, 464, 22059, 3615, 15931, 266, 5757, 7502, 259, 3250, 645, 354, 1976, 84, 1312, 71, 918, 370, 333, 17396, 77, 1976, 333, 9116, 694, 469, 365, 11840, 83, 26, 13538, 33210, 1, 1600, 4656, 1931, 4169, 14206, 11, 289, 278, 1533, 268, 318, 83, 6468, 260, 7274, 5325, 14922, 404, 13, 26943, 3932, 48405, 1702, 83, 5513, 86, 7780, 9116, 83, 328, 6184, 120, 527, 384, 500, 402, 276, 962, 268, 13, 1846, 2537, 10295, 2304, 831, 6524, 3201, 7956, 83, 1931, 11, 1175, 388, 477, 274, 523, 5513, 15448, 384, 72, 13, 360, 1236, 6184, 120, 33900, 8608, 83, 4656, 2310, 12, 73, 11033, 71, 4359, 68, 311, 11033, 782, 263, 259, 21927, 3301, 435, 82, 356, 571, 677, 372, 17431, 353, 3911, 13, 10011, 2611, 376, 13, 198, 32423, 797, 461, 5378, 268, 4587, 24964, 30, 13538, 16973, 304, 259, 370, 4224, 11, 7502, 259, 3250, 13, 21890, 298, 33467, 479, 1359, 83, 523, 299, 30830, 288, 292, 308, 272, 2736, 23999, 1, 1600, 5513, 260, 571, 83, 304, 259, 3203, 12, 49004, 9107, 13, 412, 259, 290, 11882, 407, 72, 861, 11, 1931, 387, 1350, 288, 292, 406, 798, 257, 21474, 7617, 4903, 71, 9101, 17034, 532, 3318, 1658, 266, 263, 2934, 545, 647, 5513, 293, 354, 353, 13, 843, 567, 3996, 962, 268, 264, 488, 25, 13538, 14304, 18519, 2194, 22420, 417, 318, 83, 42392, 732, 346, 328, 32203, 198, 54, 263, 479, 1236, 1658, 5513, 261, 477, 268, 302, 21474, 8352, 831, 30, 11707, 502, 11840, 435, 82, 8454, 23425, 268, 46077, 318, 83, 7502, 259, 3250, 300, 2306, 36052, 4656, 10243, 10255, 2853, 502, 396, 268, 3203, 12, 36570, 13, 6733, 12612, 361, 1211, 959, 277, 9116, 297, 268, 520, 324, 2013, 3318, 599, 8207, 268, 7123, 34380, 71, 1666, 410, 273, 9033, 71, 429, 8717, 437, 268, 13, 314, 11840, 8965, 9116, 39240, 4435, 13538, 21217, 10236, 17003, 15931, 318, 83, 288, 292, 502, 396, 332, 74, 559, 69, 660, 8965, 9116, 83, 748, 48984, 17179, 4625, 912, 13, 13794, 12644, 89, 83, 25, 13538, 33210, 1, 1911, 1406, 339, 72, 39683, 83, 4587]], \"inputs_embeds\": null, \"attention_mask\": [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], \"token_type_ids\": null, \"label\": null, \"decoder_input_ids\": null, \"decoder_inputs_embeds\": null, \"soft_token_ids\": null, \"past_key_values\": null, \"loss_ids\": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], \"guid\": [0], \"tgt_text\": [\"Linkin Park:\\u00a0Radio-Pop statt harter Gitarrenrock\"], \"encoded_tgt_text\": null, \"input_ids_len\": [512]}\n",
            "\n",
            " Berlin. Die Klagen der Fans waren laut und zahlreich. Zu soft, zu beliebig sei die neue Single, hieß es in den Sozialen Netzwerken. Für eine Band wie Linkin Park nichts Neues.\n",
            "Oft schon haben die Amerikaner mit Stilen gespielt und Erwartungen enttäuscht. Wohin steuert die Band mit ihrem neuen Album \"\"One More Light\"\" (Warner)?\n",
            "Harte Gitarrenriffs, schnörkelloser Rock? All das sucht man auf dem siebten Studioalbum vergeblich. Mit dem Vorgänger \"\"The Hunting Party\"\" waren Linkin Park noch zu ihren Wurzeln zurückgekehrt; \"\"Heavy\"\", die erste Single, hingegen ist astreiner Radiopop. Chester Bennington singt schwermütig über seine Gedanken. Im hymnischen Refrain fragt er, warum alles so schwer sei. Dann übernimmt die 21-jährige Sängerin Kiara als weiblicher Konterpart. Schema F.\n",
            "Die Reaktionen der Fans? \"\"Was ein Witz, Linkin Park. Hoffentlich klingt so nicht das ganze Album\"\", schreibt ein Facebook-Nutzer. Ein anderer notiert, er habe das Lied achtmal gehört - und es werde immer schlechter. Andere bedanken sich: \"\"Fehlender Wandel ist langweilig.\"\"\n",
            "Wer kann es schon allen recht machen? Mit mehr als 61 Millionen Likes ist Linkin Park laut Label die Band mit den meisten Facebook-Fans. Die Kalifornier füllen Stadien und spielen Headlinershows vor Zehntausenden. Ihr Debütalbum \"\"Hybrid Theory\"\" ist das meistverkaufte Debüt des Jahrhunderts. Und jetzt: \"\"Heavy\"\". So heißt der\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We are done with the setup!"
      ],
      "metadata": {
        "id": "fKcQIzuEKGMa"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training Loop for Prompt Tuning\n",
        "\n",
        "We prompt tune the model and save it 👏"
      ],
      "metadata": {
        "id": "0XB4u4fav6rj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Define a filepathe for saving the model."
      ],
      "metadata": {
        "id": "vVzBFiY_GQl2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define a filepath for saving the model\n",
        "filepath = '/content/drive/MyDrive/soft-prompt-tuned-model-1000-2.pt'"
      ],
      "metadata": {
        "id": "2ho4qoEf0szL"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Run the training loop."
      ],
      "metadata": {
        "id": "2Zwh2nuBGb4u"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"truncate rate: {}\".format(train_dataloader.tokenizer_wrapper.truncate_rate), flush=True)\n",
        "\n",
        "\n",
        "generation_arguments = {\n",
        "    \"max_new_tokens\": 30,\n",
        "}\n",
        "\n",
        "def evaluate(prompt_model, dataloader):\n",
        "    predictions = []\n",
        "    ground_truths = []\n",
        "\n",
        "    for step, inputs in enumerate(dataloader):\n",
        "        if use_cuda:\n",
        "            inputs = inputs.cuda()\n",
        "        _, output_sentence = prompt_model.generate(inputs, **generation_arguments, verbose=False)\n",
        "        predictions.extend(output_sentence)\n",
        "        ground_truths.extend(inputs['tgt_text'])\n",
        "    assert len(predictions)==len(ground_truths), (len(predictions), len(ground_truths))\n",
        "    predictions = [prediction.strip() for prediction in predictions]\n",
        "    ground_truths = [ground_truth.strip() for ground_truth in ground_truths]\n",
        "    # shown one example\n",
        "    print(f\"predictions {predictions[0]}, ground_truths {ground_truths[0]}\")\n",
        "    score = sum([prediction[:len(ground_truth)]==ground_truth for prediction, ground_truth in zip(predictions, ground_truths)])/len(ground_truths)\n",
        "    return score\n",
        "\n",
        "\n",
        "from transformers import get_linear_schedule_with_warmup,get_constant_schedule_with_warmup  # use AdamW is a standard practice for transformer\n",
        "from transformers.optimization import Adafactor  # use Adafactor is the default setting for T5\n",
        "\n",
        "# This function never gets called?!\n",
        "loss_func = torch.nn.CrossEntropyLoss()\n",
        "\n",
        "# Define Training Parameters; initital definittion from tutorial\n",
        "seed = 144\n",
        "result_file=\"../results.txt\"\n",
        "max_steps=5000 #50\n",
        "lr=1e-2\n",
        "warmup_step_prompt=500 #5\n",
        "eval_every_steps=500 #5\n",
        "tot_step = max_steps\n",
        "num_epochs = 1000000\n",
        "\n",
        "# normally we freeze the model when using soft_template. However, we keep the option to tune plm\n",
        "no_decay = ['bias', 'LayerNorm.weight'] # it's always good practice to set no decay to biase and LayerNorm parameters\n",
        "optimizer_grouped_parameters1 = [\n",
        "    {'params': [p for n, p in prompt_model.plm.named_parameters() if (not any(nd in n for nd in no_decay))], 'weight_decay': 0.01},\n",
        "    {'params': [p for n, p in prompt_model.plm.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}\n",
        "]\n",
        "\n",
        "# we should use optimizer_grouped_parameters1, for the Adam Optimizer \n",
        "#optimizer1 = AdamW(optimizer_grouped_parameters1, lr=1e-2)\n",
        "#optimizer1 = Adam8bit(model.parameters(), lr=lr, weight_decay=0.01)\n",
        "optimizer1 = Adam8bit(optimizer_grouped_parameters1, lr=lr, weight_decay=0.01)\n",
        "scheduler1 = get_linear_schedule_with_warmup(\n",
        "    optimizer1,\n",
        "    num_warmup_steps=500, num_training_steps=tot_step)\n",
        "\n",
        "tot_loss = 0\n",
        "log_loss = 0\n",
        "best_val_acc = 0\n",
        "glb_step = 0\n",
        "actual_step = 0\n",
        "leave_training = False\n",
        "gradient_accumulation_steps = 1\n",
        "\n",
        "acc_traces = []\n",
        "tot_train_time = 0\n",
        "pbar_update_freq = 50\n",
        "prompt_model.train()\n",
        "\n",
        "pbar = tqdm(total=tot_step, desc=\"Train\")\n",
        "for epoch in range(num_epochs):\n",
        "    for step, inputs in enumerate(train_dataloader):\n",
        "        if use_cuda:\n",
        "            inputs = inputs.cuda()\n",
        "        tot_train_time -= time.time()\n",
        "        loss = prompt_model(inputs)\n",
        "        loss.backward()\n",
        "        tot_loss += loss.item()\n",
        "        actual_step += 1\n",
        "\n",
        "        if actual_step % gradient_accumulation_steps == 0:\n",
        "            torch.nn.utils.clip_grad_norm_(prompt_model.parameters(), 1.0)\n",
        "            glb_step += 1\n",
        "            if glb_step % pbar_update_freq == 0:\n",
        "                aveloss = (tot_loss - log_loss)/pbar_update_freq\n",
        "                pbar.update(pbar_update_freq)\n",
        "                pbar.set_postfix({'loss': aveloss, \"epoch\": epoch})\n",
        "                log_loss = tot_loss\n",
        "\n",
        "\n",
        "                if optimizer1 is not None:\n",
        "                    optimizer1.step()\n",
        "                    optimizer1.zero_grad()\n",
        "                if scheduler1 is not None:\n",
        "                    scheduler1.step()\n",
        "\n",
        "        tot_train_time += time.time()\n",
        "\n",
        "        if actual_step % gradient_accumulation_steps == 0 and glb_step >0 and glb_step % eval_every_steps == 0:\n",
        "            val_acc = evaluate(prompt_model, validation_dataloader)\n",
        "            if val_acc >= best_val_acc:\n",
        "                # Uncomment lines below to save model checkpoints\n",
        "                print('save')\n",
        "                torch.save(prompt_model, filepath)\n",
        "                best_val_acc = val_acc\n",
        "\n",
        "            acc_traces.append(val_acc)\n",
        "            print(\"Glb_step {}, val_acc {}, average time {}\".format(glb_step, val_acc, tot_train_time/actual_step ), flush=True)\n",
        "            prompt_model.train()\n",
        "\n",
        "        if glb_step > max_steps:\n",
        "            leave_training = True\n",
        "            break\n",
        "\n",
        "    if leave_training:\n",
        "        break"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7VPU1j_hmL1d",
        "outputId": "deb45cc3-7777-4878-e7aa-43b1319fceab"
      },
      "execution_count": 15,
      "outputs": [
        {
          "metadata": {
            "tags": null
          },
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "truncate rate: 0.4858585858585859\n"
          ]
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Train:  10%|█         | 500/5000 [07:34<1:07:50,  1.11it/s, loss=0.463, epoch=0]"
          ]
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "predictions ung der Verkehrswege einsparen. Das ist ein guter Anfang. Die Kosten für die, ground_truths Straubhaar: „Weniger Sozialstaat, dafür bessere Straßen“\n",
            "save\n",
            "Glb_step 500, val_acc 0.0, average time 0.9092238235473633\n"
          ]
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Train:  20%|██        | 1000/5000 [17:10<1:02:14,  1.07it/s, loss=0.307, epoch=1]"
          ]
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "predictions ung der Verkehrswege einsetzen. Das ist eine gute Initiative, aber sie ist nicht a, ground_truths Straubhaar: „Weniger Sozialstaat, dafür bessere Straßen“\n",
            "save\n",
            "Glb_step 1000, val_acc 0.0, average time 0.9065740208625793\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Train:  30%|███       | 1500/5000 [25:28<53:27,  1.09it/s, loss=0.00948, epoch=1]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "predictions ung der Verkehrswege einsetzen.\n",
            "\n",
            "Das ist ein guter Anfang, aber nicht, ground_truths Straubhaar: „Weniger Sozialstaat, dafür bessere Straßen“\n",
            "save\n",
            "Glb_step 1500, val_acc 0.0, average time 0.9056384477615357\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Train:  40%|████      | 2000/5000 [33:46<45:45,  1.09it/s, loss=0.00722, epoch=2]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "predictions ung der Verkehrsinfrastructur einsparen.\n",
            "\n",
            "Hamburg\n",
            "\n",
            "Hamburg\n",
            "\n",
            "Hamburg, ground_truths Straubhaar: „Weniger Sozialstaat, dafür bessere Straßen“\n",
            "save\n",
            "Glb_step 2000, val_acc 0.0, average time 0.9051383047103881\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Train:  50%|█████     | 2500/5000 [42:05<38:07,  1.09it/s, loss=1.75e-6, epoch=2]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "predictions ung\n",
            "\n",
            "English: \n",
            "Hamburg\n",
            "\n",
            "German: \n",
            "Hamburg\n",
            "\n",
            "English: \n",
            "Hamburg, ground_truths Straubhaar: „Weniger Sozialstaat, dafür bessere Straßen“\n",
            "save\n",
            "Glb_step 2500, val_acc 0.0, average time 0.904837929725647\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Train:  60%|██████    | 3000/5000 [50:23<30:30,  1.09it/s, loss=2.29e-6, epoch=3]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "predictions ung\n",
            "\n",
            "English: \n",
            "Hamburg\n",
            "\n",
            "German\n",
            "\n",
            "English\n",
            "\n",
            "Hamburg\n",
            "\n",
            "English\n",
            "\n",
            "Hamburg, ground_truths Straubhaar: „Weniger Sozialstaat, dafür bessere Straßen“\n",
            "save\n",
            "Glb_step 3000, val_acc 0.0, average time 0.9046142988204956\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Train:  70%|███████   | 3500/5000 [58:42<22:52,  1.09it/s, loss=8.37e-7, epoch=3]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "predictions ung\n",
            "\n",
            "English\n",
            "\n",
            "D\n",
            "\n",
            "German\n",
            "\n",
            "05.02.2019\n",
            "\n",
            "In\n",
            "\n",
            "De\n",
            "\n",
            "\n",
            "-\n",
            "\n",
            "+++, ground_truths Straubhaar: „Weniger Sozialstaat, dafür bessere Straßen“\n",
            "save\n",
            "Glb_step 3500, val_acc 0.0, average time 0.9045298566818237\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Train:  80%|████████  | 4000/5000 [1:07:00<15:15,  1.09it/s, loss=5.51e-7, epoch=4]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "predictions ung\n",
            "\n",
            "English\n",
            "\n",
            "D\n",
            "\n",
            "German\n",
            "\n",
            "05.02.2019\n",
            "\n",
            "In\n",
            "\n",
            "De\n",
            "\n",
            "\n",
            "-\n",
            "\n",
            "+++, ground_truths Straubhaar: „Weniger Sozialstaat, dafür bessere Straßen“\n",
            "save\n",
            "Glb_step 4000, val_acc 0.0, average time 0.9044017723798752\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Train:  90%|█████████ | 4500/5000 [1:15:19<07:37,  1.09it/s, loss=6.37e-7, epoch=4]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "predictions ung\n",
            "\n",
            "English\n",
            "\n",
            "D\n",
            "\n",
            "German\n",
            "\n",
            "05.02.2019\n",
            "\n",
            "\n",
            "Sch\n",
            "\n",
            "\n",
            "\n",
            "Inf\n",
            "\n",
            "\n",
            "-, ground_truths Straubhaar: „Weniger Sozialstaat, dafür bessere Straßen“\n",
            "save\n",
            "Glb_step 4500, val_acc 0.0, average time 0.9043386662801107\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Train: 100%|██████████| 5000/5000 [1:23:37<00:00,  1.09it/s, loss=4.24e-7, epoch=5]"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "predictions ung\n",
            "\n",
            "English\n",
            "\n",
            "D\n",
            "\n",
            "German\n",
            "\n",
            "05.02.2019\n",
            "\n",
            "\n",
            "Sch\n",
            "\n",
            "\n",
            "\n",
            "Inf\n",
            "\n",
            "\n",
            "-, ground_truths Straubhaar: „Weniger Sozialstaat, dafür bessere Straßen“\n",
            "save\n",
            "Glb_step 5000, val_acc 0.0, average time 0.9043051354408265\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Evaluation\n",
        "\n",
        "To evaluate the prompt tuned model generate some headlines."
      ],
      "metadata": {
        "id": "nIDHUGHhUWtZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model.eval()\n",
        "\n",
        "for step, inputs in enumerate(validation_dataloader):\n",
        "    if use_cuda:\n",
        "        inputs = inputs.cuda()\n",
        "    _, output_sentence = prompt_model.generate(inputs, **generation_arguments, verbose=False)\n",
        "    print(output_sentence)"
      ],
      "metadata": {
        "id": "oT5ADBbTUV_9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3c5a2b3d-79f4-450c-85e3-d4923fa43229"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['ung\\n\\nEnglish\\n\\nD\\n\\nGerman\\n\\n05.02.2019\\n\\n\\nSch\\n\\n\\n\\nInf\\n\\n\\n-']\n",
            "['English\\n\\n-\\n\\n00:00:00\\n\\n-\\n\\n00:00:00\\n\\n-\\n\\nJ']\n",
            "['English\\n\\nZ\\n\\nDe\\n\\n00:00\\n\\nSR\\n\\nSW\\n\\n00\\n\\nM\\n\\nM']\n",
            "['English\\n\\nM\\n\\nA\\n\\nZ\\n\\nCategory\\n\\n(FR\\n\\nSP\\n\\n)\\n\\nJuly']\n",
            "['04.02.84']\n",
            "['ende\\n\\nN']\n",
            "['English\\n\\nBerlin\\n\\n07.02.2019\\n\\nA\\n\\n\\nns']\n",
            "['']\n",
            "['']\n",
            "['N\\n\\n###\\n\\n###\\n\\n###\\n\\n###\\n\\n###\\n\\n###\\n\\n###\\n\\n###\\n\\n###']\n"
          ]
        }
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "machine_shape": "hm"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "premium",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "da60012bab304a4699db5dfc0b9b065e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_0ff758dabc7b4dd7b82e1af62cba6ed7",
              "IPY_MODEL_30746c3f34354794a6f4c4185e0ef17f",
              "IPY_MODEL_4e8e715c770f456d89e5034e039f739d"
            ],
            "layout": "IPY_MODEL_df74ae246b35418db7156a32293bbc21"
          }
        },
        "0ff758dabc7b4dd7b82e1af62cba6ed7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5df02a3f9bba4f7c902b5f0dccf59daa",
            "placeholder": "​",
            "style": "IPY_MODEL_90b85c8bfa7a40a582c2e0ded1324242",
            "value": "Downloading config.json: 100%"
          }
        },
        "30746c3f34354794a6f4c4185e0ef17f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_364746915c8a40e883f5bf914cfd80d5",
            "max": 930,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_160152c938b1456daa6a78b0cb2a775b",
            "value": 930
          }
        },
        "4e8e715c770f456d89e5034e039f739d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e5eb6257ffd64aca9eafbc56bcba8a99",
            "placeholder": "​",
            "style": "IPY_MODEL_b799e82c985642f2ab13b61b43e6a080",
            "value": " 930/930 [00:00&lt;00:00, 66.9kB/s]"
          }
        },
        "df74ae246b35418db7156a32293bbc21": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5df02a3f9bba4f7c902b5f0dccf59daa": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "90b85c8bfa7a40a582c2e0ded1324242": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "364746915c8a40e883f5bf914cfd80d5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "160152c938b1456daa6a78b0cb2a775b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e5eb6257ffd64aca9eafbc56bcba8a99": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b799e82c985642f2ab13b61b43e6a080": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "36137d130dc94499ac9d13cd10912b92": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_1b89d549a1e2407cae34b4174c29db82",
              "IPY_MODEL_c8b5aba865bb4caaaa1d5d6fcd8b3177",
              "IPY_MODEL_3e7370d47fdd4da5acb84e535b314467"
            ],
            "layout": "IPY_MODEL_2ea8015398fa4e79b7f0f43028372dc5"
          }
        },
        "1b89d549a1e2407cae34b4174c29db82": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_655315af305e4295a42bc2953f95db16",
            "placeholder": "​",
            "style": "IPY_MODEL_79f3a1db0d84435a9c60297cbd370191",
            "value": "Downloading tokenizer_config.json: 100%"
          }
        },
        "c8b5aba865bb4caaaa1d5d6fcd8b3177": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e7eade04e7e94608bf9cd25307971bb1",
            "max": 619,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6aa16a6f06f241f79f3cdab00d774323",
            "value": 619
          }
        },
        "3e7370d47fdd4da5acb84e535b314467": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_071ea6d869ea46cb8d5a1faa34eb8c7e",
            "placeholder": "​",
            "style": "IPY_MODEL_bbcf8a3bd33c45d990d8398204ac3352",
            "value": " 619/619 [00:00&lt;00:00, 50.2kB/s]"
          }
        },
        "2ea8015398fa4e79b7f0f43028372dc5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "655315af305e4295a42bc2953f95db16": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "79f3a1db0d84435a9c60297cbd370191": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e7eade04e7e94608bf9cd25307971bb1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6aa16a6f06f241f79f3cdab00d774323": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "071ea6d869ea46cb8d5a1faa34eb8c7e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bbcf8a3bd33c45d990d8398204ac3352": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f2c6f43858884863a08743d4ad788af4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_662cc28c71f0474a964119c25df1bf5d",
              "IPY_MODEL_99841b8f20ec41f1aa96854582157d69",
              "IPY_MODEL_12648a4714aa494995059c62eaf2e02b"
            ],
            "layout": "IPY_MODEL_8e7823408cc64d8ebc8dff23d62cea0b"
          }
        },
        "662cc28c71f0474a964119c25df1bf5d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1164884164e04fe5b7d9e17848ada002",
            "placeholder": "​",
            "style": "IPY_MODEL_41fbc1338aa344cfa41769b4672e33c6",
            "value": "Downloading vocab.json: 100%"
          }
        },
        "99841b8f20ec41f1aa96854582157d69": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1599ca43c46b4f8bb6bd1f2b15ef8b45",
            "max": 798156,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_04c2f3d20d2649a0873c49350cbbc82a",
            "value": 798156
          }
        },
        "12648a4714aa494995059c62eaf2e02b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a7be14ab72ca467f944974aff7ae4531",
            "placeholder": "​",
            "style": "IPY_MODEL_4f98325a51f7424d821e107be831e5ec",
            "value": " 779k/779k [00:00&lt;00:00, 636kB/s]"
          }
        },
        "8e7823408cc64d8ebc8dff23d62cea0b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1164884164e04fe5b7d9e17848ada002": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "41fbc1338aa344cfa41769b4672e33c6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1599ca43c46b4f8bb6bd1f2b15ef8b45": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "04c2f3d20d2649a0873c49350cbbc82a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a7be14ab72ca467f944974aff7ae4531": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4f98325a51f7424d821e107be831e5ec": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e967e1cf29654a37a1af7f1155c9e568": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_5aca21caea704b49b88116c7c9617fef",
              "IPY_MODEL_15756c8474434fc9b03394b073133e71",
              "IPY_MODEL_b910a4eeeb3549c397b505e555f8d726"
            ],
            "layout": "IPY_MODEL_7d4eab9e6cce4827a84afcd392a71e64"
          }
        },
        "5aca21caea704b49b88116c7c9617fef": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a15c0d794480484f86b55fd64ab3a2ed",
            "placeholder": "​",
            "style": "IPY_MODEL_a01c1232f66f4ea59e855476c6921d7f",
            "value": "Downloading merges.txt: 100%"
          }
        },
        "15756c8474434fc9b03394b073133e71": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a1bfec5be1784818bde1a35c070c1bcc",
            "max": 456356,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8ed6bdc22f7441d3ad60ac17ca70424c",
            "value": 456356
          }
        },
        "b910a4eeeb3549c397b505e555f8d726": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_196bb2754187484fb6895366d921cc54",
            "placeholder": "​",
            "style": "IPY_MODEL_7a77cbc4fd1d44be8d6134ec3c183406",
            "value": " 446k/446k [00:00&lt;00:00, 1.22MB/s]"
          }
        },
        "7d4eab9e6cce4827a84afcd392a71e64": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a15c0d794480484f86b55fd64ab3a2ed": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a01c1232f66f4ea59e855476c6921d7f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a1bfec5be1784818bde1a35c070c1bcc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8ed6bdc22f7441d3ad60ac17ca70424c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "196bb2754187484fb6895366d921cc54": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7a77cbc4fd1d44be8d6134ec3c183406": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "615167f544294d67a666195d2e23e6a4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_b6d9514e7c7549c199a65faebd4caaa2",
              "IPY_MODEL_aeed5f319a65432a896fb2202e37476b",
              "IPY_MODEL_f8e4e3a5b6434d6d96351ebba7bd0b52"
            ],
            "layout": "IPY_MODEL_218a68e2ae1043b1804f160b7da529ef"
          }
        },
        "b6d9514e7c7549c199a65faebd4caaa2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fc37a085c1564d88b620a57862e08028",
            "placeholder": "​",
            "style": "IPY_MODEL_1fc6f05a9bcc45f9987be24fa1113418",
            "value": "Downloading added_tokens.json: 100%"
          }
        },
        "aeed5f319a65432a896fb2202e37476b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4b7385439e1945d28e02e4c588f37308",
            "max": 4039,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_e8cea6239dae47a4afbf2a05724570f1",
            "value": 4039
          }
        },
        "f8e4e3a5b6434d6d96351ebba7bd0b52": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6b54452715a9458ba807cdd32b986b67",
            "placeholder": "​",
            "style": "IPY_MODEL_e875db33e3a34501aece161f81bf90e8",
            "value": " 3.94k/3.94k [00:00&lt;00:00, 304kB/s]"
          }
        },
        "218a68e2ae1043b1804f160b7da529ef": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fc37a085c1564d88b620a57862e08028": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1fc6f05a9bcc45f9987be24fa1113418": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4b7385439e1945d28e02e4c588f37308": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e8cea6239dae47a4afbf2a05724570f1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6b54452715a9458ba807cdd32b986b67": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e875db33e3a34501aece161f81bf90e8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f6e9c393cb5a4b2fa741cf7e6aa9de32": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_1f14dfd47eba4289bde65744334a7a4d",
              "IPY_MODEL_a38c17251e7b4fda84868d54c9359845",
              "IPY_MODEL_46358fe9395e44bc8194af50a78670fa"
            ],
            "layout": "IPY_MODEL_92014f8298ad40c5a31425017c662a63"
          }
        },
        "1f14dfd47eba4289bde65744334a7a4d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_28fccfae13f64450b9915750443c428e",
            "placeholder": "​",
            "style": "IPY_MODEL_94c7a26b5bf44dc1b84e08a6db67c9f3",
            "value": "Downloading special_tokens_map.json: 100%"
          }
        },
        "a38c17251e7b4fda84868d54c9359845": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_57407b9fcd144020ae1cc1ed4e48486d",
            "max": 357,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ace1532307cd4c4285a0b7bc01f67e39",
            "value": 357
          }
        },
        "46358fe9395e44bc8194af50a78670fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e56069e3e16f47c6913d546f55e40a17",
            "placeholder": "​",
            "style": "IPY_MODEL_5aa4668fa00d4558ab16a885cfa5c599",
            "value": " 357/357 [00:00&lt;00:00, 28.3kB/s]"
          }
        },
        "92014f8298ad40c5a31425017c662a63": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "28fccfae13f64450b9915750443c428e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "94c7a26b5bf44dc1b84e08a6db67c9f3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "57407b9fcd144020ae1cc1ed4e48486d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ace1532307cd4c4285a0b7bc01f67e39": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e56069e3e16f47c6913d546f55e40a17": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5aa4668fa00d4558ab16a885cfa5c599": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "019def6362534cfc83a3410be1a26f35": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_b5edcb8ec72b4d0c9eed1e4421a5393a",
              "IPY_MODEL_d0051821eccb43aeb7d03630f3b6852d",
              "IPY_MODEL_88a2b318ef6443a09b83ddaf08f308f6"
            ],
            "layout": "IPY_MODEL_1096085eac0f457e946b62b6262304b9"
          }
        },
        "b5edcb8ec72b4d0c9eed1e4421a5393a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_508c3db3c83049998020875c2e18a1d1",
            "placeholder": "​",
            "style": "IPY_MODEL_294fff82831d44219caf5987a54a5836",
            "value": "Downloading config.json: 100%"
          }
        },
        "d0051821eccb43aeb7d03630f3b6852d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3a2075a8a5e841629b0f82736668dcc3",
            "max": 1021,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6ca2237a97b34e2d875b6dc730d64b08",
            "value": 1021
          }
        },
        "88a2b318ef6443a09b83ddaf08f308f6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f9931d7c143b4f7da5e08c427c6ac4c0",
            "placeholder": "​",
            "style": "IPY_MODEL_764131af5dbe40c9adb111e5c05aca71",
            "value": " 1.00k/1.00k [00:00&lt;00:00, 76.8kB/s]"
          }
        },
        "1096085eac0f457e946b62b6262304b9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "508c3db3c83049998020875c2e18a1d1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "294fff82831d44219caf5987a54a5836": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3a2075a8a5e841629b0f82736668dcc3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6ca2237a97b34e2d875b6dc730d64b08": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "f9931d7c143b4f7da5e08c427c6ac4c0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "764131af5dbe40c9adb111e5c05aca71": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "44e78c01090b41c3a8711b51fe28dd31": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_795d59c55e6547369d12442261f870f9",
              "IPY_MODEL_1cb7a1664da346fe9a8f7ba3ecb25a8d",
              "IPY_MODEL_ec4e2f5341284349bfb823b84b6e99e7"
            ],
            "layout": "IPY_MODEL_98dc7c8e822e4ca797edebc24ade05f3"
          }
        },
        "795d59c55e6547369d12442261f870f9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d0fc0f742acf4163aa18efea70e20216",
            "placeholder": "​",
            "style": "IPY_MODEL_e2518a40a1694fc9bab37fa85d881ed6",
            "value": "Downloading pytorch_model.bin: 100%"
          }
        },
        "1cb7a1664da346fe9a8f7ba3ecb25a8d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_344950151f6749c0abbf32ba64fbecfc",
            "max": 6177012613,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9ca2aee5bfb14a9097194443386f12ef",
            "value": 6177012613
          }
        },
        "ec4e2f5341284349bfb823b84b6e99e7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_01e1f611141846aaa429d0c47f7b7166",
            "placeholder": "​",
            "style": "IPY_MODEL_82460c68e84c48ac8a667ca3bf8b1f6c",
            "value": " 5.75G/5.75G [01:28&lt;00:00, 74.9MB/s]"
          }
        },
        "98dc7c8e822e4ca797edebc24ade05f3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d0fc0f742acf4163aa18efea70e20216": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e2518a40a1694fc9bab37fa85d881ed6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "344950151f6749c0abbf32ba64fbecfc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9ca2aee5bfb14a9097194443386f12ef": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "01e1f611141846aaa429d0c47f7b7166": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "82460c68e84c48ac8a667ca3bf8b1f6c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}