Tutorial_RAAI_Summer_School.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Tutorial_RAAI Summer School.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.5"
    },
    "kernelspec": {
      "display_name": "Py3.6",
      "language": "python",
      "name": "py36"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NGTO9GREF8Fw",
        "colab_type": "text"
      },
      "source": [
        "# Tutorial \"How to solve NLP tasks with DeepPavlov\"\n",
        "\n",
        "This tutorial is aimed to make participants familiar with solving NLP tasks using `DeepPavlov`.\n",
        "We are going to use **BERT-based models** in this tutorial.\n",
        "\n",
        "The tutorial has the following structure:\n",
        "\n",
        "* [BERT input representation](#BERT-input-representation)\n",
        "\n",
        "* [DeepPavlov Installation](#DeepPavlov-Installation)\n",
        "\n",
        "* [Configs](#Configs)\n",
        "\n",
        "* [Command line interface](#Command-line-interface)\n",
        "\n",
        "* [Python code interface](#Python-code-interface)\n",
        "\n",
        "* [BERT for text classification](#BERT-for-text-classification)\n",
        "\n",
        "* [BERT for tagging](#BERT-for-tagging)\n",
        "\n",
        "* [BERT for Question Answering](#BERT-for-Question-Answering)\n",
        "\n",
        "* [Zero-shot Transfer from English to 103 languages](#Zero-shot-Transfer-from-English-to-103-languages)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sEyksCwPQyDT",
        "colab_type": "text"
      },
      "source": [
        "## BERT input representation\n",
        "Text preprocessing for BERT relies on tokenizing text on subtokens (or WordPieces). Then BERT internally represents each subtoken as sum of three vectors:\n",
        "\n",
        "* subtoken embedding\n",
        "* segment embedding\n",
        "* position embedding\n",
        "\n",
        "<img src=\"https://github.com/deepmipt/dp_tutorials/blob/master/img/BERT_input.png?raw=1\" width=\"75%\" />"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3Uut9t-OF8Fz",
        "colab_type": "text"
      },
      "source": [
        "## DeepPavlov Installation\n",
        "\n",
        "The following command installs basic requirements of `DeepPavlov`. Please, pay attention that in case of using some particular model you will probably have to install some additional dependencies. Please, for versions coincidence install additional requirements also using `DeepPavlov`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WaBQeRjXF8F0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install deeppavlov"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c9_kXqGRF8F6",
        "colab_type": "text"
      },
      "source": [
        "## Configs\n",
        "\n",
        "One of the main conceptions of `DeepPavlov` is that each model is being defined by the configuration file, so called `config`. `config` is just a `json` file containing dictionary with dataset reader, dataset iterator, model pipeline, training parameters and some metadata. \n",
        "\n",
        "So, in case you want to use pre-defined model, just find [here](https://github.com/deepmipt/DeepPavlov/tree/master/deeppavlov/configs) the config of your interest.\n",
        "\n",
        "If you want to compose your own model pipeline or somehow change presented model, the best idea will be to take one of the presented [here](https://github.com/deepmipt/DeepPavlov/tree/master/deeppavlov/configs) configs and change data/pipeline elements/training parameters etc.\n",
        "\n",
        "For example, now we want to use BERT-based classification model for insults classification in [Social Commentary](https://www.kaggle.com/c/detecting-insults-in-social-commentary).\n",
        "\n",
        "Let's look into the config.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kRdMZIkWF8F8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import json\n",
        "from deeppavlov import configs\n",
        "\n",
        "config_path = configs.classifiers.insults_kaggle_bert\n",
        "print(\"Path to config: {}\".format(config_path))\n",
        "\n",
        "with open(config_path, \"r\") as f:\n",
        "    config = json.load(f)\n",
        "    \n",
        "print(json.dumps(config, indent=2))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ssJ6XUe5F8F_",
        "colab_type": "text"
      },
      "source": [
        "There are two possible ways to work with configs in `DeepPavlov`: from command line or `python` code."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hd39OpJyF8GA",
        "colab_type": "text"
      },
      "source": [
        "## Command line interface\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H7wMSrXfF8GB",
        "colab_type": "text"
      },
      "source": [
        "So, let's firtsly install additional dependencies for BERT-based classification model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ALhkO3rYF8GD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! python -m deeppavlov install insults_kaggle_bert"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AXOE8n7sF8GH",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "From command line one may call model for interact in the following way (flag `-d` for downloading model, if files were already downloaded and were not modified, they won't be downloaded again):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iMTL_2-LF8GI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! python -m deeppavlov interact -d insults_kaggle_bert"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MXbEC1H8F8GN",
        "colab_type": "text"
      },
      "source": [
        "Configs can be also called for training and evaluating (calculating scores) using modes `train` and `evaluate`. Flag `-d` is also optional.\n",
        "\n",
        "```\n",
        "! python -m deeppavlov train [-d] insults_kaggle_bert\n",
        "```\n",
        "and\n",
        "```\n",
        "! python -m deeppavlov evaluate [-d] insults_kaggle_bert\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NoMy9y2vF8GO",
        "colab_type": "text"
      },
      "source": [
        "Take into account that `insults_kaggle_bert` in the examples above is not a special keyword but just stem part of the corresponding [config file name](https://github.com/deepmipt/DeepPavlov/tree/master/deeppavlov/configs/classifiers/insults_kaggle_bert.json). So, any config file from the folder [`deeppavlov/configs/`](https://github.com/deepmipt/DeepPavlov/tree/master/deeppavlov/) can be given to command line only as a part of the name of file without extension. Anyway one still can save config file anywhere and specify a full path to config file."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C06w1OudOrYv",
        "colab_type": "text"
      },
      "source": [
        "## Python code interface\n",
        "\n",
        "In `python` code interface one can specify to functions `build_model` and `train_evaluate_model_from_config` either **path to config file or config dictionary itself**.\n",
        "\n",
        "`DeepPavlov` models can be also used in `python` interface in the following way:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mBPGVhobF8GT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from deeppavlov import build_model, configs\n",
        "\n",
        "model = build_model(configs.classifiers.insults_kaggle_bert, \n",
        "                    download=False) # download=True if model is not downloaded yet"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aGuYvcNaF8GW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model(['Hey, you are stupid!', \n",
        "       'Hey, you are smart!'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SmFEMw2DTPXX",
        "colab_type": "text"
      },
      "source": [
        "Configs can be also called for training and/or evaluating (calculating scores) using python commands `train_evaluate_model_from_config`. Parameter `download` is also optional.\n",
        "\n",
        "```python\n",
        "from deeppavlov import train_evaluate_model_from_config\n",
        "\n",
        "train_evaluate_model_from_config(configs.classifiers.insults_kaggle_bert,  \n",
        "                                 to_train=False,  # set to True to train the model\n",
        "                                 to_validate=True,\n",
        "                                 download=True  # download=True if model is not downloaded yet\n",
        "                                )\n",
        "\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "maZPqgsJQ1CQ",
        "colab_type": "text"
      },
      "source": [
        "## BERT for text classification\n",
        "When we want to use BERT model for text classification task we can add only one dense layer on top of the output from the last BERT Transformer layer for special `[CLS]` token.\n",
        "\n",
        "<img src=\"https://github.com/deepmipt/dp_tutorials/blob/master/img/BERT_classification.png?raw=1\" width=\"75%\" />\n",
        "\n",
        "Let's just recap and discuss how the config for BERT-based text classification looks like:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EKeyvZwQPu5y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import json\n",
        "from deeppavlov import configs\n",
        "\n",
        "config_path = configs.classifiers.insults_kaggle_bert\n",
        "print(\"Path to config: {}\".format(config_path))\n",
        "\n",
        "with open(config_path, \"r\") as f:\n",
        "    config = json.load(f)\n",
        "    \n",
        "print(json.dumps(config, indent=2))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ea6q12JPhwdJ",
        "colab_type": "text"
      },
      "source": [
        "## BERT for tagging\n",
        "\n",
        "BERT model can be used for tagging tasks such like Named Entity Recognition and Part of Speech tagging.\n",
        "We train only one dense layer on top of the output from the last BERT Transformer layer for each token. You can optionally add CRF layer on top the dense layer like in most common architecture BiLSTM + CRF for tagging.\n",
        "\n",
        "Named Entity Recognition:\n",
        "\n",
        "For example, we want to extract persons' and organizations' names from the text. Then for the input text:\n",
        "\n",
        "    Yan Goodfellow works for Google Brain\n",
        "\n",
        "a NER model needs to provide the following sequence of tags:\n",
        "\n",
        "    B-PER I-PER    O     O   B-ORG  I-ORG\n",
        "\n",
        "Where *B-* and *I-* prefixes stand for the beginning and inside of the entity, while *O* stands for out of tag or no tag. Markup with the prefix scheme is called *BIO markup*. This markup is introduced for distinguishing of consequent entities with similar types.\n",
        "\n",
        "Here is how input is preprocessed for tagging:\n",
        "\n",
        "<img src=\"https://github.com/deepmipt/dp_tutorials/blob/master/img/BERT_NER.png?raw=1\" width=\"75%\" />"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Utctb3YLhqQF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from deeppavlov import build_model, configs\n",
        "\n",
        "model = build_model(configs.ner.ner_ontonotes_bert, \n",
        "                    download=True) # download=True if model is not downloaded yet"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xb1Iy-uSik2h",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model(['Moscow Institute of Physics and Technology is aimed to win Alexa Prize Challenge'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yMGIFpCwiKhF",
        "colab_type": "text"
      },
      "source": [
        "Data for Named Enitity Recognition task is usually stored in CoNLL files.\n",
        "Typical CoNLL file with NER data contains lines with pairs of tokens (word/punctuation symbol) and tags, separated by a whitespace. In many cases additional information such as POS tags included between  Different documents are separated by lines **started** with **-DOCSTART-** token. Different sentences are separated by an empty line. Example\n",
        "\n",
        "    -DOCSTART- -X- -X- O\n",
        "\n",
        "    EU NNP B-NP B-ORG\n",
        "    rejects VBZ B-VP O\n",
        "    German JJ B-NP B-MISC\n",
        "    call NN I-NP O\n",
        "    to TO B-VP O\n",
        "    boycott VB I-VP O\n",
        "    British JJ B-NP B-MISC\n",
        "    lamb NN I-NP O\n",
        "    . . O O\n",
        "\n",
        "    Peter NNP B-NP B-PER\n",
        "    Blackburn NNP I-NP I-PER\n",
        "    \n",
        "    \n",
        "If you wants to train model on your own data you can convert it to this CoNLL format or implement your version of `dataset_reader`. \n",
        "\n",
        "Now let's look into the config for BERT-based NER."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p3zUMc2KiLAN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import json\n",
        "from deeppavlov import configs\n",
        "\n",
        "config_path = configs.ner.ner_ontonotes_bert\n",
        "print(\"Path to config: {}\".format(config_path))\n",
        "\n",
        "with open(config_path, \"r\") as f:\n",
        "    config = json.load(f)\n",
        "    \n",
        "print(json.dumps(config, indent=2))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HOIUkg70j0xY",
        "colab_type": "text"
      },
      "source": [
        "## BERT for Question Answering \n",
        "\n",
        "One can use BERT model for extractive Question Answering, e.g.,\n",
        "context:\n",
        "```markdown\n",
        "In meteorology, precipitation is any product of the condensation of atmospheric water vapor that falls under gravity. The main forms of precipitation include drizzle, rain, sleet, snow, graupel and hail… Precipitation forms as smaller droplets coalesce via collision with other rain drops or ice crystals **within a cloud**. Short, intense periods of rain in scattered locations are called “showers”.\n",
        "```\n",
        "and question:\n",
        "```\n",
        "Where do water droplets collide with ice crystals to form precipitation?\n",
        "```\n",
        "Answer is always a span from context.\n",
        "\n",
        "To solve this task with BERT model all we need is to train two dense layes to predict answer start and answer end positions:\n",
        "\n",
        "<img src=\"https://github.com/deepmipt/dp_tutorials/blob/master/img/BERT_QA.png?raw=1\" width=\"50%\" />"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hMH7FednjP2y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from deeppavlov import build_model, configs\n",
        "\n",
        "model = build_model(configs.squad.squad_bert,\n",
        "                    download=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XhFw_BVPj9q6",
        "colab_type": "text"
      },
      "source": [
        "Model returns an answer, position in characters and confidence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h1D9Vk2Kj7T3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model(['In meteorology, precipitation is any product of the condensation of atmospheric water vapor that falls under gravity. The main forms of precipitation include drizzle, rain, sleet, snow, graupel and hail… Precipitation forms as smaller droplets coalesce via collision with other rain drops or ice crystals within a cloud. Short, intense periods of rain in scattered locations are called “showers”.'], \n",
        "      ['Where do water droplets collide with ice crystals to form precipitation?'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "slD1aI2_kAyW",
        "colab_type": "text"
      },
      "source": [
        "To train model on your data you should put it json files in SQuAD format: https://rajpurkar.github.io/SQuAD-explorer/\n",
        "\n",
        "These json files contain paragraphs, questions and answers."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h8aeRm-BlM3_",
        "colab_type": "text"
      },
      "source": [
        "## Zero-shot Transfer from English to 103 languages\n",
        "\n",
        "BERT model was originaly trained only for English language, but lately multilingual model trained on 103 was released. It gives ability to train models on language and use them for 103 other language. This technique is called zero-shot transfer as we don't use any training data for target language.\n",
        "\n",
        "<img src=\"https://github.com/deepmipt/dp_tutorials/blob/master/img/BERT_multilingual.png?raw=1\" width=\"75%\" />\n",
        "\n",
        "We will cover two examples:\n",
        " * NER transfer from Ontonotes dataset (English -> 103)\n",
        " * QA transfer from SQuAD dataset (English -> 103)\n",
        " \n",
        " These models are also available at [demo.ipavlov.ai](https://demo.ipavlov.ai/#multiLang)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "shVjMUprl3gf",
        "colab_type": "text"
      },
      "source": [
        "#### Zero-shot multilingual NER\n",
        "\n",
        "Download and interact the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cprV0izNkBx7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from deeppavlov import build_model, configs\n",
        "\n",
        "model = build_model(configs.ner.ner_ontonotes_bert_mult, \n",
        "                    download=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Djp1ywWl8tl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model(['Curling World Championship will be held in Antananarivo'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VtYg-Nijl-cV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model(['Чемпионат мира по кёрлингу пройдёт в Антананариву']) # Чемпионат мира по кёрлингу == Curling World Championship"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PYwwfqSxmDjs",
        "colab_type": "text"
      },
      "source": [
        "#### Zero-shot multilingual QA\n",
        "\n",
        "Get configuration file, download and interact the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G06dSeUmmAD8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! wget https://raw.githubusercontent.com/deepmipt/DeepPavlov/squad_multilingual_configs/deeppavlov/configs/squad/squad_bert_multilingual_freezed_emb.json"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3q7QacpNmIBq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from deeppavlov import build_model, configs\n",
        "\n",
        "model = build_model('./squad_bert_multilingual_freezed_emb.json', download=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1xURHgOamN8l",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model(['Su área de distribución comprende casi toda Sudamérica al este de los Andes en las \\\n",
        "       cuencas del río Orinoco, del Amazonas y del Río de la Plata; cubriendo desde el este \\\n",
        "       de Venezuela y la Guyana hasta Uruguay y el norte y centro de Argentina. Pueden vivir \\\n",
        "       en diferentes tipos de hábitat, pero muestran preferencia por algunos en concreto. \\\n",
        "       Suelen encontrarse cerca de lagos, ríos, marismas o manglares.'], \n",
        "      ['What countries do capybara live in?'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I11y1WpSmSAK",
        "colab_type": "text"
      },
      "source": [
        "As you can see model can work even if context and question languages are different!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oba6RyyMmSEB",
        "colab_type": "text"
      },
      "source": [
        "### Zero-shot transfer performance\n",
        "\n",
        "Results for Zero-Shot NER from English to Russian:\n",
        "\n",
        "| model                            | Overall (Span F-1)   | PER (Span F-1)    | LOC (Span F-1)   | ORG (Span F-1) |\n",
        "|----------------------------------|-------|----------|----|----|\n",
        "| RuBERT NER | 97.7 |98.3   | 99.7 | 94.9|\n",
        "| Zero-shot Multilingual BERT NER   | 79.4 | 95.7   |82.6 | 55.7|\n",
        "\n",
        "Results for Zero-Shot QA from English to Russian:\n",
        "\n",
        "| model                            | F-1   |\n",
        "|----------------------------------|-------|\n",
        "| RuBERT QA | 84.6 |\n",
        "| Zero-shot Multilingual BERT QA   | 77.36 |\n",
        "\n",
        "Results for Zero-Shot QA from Russian to English:\n",
        "\n",
        "| model                            | F-1   |\n",
        "|----------------------------------|-------|\n",
        "| BERT QA | 88.49 |\n",
        "| Zero-shot Multilingual BERT QA   | 75.26 |"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ESfWRJlNmTul",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}