other_examples/chunker_english.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CoNLL 2000 Syntactic Chunker Task\n",
    "The  **`CoNLL 2000`** chunking task consists of data with annotations for 11 types of syntactic chunks (Noun Phrases, Verb phrases, etc). The data is in a [IOB2](https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)) format.  Each token enitity has a `'B-'` or `'I-'` tag indicating if it is the start of the entity or if the token is inside the annotation. \n",
    "\n",
    "Tags include:\n",
    "\n",
    "* **Noun Phrase**: `'B-NP', 'I-NP'`\n",
    "\n",
    "* **Verb Phrase**: `'B-VP','I-VP'`\n",
    "\n",
    "* **Adjective Phrase**:`'B-ADJP', 'I-ADJP'`\n",
    "\n",
    "* **Adverb Phrase**:`'B-ADVP', 'I-ADVP'`\n",
    "\n",
    "* **Prepositional Phrase**:`'B-PP', 'I-PP`\n",
    "\n",
    "* **Conjunctions Phrase**:`'B-CONJP', 'I-CONJP'`\n",
    "\n",
    "* **Interjection Phrase**:`B-INTJ', 'I-INTJ'`\n",
    "\n",
    "* **Verb Particle  Phrase**:`'B-PRT', 'I-PRT'`\n",
    "\n",
    "* **List Marker Phrase**:`'B-LST', 'I-LST'`\n",
    "\n",
    "* **Unlike Coordinated  phrase**:`'B-UCP', 'I-UCP'`\n",
    "\n",
    "* **Complementizer Phrase**:`'B-SBAR', 'I-SBAR'`\n",
    "\n",
    "* **Outside Tokens**:`'O'`\n",
    "\n",
    "\n",
    "\n",
    "See [website](https://www.clips.uantwerpen.be/conll2000/chunking/) and [paper](https://www.clips.uantwerpen.be/conll2000/pdf/12732tjo.pdf) for more info."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is already tokenized and tagged with Part of Speech(POS) and Chunk labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# token         POS Chunk  \n",
    "#------------------------\n",
    "# Rockwell      NNP B-NP\n",
    "# International NNP I-NP\n",
    "# Corp.         NNP I-NP\n",
    "# 's            POS B-NP\n",
    "# Tulsa         NNP I-NP\n",
    "# unit          NN  I-NP\n",
    "# said          VBD B-VP\n",
    "# it            PRP B-NP\n",
    "# signed        VBD B-VP\n",
    "# a             DT  B-NP\n",
    "# tentative     JJ  I-NP\n",
    "# agreement     NN  I-NP\n",
    "# extending     VBG B-VP\n",
    "# its           PRP B-NP\n",
    "# contract      NN  I-NP\n",
    "# with          IN  B-PP\n",
    "# Boeing        NNP B-NP\n",
    "# Co.           NNP I-NP\n",
    "# to            TO  B-VP\n",
    "# provide       VB  I-VP\n",
    "# structural    JJ  B-NP\n",
    "# parts         NNS I-NP\n",
    "# for           IN  B-PP\n",
    "# Boeing        NNP B-NP\n",
    "# 's            POS B-NP\n",
    "# 747           CD  I-NP\n",
    "# jetliners     NNS I-NP\n",
    "# .             .   O"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first column is the token, the second column is Part of Speech(POS) tag, the third is syntactic chunk tag.\n",
    "\n",
    "So for the chunking task the data consists of features:`X`and labels:`y`\n",
    "\n",
    "\n",
    "* **`X`** :  a list of list of tokens \n",
    "\n",
    "\n",
    "* **`y`** :  a list of list of syntactic chunk tags\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "DATADIR=\"chunker_english\"\n",
    "if test ! -d \"$DATADIR\";then\n",
    "    echo \"Creating $DATADIR dir\"\n",
    "    mkdir \"$DATADIR\"\n",
    "    cd \"$DATADIR\"\n",
    "    wget https://www.clips.uantwerpen.be/conll2000/chunking/train.txt.gz\n",
    "    gunzip train.txt.gz\n",
    "    wget https://www.clips.uantwerpen.be/conll2000/chunking/test.txt.gz\n",
    "    gunzip test.txt.gz\n",
    "fi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data: 8936 sentences, 211727 tokens\n",
      "Test data: 2012 sentences, 47377 tokens\n",
      "\n",
      "Chunker tags/labels:\n",
      " ['I-UCP', 'O', 'B-LST', 'I-ADJP', 'B-INTJ', 'I-CONJP', 'I-VP', 'I-LST', 'B-CONJP', 'I-INTJ', 'B-PP', 'I-SBAR', 'I-PP', 'B-UCP', 'B-ADVP', 'B-VP', 'I-PRT', 'B-NP', 'I-ADVP', 'B-ADJP', 'B-PRT', 'I-NP', 'B-SBAR']\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "Train data: 8936 sentences, 211727 tokens\n",
    "Test data: 2012 sentences, 47377 tokens\n",
    "\"\"\"\n",
    "import os\n",
    "import sys\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "sys.path.append(\"../\") \n",
    "from bert_sklearn import BertTokenClassifier, load_model\n",
    "\n",
    "def flatten(l):\n",
    "    return [item for sublist in l for item in sublist]\n",
    "\n",
    "def read_CoNLL2003_format(filename, idx=3):\n",
    "    \"\"\"Read file in CoNLL-2003 shared task format\"\"\" \n",
    "    \n",
    "    lines =  open(filename).read().strip()\n",
    "    \n",
    "    # find sentence-like boundaries\n",
    "    lines = lines.split(\"\\n\\n\")  \n",
    "    \n",
    "    # throw out -DOCSTART- lines \n",
    "    lines = [line for line in lines if not line.startswith(\"-DOCSTART-\")]\n",
    "    \n",
    "     # split on newlines\n",
    "    lines = [line.split(\"\\n\") for line in lines]\n",
    "    \n",
    "    # get tokens\n",
    "    tokens = [[l.split()[0] for l in line] for line in lines]\n",
    "    \n",
    "    # get labels/tags\n",
    "    labels = [[l.split()[idx] for l in line] for line in lines]\n",
    "    \n",
    "    data= {'tokens': tokens, 'labels': labels}\n",
    "    df=pd.DataFrame(data=data)\n",
    "    \n",
    "    return df\n",
    "\n",
    "DATADIR = \"./chunker_english/\"\n",
    "\n",
    "def get_data(trainfile=DATADIR + \"train.txt\",\n",
    "            testfile=DATADIR + \"test.txt\"):\n",
    "\n",
    "    train = read_CoNLL2003_format(trainfile,idx=2)\n",
    "    print(\"Train data: %d sentences, %d tokens\"%(len(train),len(flatten(train.tokens))))\n",
    "\n",
    "    test = read_CoNLL2003_format(testfile,idx=2)\n",
    "    print(\"Test data: %d sentences, %d tokens\"%(len(test),len(flatten(test.tokens))))\n",
    "    \n",
    "    return train, test\n",
    "\n",
    "train, test = get_data()\n",
    "X_train, y_train = train.tokens, train.labels\n",
    "X_test, y_test = test.tokens, test.labels\n",
    "\n",
    "# chunk tags may be spread out among the different sets i.e we cant rely on all \n",
    "# the chunk tags being present in the training set\n",
    "label_list = set(flatten(y_train)).union(set(flatten(y_test)))\n",
    "label_list = list(label_list)\n",
    "print(\"\\nChunker tags/labels:\\n\", label_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tokens</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[Confidence, in, the, pound, is, widely, expec...</td>\n",
       "      <td>[B-NP, B-PP, B-NP, I-NP, B-VP, I-VP, I-VP, I-V...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[Chancellor, of, the, Exchequer, Nigel, Lawson...</td>\n",
       "      <td>[O, B-PP, B-NP, I-NP, B-NP, I-NP, B-NP, I-NP, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[But, analysts, reckon, underlying, support, f...</td>\n",
       "      <td>[O, B-NP, B-VP, B-NP, I-NP, B-PP, B-NP, B-VP, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[This, has, increased, the, risk, of, the, gov...</td>\n",
       "      <td>[B-NP, B-VP, I-VP, B-NP, I-NP, B-PP, B-NP, I-N...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[``, The, risks, for, sterling, of, a, bad, tr...</td>\n",
       "      <td>[O, B-NP, I-NP, B-PP, B-NP, B-PP, B-NP, I-NP, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              tokens  \\\n",
       "0  [Confidence, in, the, pound, is, widely, expec...   \n",
       "1  [Chancellor, of, the, Exchequer, Nigel, Lawson...   \n",
       "2  [But, analysts, reckon, underlying, support, f...   \n",
       "3  [This, has, increased, the, risk, of, the, gov...   \n",
       "4  [``, The, risks, for, sterling, of, a, bad, tr...   \n",
       "\n",
       "                                              labels  \n",
       "0  [B-NP, B-PP, B-NP, I-NP, B-VP, I-VP, I-VP, I-V...  \n",
       "1  [O, B-PP, B-NP, I-NP, B-NP, I-NP, B-NP, I-NP, ...  \n",
       "2  [O, B-NP, B-VP, B-NP, I-NP, B-PP, B-NP, B-VP, ...  \n",
       "3  [B-NP, B-VP, I-VP, B-NP, I-NP, B-PP, B-NP, I-N...  \n",
       "4  [O, B-NP, I-NP, B-PP, B-NP, B-PP, B-NP, I-NP, ...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And lets look at an observation on the tokens,labels pair and make sure it makes sense:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "His meals are most often prepared by women he trusts -- his full-time mistress , Vicky Amado , and her mother , Norma .\n",
      "B-NP I-NP B-VP I-VP I-VP I-VP B-PP B-NP B-NP B-VP O B-NP I-NP I-NP O B-NP I-NP O O B-NP I-NP O B-NP O\n"
     ]
    }
   ],
   "source": [
    "i = 151\n",
    "tokens = X_test[i]\n",
    "labels = y_test[i]\n",
    "\n",
    "print(\" \".join(tokens))\n",
    "print(\" \".join(labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets define our model using the **`BertTokenClassifier`** class\n",
    "\n",
    "* We will include an `ignore_label` option to ignore the `'O'` labels.\n",
    "\n",
    "\n",
    "* We will also use the cased model as we did with NER."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building sklearn token classifier...\n"
     ]
    }
   ],
   "source": [
    "# define model\n",
    "model = BertTokenClassifier(bert_model='bert-base-cased',\n",
    "                            epochs=3,\n",
    "                            label_list = label_list,\n",
    "                            learning_rate=2e-5,\n",
    "                            train_batch_size=16,\n",
    "                            eval_batch_size=16,\n",
    "                            ignore_label=['O'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One issue that we need to be mindful of is the max token length in the token lists. \n",
    "There are 2 complications:\n",
    "    \n",
    "* We have a **`max_seq_length`** parameter  with BERT that will dictate how long a token sequence we can handle. All input tokens will be truncaed based on this. The limit on this is 512, but we would like smaller sequences since they are much faster and consume less memory on the GPU. \n",
    "    \n",
    "    \n",
    "* Each token will be tokenized again by the BERT wordpiece tokenizer. This will result in longer token sequences than the input token lists. \n",
    "    \n",
    "Let's check our bert token lengths by running the data through the BERT wordpiece tokenizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bert wordpiece tokenizer max token length in train: 109 tokens\n",
      "Bert wordpiece tokenizer max token length in test: 88 tokens\n",
      "CPU times: user 3.59 s, sys: 20 ms, total: 3.61 s\n",
      "Wall time: 3.97 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "print(\"Bert wordpiece tokenizer max token length in train: %d tokens\"% model.get_max_token_len(X_train))\n",
    "print(\"Bert wordpiece tokenizer max token length in test: %d tokens\"% model.get_max_token_len(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So based on this we will set the max_seq_length to at least 111 = 109 + 2( for the `'[CLS]'` and `'[SEP]'` tokens that Bert uses)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## finetune model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading bert-base-cased model...\n",
      "Defaulting to linear classifier/regressor\n",
      "train data size: 8043, validation data size: 893\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training: 100%|██████████| 503/503 [03:30<00:00,  2.45it/s, loss=0.0724]\n",
      "                                                           "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1, Train loss: 0.0724, Val loss: 0.0168, Val accy: 97.92%, f1: 97.92\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training: 100%|██████████| 503/503 [04:25<00:00,  2.19it/s, loss=0.0132]\n",
      "                                                           "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2, Train loss: 0.0132, Val loss: 0.0148, Val accy: 98.05%, f1: 98.07\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training: 100%|██████████| 503/503 [05:08<00:00,  1.80it/s, loss=0.00871]\n",
      "                                                           "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3, Train loss: 0.0087, Val loss: 0.0144, Val accy: 98.15%, f1: 98.17\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Predicting:   0%|          | 0/126 [00:00<?, ?it/s]          "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test f1: 97.76\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                             \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "      B-ADJP       0.86      0.83      0.85       438\n",
      "      B-ADVP       0.87      0.87      0.87       866\n",
      "     B-CONJP       0.55      0.67      0.60         9\n",
      "      B-INTJ       0.00      0.00      0.00         2\n",
      "       B-LST       0.00      0.00      0.00         5\n",
      "        B-NP       0.99      0.98      0.99     12422\n",
      "        B-PP       0.98      0.99      0.99      4811\n",
      "       B-PRT       0.78      0.92      0.84       106\n",
      "      B-SBAR       0.94      0.96      0.95       535\n",
      "        B-VP       0.98      0.98      0.98      4658\n",
      "      I-ADJP       0.85      0.74      0.79       167\n",
      "      I-ADVP       0.81      0.71      0.75        89\n",
      "     I-CONJP       0.62      0.77      0.69        13\n",
      "       I-LST       0.00      0.00      0.00         2\n",
      "        I-NP       0.98      0.99      0.99     14376\n",
      "        I-PP       0.84      0.77      0.80        48\n",
      "      I-SBAR       0.20      0.75      0.32         4\n",
      "        I-VP       0.97      0.97      0.97      2646\n",
      "           O       0.98      0.97      0.98      6180\n",
      "\n",
      "   micro avg       0.98      0.98      0.98     47377\n",
      "   macro avg       0.69      0.73      0.70     47377\n",
      "weighted avg       0.98      0.98      0.98     47377\n",
      "\n",
      "CPU times: user 9min 20s, sys: 5min 38s, total: 14min 58s\n",
      "Wall time: 14min 59s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# set max_seq_length\n",
    "model.max_seq_length = 111\n",
    "\n",
    "# finetune model on train data\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# score model on test data\n",
    "f1_test = model.score(X_test, y_test)\n",
    "print(\"Test f1: %0.02f\"%(f1_test))\n",
    "\n",
    "# get predictions on test data\n",
    "y_preds = model.predict(X_test)\n",
    "\n",
    "# calculate the probability of each class\n",
    "y_probs = model.predict_proba(X_test)\n",
    "\n",
    "# print report on tag classifier stats\n",
    "print(classification_report(flatten(y_test), flatten(y_preds)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I-CONJP',\n",
       " 'B-SBAR',\n",
       " 'B-PP',\n",
       " 'I-LST',\n",
       " 'B-PRT',\n",
       " 'B-NP',\n",
       " 'I-ADVP',\n",
       " 'I-PP',\n",
       " 'I-SBAR',\n",
       " 'B-INTJ',\n",
       " 'I-PRT',\n",
       " 'I-ADJP',\n",
       " 'B-UCP',\n",
       " 'I-VP',\n",
       " 'B-CONJP',\n",
       " 'B-ADJP',\n",
       " 'I-INTJ',\n",
       " 'B-VP',\n",
       " 'B-LST',\n",
       " 'I-NP',\n",
       " 'B-ADVP',\n",
       " 'I-UCP']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels = list(set(label_list) - set(model.ignore_label))\n",
    "labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want span level precision, recall, and f1 then install this helpful utility:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting seqeval\n",
      "  Downloading https://files.pythonhosted.org/packages/55/dd/3bf1c646c310daabae47fceb84ea9ab66df7f518a31a89955290d82b8100/seqeval-0.0.10-py3-none-any.whl\n",
      "Requirement already satisfied: Keras>=2.2.4 in /root/miniconda3/lib/python3.6/site-packages (from seqeval) (2.2.4)\n",
      "Requirement already satisfied: numpy>=1.14.0 in /root/miniconda3/lib/python3.6/site-packages (from seqeval) (1.15.4)\n",
      "Requirement already satisfied: six>=1.9.0 in /root/miniconda3/lib/python3.6/site-packages (from Keras>=2.2.4->seqeval) (1.11.0)\n",
      "Requirement already satisfied: keras-applications>=1.0.6 in /root/miniconda3/lib/python3.6/site-packages (from Keras>=2.2.4->seqeval) (1.0.6)\n",
      "Requirement already satisfied: keras-preprocessing>=1.0.5 in /root/miniconda3/lib/python3.6/site-packages (from Keras>=2.2.4->seqeval) (1.0.5)\n",
      "Requirement already satisfied: h5py in /root/miniconda3/lib/python3.6/site-packages (from Keras>=2.2.4->seqeval) (2.8.0)\n",
      "Requirement already satisfied: scipy>=0.14 in /root/miniconda3/lib/python3.6/site-packages (from Keras>=2.2.4->seqeval) (1.1.0)\n",
      "Requirement already satisfied: pyyaml in /root/miniconda3/lib/python3.6/site-packages (from Keras>=2.2.4->seqeval) (3.13)\n",
      "\u001b[31mallennlp 0.7.3-unreleased has requirement pytorch-pretrained-bert==0.3.0, but you'll have pytorch-pretrained-bert 0.4.0 which is incompatible.\u001b[0m\n",
      "Installing collected packages: seqeval\n",
      "Successfully installed seqeval-0.0.10\n",
      "\u001b[33mYou are using pip version 10.0.1, however version 19.1.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install seqeval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "           precision    recall  f1-score   support\n",
      "\n",
      "       NP       0.97      0.97      0.97     12422\n",
      "     ADJP       0.82      0.81      0.81       438\n",
      "     ADVP       0.86      0.86      0.86       866\n",
      "       PP       0.98      0.99      0.98      4811\n",
      "       VP       0.96      0.97      0.97      4658\n",
      "      PRT       0.78      0.92      0.84       106\n",
      "     SBAR       0.94      0.96      0.95       535\n",
      "    CONJP       0.38      0.56      0.45         9\n",
      "      LST       0.00      0.00      0.00         5\n",
      "     INTJ       0.00      0.00      0.00         2\n",
      "\n",
      "micro avg       0.96      0.97      0.96     23852\n",
      "macro avg       0.96      0.97      0.96     23852\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from seqeval.metrics import classification_report as seqeval_report\n",
    "print(seqeval_report(flatten(y_test),flatten(y_preds)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also take a look at the example from the test set we looked at before and compare the predicted tags with the actuals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        token label predict\n",
      "0         His  B-NP    B-NP\n",
      "1       meals  I-NP    I-NP\n",
      "2         are  B-VP    B-VP\n",
      "3        most  I-VP    I-VP\n",
      "4       often  I-VP    I-VP\n",
      "5    prepared  I-VP    I-VP\n",
      "6          by  B-PP    B-PP\n",
      "7       women  B-NP    B-NP\n",
      "8          he  B-NP    B-NP\n",
      "9      trusts  B-VP    B-VP\n",
      "10         --     O       O\n",
      "11        his  B-NP    B-NP\n",
      "12  full-time  I-NP    I-NP\n",
      "13   mistress  I-NP    I-NP\n",
      "14          ,     O       O\n",
      "15      Vicky  B-NP    B-NP\n",
      "16      Amado  I-NP    I-NP\n",
      "17          ,     O       O\n",
      "18        and     O       O\n",
      "19        her  B-NP    B-NP\n",
      "20     mother  I-NP    I-NP\n",
      "21          ,     O       O\n",
      "22      Norma  B-NP    B-NP\n",
      "23          .     O       O\n"
     ]
    }
   ],
   "source": [
    "i = 151\n",
    "tokens = X_test[i]\n",
    "labels = y_test[i]\n",
    "preds  = y_preds[i]\n",
    "prob   = y_probs[i]\n",
    "\n",
    "data = {\"token\": tokens,\"label\": labels,\"predict\": preds}\n",
    "df=pd.DataFrame(data=data)\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets calculate tthe probability of each class labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        token  I-CONJP  B-SBAR  B-PP  I-LST  B-PRT  B-NP    O  I-ADVP  I-PP  \\\n",
      "0         His     0.00    0.00  0.00   0.00   0.00  1.00 0.00    0.00  0.00   \n",
      "1       meals     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "2         are     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "3        most     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "4       often     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.14  0.00   \n",
      "5    prepared     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "6          by     0.00    0.00  1.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "7       women     0.00    0.00  0.00   0.00   0.00  1.00 0.00    0.00  0.00   \n",
      "8          he     0.00    0.00  0.00   0.00   0.00  1.00 0.00    0.00  0.00   \n",
      "9      trusts     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "10         --     0.00    0.00  0.00   0.00   0.00  0.00 1.00    0.00  0.00   \n",
      "11        his     0.00    0.00  0.00   0.00   0.00  1.00 0.00    0.00  0.00   \n",
      "12  full-time     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "13   mistress     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "14          ,     0.00    0.00  0.00   0.00   0.00  0.00 1.00    0.00  0.00   \n",
      "15      Vicky     0.00    0.00  0.00   0.00   0.00  0.99 0.00    0.00  0.00   \n",
      "16      Amado     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "17          ,     0.00    0.00  0.00   0.00   0.00  0.00 1.00    0.00  0.00   \n",
      "18        and     0.00    0.00  0.00   0.00   0.00  0.00 1.00    0.00  0.00   \n",
      "19        her     0.00    0.00  0.00   0.00   0.00  1.00 0.00    0.00  0.00   \n",
      "20     mother     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "21          ,     0.00    0.00  0.00   0.00   0.00  0.00 1.00    0.00  0.00   \n",
      "22      Norma     0.00    0.00  0.00   0.00   0.00  0.99 0.00    0.00  0.00   \n",
      "23          .     0.00    0.00  0.00   0.00   0.00  0.00 1.00    0.00  0.00   \n",
      "\n",
      "    ...    B-UCP  I-VP  B-CONJP  B-ADJP  I-INTJ  B-VP  B-LST  I-NP  B-ADVP  \\\n",
      "0   ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "1   ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  1.00    0.00   \n",
      "2   ...     0.00  0.00     0.00    0.00    0.00  1.00   0.00  0.00    0.00   \n",
      "3   ...     0.00  0.82     0.00    0.01    0.00  0.01   0.00  0.00    0.14   \n",
      "4   ...     0.00  0.84     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "5   ...     0.00  1.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "6   ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "7   ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "8   ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "9   ...     0.00  0.00     0.00    0.00    0.00  1.00   0.00  0.00    0.00   \n",
      "10  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "11  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "12  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  1.00    0.00   \n",
      "13  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  1.00    0.00   \n",
      "14  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "15  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.01    0.00   \n",
      "16  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  1.00    0.00   \n",
      "17  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "18  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "19  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "20  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  1.00    0.00   \n",
      "21  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "22  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.01    0.00   \n",
      "23  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "\n",
      "    I-UCP  \n",
      "0    0.00  \n",
      "1    0.00  \n",
      "2    0.00  \n",
      "3    0.00  \n",
      "4    0.00  \n",
      "5    0.00  \n",
      "6    0.00  \n",
      "7    0.00  \n",
      "8    0.00  \n",
      "9    0.00  \n",
      "10   0.00  \n",
      "11   0.00  \n",
      "12   0.00  \n",
      "13   0.00  \n",
      "14   0.00  \n",
      "15   0.00  \n",
      "16   0.00  \n",
      "17   0.00  \n",
      "18   0.00  \n",
      "19   0.00  \n",
      "20   0.00  \n",
      "21   0.00  \n",
      "22   0.00  \n",
      "23   0.00  \n",
      "\n",
      "[24 rows x 24 columns]\n"
     ]
    }
   ],
   "source": [
    "## pprint out probs for this observation\n",
    "tokens_prob = model.tokens_proba(tokens, prob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, lets predict the tags and tag probabilities on some new text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Predicting:   0%|          | 0/1 [00:00<?, ?it/s]        "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     token predicted tags\n",
      "0        I           B-NP\n",
      "1   really         B-ADVP\n",
      "2     want           B-VP\n",
      "3       to           I-VP\n",
      "4       go           I-VP\n",
      "5       to           B-PP\n",
      "6      the           B-NP\n",
      "7   museum           I-NP\n",
      "8    today           B-NP\n",
      "9        ,              O\n",
      "10     but              O\n",
      "11       I           B-NP\n",
      "12   first         B-ADVP\n",
      "13    need           B-VP\n",
      "14      to           I-VP\n",
      "15   visit           I-VP\n",
      "16      my           B-NP\n",
      "17     mom           I-NP\n",
      "18     and           I-NP\n",
      "19     dad           I-NP\n",
      "20       .              O\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                         "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     token  I-CONJP  B-SBAR  B-PP  I-LST  B-PRT  B-NP    O  I-ADVP  I-PP  \\\n",
      "0        I     0.00    0.00  0.00   0.00   0.00  1.00 0.00    0.00  0.00   \n",
      "1   really     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "2     want     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "3       to     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "4       go     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "5       to     0.00    0.00  1.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "6      the     0.00    0.00  0.00   0.00   0.00  1.00 0.00    0.00  0.00   \n",
      "7   museum     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "8    today     0.00    0.00  0.00   0.00   0.00  1.00 0.00    0.00  0.00   \n",
      "9        ,     0.00    0.00  0.00   0.00   0.00  0.00 1.00    0.00  0.00   \n",
      "10     but     0.00    0.00  0.00   0.00   0.00  0.00 1.00    0.00  0.00   \n",
      "11       I     0.00    0.00  0.00   0.00   0.00  1.00 0.00    0.00  0.00   \n",
      "12   first     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "13    need     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "14      to     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "15   visit     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "16      my     0.00    0.00  0.00   0.00   0.00  1.00 0.00    0.00  0.00   \n",
      "17     mom     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "18     and     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "19     dad     0.00    0.00  0.00   0.00   0.00  0.00 0.00    0.00  0.00   \n",
      "20       .     0.00    0.00  0.00   0.00   0.00  0.00 1.00    0.00  0.00   \n",
      "\n",
      "    ...    B-UCP  I-VP  B-CONJP  B-ADJP  I-INTJ  B-VP  B-LST  I-NP  B-ADVP  \\\n",
      "0   ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "1   ...     0.00  0.00     0.00    0.00    0.00  0.01   0.00  0.00    0.98   \n",
      "2   ...     0.00  0.01     0.00    0.00    0.00  0.99   0.00  0.00    0.00   \n",
      "3   ...     0.00  1.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "4   ...     0.00  1.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "5   ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "6   ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "7   ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  1.00    0.00   \n",
      "8   ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "9   ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "10  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "11  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "12  ...     0.00  0.00     0.00    0.00    0.00  0.02   0.00  0.00    0.98   \n",
      "13  ...     0.00  0.01     0.00    0.00    0.00  0.99   0.00  0.00    0.00   \n",
      "14  ...     0.00  1.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "15  ...     0.00  1.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "16  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "17  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  1.00    0.00   \n",
      "18  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  1.00    0.00   \n",
      "19  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  1.00    0.00   \n",
      "20  ...     0.00  0.00     0.00    0.00    0.00  0.00   0.00  0.00    0.00   \n",
      "\n",
      "    I-UCP  \n",
      "0    0.00  \n",
      "1    0.00  \n",
      "2    0.00  \n",
      "3    0.00  \n",
      "4    0.00  \n",
      "5    0.00  \n",
      "6    0.00  \n",
      "7    0.00  \n",
      "8    0.00  \n",
      "9    0.00  \n",
      "10   0.00  \n",
      "11   0.00  \n",
      "12   0.00  \n",
      "13   0.00  \n",
      "14   0.00  \n",
      "15   0.00  \n",
      "16   0.00  \n",
      "17   0.00  \n",
      "18   0.00  \n",
      "19   0.00  \n",
      "20   0.00  \n",
      "\n",
      "[21 rows x 24 columns]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    }
   ],
   "source": [
    "text = \"I really want to go to the museum today, but I first need to visit my mom and dad.\"       \n",
    "\n",
    "tag_predicts  = model.tag_text(text)       \n",
    "prob_predicts = model.tag_text_proba(text)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}