Sunoikisis - Named Entity Extraction 2a-FM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "toc": "true"
   },
   "source": [
    "# Table of Contents\n",
    " <p><div class=\"lev1 toc-item\"><a href=\"#Summary-of-the-previous-lecture\" data-toc-modified-id=\"Summary-of-the-previous-lecture-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Summary of the previous lecture</a></div><div class=\"lev2 toc-item\"><a href=\"#Libraries-and-import-statements\" data-toc-modified-id=\"Libraries-and-import-statements-11\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Libraries and import statements</a></div><div class=\"lev2 toc-item\"><a href=\"#Data-types\" data-toc-modified-id=\"Data-types-12\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Data types</a></div><div class=\"lev2 toc-item\"><a href=\"#Data-collections-(and-variables)\" data-toc-modified-id=\"Data-collections-(and-variables)-13\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Data collections (and variables)</a></div><div class=\"lev2 toc-item\"><a href=\"#For-loops-and-if-statements\" data-toc-modified-id=\"For-loops-and-if-statements-14\"><span class=\"toc-item-num\">1.4&nbsp;&nbsp;</span>For loops and if statements</a></div><div class=\"lev2 toc-item\"><a href=\"#Functions\" data-toc-modified-id=\"Functions-15\"><span class=\"toc-item-num\">1.5&nbsp;&nbsp;</span>Functions</a></div><div class=\"lev2 toc-item\"><a href=\"#Handling-exceptions\" data-toc-modified-id=\"Handling-exceptions-16\"><span class=\"toc-item-num\">1.6&nbsp;&nbsp;</span>Handling exceptions</a></div><div class=\"lev2 toc-item\"><a href=\"#A-bonus:-objects\" data-toc-modified-id=\"A-bonus:-objects-17\"><span class=\"toc-item-num\">1.7&nbsp;&nbsp;</span>A bonus: objects</a></div><div class=\"lev1 toc-item\"><a href=\"#Regular-expressions\" data-toc-modified-id=\"Regular-expressions-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Regular expressions</a></div><div class=\"lev1 toc-item\"><a href=\"#Extracting-dates-and-persons-from-texts\" data-toc-modified-id=\"Extracting-dates-and-persons-from-texts-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Extracting dates and persons from texts</a></div><div class=\"lev2 toc-item\"><a href=\"#A-modern-text-in-English\" data-toc-modified-id=\"A-modern-text-in-English-31\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>A modern text in English</a></div><div class=\"lev2 toc-item\"><a href=\"#Part-Of-Speech-(POS)-and-Named-Entity-(NE)-Tagging\" data-toc-modified-id=\"Part-Of-Speech-(POS)-and-Named-Entity-(NE)-Tagging-32\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Part-Of-Speech (POS) and Named-Entity (NE) Tagging</a></div><div class=\"lev2 toc-item\"><a href=\"#Chunking\" data-toc-modified-id=\"Chunking-33\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>Chunking</a></div><div class=\"lev2 toc-item\"><a href=\"#Export-to-IOB-notation\" data-toc-modified-id=\"Export-to-IOB-notation-34\"><span class=\"toc-item-num\">3.4&nbsp;&nbsp;</span>Export to IOB notation</a></div><div class=\"lev1 toc-item\"><a href=\"#Regex-tagger\" data-toc-modified-id=\"Regex-tagger-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Regex tagger</a></div><div class=\"lev1 toc-item\"><a href=\"#Exercise\" data-toc-modified-id=\"Exercise-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Exercise</a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Summary of the previous lecture "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In our previous common session, we have introduced some fundamental notions of the Python language. Let's review some of them!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Libraries and import statements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from idai_journals import nlp as dainlp\n",
    "import re\n",
    "from treetagger import TreeTagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from nltk.tag import StanfordNERTagger\n",
    "from nltk.chunk.util import tree2conlltags\n",
    "from nltk.chunk import RegexpParser\n",
    "from nltk.tree import Tree\n",
    "from nltk.tag import StanfordNERTaggelr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Data types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#interges and floats\n",
    "3 + 0.5\n",
    "#strings\n",
    "\"hello\"\n",
    "#Booleans\n",
    "True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Data collections (and variables)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#lists (can also contain multiple different data types)\n",
    "li  = [\"Leipzig\", \"London\", \"Berlin\", \"Boston\", 4, False]\n",
    "#tuples (like lists, but immutable)\n",
    "tu = (\"tuple\", \"list\", \"dictionary\")\n",
    "#dictionaries (key : value pairs)\n",
    "di = {\"key\" : \"value\", \"other-key\" : \"second value\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## For loops and if statements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Leipzig\n",
      "London\n",
      "Berlin\n",
      "Boston\n"
     ]
    }
   ],
   "source": [
    "#home assignment: try to figure out what the if statement (line 2) does\n",
    "for l in li:\n",
    "    if isinstance(l, str):\n",
    "        print(l)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello, world!\n",
      "goodbye...\n"
     ]
    }
   ],
   "source": [
    "def printMe(message):\n",
    "    print(message)\n",
    "    \n",
    "printMe(\"Hello, world!\")\n",
    "printMe(\"goodbye...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Handling exceptions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-44-9c6d14a570dc>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0ml\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m\"zero\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"one\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"two\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"three\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0ml\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "l = [\"zero\", \"one\", \"two\", \"three\"]\n",
    "l[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hey, your index is way too high!\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    l[10]\n",
    "except IndexError:\n",
    "    print(\"hey, your index is way too high!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## A bonus: objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Objects might be a bit complicated, but they're very important to understand the code written by other people, while most of the programs that you'll find around is written using classes and objects. Oh, and the good news is... you've already met them!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "What are \"objects\" in a programming language like Python? Well, I like to think about them as... **magical, animated tools**!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "![caption](https://s.aolcdn.com/dims-shared/dims3/GLOB/crop/2048x1086+0+384/resize/660x350!/format/jpg/quality/85/https://s.aolcdn.com/hss/storage/midas/6df6ff55a4f197cf759a1027d3308654/202946939/Mickey_Apprentice.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Say that you want to fetch water from a well (and maybe clean some of the mess...). Well, the **object-oriented** approach to this tak consists in creating one or more magic brooms that go and fetch the water for you! In order to create them, you have to conceptualize the broom in terms of:\n",
    "\n",
    "* the special features it has (e.g. number of buckets carried, speed...)\n",
    "* the actions that it can execute (fetch water, clean the floor)\n",
    "\n",
    "That's it! In programming parlance, the features are called **properties** of the object; the actions are called **methods**.\n",
    "\n",
    "When you want to build your own magic brooms you first create a sort of prototype for each of them (which is called the **class** of magic brooms); then you can go on and create as many brooms as you want...\n",
    "\n",
    "Here's how to do it! (very simplified)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class MagicBroom():\n",
    "    #this is called \"constructor\"; it's a special method\n",
    "    def __init__(self, name, speed=20):\n",
    "        self.name = name\n",
    "        self.buckets = 2\n",
    "        self.speed = speed\n",
    "\n",
    "    def greet(self):\n",
    "        print(\"Hello, my name is %s! What can I do for you?\" % self.name)\n",
    "    \n",
    "    def fetchWater(self):\n",
    "        if self.speed >= 20:\n",
    "            print(\"Yes, sir! I'll be back in a sec!\")\n",
    "        else:\n",
    "            print(\"Allright, but I am taking my time!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello, my name is Mickey! What can I do for you?\n"
     ]
    }
   ],
   "source": [
    "mickey = MagicBroom(\"Mickey\")\n",
    "mickey.greet()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "peter = MagicBroom(\"Peter\", speed=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mickey.speed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Yes, sir! I'll be back in a sec!\n"
     ]
    }
   ],
   "source": [
    "mickey.fetchWater()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Allright, but I am taking my time!\n"
     ]
    }
   ],
   "source": [
    "peter.fetchWater()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Regular expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "How would you find all the numbers in [this sentence](https://en.wikipedia.org/wiki/Integer)?\n",
    "> The set of integers consists of zero (**0**), the positive natural numbers (**1, 2, 3**, …), also called whole numbers or counting numbers,**[1][2]** and their additive inverses (the negative integers, i.e., **−1, −2, −3,** …). This is often denoted by a boldface Z (\"Z\") or blackboard bold Z {\\displaystyle \\mathbb {Z} } \\mathbb {Z} (Unicode **U+2124** ℤ) standing for the German word Zahlen ([ˈtsaːlən], \"numbers\").**[3][4]** ℤ is a subset of the sets of rational and real numbers and, like the natural numbers, is countably infinite."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "wiki = 'The set of integers consists of zero (0), the positive natural numbers (1, 2, 3, …), also called whole numbers or counting numbers,[1][2] and their additive inverses (the negative integers, i.e., −1, −2, −3, …). This is often denoted by a boldface Z (\"Z\") or blackboard bold Z {\\displaystyle \\mathbb {Z} } \\mathbb {Z} (Unicode U+2124 ℤ) standing for the German word Zahlen ([ˈtsaːlən], \"numbers\").[3][4] ℤ is a subset of the sets of rational and real numbers and, like the natural numbers, is countably infinite.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We'd need a way to tell our machine not to look for specific strings, bur rather for **classes of strings**, i.e. using some sort of **meta-character** to catch a whole group of signs (e.g. the numbers); then we'd need to tell to optionally include/exclude some other signs, or to catch the numbers only if they're not preceeded/followed by other signs..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "That's precisely what **Regular Expressions** do! They allow you to express a query as a string of metacharacters (or groups of metacharacters).\n",
    "\n",
    "How do we use them in Python? First, we need to import a module from the Standard Library (i.e. you already have them with Python: no need to install external libraries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A cool feature of RegExp in Python is that you can create your complicated patterns as objects (and assign them to variables)! That's right, **RegExp patterns are your magic brooms**..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "_sre.SRE_Pattern"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#here is one to catch all numbers\n",
    "reg = re.compile(r'[0-9]+') #or: r'\\d+'\n",
    "type(reg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The Pattern object has a number of interesting methods to search and replace the pattern. Generally, you use them with the text that must be searched as an argument. For instance, **findall** returns all matches as a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0', '1', '2', '3', '1', '2', '1', '2', '3', '2124', '3', '4']"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg.findall(wiki)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Kind of a sloppy job we did! The negative numbers are not captured as negative; the footnote reference (e.g. [1], [4]) are also captured and we don't want them... We can do better. Let's improve our pattern so that we include the '-' signs (if present) and we get rid of the footnotes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0', '1', '2', '3', '−1', '−2', '−3', '2124']"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg = re.compile(r'(?<!\\[)−?\\d+(?!\\])') # the 'r' is there to make sure that we don't have to \"escape the escape\" sing (\\)\n",
    "reg.findall(wiki)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now it's time to go back to our task of **(Named) Entity** recognition and extraction task. But we're going to use RegExp patterns and syntax quite a few times now..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Extracting dates and persons from texts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "As Matteo said last time, the concept of \"named entity\" is domain- and task- specific. While a person's or a place's name will more or less always fall under the definition, in some contexts of information extraction people might be interested in other kinds of real-life \"entities\", such as time references (months, days, dates) or museum objects, which are not relevant to others.\n",
    "\n",
    "In this exercise, we are going to expand on what Matteo did last time with proper names in Latin and look at two specific classes of \"entities\" mentioned in a modern scientific text about ancient history: dates and persons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## A modern text in English"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First, let's grab a text.\n",
    "\n",
    "We will be working with an English [article](https://publications.dainst.org/journals/index.php/chiron/article/view/466/2668) on Roman history. The article is: Frederik Juliaan Vervaet, The Praetorian Proconsuls of the Roman Republic (211–52 BCE). A Constitutional Survey, *Chiron* 42(2012): 45-96.\n",
    "\n",
    "Let's start by loading the text and inspect the first 10.000 characters (we'll be working with just the first 10k words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "with open(\"data/txt/article446_10k.txt\") as f:\n",
    "    txt = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\x0cThe Praetorian Proconsuls of the Roman Republic 45 \\nFREDERIK JULIAAN VERVAET \\nThe Praetorian Proconsuls of the Roman Republic (211–52 BCE). \\nA Constitutional Survey \\n1. Introduction \\nThe republican administrative procedure of sending out praetors with consular imperium is reasonably well-known but little understood. To the best of my knowledge, \\nnot a single study or book chapter has been devoted exclusively to a gubernatorial \\npractice that rapidly gained importance from the Second Punic War. This bipartite \\nstudy aims at bridging this remarkable gap. The first component of this inquiry endeavours to offer an overall constitutional survey of the institutional phenomenon of \\nthe praetura pro consule by discussing its origins, nature and historical development. \\nThe second part is conducted by F. Hurlet and scrutinizes this practice as recorded \\nin the fasti of Africa, Sicily and Corsica-Sardinia. After highlighting the significance of \\nthe Metilian Law from 217 BCE as a precedent, the'"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "txt[:1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Part-Of-Speech (POS) and Named-Entity (NE) Tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Most of the time POS tagging is the precondition before you can perform any other advanced operation on a text\n",
    "\n",
    "As we did with Matteo last time, by \"tagging\" we mean the coupling of each word with a **tag** that describe some property of the word itself. Part-of-speech tags define what word class (e.g. \"verb\", or \"proper noun\") a text token belongs to.\n",
    "\n",
    "There are several tagset used for each language, and several software (pos taggers) who can tag your text automatically. One of the most used is [TreeTagger](http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/), which has pretrained classifiers for many languages.\n",
    "\n",
    "Let's run it from Python, using one of the few \"wrappers\" available"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#first we load the library\n",
    "from treetagger import TreeTagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#That's right! we start by creating a Tagger \"magic broom\" (a Tagger object)\n",
    "tt = TreeTagger(language=\"english\")\n",
    "\n",
    "#then we tag our text\n",
    "tagged = tt.tag(txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['The', 'DT', 'the'],\n",
       " ['Praetorian', 'JJ', 'Praetorian'],\n",
       " ['Proconsuls', 'NNS', 'proconsul'],\n",
       " ['of', 'IN', 'of'],\n",
       " ['the', 'DT', 'the'],\n",
       " ['Roman', 'NP', 'Roman'],\n",
       " ['Republic', 'NP', 'Republic'],\n",
       " ['45', 'CD', '@card@'],\n",
       " ['FREDERIK', 'NP', 'Frederik'],\n",
       " ['JULIAAN', 'NP', '<unknown>'],\n",
       " ['VERVAET', 'NP', '<unknown>'],\n",
       " ['The', 'DT', 'the'],\n",
       " ['Praetorian', 'JJ', 'Praetorian'],\n",
       " ['Proconsuls', 'NNS', 'proconsul'],\n",
       " ['of', 'IN', 'of'],\n",
       " ['the', 'DT', 'the'],\n",
       " ['Roman', 'NP', 'Roman'],\n",
       " ['Republic', 'NP', 'Republic'],\n",
       " ['(', '(', '('],\n",
       " ['211–52', 'JJ', '<unknown>']]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagged[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Named Entity Recognition (using a tool like the Stanford NER that we saw in our last lecture) is also a way of tagging the text, this time using information not on the word class but on a different level of classification (place, person, organization or none of the above).\n",
    "\n",
    "Let's do this too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#first, we define the path to the English classifier for Stanford NER\n",
    "english_classifier = 'english.all.3class.distsim.crf.ser.gz'\n",
    "twords = [w[0] for w in tagged]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#then... guess what? Yes, we create a NER-tagger Magic Broom ;-)\n",
    "from nltk.tag import StanfordNERTagger\n",
    "\n",
    "ner_tagger = StanfordNERTagger(english_classifier)\n",
    "ners = ner_tagger.tag(twords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('The', 'O'),\n",
       " ('Praetorian', 'O'),\n",
       " ('Proconsuls', 'O'),\n",
       " ('of', 'O'),\n",
       " ('the', 'O'),\n",
       " ('Roman', 'LOCATION'),\n",
       " ('Republic', 'LOCATION'),\n",
       " ('45', 'O'),\n",
       " ('FREDERIK', 'O'),\n",
       " ('JULIAAN', 'O'),\n",
       " ('VERVAET', 'O'),\n",
       " ('The', 'O'),\n",
       " ('Praetorian', 'O'),\n",
       " ('Proconsuls', 'O'),\n",
       " ('of', 'O'),\n",
       " ('the', 'O'),\n",
       " ('Roman', 'LOCATION'),\n",
       " ('Republic', 'LOCATION'),\n",
       " ('(', 'O'),\n",
       " ('211–52', 'O')]"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#not very pretty...\n",
    "ners[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Chunking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "As we saw, when we analyze a text we proceed word by word (more exactly: token by token). However, Named Entities (now including dates) often span over more than one token. The task of sub-dividing a section of text into phrases and/or meaningful constituents (which may include 1 or more text tokens) is called [chunking](http://www.nltk.org/book/ch07.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "![chunking](http://www.nltk.org/images/chunk-segmentation.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In the image above, the tokens are [We, saw, the, yellow, dog]. Two Noun Phrases (NP) can be chunked:\n",
    "* \"we\" (1 token)\n",
    "* \"the yellow dog\" (3 tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The **IOB notation** that Matteo introduced last time is a popular way to store the information about chunks in a word-by-word format. In the case of \"the yellow dog\", we will have:\n",
    "* saw = not in a chunk --> O\n",
    "* the = beginning of the chunk --> B-NP\n",
    "* yellow = internal part of the chunk --> I-NP\n",
    "* dog = internal part of the chunk --> I-NP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The easiest method for chunking a sentence in Python is to use the information in the Tag and a regexp syntax.\n",
    "\n",
    "For example, if we have:\n",
    "\n",
    "``\n",
    "in O\n",
    "New LOCATION\n",
    "York LOCATION\n",
    "City LOCATION\n",
    "``\n",
    "\n",
    "We easily see that the 3 tokens tagged as LOCATION go together. We may thus write a grammar rule that chunks the LOC together:\n",
    "\n",
    "``\n",
    "LOC:\n",
    "  {<LOCATION><LOCATION>*}\n",
    "``\n",
    "\n",
    "Which means group in a chunk named `LOC` every token tagged as `LOCATION`, including any token tagged as `LOCATION` that might optionally come after.\n",
    "\n",
    "And the same goes also for `PERSONS` and `ORGANIZATIONS`. We may even use RegExp syntax to be more tollerant and make room for annotation errors, in case e.g. the two tokens *Geore Washington* are wrongly tagged as `PERSON` and `LOCATION`.\n",
    "\n",
    "Here's how I'd do it (it's not perfect at all but it should work in most cases)..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from nltk.chunk import RegexpParser\n",
    "\n",
    "english_chunker = RegexpParser(r'''\n",
    "LOC:\n",
    "    {<LOCATION><(PERSON|LOCATION|MISC|ORGANIZATION)>*}\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's see it in action with the first few words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(S\n",
      "  The/O\n",
      "  Praetorian/O\n",
      "  Proconsuls/O\n",
      "  of/O\n",
      "  the/O\n",
      "  (LOC Roman/LOCATION Republic/LOCATION)\n",
      "  45/O\n",
      "  FREDERIK/O\n",
      "  JULIAAN/O\n",
      "  VERVAET/O\n",
      "  The/O\n",
      "  Praetorian/O\n",
      "  Proconsuls/O\n",
      "  of/O\n",
      "  the/O\n",
      "  (LOC Roman/LOCATION Republic/LOCATION)\n",
      "  (/O\n",
      "  211–52/O)\n"
     ]
    }
   ],
   "source": [
    "tree = english_chunker.parse(ners[:20])\n",
    "print(tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Well... OK, \"Roman Republic\" is not a location, but at least the chunking is exactly what we wanted to have, right?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Export to IOB notation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "OK, but now how do we convert this to the IOB notation?\n",
    "\n",
    "Luckily, there's a ready-made function in a module from the NLTK library! Let's load and use it\n",
    "\n",
    "(just in case, there is also a function that does the reverse: from IOB to tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from nltk.chunk.util import tree2conlltags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "iobs = tree2conlltags(tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('The', 'O', 'O'),\n",
       " ('Praetorian', 'O', 'O'),\n",
       " ('Proconsuls', 'O', 'O'),\n",
       " ('of', 'O', 'O'),\n",
       " ('the', 'O', 'O'),\n",
       " ('Roman', 'LOCATION', 'B-LOC'),\n",
       " ('Republic', 'LOCATION', 'I-LOC'),\n",
       " ('45', 'O', 'O'),\n",
       " ('FREDERIK', 'O', 'O'),\n",
       " ('JULIAAN', 'O', 'O'),\n",
       " ('VERVAET', 'O', 'O'),\n",
       " ('The', 'O', 'O'),\n",
       " ('Praetorian', 'O', 'O'),\n",
       " ('Proconsuls', 'O', 'O'),\n",
       " ('of', 'O', 'O'),\n",
       " ('the', 'O', 'O'),\n",
       " ('Roman', 'LOCATION', 'B-LOC'),\n",
       " ('Republic', 'LOCATION', 'I-LOC'),\n",
       " ('(', 'O', 'O'),\n",
       " ('211–52', 'O', 'O')]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iobs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Regex tagger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, to go back to our original task, how do we use all this to annotate the dates and export them to IOB?\n",
    "\n",
    "Dates are often just numbers (e.g. \"2017\"); sometimes they come in more complex formats like: \"14 September 2017\" or \"14-09-2017\". \n",
    "\n",
    "One very simple solutions to find them and annotate them with a chunking notation might be to tag the tokens of our text with a very simple custom tagset that we design for dates. We assign \"O\" to all tokens, save the numbers (that we tag \"CD\") and some selected time formats or expressions, like the months of the year or the sequence number-number. We use the tag \"Date\" for them.\n",
    "\n",
    "In order to do this, we need:\n",
    "\n",
    "* regular expression syntax\n",
    "* a tagger that works with RegExp patterns\n",
    "\n",
    "A module of NLTK provides with exactly that tagger that can work with RegExp syntax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from nltk.tag import RegexpTagger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#here is our list of patterns\n",
    "patterns = [\n",
    "    (r'\\d+$', 'CD'),\n",
    "    (r'\\d+[-–]\\d+$', \"Date\"),\n",
    "    (r'\\d{1,2}[-\\.\\/]\\d{1,2}[-\\.\\/]\\d{2,4}', \"Date\"),\n",
    "    (r'January|February|March|April|May|June|July|August|September|October|November|December', \"Date\"),\n",
    "    (r'\\d{4}$', \"Date\"),\n",
    "    (r'BCE|BC|AD', \"Date\"),\n",
    "    (r'.*', \"O\")\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#Our RegexpTagger magic broom! We initialize it with our pattern list\n",
    "tagger = RegexpTagger(patterns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('I', 'O'),\n",
       " ('was', 'O'),\n",
       " ('born', 'O'),\n",
       " ('on', 'O'),\n",
       " ('September', 'Date'),\n",
       " ('14', 'CD'),\n",
       " (',', 'O'),\n",
       " ('or', 'O'),\n",
       " ('14-09', 'Date')]"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#let's test it with a trivial example\n",
    "tagger.tag(\"I was born on September 14 , or 14-09\".split(\" \"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now let's see it in action on the real stuff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reg_tag = tagger.tag(twords)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('The', 'O'),\n",
       " ('Praetorian', 'O'),\n",
       " ('Proconsuls', 'O'),\n",
       " ('of', 'O'),\n",
       " ('the', 'O'),\n",
       " ('Roman', 'O'),\n",
       " ('Republic', 'O'),\n",
       " ('45', 'CD'),\n",
       " ('FREDERIK', 'O'),\n",
       " ('JULIAAN', 'O'),\n",
       " ('VERVAET', 'O'),\n",
       " ('The', 'O'),\n",
       " ('Praetorian', 'O'),\n",
       " ('Proconsuls', 'O'),\n",
       " ('of', 'O'),\n",
       " ('the', 'O'),\n",
       " ('Roman', 'O'),\n",
       " ('Republic', 'O'),\n",
       " ('(', 'O'),\n",
       " ('211–52', 'Date'),\n",
       " ('BCE', 'Date'),\n",
       " (')', 'O'),\n",
       " ('.', 'O'),\n",
       " ('A', 'O'),\n",
       " ('Constitutional', 'O'),\n",
       " ('Survey', 'O'),\n",
       " ('1', 'CD'),\n",
       " ('.', 'O'),\n",
       " ('Introduction', 'O'),\n",
       " ('The', 'O'),\n",
       " ('republican', 'O'),\n",
       " ('administrative', 'O'),\n",
       " ('procedure', 'O'),\n",
       " ('of', 'O'),\n",
       " ('sending', 'O'),\n",
       " ('out', 'O'),\n",
       " ('praetors', 'O'),\n",
       " ('with', 'O'),\n",
       " ('consular', 'O'),\n",
       " ('imperium', 'O'),\n",
       " ('is', 'O'),\n",
       " ('reasonably', 'O'),\n",
       " ('well-known', 'O'),\n",
       " ('but', 'O'),\n",
       " ('little', 'O'),\n",
       " ('understood', 'O'),\n",
       " ('.', 'O'),\n",
       " ('To', 'O'),\n",
       " ('the', 'O'),\n",
       " ('best', 'O')]"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg_tag[:50]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we just need to chunk it and export it to IOB. Then we are ready to evaluate the manual annotation...\n",
    "\n",
    "First, we have to define a chunker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "date_chunker = RegexpParser(r'''\n",
    "DATE:\n",
    "    {<CD>*<Date><Date|CD>*}\n",
    "DATE:\n",
    "    {<CD>+}\n",
    "''')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "t = date_chunker.parse(reg_tag)\n",
    "\n",
    "#we use that function to make sure that the tree is not too complex to be converted\n",
    "flat = dainlp.flatten_tree(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "iob_list = tree2conlltags(flat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('The', 'O', 'O'),\n",
       " ('Praetorian', 'O', 'O'),\n",
       " ('Proconsuls', 'O', 'O'),\n",
       " ('of', 'O', 'O'),\n",
       " ('the', 'O', 'O'),\n",
       " ('Roman', 'O', 'O'),\n",
       " ('Republic', 'O', 'O'),\n",
       " ('45', 'CD', 'B-DATE'),\n",
       " ('FREDERIK', 'O', 'O'),\n",
       " ('JULIAAN', 'O', 'O'),\n",
       " ('VERVAET', 'O', 'O'),\n",
       " ('The', 'O', 'O'),\n",
       " ('Praetorian', 'O', 'O'),\n",
       " ('Proconsuls', 'O', 'O'),\n",
       " ('of', 'O', 'O'),\n",
       " ('the', 'O', 'O'),\n",
       " ('Roman', 'O', 'O'),\n",
       " ('Republic', 'O', 'O'),\n",
       " ('(', 'O', 'O'),\n",
       " ('211–52', 'Date', 'B-DATE'),\n",
       " ('BCE', 'Date', 'I-DATE'),\n",
       " (')', 'O', 'O'),\n",
       " ('.', 'O', 'O'),\n",
       " ('A', 'O', 'O'),\n",
       " ('Constitutional', 'O', 'O'),\n",
       " ('Survey', 'O', 'O'),\n",
       " ('1', 'CD', 'B-DATE'),\n",
       " ('.', 'O', 'O'),\n",
       " ('Introduction', 'O', 'O'),\n",
       " ('The', 'O', 'O'),\n",
       " ('republican', 'O', 'O'),\n",
       " ('administrative', 'O', 'O'),\n",
       " ('procedure', 'O', 'O'),\n",
       " ('of', 'O', 'O'),\n",
       " ('sending', 'O', 'O'),\n",
       " ('out', 'O', 'O'),\n",
       " ('praetors', 'O', 'O'),\n",
       " ('with', 'O', 'O'),\n",
       " ('consular', 'O', 'O'),\n",
       " ('imperium', 'O', 'O'),\n",
       " ('is', 'O', 'O'),\n",
       " ('reasonably', 'O', 'O'),\n",
       " ('well-known', 'O', 'O'),\n",
       " ('but', 'O', 'O'),\n",
       " ('little', 'O', 'O'),\n",
       " ('understood', 'O', 'O'),\n",
       " ('.', 'O', 'O'),\n",
       " ('To', 'O', 'O'),\n",
       " ('the', 'O', 'O'),\n",
       " ('best', 'O', 'O')]"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iob_list[:50]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#then we can write it on an output file\n",
    "with open(\"data/iob/article_446_date_aut.iob\", \"w\") as out:\n",
    "    for i in iob_list:\n",
    "        out.write(\"\\t\".join(i)+\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In the practical exercise, you are requested to extract the **person names** from the same article that we used for dates. You will annotate them using the Stanford NER with the pre-trained classifier for English that come with the software; extract the Person chunks; evaluate the results against a golden standard.\n",
    "\n",
    "Here is a **summary of the steps** that you will have to execute in order to solve the exercise:\n",
    "\n",
    "* load the file: `data/txt/article446_10k.txt` and read its content\n",
    "* annotate the Named Entities using Stanford NER\n",
    "* define an appropriate chunker for Persons\n",
    "* chunk the extracted Named Entities\n",
    "* convert the chunked Tree into IOB format\n",
    "* evaluate the IOB annotation using the appropriate functions\n",
    "    * use the file: `data/iob/article_446_person_GOLD.iob` as gold standard\n",
    "* report the final evaluation metrics (precision, recall, F-score)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#just remember that the path to the English pre-trained classifier for Stanfor NER is\n",
    "english_classifier = 'english.all.3class.distsim.crf.ser.gz'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the evaluation of the accuracy of your classifier, you can adapt the following lines of code:\n",
    "\n",
    "```python\n",
    "\n",
    "from sklearn.metrics import precision_recall_fscore_support\n",
    "precision, recall, fscore, support = precision_recall_fscore_support(gold_labels\n",
    "                                                                     , auto_labels\n",
    "                                , average=\"micro\"\n",
    "                                , labels=[\"B-DATE\",\"I-DATE\"])\n",
    "print(\"Precision: {0:.2f}\".format(precision))\n",
    "print(\"Recall: {0:.2f}\".format(recall))\n",
    "print(\"F1-score: {0:.2f}\".format(fscore))\n",
    "\n",
    "```\n",
    "\n",
    "Things you'll need to change/provide:\n",
    "- list of positive labels (variable `labels`)\n",
    "- `gold_labels`: a list with the correct labels\n",
    "- `auto_labels`: a similar list with the labels output by your classifier.\n",
    "\n",
    "**NB**: make sure that `gold_labels` and `auto_labels` are of the same lenght, i.e. that both labels at position `n` in  both lists refer to the same token.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "300px",
    "width": "251px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": true,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}