Lectures/Week-14-RegEx.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spring 2018 Status: UNREVIEWED\n",
    "\n",
    "# Regular expressions\n",
    "\n",
    "For the past two weeks we've been learning how to extract text out of XML documents via XPath statements.  Now we're going to turn our attention to pure text.  So whereas XPath statements describe patterns of locations in XML documents, regular expressions describe how text might look.\n",
    "\n",
    "For example, if we're looking for instances in a document where a year is noted (e.g. 2017), we could search for the year we expected.  But what if there a range of years?  We could search for \"201\" to catch 2010-2019, but we'd need to switch patterns for anything outside that range.  Likewise, if you're looking for a year anywhere between 1900-1999 you could search for just \"19\" but now you might get ages, days, or other numerical values.\n",
    "\n",
    "This kind of search is exactly what regular expressions are designed to do.  Instead of starting with constituant numbers, we could build up a more specific pattern of what that year might look like.  \n",
    "\n",
    "For the range: 2010-2019:\n",
    "\n",
    "* We know that a year is composed of four integer values all together. Yes, sometimes years might be in \"'nn\" or \"nn\" format, but let's say there will be 4 numbers in the document we're searching on.\n",
    "* We know that the first three numbers will be constant:  a literal 2, a literal 0, and a literal 1.  \n",
    "* The last number is what can vary, and we can say it is any integer value between 0 and 9.\n",
    "\n",
    "These three rules can be described within a regular expression. \n",
    "\n",
    "This is a good time to skip off and do some reading before continuing.  Here are some things to skim over for background:  https://en.wikipedia.org/wiki/Regular_expression  You can skim some of the narrative for background, but I'd like you to do more focused reading on:\n",
    "\n",
    "* Basic concepts: https://en.wikipedia.org/wiki/Regular_expression#Basic_concepts\n",
    "* Formal langugae: https://en.wikipedia.org/wiki/Regular_expression#Formal_language_theory\n",
    "    * Gloss over the math notation, focus on the examples and context\n",
    "* Syntax:  https://en.wikipedia.org/wiki/Regular_expression#Syntax\n",
    "    * Head for that table and focus on reading the examples rather than understanding the technical jargon.\n",
    "    \n",
    "Don't worry about the other sections.\n",
    "\n",
    "Many software programs dealing with text have support for regular expression searches.  Even PyCharm!  When you open up a text file, start a search inside of it, and you'll see a check box for Regex.  So you can practice on this without needing to use Python.\n",
    "\n",
    "In fact, that's what I recommend.  Particularly when you're trying to do known item searches within a text document, you often want to iteratively experiment with your expressions directly on the document before you bring it into python to extract those results.  This will give you instant feedback of what it is finding.\n",
    "\n",
    "For example, say you have a document with 100 records.  The data is semi-structured, so you've decided to use a regular expression to extract out a certain data point.  As your query, there will be a result count.  When you think you have your expression done, check the count.  If you see something other than 100, you know that you need to change it.  \n",
    "\n",
    "* A number less than your known result count means that you've made your expression too restrictive.  You're falsely rejecting some data.  \n",
    "* A number more than your known result count means that you've made your expression too permissive.  You're flasely accepting some data.\n",
    "\n",
    "There will be times that you cannot get exactly what you need with a single regular expression.  That's usually because the data is too unstructured and the rules are too complex or broad to be applied over the entire document.  This is usually a good point, and a valuable place, to open your string processing skills to subdivide the document.  For example, if you have a very broad search, you might want to do make subdivisions (remember splitting Dracula apart?) and apply the broad search to just the sections that you know apply.\n",
    "\n",
    "For example, say you have a long report of 100,000 summary records in one document.  These are records on snake species and their field measurements.  You want the length field, which is present within each record, but you only want it for the boa species.  This is a pretty classic data query, and maybe you can imagine how easy it would be to construct in SQL.  But instead of a lovely database, you have an unlovely semistructured text report.\n",
    "\n",
    "So trying to write a regular expression to get the length value is going to be overly permissive.  You're going to get it for all 100,000 records.  But say you know that there are 45,000 boa records.  You know this because you've also run a regular expression to detect how many species records are classified as boa.  It might be possible to include that subdivision in your regular expression, but there's a good chance it'll be so complex and unweidly that you won't be able to contain it or make a good slice.  But, you very likely could split the document apart, such that you have all 100,000 records as separate strings, then you can filter out just the boa records, and then apply your length expression to just that subset of strings.  \n",
    "\n",
    "This kind of situation is when using regular expressions in the context of Python is very valuable.  \n",
    "\n",
    "There are also situations when using other mixed methods, such as a combination of regular expressions and xpath statements make a lot of sense.  For example, when doing web scraping, sometimes there will be fields contained in single HTML elements that actually have multiple data points.  Those data points are only separated by text delimiters (words or symbols).  So you use xpath to cleanly extract the element text value, and then you throw that text into regular expressions for splitting.  This is another example of subsetting known data before feeding it into regular expressions.  The more you can clean away noise from your source text before applying a regular expression, the better.\n",
    "\n",
    "# Practice is important\n",
    "\n",
    "There are two components of learning regex:\n",
    "\n",
    "* recall/memorization of the syntax and metacharacter meaning\n",
    "* practice to improve recognition of these things.\n",
    "\n",
    "So have a cheet sheet somewhere of the things I've covered here, and then zip off to https://regexcrossword.com/.  Going through the tutorial, beginner, and intermediate would be great prep to get more used to things."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regular expressions in PyCharm\n",
    "\n",
    "Let's practice a few expressions on our Dracula document.  Yes, this is an xml document, but it is just plain text so we can do whatever we want with it.\n",
    "\n",
    "## Character classes\n",
    "\n",
    "Let's practice our year query first.  We're going to use a character class here to specify the numbers that are applicable.  Character classes are contained within `[]`.  \n",
    "\n",
    "* You can specify multiple things, such as `[057]` to only find the numbers 0, 5, and 7.\n",
    "* You can also put in ranges (that are ranged over the ASCII table, not our human understanding of ranges).  Such as `[0-9]` which will match 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9.\n",
    "* You can combinge these, `[0-278]` for 0, 1, 2, 7, or 8.\n",
    "\n",
    "This also works with letters (well, latin 1)\n",
    "\n",
    "* `[a-z]` gets all the lower case ones\n",
    "* `[A-Z]` gets all the upper case letters \n",
    "* `[abcxyz]` will match any instance of those\n",
    "\n",
    "Caution!  `[a-Z]` includes more than letters!  There's some punctuation between the two groups of letters.\n",
    "\n",
    "So to match a year:\n",
    "\n",
    "* What is a year?  Really depends on how your data defines a year to be.  \n",
    "* Lets' say that we want to match any date that's between 1700 and 1899.  Think about these dates in terms of positions and optional values.\n",
    "    * `[1][7-8][0-9][0-9]`\n",
    "* More generally, we could say that a 4 digit common year would be `[12][0-9][0-9][0-9]`.  Again, depends on what your minimum might be.\n",
    "\n",
    "### Negative character classes\n",
    "\n",
    "Sometimes you can't enumerate all the things that you want, but you clearly know the things that you don't want.  You can create a character class that reads roughly like \"none of these things\" instead of \"any of these\".  Simply place a `^` at the beginning of the class list to make it a negative class.\n",
    "\n",
    "So `[^45]` would mean \"match anything that isn't 4 or 5\".  \n",
    "\n",
    "## Repetition\n",
    "\n",
    "See that we had to repeat a class 3 times in there?  We may be in a position where we can easily type it again, but often that can be really annoying.  There are several ways to indicate repetition.\n",
    "\n",
    "Each of these operates on the single preeceeing element and not the entire content.\n",
    "\n",
    "### single optional item: `?`\n",
    "\n",
    "This is for when you want to indicate that something could appear once or not. \n",
    "\n",
    "You can use `?` directly after that to make it optional for 0 or 1 times.  For example, `cats?` would match `cat` and `cats`.\n",
    "\n",
    "So going back to our date, we can say that the first digit could be 1, 2 or 3 optianally, followed by 0-9.  `[0-3]?[0-9]`  Well, this would still match a 0, which may or may not be acceptable.\n",
    "\n",
    "\n",
    "### known min and max repeats:  `{}`\n",
    "\n",
    "We can use the `{}` directly after an item to specify repetition with precision.  There are two forms of this:\n",
    "\n",
    "* One argument:  the exact number of times it is allowed to repeat.  For example:  `[12][0-9]{3}`\n",
    "* Two arguments:  first the minimum number of times and the second the maximum number of times it could match.  For example, if you were looking for a day number (which can have 1 or 2 digits), you could say:  `[0-9]{1,2}` Which might be too loose of a pattern (matching 99 for example).\n",
    "\n",
    "### to infinity and beyond:  `*` and `+`\n",
    "\n",
    "Meanwhile, you may not care at all how many times something matches.  You can use `*` or `+` to help you with that, but be careful!  You have two choices:\n",
    "\n",
    "* Optionally to infinity:  `*` will match between 0 and infinity times.\n",
    "* At least once to infinity:  `+` will match between 1 and infinity times.\n",
    "\n",
    "Yes, these will go as far as they can until they stop.  You can add `?` after them to force it to be less greedy.\n",
    "\n",
    "\n",
    "### repetition in sum:\n",
    "\n",
    "* `?` will match that element 0 or 1 times\n",
    "    * Example:  \"cats?\" will match cat or cats\n",
    "* `*` will match that element 0-infinity times\n",
    "    * Example: \"cats*\" will match cat, cats, or catssssssssssssssssss (etc.)\n",
    "* `+` will match that element 1-infinity times\n",
    "    * Example:  cats+ will match cats or catssssssssssss (etc.)\n",
    "* `{}` can be used in two ways:\n",
    "    * `{min_times, max_times}` will match that element at least min_times but no more than max_times\n",
    "    * `{exact_number_of_times}` will match that element the specific number of times declared \n",
    "    * Example: cats!{1,3} will match cats!, cats!!, or cats!!! and cats!{2} will match only cats!!\n",
    "\n",
    "All of these symbols operate on the single preceeding element.  So for `cats?`, only the `s` is being operated on.  Likewise, individual character classes count as a single element.  You can force multiple items to be counted for the repetition operation by surounding those items with `()` and placing the items you desire in there.\n",
    "\n",
    "`[0-9]{4}` means \"match any integer between 0 and 9 (inclusive) in a group 4 times.\"  So this will find any group of 4 consecutive integer numbers.  \n",
    "\n",
    "\n",
    "## Adding more speficicity\n",
    "\n",
    "Try a few of these things on the dracula text and see what kind of extra results you get.  Particularly for years, even restricting the start to be 17 or 18, you are still matching portions of zip codes and other numbers.  Often times we will try to add optionality into our expressions that is only necessary for a subset of options there.  Think back to the day thing.\n",
    "\n",
    "The first digit can only start with a 1, 2, or 3, but may not appear at all.  The last digit can be anything between 0 and 9, but can only be 0 when the first digit is present.  Likewise, the second digit can only be greater than 1 when the first digit is 3.  These sort of business rules may not apply or they may be vitally important.  Only you will know how to apply things if you know your data.\n",
    "\n",
    "So, depending on our data, saying `[0-9]{1,2}` (which would match 0-99) might be completely sufficent.  However, you can start adding in more complexity.  The first step is to come up with all your conditions.\n",
    "\n",
    "* `[1-9]` when only 1 digit.  Or `0[1-9]` if you are expecting 2 digits no matter what.  The other problem with this is that you might match a part of a number.  \n",
    "    * For example, given this text:  `_17 June._` The query `[1-9] June` would match `7 June`, which would give us an incorrect answer.\n",
    "    * Solving this problem can be tricky and also requires that you know your data well.  Sometimes there's another delimiter that you could add in and be just fine, or you could add in some negation.  But the negation won't be perfect.  \n",
    "    * So you could say `[^0-9][1-9] June` to state that there needs to be something before the digit that you want, but it cannot be another digit.  This would match `_5 June._`.\n",
    "    * Alternatively, you could use the `\\b` flag to say \"at the beginning of a word\".  So now there are no results because there are no fields that work.  When we change this to: `[1-9] July` we now get \"6 July\" matching, but not \"11 July\".\n",
    "* `[12][0-9]` will cover 10-29\n",
    "* `3[01]` will cover 30 and 31.\n",
    "\n",
    "### Saying \"or\"\n",
    "\n",
    "The pipe `|` symbol is how we specify or.  BEWARE, \"or\" is very very aggressive.  If you make a pattern that says something like 'stuff|yoiamotheroptionsandstuffallovertheplace', it'll go entire for \"stuff\" or \"yoiamotheroptionsandstuffallovertheplace\".\n",
    "\n",
    "So putting our 3 things together we get `([1-9]|[12][0-9]|3[01])`.  In Dracula, `([1-9]|[12][0-9]|3[01]) June` gives us 7 matches, but so does `[0-9]{1,2} June`.  So in this dataset, we apparently don't need to be that specific.\n",
    "\n",
    "# Is that it?\n",
    "\n",
    "There's a lot to regular expressions, but character classes, (), |, and a variety of repetitions are pretty much \n",
    "\n",
    "There are many other symbols, including those to declare that a match must start at the beginning of the line, be against the end of a line, etc.  \n",
    "\n",
    "# Now go do those crosswords!\n",
    "\n",
    "This is a good time to stop and go play with those crossword puzzles."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using regex in python\n",
    "\n",
    "Just as we explored with XPath in a GUI tool and then transitioned to a tool in Python, the regex tool in python will work very similarly.  The Python standard library has a module called `re`.  You can see the documentation here:  https://docs.python.org/3.6/library/re.html#contents-of-module-re\n",
    ".\n",
    "\n",
    "This module has many features, and we'll only cover a few.  However, skimming through some of the functions offered by the module will be valuable.\n",
    "\n",
    "Tools in this module can use regex to:  match text and give you position numbers for those matches, give a boolean to see if there are any matches, match and extract content from the match, replace content according to matching rules, and many more.  What you'll normally do with a novel problem you are facing is look at the documentation and shop around for what kind of tool best matches what you need to get done.  Your homework for this week isn't going to require this, so this is more of a recommendation for when you are working on your own projects.\n",
    "\n",
    "## import the module\n",
    "\n",
    "``` python\n",
    "import re\n",
    "```\n",
    "\n",
    "This is the canonical way to import this package, which means that you'll need to use `re.` before all the functions you use from it.\n",
    "\n",
    "\n",
    "## Compiling patterns\n",
    "\n",
    "Most of the regex functions in this module will take a regex pattern just as a string.  However, you can use `re.compile()` to \"compile\" your pattern into a named object.  The biggest advantage that you will get is storing your pattern as a variable, so you don't have to repeat yourself.  There are other advantages, that we will not be using, but it's a good habit to get into.\n",
    "\n",
    "Anyhow, it looks like this:\n",
    "\n",
    "``` python\n",
    "youvarname = re.compile('yourpatterngoeshere')\n",
    "```\n",
    "\n",
    "You do have the option too put flags and other things into this.  For example, `re.IGNORECASE` will cause the pattern to ignore and case when finding matches.  You would put that together as such:\n",
    "\n",
    "``` python\n",
    "find_text_ignore_case = re.compile('[a-z]+', flags = re.IGNORECASE)\n",
    "```\n",
    "\n",
    "## getting matches\n",
    "\n",
    "The `re.findall()` function is going to be a good place to start for you, and will feel similar to how you worked with the xpath functions.  You give it a pattern and it gives you results back in a list.  No matches give you an empty list.\n",
    "\n",
    "It looks like this:\n",
    "\n",
    "``` python\n",
    "results = re.findall(yourpattern, yourtext)\n",
    "```\n",
    "\n",
    "Let's see this in action and find all words that begin with 'blood' or 'bleed'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "infile = open('dracula.txt', 'r')\n",
    "text = infile.read()\n",
    "infile.close()\n",
    "\n",
    "bloodpattern = re.compile('bl[oe]{2}d[a-z]*', flags = re.IGNORECASE) \n",
    "# yes this allows bloed, but I tested it and it's fine\n",
    "\n",
    "results = re.findall(bloodpattern, text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['blood', 'blood', 'blood', 'blood', 'blood', 'bloody', 'bloody', 'blood', 'blood', 'Blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'bloodless', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'bleeding', 'blood', 'blood', 'bleeding', 'bleed', 'blood', 'blood', 'blood', 'bleed', 'blood', 'Bleeding', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'bloody', 'blood', 'blood', 'blood', 'blood', 'bloodedness', 'blood', 'bloodstained', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'blood', 'bloody', 'blood', 'bloody', 'blood', 'blood']\n"
     ]
    }
   ],
   "source": [
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And counter to the rescue!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counter({'blood': 112, 'bloody': 5, 'bleeding': 2, 'bleed': 2, 'Blood': 1, 'bloodless': 1, 'Bleeding': 1, 'bloodedness': 1, 'bloodstained': 1})\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "print(Counter(results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting boolean results\n",
    "\n",
    "What if you just want to check that a match is present?  You can use `re.search()` to check if a match exists.  Now, this isn't exactly a boolean result, but the result will evaluate to `True` so you can use it in an if statement.\n",
    "\n",
    "So here's a string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bloody = \"I hate that bloody shirt.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now I can get use `re.search` to see if my pattern hits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<_sre.SRE_Match object; span=(12, 18), match='bloody'>\n"
     ]
    }
   ],
   "source": [
    "result = re.search(bloodpattern, bloody)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's kind of ugly, but it'sa match object with a lot of information in it.  Here's the documentation on the match objects:  https://docs.python.org/3.6/library/re.html#match-objects\n",
    "\n",
    "This object is designed to be used in a boolean query, and we can do just that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I hate that bloody shirt.\n",
      "bloody\n"
     ]
    }
   ],
   "source": [
    "if result:\n",
    "    print(result.string)\n",
    "    print(result.group())\n",
    "else:\n",
    "    print(\"No match\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can rethink our approach here.  It's one thing to find all the instances of the word, but we might want to see it in context.  This context will have many definitions, but we could think first of looking at each line that contains a match.  This means that we can split the text up, then check each line for a match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "blood: in all this region that has not been enriched by the blood of men,\n",
      "blood: the instant I saw that the cut had bled a little, and the blood was\n",
      "blood: \"We Szekelys have a right to be proud, for in our veins flows the blood\n",
      "blood: till the dying peoples held that in their veins ran the blood of those\n",
      "blood: Attila, whose blood is in these veins?\" He held up his arms. \"Is it a\n",
      "bloody: throughout the Four Nations received the 'bloody sword,' or at its\n",
      "bloody: to come alone from the bloody field where his troops were being\n",
      "blood: Hungarian yoke, we of the Dracula blood were amongst their leaders, for\n",
      "blood: Szekelys--and the Dracula as their heart's blood, their brains, and\n",
      "Blood: Blood is too precious a thing in these days of dishonourable peace; and\n",
      "blood: underlying the sweet, a bitter offensiveness, as one smells in blood.\n",
      "blood: were--who _are_--waiting to suck my blood.\n"
     ]
    }
   ],
   "source": [
    "lines = text.split('\\n')\n",
    "for line in lines[:2000]: #2k for short printing\n",
    "    result = re.search(bloodpattern, line)\n",
    "    if result:\n",
    "        print(result.group() + \":\", result.string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## start and end of things\n",
    "\n",
    "Sometimes the start of the line is a really important clue that you have found what you want.  You can use that to your advantage.  Now, take a moment to think about how python interprets what the beginning of a line.  Say that we want to find all the lines that start or end with `_` and then a number because that's how we can tell certain diary lines apart.\n",
    "\n",
    "You can use `^` to indicate the beginning of a line and `$` to indicate the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "underscores = re.compile('^_.+_$')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "print(re.findall(underscores, text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how that's coming up with nothing?  That's because it's really seeing the beginning of the file and then some text with newlines.  What we need to do here is use a flag to indicate that the newlines are in fact meaningful and should be used as the beginning or ending of lines.  The `re.MULTILINE` flag will do this for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['_Letter from Miss Mina Murray to Miss Lucy Westenra._', \"_Dr. Seward's Diary._\", '_Letter, Quincey P. Morris to Hon. Arthur Holmwood._', '_Telegram from Arthur Holmwood to Quincey P. Morris._', \"_Dr. Seward's Diary._\", \"_Mina Murray's Journal._\", '_Whitby_', '_Varna to Whitby._', \"_Mina Murray's Journal._\", \"_Mina Murray's Journal._\", \"_Dr. Seward's Diary._\", '_Letter, Mina Harker to Lucy Westenra._', '_Letter, Lucy Westenra to Mina Harker._', \"_Dr. Seward's Diary._\", \"_Lucy Westenra's Diary_\", '_Letter, Arthur Holmwood to Dr. Seward._', '_Telegram, Arthur Holmwood to Seward._', '_Letter from Dr. Seward to Arthur Holmwood._', '_Letter, Dr. Seward to Hon. Arthur Holmwood._', \"_Dr. Seward's Diary._\", '_Telegram, Seward, London, to Van Helsing, Amsterdam._', '_Telegram, Seward, London, to Van Helsing, Amsterdam._', '_Telegram, Seward, London, to Van Helsing, Amsterdam._', '_Letter, Dr. Seward to Hon. Arthur Holmwood._', \"_Dr. Seward's Diary._\", \"_Dr. Seward's Diary--continued._\", \"_Lucy Westenra's Diary._\", \"_Dr. Seward's Diary._\", \"_Lucy Westenra's Diary._\", \"_Dr. Seward's Diary._\", \"_Lucy Westenra's Diary._\", '_\"The Pall Mall Gazette,\" 18 September._', \"_Dr. Seward's Diary._\", '_Telegram, Van Helsing, Antwerp, to Seward, Carfax._', \"_Dr. Seward's Diary._\", '_Memorandum left by Lucy Westenra._', '_Letter, Mina Harker to Lucy Westenra._', \"_Dr. Seward's Diary._\", \"_Mina Harker's Journal._\", \"_Dr. Seward's Diary._\", '_\"The Westminster Gazette,\" 25 September._', '_Letter, Van Helsing to Mrs. Harker._', '_Telegram, Mrs. Harker to Van Helsing._', '_Letter (by hand), Van Helsing to Mrs. Harker._', '_Letter, Mrs. Harker to Van Helsing._', \"_Jonathan Harker's Journal._\", \"_Dr. Seward's Diary._\", \"_Dr. Seward's Diary._\", \"_Mina Harker's Journal._\", \"_Dr. Seward's Diary._\", \"_Mina Harker's Journal._\", \"_Dr. Seward's Diary._\", \"_Jonathan Harker's Journal._\", \"_Mina Harker's Journal_\", \"_Mina Harker's Journal._\", \"_Dr. Seward's Diary._\", \"_Dr. Seward's Diary._\", \"_Mina Harker's Journal._\", \"_Dr. Seward's Diary._\", '_Letter, Mitchell, Sons and Candy to Lord Godalming._', '_\"1 October._', \"_Dr. Seward's Diary._\", \"_Jonathan Harker's Journal._\", \"_Jonathan Harker's Journal._\", \"_Mina Harker's Journal._\", \"_Dr. Seward's Diary._\", \"_Jonathan Harker's Journal._\", \"_Jonathan Harker's Journal._\", '_Telegram, October 24th._', \"_Dr. Seward's Diary._\", \"_Dr. Seward's Diary._\", \"_Mina Harker's Journal._\", \"_Jonathan Harker's Journal._\", \"_Mina Harker's Journal._\", \"_Mina Harker's Memorandum._\", \"_Mina Harker's Journal--continued._\", \"_Jonathan Harker's Journal._\", \"_Dr. Seward's Diary._\", \"_Mina Harker's Journal._\", '_Memorandum by Abraham Van Helsing._', \"_Jonathan Harker's Journal._\", \"_Dr. Seward's Diary._\", \"_Dr. Van Helsing's Memorandum._\", \"_Mina Harker's Journal._\", '_Ask for Complete free list of G. & D. Popular Copyrighted Fiction_']\n"
     ]
    }
   ],
   "source": [
    "underscores = re.compile('^_.+_$', flags=re.MULTILINE)\n",
    "underresult = re.findall(underscores, text)\n",
    "print(underresult)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can ask questions of our data.  What are the differences between content that is between `_` characters, and lines that begin and end with `_`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['_by_', '_Publishers_', '_All rights reserved._', '_Kept in shorthand._', '_3 May. Bistritz._', '_Mem._', '_Mem._', '_Mem._', '_4 May._', '_5 May. The Castle._', '_Mem._', '_continued_', '_5 May._', '_7 May._', '_boyar_', '_8 May._', '_continued_', '_Midnight._', '_boyar_', '_Mem._', '_12 May._', '_Later._', '_face down_', '_15 May._', '_Later: the Morning of 16 May._', '_continued_', '_are_', '_18 May._', '_must_', '_19 May._', '_28 May._', '_boyar_', '_31 May._', '_17 June._', '_24 June, before morning._', '_25 June, morning._', '_Same day, later._', '_29 June._', '_boyars_', '_30 June, morning._', '_Letter from Miss Mina Murray to Miss Lucy Westenra._', '_9 May._', '_Letter, Lucy Westenra to Mina Murray_', '_17, Chatham Street_', '_Wednesday_', '_very_', '_twice_', '_second_', '_do for you_', '_parti_', '_I do_', '_children_', '_think_', '_do_', '_at once_', '_Letter, Lucy Westenra to Mina Murray_', '_24 May_', '_every one_', '_Evening._', '_make_', '_was_', \"_Dr. Seward's Diary._\", '_25 May._', '_Mem._, under what circumstances would I _not_', '_Omnia Romæ venalia sunt._ Hell has its price! _verb. sap._', '_accurately_', '_Letter, Quincey P. Morris to Hon. Arthur Holmwood._', '_25 May._', '_Telegram from Arthur Holmwood to Quincey P. Morris._', '_26 May._', '_24 July. Whitby._', '_1 August._', '_Lively_', '_The same day._', \"_Dr. Seward's Diary._\", '_5 June._', '_18 June._', '_1 July._', '_8 July._', '_19 July._', '_10 p. m._', '_20 July._', '_11 a. m._', '_11 p. m._', \"_Mina Murray's Journal._\", '_26 July._', '_27 July._', '_3 August._', '_6 August._', \"_Pasted in Mina Murray's Journal._\", '_Whitby_', '_Emma_ and _Scarborough_', '_mirabile dictu_', '_dead hand_', '_Whitby_', '_9 August._', '_Demeter_', '_Later._', '_Demeter_', '_cum grano_']\n"
     ]
    }
   ],
   "source": [
    "underscores = re.compile('_.+_', flags=re.MULTILINE)\n",
    "underresult = re.findall(underscores, text)\n",
    "print(underresult[:100]) # for short printing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}