text_mining_edx.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Working with text!</h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Sentiment Analysis</h2>\n",
    "Identify entities and emotions in a sentence and use these to determine if the entity is being viewed positively or negatively"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Easy examples</h3>\n",
    "<li>I had an <b style=\"color:green\">excellent</b> souffle at the restaurant Cavity Maker</li>\n",
    "<li>Excellent is a positive word for both the souffle as well as for the restaurant</li>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Not so easy examples</h3>\n",
    "<h4>Often, looking at words alone is not enough to figure out the sentiment</h4>\n",
    "<li><i>The Girl on the Train is an <span style=\"color:green\">excellent</span> book for a ‘stuck at home’ snow day</i></li> This one is easy since it includes an explicit positive opinion using a positive word\n",
    "<li><i>The Girl on the Train is an <span style=\"color:green\">excellent</span> book for using as a liner for your cat’s litter box</i></li> Not so simple! The positive word \"excellent\" is used with a negative connotation. \n",
    "<li><i>The Girl on the Train is <span style=\"color:green\">better</span> than Gone Girl</i></li> The positive word is used as a comparator. Whether the writer likes The Girl on the Train or not depends on what he or she thinks of Gone Girl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Bottom line</h4>\n",
    "Sentiment analysis is generally a starting point in analyzing a text and is then coupled with other techniques (e.g., topic analysis)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Sentiment analysis is usually done using a corpus of positive and negative words</h2>\n",
    "<li>Some sources compile lists of positive and negative words\n",
    "<li>Others include the polarity - the degree of positivity or negativity - of each word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Sources of sentiment coded words</h2>\n",
    "<ol>\n",
    "<li>Hu and Liu's sentiment analysis lexicon: words coded as either positive or negative</li>\n",
    "<ul>\n",
    "<li>http://ptrckprry.com/course/ssd/data/positive-words.txt\n",
    "<li>http://ptrckprry.com/course/ssd/data/negative-words.txt\n",
    "</ul>\n",
    "<li>NRC Emotion Lexicon: words coded into emotional categories (many languages)</li>\n",
    "<ul>\n",
    "<li>http://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm</li>\n",
    "</ul>\n",
    "<li>SentiWordNet: Lists of words weighted by positive or negative sentiment. Includes guidance on how to use the words</li>\n",
    "<ul>\n",
    "<li>http://sentiwordnet.isti.cnr.it/</li>\n",
    "</ul>\n",
    "<li>Vadar Sentiment tool: 7800 words with positive or negative polarity</li>\n",
    "<ul>\n",
    "<li>Included with python nltk</li>\n",
    "</ul>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Our examples</h2>\n",
    "<li>Compiled set of 15 reviews each of four neighborhood restaurants\n",
    "<li>Presidential inaugural addresses (from Washington to Trump)\n",
    "<li>Some data from yelp (very limited!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Simple sentiment analysis</h3>\n",
    "Compute the proportion of positive and negative words in a text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_words(url):\n",
    "    import requests\n",
    "    words = requests.get(url).content.decode('latin-1')\n",
    "    word_list = words.split('\\n')\n",
    "    index = 0\n",
    "    while index < len(word_list):\n",
    "        word = word_list[index]\n",
    "        if ';' in word or not word:\n",
    "            word_list.pop(index)\n",
    "        else:\n",
    "            index+=1\n",
    "    return word_list\n",
    "\n",
    "#Get lists of positive and negative words\n",
    "p_url = 'http://ptrckprry.com/course/ssd/data/positive-words.txt'\n",
    "n_url = 'http://ptrckprry.com/course/ssd/data/negative-words.txt'\n",
    "positive_words = get_words(p_url)\n",
    "negative_words = get_words(n_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Read the text being analyzed and count the proportion of positive and negative words in the text</h4>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('data/community.txt','r') as f:\n",
    "    community = f.read()\n",
    "with open('data/le_monde.txt','r') as f:\n",
    "    le_monde = f.read()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Compute sentiment by looking at the proportion of positive and negative words in the text</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk import word_tokenize\n",
    "cpos = cneg = lpos = lneg = 0\n",
    "for word in word_tokenize(community):\n",
    "    if word in positive_words:\n",
    "        cpos+=1\n",
    "    if word in negative_words:\n",
    "        cneg+=1\n",
    "for word in word_tokenize(le_monde):\n",
    "    if word in positive_words:\n",
    "        lpos+=1\n",
    "    if word in negative_words:\n",
    "        lneg+=1\n",
    "print(\"community {0:1.2f}%\\t {1:1.2f}%\\t {2:1.2f}%\".format(cpos/len(word_tokenize(community))*100,\n",
    "                                                        cneg/len(word_tokenize(community))*100,\n",
    "                                                        (cpos-cneg)/len(word_tokenize(community))*100))\n",
    "print(\"le monde  {0:1.2f}%\\t {1:1.2f}%\\t {2:1.2f}%\".format(lpos/len(word_tokenize(le_monde))*100,\n",
    "                                                        lneg/len(word_tokenize(le_monde))*100,\n",
    "                                                        (lpos-lneg)/len(word_tokenize(le_monde))*100))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Simple sentiment analysis using NRC data</h2>\n",
    "<li>NRC data codifies words with emotions</li>\n",
    "<li>14,182 words are coded into 2 sentiments and 8 emotions</li>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>For example, the word abandonment is associated with anger, fear, sadness and has a negative sentiment</h4>\n",
    "<li>abandoned\tanger\t1\n",
    "<li>abandoned\tanticipation\t0\n",
    "<li>abandoned\tdisgust\t0\n",
    "<li>abandoned\tfear\t1\n",
    "<li>abandoned\tjoy\t0\n",
    "<li>abandoned\tnegative\t1\n",
    "<li>abandoned\tpositive\t0\n",
    "<li>abandoned\tsadness\t1\n",
    "<li>abandoned\tsurprise\t0\n",
    "<li>abandoned\ttrust\t0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Read the NRC sentiment data</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nrc = \"data/NRC-emotion-lexicon-wordlevel-alphabetized-v0.92.txt\"\n",
    "count=0\n",
    "emotion_dict=dict()\n",
    "with open(nrc,'r') as f:\n",
    "    all_lines = list()\n",
    "    for line in f:\n",
    "        if count < 46:\n",
    "            count+=1\n",
    "            continue\n",
    "        line = line.strip().split('\\t')\n",
    "        if int(line[2]) == 1:\n",
    "            if emotion_dict.get(line[0]):\n",
    "                emotion_dict[line[0]].append(line[1])\n",
    "            else:\n",
    "                emotion_dict[line[0]] = [line[1]]\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Functionalize this</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_nrc_data():\n",
    "    nrc = \"data/NRC-emotion-lexicon-wordlevel-alphabetized-v0.92.txt\"\n",
    "    count=0\n",
    "    emotion_dict=dict()\n",
    "    with open(nrc,'r') as f:\n",
    "        all_lines = list()\n",
    "        for line in f:\n",
    "            if count < 46:\n",
    "                count+=1\n",
    "                continue\n",
    "            line = line.strip().split('\\t')\n",
    "            if int(line[2]) == 1:\n",
    "                if emotion_dict.get(line[0]):\n",
    "                    emotion_dict[line[0]].append(line[1])\n",
    "                else:\n",
    "                    emotion_dict[line[0]] = [line[1]]\n",
    "    return emotion_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "emotion_dict = get_nrc_data()\n",
    "emotion_dict['abandoned']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Yelp API</h1>\n",
    "<li>We'll use data from yelp to figure out useful things about restaurants near Columbia\n",
    "<li>https://www.yelp.com/developers/documentation/v3\n",
    "<li>log into yelp (top right hand corner of the page)\n",
    "<li>Click <span style=\"color:blue\">Create App</span> on the left hand menu bar\n",
    "<li>Enter app info (leave optional stuff blank)\n",
    "<li>Copy the client id and API key to a secure place (this notebook should do the trick or use a text file!)\n",
    "<li>You might need to generate the API key by clicking on Manage App after creating an App. Make sure you get one!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>If you can't get the Yelp API to work, scroll down to the cell marked \"YELP FALLBACK\" and use the fallback option to get the data and continue with this exercise!</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Important note</h2>\n",
    "<li>The Yelp API has changed since the video accompanying this lecture was recorded\n",
    "<li>There will be some differences between what the video says and what you see here, particularly when refering to the get_reviews function\n",
    "<li>But they are fairly small and you should be able to figure out what's what\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "CLIENT_ID = \"\"\n",
    "API_KEY = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>I've saved my keys in a file and will use those. Don't run the next cell!</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('yelp_fusion.txt','r') as f:\n",
    "    count = 0\n",
    "    for line in f:\n",
    "        if count == 0:\n",
    "            CLIENT_ID = line.strip()\n",
    "        if count == 1:\n",
    "            API_KEY = line.strip()\n",
    "        count+=1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check that you have the keys\n",
    "print(CLIENT_ID,API_KEY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# API constants, you shouldn't have to change these.\n",
    "API_HOST = 'https://api.yelp.com' #The API url header\n",
    "SEARCH_PATH = '/v3/businesses/search' #The path for an API request to find businesses\n",
    "BUSINESS_PATH = '/v3/businesses/'  # The path to get data for a single business"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Now we can get reviews</h3>\n",
    "<li>get_reviews(location,number=15) returns the reviews of \"number\" (default=15) restaurants in the vicinity of \"location\"\n",
    "<li>First, we'll write a function that gets  restaurants in the vicinity of location\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_restaurants(api_key,location,number=15):\n",
    "    import requests\n",
    "    \n",
    "    #First we get the access token\n",
    "    #Set up the search data dictionary\n",
    "    search_data = {\n",
    "    'term': \"restaurant\",\n",
    "    'location': location.replace(' ', '+'),\n",
    "    'limit': number\n",
    "    }\n",
    "    url = API_HOST + SEARCH_PATH\n",
    "    headers = {\n",
    "        'Authorization': 'Bearer %s' % api_key,\n",
    "    }\n",
    "    response = requests.request('GET', url, headers=headers, params=search_data).json()\n",
    "    businesses = response.get('businesses')\n",
    "    return businesses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "get_restaurants(API_KEY,\"Columbia University, New York, NY\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Then a function that, given a business id, returns a string containing the reviews</h4>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_business_review(api_key,business_id):\n",
    "    import json\n",
    "    import requests\n",
    "    business_path = BUSINESS_PATH + business_id+\"/reviews\"\n",
    "    url = API_HOST + business_path\n",
    "\n",
    "    headers = {\n",
    "        'Authorization': 'Bearer %s' % api_key,\n",
    "    }\n",
    "\n",
    "\n",
    "    response = requests.request('GET', url, headers=headers).json()\n",
    "   \n",
    "    review_text = ''\n",
    "    for review in response['reviews']:\n",
    "        review_text += review['text']\n",
    "    return review_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "get_business_review(API_KEY,'flat-top-new-york')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Finally, put all this together to get review data for the set of restaurants</h4>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_reviews(location,number=15):\n",
    "\n",
    "    restaurants = get_restaurants(API_KEY,location,number)\n",
    "\n",
    "    if not restaurants:\n",
    "        return None\n",
    "    review_list = list()\n",
    "    for restaurant in restaurants:\n",
    "        restaurant_name = restaurant['name']\n",
    "        restaurant_id = restaurant['id']\n",
    "        review_text = get_business_review(API_KEY,restaurant_id)\n",
    "        \n",
    "        review_list.append((restaurant_id,restaurant_name,review_text))\n",
    "    return review_list\n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_snippets = get_reviews(\"Columbia University, New York, NY\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check to see if we got all the reviews\n",
    "all_snippets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>YELP FALLBACK</h2>\n",
    "<li>Make sure you've downloaded the file \"yelp_data.pickle\" (look under files for Text Mining)\n",
    "<li>Make sure it is in the same directory as your notebook\n",
    "<li>Then run the next cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "with open ('yelp_data.pickle', 'rb') as fp:\n",
    "    all_snippets = pickle.load(fp)\n",
    "#test that it has loaded correctly\n",
    "all_snippets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>A function that analyzes emotions</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def emotion_analyzer(text,emotion_dict=emotion_dict):\n",
    "    #Set up the result dictionary\n",
    "    emotions = {x for y in emotion_dict.values() for x in y}\n",
    "    emotion_count = dict()\n",
    "    for emotion in emotions:\n",
    "        emotion_count[emotion] = 0\n",
    "\n",
    "    #Analyze the text and normalize by total number of words\n",
    "    total_words = len(text.split())\n",
    "    for word in text.split():\n",
    "        if emotion_dict.get(word):\n",
    "            for emotion in emotion_dict.get(word):\n",
    "                emotion_count[emotion] += 1/len(text.split())\n",
    "    return emotion_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Now we can analyze the emotional content of the review snippets</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"%-12s %1s\\t%1s %1s %1s %1s   %1s %1s %1s %1s\"%(\n",
    "        \"restaurant\",\"fear\",\"trust\",\"negative\",\"positive\",\"joy\",\"disgust\",\"anticip\",\n",
    "        \"sadness\",\"surprise\"))\n",
    "        \n",
    "for snippet in all_snippets:\n",
    "    text = snippet[2]\n",
    "    result = emotion_analyzer(text)\n",
    "    print(\"%-12s %1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\"%(\n",
    "        snippet[1][0:10],result['fear'],result['trust'],\n",
    "          result['negative'],result['positive'],result['joy'],result['disgust'],\n",
    "          result['anticipation'],result['sadness'],result['surprise']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Let's functionalize this</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def comparative_emotion_analyzer(text_tuples):\n",
    "    print(\"%-20s %1s\\t%1s %1s %1s %1s   %1s %1s %1s %1s\"%(\n",
    "            \"restaurant\",\"fear\",\"trust\",\"negative\",\"positive\",\"joy\",\"disgust\",\"anticip\",\n",
    "            \"sadness\",\"surprise\"))\n",
    "        \n",
    "    for text_tuple in text_tuples:\n",
    "        text = text_tuple[2] \n",
    "        result = emotion_analyzer(text)\n",
    "        print(\"%-20s %1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\"%(\n",
    "            text_tuple[1][0:20],result['fear'],result['trust'],\n",
    "              result['negative'],result['positive'],result['joy'],result['disgust'],\n",
    "              result['anticipation'],result['sadness'],result['surprise']))\n",
    "        \n",
    "#And test it        \n",
    "comparative_emotion_analyzer(all_snippets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>And let's functionalize the yelp stuff as well</h4>\n",
    "<li>Comment out the line get_reviews(address,number) if the yelp api is not working for you"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def analyze_nearby_restaurants(address,number=15):\n",
    "    snippets = get_reviews(address,number)\n",
    "    comparative_emotion_analyzer(snippets)\n",
    "\n",
    "#And test it    \n",
    "analyze_nearby_restaurants(\"Community Food and Juice, New York, NY\",15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Test it on some other place\n",
    "#This won't work if your yelp api is not working!\n",
    "analyze_nearby_restaurants(\"221 Baker Street\",15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Simple analysis: Word Clouds</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Let's see what sort of words the snippets use</h4>\n",
    "<li>First we'll combine all snippets into one string\n",
    "<li>Then we'll generate a word cloud using the words in the string\n",
    "<li>You may need to install wordcloud using pip\n",
    "<li>pip install wordcloud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_snippets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text=''\n",
    "for snippet in all_snippets:\n",
    "    text+=snippet[2]\n",
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud, STOPWORDS\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "wordcloud = WordCloud(stopwords=STOPWORDS,background_color='white',width=3000,height=3000).generate(text)\n",
    "\n",
    "\n",
    "plt.imshow(wordcloud)\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Let's do a detailed comparison of local restaurants</h2>\n",
    "<h4>I've saved a few reviews for each restaurant in four directories</h4>\n",
    "<h4>We'll use the PlainTextCorpusReader to read these directories</h4>\n",
    "<li>PlainTextCorpusReader reads all matching files in a directory and saves them by file-ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus import PlaintextCorpusReader\n",
    "community_root = \"data/community\"\n",
    "le_monde_root = \"data/le_monde\"\n",
    "community_files = \"community.*\"\n",
    "le_monde_files = \"le_monde.*\"\n",
    "heights_root = \"data/heights\"\n",
    "heights_files = \"heights.*\"\n",
    "amigos_root = \"data/amigos\"\n",
    "amigos_files = \"amigos.*\"\n",
    "community_data = PlaintextCorpusReader(community_root,community_files)\n",
    "le_monde_data = PlaintextCorpusReader(le_monde_root,le_monde_files)\n",
    "heights_data = PlaintextCorpusReader(heights_root,heights_files)\n",
    "amigos_data = PlaintextCorpusReader(amigos_root,amigos_files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amigos_data.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amigos_data.raw()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>We need to modify comparitive_emotion_analyzer to tell it where the restaurant name and the text is in the tuple</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def comparative_emotion_analyzer(text_tuples,name_location=1,text_location=1):\n",
    "    print(\"%-20s %1s\\t%1s %1s %1s %1s   %1s %1s %1s %1s\"%(\n",
    "            \"restaurant\",\"fear\",\"trust\",\"negative\",\"positive\",\"joy\",\"disgust\",\"anticip\",\n",
    "            \"sadness\",\"surprise\"))\n",
    "        \n",
    "    for text_tuple in text_tuples:\n",
    "        text = text_tuple[text_location] \n",
    "        result = emotion_analyzer(text)\n",
    "        print(\"%-20s %1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\\t%1.2f\"%(\n",
    "            text_tuple[name_location][0:20],result['fear'],result['trust'],\n",
    "              result['negative'],result['positive'],result['joy'],result['disgust'],\n",
    "              result['anticipation'],result['sadness'],result['surprise']))\n",
    "        \n",
    "#And test it        \n",
    "comparative_emotion_analyzer(all_snippets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "restaurant_data = [('community',community_data.raw()),('le monde',le_monde_data.raw())\n",
    "                  ,('heights',heights_data.raw()), ('amigos',amigos_data.raw())]\n",
    "comparative_emotion_analyzer(restaurant_data,0,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Simple Analysis: Complexity</h2>\n",
    "<h4>We'll look at four complexity factors</h4>\n",
    "<li>average word length: longer words adds to complexity\n",
    "<li>average sentence length: longer sentences are more complex (unless the text is rambling!)\n",
    "<li>vocabulary: the ratio of unique words used to the total number of words (more variety, more complexity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>token:</b> A sequence (or group) of characters of interest. For e.g., in the below analysis, a token = a word\n",
    "<li>Generally: A token is the base unit of analysis</li>\n",
    "<li>So, the first step is to convert text into tokens and nltk text object</li>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Construct tokens (words/sentences) from the text\n",
    "text = le_monde_data.raw()\n",
    "import nltk\n",
    "from nltk import sent_tokenize,word_tokenize \n",
    "sentences = nltk.Text(sent_tokenize(text))\n",
    "print(len(sentences))\n",
    "words = nltk.Text(word_tokenize(text))\n",
    "print(len(words))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_chars=len(text)\n",
    "num_words=len(word_tokenize(text))\n",
    "num_sentences=len(sent_tokenize(text))\n",
    "vocab = {x.lower() for x in word_tokenize(text)}\n",
    "print(num_chars,int(num_chars/num_words),int(num_words/num_sentences),(len(vocab)/num_words))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Functionalize this</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_complexity(text):\n",
    "    num_chars=len(text)\n",
    "    num_words=len(word_tokenize(text))\n",
    "    num_sentences=len(sent_tokenize(text))\n",
    "    vocab = {x.lower() for x in word_tokenize(text)}\n",
    "    return len(vocab),int(num_chars/num_words),int(num_words/num_sentences),len(vocab)/num_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "get_complexity(le_monde_data.raw())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for text in restaurant_data:\n",
    "    (vocab,word_size,sent_size,vocab_to_text) = get_complexity(text[1])\n",
    "    print(\"{0:15s}\\t{1:1.2f}\\t{2:1.2f}\\t{3:1.2f}\\t{4:1.2f}\".format(text[0],vocab,word_size,sent_size,vocab_to_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>We could do a word cloud comparison</h4>\n",
    "We'll remove short words and look only at words longer than 6 letters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "texts = restaurant_data\n",
    "from wordcloud import WordCloud, STOPWORDS\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "#Remove unwanted words\n",
    "#As we look at the cloud, we can get rid of words that don't make sense by adding them to this variable\n",
    "DELETE_WORDS = []\n",
    "def remove_words(text_string,DELETE_WORDS=DELETE_WORDS):\n",
    "    for word in DELETE_WORDS:\n",
    "        text_string = text_string.replace(word,' ')\n",
    "    return text_string\n",
    "\n",
    "#Remove short words\n",
    "MIN_LENGTH = 0\n",
    "def remove_short_words(text_string,min_length = MIN_LENGTH):\n",
    "    word_list = text_string.split()\n",
    "    for word in word_list:\n",
    "        if len(word) < min_length:\n",
    "            text_string = text_string.replace(' '+word+' ',' ',1)\n",
    "    return text_string\n",
    "\n",
    "\n",
    "#Set up side by side clouds\n",
    "COL_NUM = 2\n",
    "ROW_NUM = 2\n",
    "fig, axes = plt.subplots(ROW_NUM, COL_NUM, figsize=(12,12))\n",
    "\n",
    "for i in range(0,len(texts)):\n",
    "    text_string = remove_words(texts[i][1])\n",
    "    text_string = remove_short_words(text_string)\n",
    "    ax = axes[i%2]\n",
    "    ax = axes[i//2, i%2] #Use this if ROW_NUM >=2\n",
    "    ax.set_title(texts[i][0])\n",
    "    wordcloud = WordCloud(stopwords=STOPWORDS,background_color='white',width=1200,height=1000,max_words=20).generate(text_string)\n",
    "    ax.imshow(wordcloud)\n",
    "    ax.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Comparing complexity of restaurant reviews won't get us anything useful</h3>\n",
    "<h3>Let's look at something more useful</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>nltk: Python's natural language toolkit</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>ntlk documentation link:</h3> http://www.nltk.org/api/nltk.html\n",
    "<h3>Commands cheat sheet</h3> https://blogs.princeton.edu/etc/files/2014/03/Text-Analysis-with-NLTK-Cheatsheet.pdf\n",
    "<h3>nltk book</h3>http://www.nltk.org/book/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>nltk contains a large corpora of pre-tokenized text</h2>\n",
    "Load it using the command:<p>\n",
    "nltk.download()\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Import the corpora</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.book import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Often, a comparitive analysis helps us understand text better</h1>\n",
    "<h2>Let's look at US Presidentinaugural speeches</h2>\n",
    "<h4>Copy the files 2013-Obama.txt and 2017-Trump.txt to the nltk_data/corpora/inaugural directory. nltk_data should be under your home directory</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "inaugural.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "inaugural.raw('1861-Lincoln.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Let's look at the complexity of the speeches by four presidents</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "texts = [('trump',inaugural.raw('2017-Trump.txt')),\n",
    "         ('obama',inaugural.raw('2009-Obama.txt')+inaugural.raw('2013-Obama.txt')),\n",
    "         ('jackson',inaugural.raw('1829-Jackson.txt')+inaugural.raw('1833-Jackson.txt')),\n",
    "         ('washington',inaugural.raw('1789-Washington.txt')+inaugural.raw('1793-Washington.txt'))]\n",
    "for text in texts:\n",
    "    (vocab,word_size,sent_size,vocab_to_text) = get_complexity(text[1])\n",
    "    print(\"{0:15s}\\t{1:1.2f}\\t{2:1.2f}\\t{3:1.2f}\\t{4:1.2f}\".format(text[0],vocab,word_size,sent_size,vocab_to_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Analysis over time</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>The files are arranged over time so we can analyze how complexity has changed between Washington and Trump</h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import inaugural\n",
    "sentence_lengths = list()\n",
    "for fileid in inaugural.fileids():\n",
    "    sentence_lengths.append(get_complexity(' '.join(inaugural.words(fileid)))[2])\n",
    "plt.plot(sentence_lengths)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>dispersion plots</h1>\n",
    "<h2>Dispersion plots show the relative frequency of words over the text</h2>\n",
    "<h3>Let's see how the frequency of some words has changed over the course of the republic</h3>\n",
    "<h3>That should give us some idea of how the focus of the nation has changed</h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text4.dispersion_plot([\"government\", \"citizen\", \"freedom\", \"duties\", \"America\",'independence','God','patriotism'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>We may want to use word stems rather than the part of speect form</h4>\n",
    "<li>For example: patriot, patriotic, patriotism all express roughly the same idea\n",
    "<li>nltk has a stemmer that implements the \"Porter Stemming Algorithm\" (https://tartarus.org/martin/PorterStemmer/)\n",
    "<li>We'll push everything to lowercase as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "p_stemmer = PorterStemmer()\n",
    "text = inaugural.raw()\n",
    "striptext = text.replace('\\n\\n', ' ')\n",
    "striptext = striptext.replace('\\n', ' ')\n",
    "sentences = sent_tokenize(striptext)\n",
    "words = word_tokenize(striptext)\n",
    "text = nltk.Text([p_stemmer.stem(i).lower() for i in words])\n",
    "text.dispersion_plot([\"govern\", \"citizen\", \"free\", \"america\",'independ','god','patriot'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Weighted word analysis using Vader</h2>\n",
    "<h4>Vader contains a list of 7500 features weighted by how positive or negative they are</h4>\n",
    "<h4>It uses these features to calculate stats on how positive, negative and neutral a passage is</h4>\n",
    "<h4>And combines these results to give a compound sentiment (higher = more positive) for the passage</h4>\n",
    "<h4>Human trained on twitter data and generally considered good for informal communication</h4>\n",
    "<h4>10 humans rated each feature in each tweet in context from -4 to +4</h4>\n",
    "<h4>Calculates the sentiment in a sentence using word order analysis</h4>\n",
    "<li>\"marginally good\" will get a lower positive score than \"extremely good\"\n",
    "<h4>Computes a \"compound\" score based on heuristics (between -1 and +1)</h4>\n",
    "<h4>Includes sentiment of emoticons, punctuation, and other 'social media' lexicon elements</h4>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!pip install vaderSentiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "headers = ['pos','neg','neu','compound']\n",
    "texts = restaurant_data\n",
    "analyzer = SentimentIntensityAnalyzer()\n",
    "for i in range(len(texts)):\n",
    "    name = texts[i][0]\n",
    "    sentences = sent_tokenize(texts[i][1])\n",
    "    pos=compound=neu=neg=0\n",
    "    for sentence in sentences:\n",
    "        vs = analyzer.polarity_scores(sentence)\n",
    "        pos+=vs['pos']/(len(sentences))\n",
    "        compound+=vs['compound']/(len(sentences))\n",
    "        neu+=vs['neu']/(len(sentences))\n",
    "        neg+=vs['neg']/(len(sentences))\n",
    "    print(name,pos,neg,neu,compound)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>And functionalize this as well</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def vader_comparison(texts):\n",
    "    from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer\n",
    "    headers = ['pos','neg','neu','compound']\n",
    "    print(\"Name\\t\",'  pos\\t','neg\\t','neu\\t','compound')\n",
    "    analyzer = SentimentIntensityAnalyzer()\n",
    "    for i in range(len(texts)):\n",
    "        name = texts[i][0]\n",
    "        sentences = sent_tokenize(texts[i][1])\n",
    "        pos=compound=neu=neg=0\n",
    "        for sentence in sentences:\n",
    "            vs = analyzer.polarity_scores(sentence)\n",
    "            pos+=vs['pos']/(len(sentences))\n",
    "            compound+=vs['compound']/(len(sentences))\n",
    "            neu+=vs['neu']/(len(sentences))\n",
    "            neg+=vs['neg']/(len(sentences))\n",
    "        print('%-10s'%name,'%1.2f\\t'%pos,'%1.2f\\t'%neg,'%1.2f\\t'%neu,'%1.2f\\t'%compound)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vader_comparison(restaurant_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<h2>Named Entities</h2>\n",
    "<h4>People, places, organizations</h4>\n",
    "Named entities are often the subject of sentiments so identifying them can be very useful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Named entity detection is based on Part-of-speech tagging of words and chunks (groups of words)</h4>\n",
    "<li>Start with sentences (using a sentence tokenizer)\n",
    "<li>tokenize words in each sentence\n",
    "<li>chunk them. ne_chunk identifies likely chunked candidates (ne = named entity)\n",
    "<li>Finally build chunks using nltk's guess on what members of chunk represent (people, place, organization)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "en={}\n",
    "try:\n",
    "    sent_detector = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "    sentences = sent_detector.tokenize(community_data.raw().strip())\n",
    "    for sentence in sentences:\n",
    "            tokenized = nltk.word_tokenize(sentence)\n",
    "            tagged = nltk.pos_tag(tokenized)\n",
    "            chunked = nltk.ne_chunk(tagged)\n",
    "            for tree in chunked:\n",
    "                if hasattr(tree, 'label'):\n",
    "                    ne = ' '.join(c[0] for c in tree.leaves())\n",
    "                    en[ne] = [tree.label(), ' '.join(c[1] for c in tree.leaves())]\n",
    "except Exception as e:\n",
    "    print(str(e))\n",
    "import pprint\n",
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "pp.pprint(en)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Assuming we've done a good job of identifying named entities, we can get an affect score on entities</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "meaningful_sents = list()\n",
    "i=0\n",
    "for sentence in sentences:\n",
    "    if 'service' in sentence:\n",
    "        i+=1\n",
    "        meaningful_sents.append((i,sentence))\n",
    "\n",
    "vader_comparison(meaningful_sents)       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>We could also develop a affect calculator for common terms in our domain (e.g., food items)</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_affect(text,word,lower=False):\n",
    "    import nltk\n",
    "    from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer\n",
    "    analyzer = SentimentIntensityAnalyzer()\n",
    "    sent_detector = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "    sentences = sent_detector.tokenize(text.strip())\n",
    "    sentence_count = 0\n",
    "    running_total = 0\n",
    "    for sentence in sentences:\n",
    "        if lower: sentence = sentence.lower()\n",
    "        if word in sentence:\n",
    "            vs = analyzer.polarity_scores(sentence) \n",
    "            running_total += vs['compound']\n",
    "            sentence_count += 1\n",
    "    if sentence_count == 0: return 0\n",
    "    return running_total/sentence_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "get_affect(community_data.raw(),'service',True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>The nltk function concordance returns text fragments around a word</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nltk.Text(community_data.words()).concordance('service',100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Text summarization</h2>\n",
    "<h4>Text summarization is useful because you can generate a short summary of a large piece of text automatically</h4>\n",
    "<h4>Then, these summaries can serve as an input into a topic analyzer to figure out what the main topic of the text is</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A naive form of summarization is to identify the most frequent words in a piece of text and use the occurrence of these words in sentences to rate the importance of a sentence. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>First the imports</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.tokenize import sent_tokenize\n",
    "from nltk.probability import FreqDist\n",
    "from nltk.corpus import stopwords\n",
    "from collections import OrderedDict\n",
    "import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Then prep the text. Get did of end of line chars</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text = community_data.raw()\n",
    "summary_sentences = []\n",
    "candidate_sentences = {}\n",
    "candidate_sentence_counts = {}\n",
    "striptext = text.replace('\\n\\n', ' ')\n",
    "striptext = striptext.replace('\\n', ' ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Construct a list of words after getting rid of unimportant ones and numbers</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "words = word_tokenize(striptext)\n",
    "lowercase_words = [word.lower() for word in words\n",
    "                  if word not in stopwords.words() and word.isalpha()]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Construct word frequencies and choose the most common n (20)</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "word_frequencies = FreqDist(lowercase_words)\n",
    "most_frequent_words = FreqDist(lowercase_words).most_common(20)\n",
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "pp.pprint(most_frequent_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>lowercase the sentences</h4>\n",
    "candidate_sentences is a dictionary with the original sentence as the key, and its lowercase version as the value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sentences = sent_tokenize(striptext)\n",
    "for sentence in sentences:\n",
    "    candidate_sentences[sentence] = sentence.lower()\n",
    "candidate_sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for long, short in candidate_sentences.items():\n",
    "    count = 0\n",
    "    for freq_word, frequency_score in most_frequent_words:\n",
    "        if freq_word in short:\n",
    "            count += frequency_score\n",
    "            candidate_sentence_counts[long] = count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sorted_sentences = OrderedDict(sorted(\n",
    "                    candidate_sentence_counts.items(),\n",
    "                    key = lambda x: x[0],\n",
    "                    reverse = True)[:4])\n",
    "pp.pprint(sorted_sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Packaging all this into a function</h4>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def build_naive_summary(text):\n",
    "    from nltk.tokenize import word_tokenize\n",
    "    from nltk.tokenize import sent_tokenize\n",
    "    from nltk.probability import FreqDist\n",
    "    from nltk.corpus import stopwords\n",
    "    from collections import OrderedDict\n",
    "    summary_sentences = []\n",
    "    candidate_sentences = {}\n",
    "    candidate_sentence_counts = {}\n",
    "    striptext = text.replace('\\n\\n', ' ')\n",
    "    striptext = striptext.replace('\\n', ' ')\n",
    "    words = word_tokenize(striptext)\n",
    "    lowercase_words = [word.lower() for word in words\n",
    "                      if word not in stopwords.words() and word.isalpha()]\n",
    "    word_frequencies = FreqDist(lowercase_words)\n",
    "    most_frequent_words = FreqDist(lowercase_words).most_common(20)\n",
    "    sentences = sent_tokenize(striptext)\n",
    "    for sentence in sentences:\n",
    "        candidate_sentences[sentence] = sentence.lower()\n",
    "    for long, short in candidate_sentences.items():\n",
    "        count = 0\n",
    "        for freq_word, frequency_score in most_frequent_words:\n",
    "            if freq_word in short:\n",
    "                count += frequency_score\n",
    "                candidate_sentence_counts[long] = count   \n",
    "    sorted_sentences = OrderedDict(sorted(\n",
    "                        candidate_sentence_counts.items(),\n",
    "                        key = lambda x: x[1],\n",
    "                        reverse = True)[:4])\n",
    "    return sorted_sentences   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "summary = '\\n'.join(build_naive_summary(community_data.raw()))\n",
    "print(summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "summary = '\\n'.join(build_naive_summary(le_monde_data.raw()))\n",
    "print(summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>We can summarize George Washington's first inaugural speech<h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "build_naive_summary(inaugural.raw('1789-Washington.txt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>gensim: another text summarizer</h3>\n",
    "Gensim uses a network with sentences as nodes and 'lexical similarity' as weights on the arcs between nodes<p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from wordcloud import WordCloud, STOPWORDS\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import nltk\n",
    "from nltk.corpus import PlaintextCorpusReader\n",
    "from nltk import sent_tokenize,word_tokenize \n",
    "from nltk.book import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus import PlaintextCorpusReader\n",
    "community_root = \"data/community\"\n",
    "le_monde_root = \"data/le_monde\"\n",
    "community_files = \"community.*\"\n",
    "le_monde_files = \"le_monde.*\"\n",
    "heights_root = \"data/heights\"\n",
    "heights_files = \"heights.*\"\n",
    "amigos_root = \"data/amigos\"\n",
    "amigos_files = \"amigos.*\"\n",
    "community_data = PlaintextCorpusReader(community_root,community_files)\n",
    "le_monde_data = PlaintextCorpusReader(le_monde_root,le_monde_files)\n",
    "heights_data = PlaintextCorpusReader(heights_root,heights_files)\n",
    "amigos_data = PlaintextCorpusReader(amigos_root,amigos_files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "type(community_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text = community_data.raw()\n",
    "summary_sentences = []\n",
    "candidate_sentences = {}\n",
    "candidate_sentence_counts = {}\n",
    "striptext = text.replace('\\n\\n', ' ')\n",
    "striptext = striptext.replace('\\n', ' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import gensim.summarization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!pip install gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import gensim.summarization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "summary = gensim.summarization.summarize(striptext, word_count=100) \n",
    "print(summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(gensim.summarization.keywords(striptext,words=10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "summary = '\\n'.join(build_naive_summary(community_data.raw()))\n",
    "print(summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text = le_monde_data.raw()\n",
    "summary_sentences = []\n",
    "candidate_sentences = {}\n",
    "candidate_sentence_counts = {}\n",
    "striptext = text.replace('\\n\\n', ' ')\n",
    "striptext = striptext.replace('\\n', ' ')\n",
    "summary = gensim.summarization.summarize(striptext, word_count=100) \n",
    "print(summary)\n",
    "#print(gensim.summarization.keywords(striptext,words=10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Topic modeling</h1>\n",
    "<h4>The goal of topic modeling is to identify the major concepts underlying a piece of text</h4>\n",
    "<h4>Topic modeling uses \"Unsupervised Learning\". No apriori knowledge is necessary\n",
    "<li>Though it is helpful in cleaning up results!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>LDA: Latent Dirichlet Allocation Model</h3>\n",
    "<li>Identifies potential topics using pruning techniques like 'upward closure'\n",
    "<li>Computes conditional probabilities for topic word sets\n",
    "<li>Identifies the most likely topics\n",
    "<li>Does this over multiple passes probabilistically picking topics in each pass\n",
    "<li>Good intuitive explanation: http://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-allocation/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gensim import corpora\n",
    "from gensim.models.ldamodel import LdaModel\n",
    "from gensim.parsing.preprocessing import STOPWORDS\n",
    "import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Prepare the text</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text = PlaintextCorpusReader(\"data/\",\"Nikon_coolpix_4300.txt\").raw()\n",
    "striptext = text.replace('\\n\\n', ' ')\n",
    "striptext = striptext.replace('\\n', ' ')\n",
    "sentences = sent_tokenize(striptext)\n",
    "#words = word_tokenize(striptext)\n",
    "#tokenize each sentence into word tokens\n",
    "texts = [[word for word in sentence.lower().split()\n",
    "        if word not in STOPWORDS and word.isalnum()]\n",
    "        for sentence in sentences]\n",
    "len(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Create a (word,frequency) dictionary for each word in the text</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dictionary = corpora.Dictionary(texts) #(word_id,frequency) pairs\n",
    "corpus = [dictionary.doc2bow(text) for text in texts] #(word_id,freq) pairs by sentence\n",
    "#print(dictionary.token2id)\n",
    "#print(dictionary.keys())\n",
    "#print(corpus[9])\n",
    "#print(texts[9])\n",
    "#print(dictionary[73])\n",
    "#dictionary[4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Do the LDA</h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Parameters:</h4>\n",
    "<li>Number of topics: The number of topics you want generated. The larger the document, the more the desirable topics\n",
    "<li>Passes: The LDA model makes through the document. More passes, slower analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Set parameters\n",
    "num_topics = 5 #The number of topics that should be generated\n",
    "passes = 10 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lda = LdaModel(corpus,\n",
    "              id2word=dictionary,\n",
    "              num_topics=num_topics,\n",
    "              passes=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>See results</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "pp.pprint(lda.print_topics(num_words=3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Matching topics to documents</h2>\n",
    "<h3>Sort topics by probability</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>We're using sentences as documents here, so this is less than ideal</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from operator import itemgetter\n",
    "lda.get_document_topics(corpus[0],minimum_probability=0.05,per_word_topics=False)\n",
    "sorted(lda.get_document_topics(corpus[0],minimum_probability=0,per_word_topics=False),key=itemgetter(1),reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Making sense of the topics</h3>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Draw wordclouds</h4>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def draw_wordcloud(lda,topicnum,min_size=0,STOPWORDS=[]):\n",
    "    word_list=[]\n",
    "    prob_total = 0\n",
    "    for word,prob in lda.show_topic(topicnum,topn=50):\n",
    "        prob_total +=prob\n",
    "    for word,prob in lda.show_topic(topicnum,topn=50):\n",
    "        if word in STOPWORDS or  len(word) < min_size:\n",
    "            continue\n",
    "        freq = int(prob/prob_total*1000)\n",
    "        alist=[word]\n",
    "        word_list.extend(alist*freq)\n",
    "\n",
    "    from wordcloud import WordCloud, STOPWORDS\n",
    "    import matplotlib.pyplot as plt\n",
    "    %matplotlib inline\n",
    "    text = ' '.join(word_list)\n",
    "    wordcloud = WordCloud(stopwords=STOPWORDS,background_color='white',width=3000,height=3000).generate(' '.join(word_list))\n",
    "\n",
    "\n",
    "    plt.imshow(wordcloud)\n",
    "    plt.axis('off')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "draw_wordcloud(lda,2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h4>Roughly,</h4>\n",
    "<li>lda looks for candidate topics assuming that there are many such candidates\n",
    "<li>looks for words related to the candidate topics\n",
    "<li>assign probablilites to those words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Let's look at Presidential addresses to see what sorts of topics emerge from there</h3>\n",
    "<li>Each document will be analyzed for topic</li>\n",
    "<li>The corpus will consist of 58 documents, one per presidential address"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "REMOVE_WORDS = {'shall','generally','spirit','country','people','nation','nations','great','better'}\n",
    "#Create a word dictionary (id, word)\n",
    "texts = [[word for word in sentence.lower().split()\n",
    "        if word not in STOPWORDS and word not in REMOVE_WORDS and word.isalnum()]\n",
    "        for sentence in sentences]\n",
    "dictionary = corpora.Dictionary(texts)\n",
    "\n",
    "#Create a corpus of documents\n",
    "text_list = list()\n",
    "for fileid in inaugural.fileids():\n",
    "    text = inaugural.words(fileid)\n",
    "    doc=list()\n",
    "    for word in text:\n",
    "        if word in STOPWORDS or word in REMOVE_WORDS or not word.isalpha() or len(word) <5:\n",
    "            continue\n",
    "        doc.append(word)\n",
    "    text_list.append(doc)\n",
    "by_address_corpus = [dictionary.doc2bow(text) for text in text_list]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Create the model</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lda = LdaModel(by_address_corpus,\n",
    "              id2word=dictionary,\n",
    "              num_topics=20,\n",
    "              passes=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "pp.pprint(lda.print_topics(num_words=10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>We can now compare presidential addresses by topic</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "len(by_address_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from operator import itemgetter\n",
    "sorted(lda.get_document_topics(by_address_corpus[0],minimum_probability=0,per_word_topics=False),key=itemgetter(1),reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "draw_wordcloud(lda,18)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(lda.show_topic(12,topn=5))\n",
    "print(lda.show_topic(18,topn=5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<h1>Similarity</h1>\n",
    "<h2>Given a corpus of documents, when a new document arrives, find the document that is the most similar</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "doc_list = [community_data,le_monde_data,amigos_data,heights_data]\n",
    "all_text = community_data.raw() + le_monde_data.raw() + amigos_data.raw() + heights_data.raw()\n",
    "\n",
    "documents = [doc.raw() for doc in doc_list]\n",
    "texts = [[word for word in document.lower().split()\n",
    "        if word not in STOPWORDS and word.isalnum()]\n",
    "        for document in documents]\n",
    "dictionary = corpora.Dictionary(texts)\n",
    "corpus = [dictionary.doc2bow(text) for text in texts]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gensim.similarities.docsim import Similarity\n",
    "from gensim import corpora, models, similarities\n",
    "lsi = models.LsiModel(corpus, id2word=dictionary, num_topics=2)\n",
    "doc = \"\"\"\n",
    "Many, many years ago, I used to frequent this place for their amazing french toast. \n",
    "It's been a while since then and I've been hesitant to review a place I haven't been to in 7-8 years... \n",
    "but I passed by French Roast and, feeling nostalgic, decided to go back.\n",
    "\n",
    "It was a great decision.\n",
    "\n",
    "Their Bloody Mary is fantastic and includes bacon (which was perfectly cooked!!), olives, \n",
    "cucumber, and celery. The Irish coffee is also excellent, even without the cream which is what I ordered.\n",
    "\n",
    "Great food, great drinks, a great ambiance that is casual yet familiar like a tiny little French cafe. \n",
    "I highly recommend coming here, and will be back whenever I'm in the area next.\n",
    "\n",
    "Juan, the bartender, is great!! One of the best in any brunch spot in the city, by far.\n",
    "\"\"\"\n",
    "vec_bow = dictionary.doc2bow(doc.lower().split())\n",
    "vec_lsi = lsi[vec_bow]\n",
    "index = similarities.MatrixSimilarity(lsi[corpus])\n",
    "sims = index[vec_lsi]\n",
    "sims = sorted(enumerate(sims), key=lambda item: -item[1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sims"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "doc=\"\"\"\n",
    "I went to Mexican Festival Restaurant for Cinco De Mayo because I had been there years \n",
    "prior and had such a good experience. This time wasn't so good. The food was just \n",
    "mediocre and it wasn't hot when it was brought to our table. They brought my friends food out \n",
    "10 minutes before everyone else and it took forever to get drinks. We let it slide because the place was \n",
    "packed with people and it was Cinco De Mayo. Also, the margaritas we had were slamming! Pure tequila. \n",
    "\n",
    "But then things took a turn for the worst. As I went to get something out of my purse which was on \n",
    "the back of my chair, I looked down and saw a huge water bug. I had to warn the lady next to me because \n",
    "it was so close to her chair. We called the waitress over and someone came with a broom and a dustpan and \n",
    "swept it away like it was an everyday experience. No one seemed phased.\n",
    "\n",
    "Even though our waitress was very nice, I do not think we will be returning to Mexican Festival again. \n",
    "It seems the restaurant is a shadow of its former self.\n",
    "\"\"\"\n",
    "vec_bow = dictionary.doc2bow(doc.lower().split())\n",
    "vec_lsi = lsi[vec_bow]\n",
    "index = similarities.MatrixSimilarity(lsi[corpus])\n",
    "sims = index[vec_lsi]\n",
    "sims = sorted(enumerate(sims), key=lambda item: -item[1])\n",
    "sims"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}