23. NLP Introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 自然语言处理介绍及实践"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. 基本概念"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/nlp.jpg' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "自然语言处理是计算机科学领域与人工智能领域中的一个重要方向。它研究能实现人与计算机之间用自然语言进行有效通信的各种理论和方法。自然语言处理是一门融语言学、计算机科学、数学于一体的科学。因此，这一领域的研究将涉及自然语言，即人们日常使用的语言，所以它与语言学的研究有着密切的联系，但又有重要的区别。自然语言处理并不是一般地研究自然语言，而在于研制能有效地实现自然语言通信的计算机系统，特别是其中的软件系统。因而它是计算机科学的一部分。\n",
    "\n",
    "自然语言处理（NLP）是计算机科学，人工智能，语言学关注计算机和人类（自然）语言之间的相互作用的领域。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "作为data analyst，我们日常中的工作，很大一部分就是将信息从交易所、上市公司、基金公司公布的金融文档中提取出来。\n",
    "\n",
    "比如基金名称，具体的林林总总的金融数据等，如果掌握自然语言处理技巧，或许能够对日常工作如虎添翼。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. 主要范畴"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "文本朗读（Text to speech）/语音合成（Speech synthesis）\n",
    "\n",
    "语音识别（Speech recognition）\n",
    "\n",
    "中文自动分词（Chinese word segmentation）\n",
    "\n",
    "词性标注（Part-of-speech tagging）\n",
    "\n",
    "句法分析（Parsing）\n",
    "\n",
    "自然语言生成（Natural language generation）\n",
    "\n",
    "文本分类（Text categorization）\n",
    "\n",
    "信息检索（Information retrieval）\n",
    "\n",
    "信息抽取（Information extraction）\n",
    "\n",
    "文字校对（Text-proofing）\n",
    "\n",
    "问答系统（Question answering）\n",
    "\n",
    "机器翻译（Machine translation）\n",
    "\n",
    "自动摘要（Automatic summarization）\n",
    "\n",
    "文字蕴涵（Textual entailment）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/nlparc.jpg' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. 常用套路"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 收集数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于我们analyst来说，就是从我们文档库里面，把我们关心的filing收集起来，然后最好按照句子为单位作为样本进行堆叠。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们拿如下这一段话，进行分句：\n",
    "\n",
    "This prospectus offers variable annuity contract allowing you to accumulate values and paying you benefits on a variable and/or fixed basis. This prospectus provides information regarding the material provisions of your variable annuity contract. We may restrict the availability of this contract to certain broker-dealers. National Security Life V.I. and Annuity Company (\"National Security\") issues the contract. This contract is only available in New York."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def splitparagraph2sentence(paragraph):\n",
    "    doc = nlp(paragraph)\n",
    "    return [sentence.text for sentence in doc.sents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This prospectus offers variable annuity contract allowing you to accumulate values and paying you benefits on a variable and/or fixed basis.\n",
      "This prospectus provides information regarding the material provisions of your variable annuity contract.\n",
      "We may restrict the availability of this contract to certain broker-dealers.\n",
      "National Security Life V.I. and Annuity Company (\"National Security\") issues the contract.\n",
      "This contract is only available in New York.\n"
     ]
    }
   ],
   "source": [
    "sentences = splitparagraph2sentence('This prospectus offers variable annuity contract allowing you to accumulate values and paying you benefits on a variable and/or fixed basis. This prospectus provides information regarding the material provisions of your variable annuity contract. We may restrict the availability of this contract to certain broker-dealers. National Security Life V.I. and Annuity Company (\"National Security\") issues the contract. This contract is only available in New York.')\n",
    "for sentence in sentences:\n",
    "    print(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意：National Security Life V.I.中的点，没有被无脑作为分句的依据，而是真正根据语义分句。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>如果能够做有监督的分类，就顺手打上标签</b>，因为做无监督的聚类操作，然后根据相似度判断文本类型，耗时耗力，而且效果不是很好。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 清洗数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们遵循的第一原则是：“再好的模型也拯救不了shi一样的数据”。所以，先来清洗一下数据吧！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们做以下处理：\n",
    "准则：去除变量，只留常量，或者可常量化。\n",
    "\n",
    "1. 删除所有不相关的字符，如任何非字母数字字符\n",
    "\n",
    "2. 通过文本分隔分成单独的单词来标记你的文章\n",
    "\n",
    "3. 删除不相关的字词，例如“@”推特或网址\n",
    "\n",
    "4. 将所有字符转换为小写字母，以便将诸如“hello”，“Hello”和“HELLO”等单词看做相同单词\n",
    "\n",
    "5. 考虑整合拼写错误或多种拼写的单词，用一个单词代表（例如“cool”/“kewl”/“cooool”）相结合\n",
    "\n",
    "6. 考虑词形还原（把“am”，“are”，“is”等词语缩小为“be”这样的常见形式）\n",
    "\n",
    "7. 将所有专有名词转换为propn这个语义标注词，即变量转换为常量！\n",
    "\n",
    "8. 去除停用词，比如for a an of the and to about after in among as..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "具体实现方式："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 删除所有非字母的字符\n",
    "如这句话：Also assume that, when the Owner is age 76, a step up occurs and the highest quarterly Contract Value is greater than the BDB; in that case, the GAWA percentage will be re determined based on the Owner's attained age of 76, resulting in a new GAWA percentage of 6%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Also assume that when the Owner is age a step up occurs and the highest quarterly Contract Value is greater than the BDB in that case the GAWA percentage will be re determined based on the Owner s attained age of resulting in a new GAWA percentage of\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "text = '''Also assume that, when the Owner is age 76, a step up occurs and the highest quarterly Contract Value is greater than the BDB; in that case, the GAWA percentage will be re determined based on the Owner's attained age of 76, resulting in a new GAWA percentage of 6%.'''\n",
    "text = re.sub(r'\\W', ' ', text)\n",
    "text = re.sub(r'\\d', ' ', text)\n",
    "text = re.sub(r'( ){2,}', ' ', text).strip()\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 词性还原"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 什么是词性？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "词性指以词的特点作为划分词类的根据，比如：\n",
    "ADV: 副词；sample：very, well, exactly, tomorrow, up, down\n",
    "\n",
    "VERB: 动词；sample: run, eat, ate, running, eats\n",
    "\n",
    "ADJ: 形容词；sample: big, old, green\n",
    "\n",
    "DET: 限定词；sample: a, an, this, this, no\n",
    "\n",
    "NOUN: 名词；sample: girl, boy, cat, tree\n",
    "\n",
    "ADP: 介词；sample: in, to, during\n",
    "\n",
    "PROPN: 专属名词；sample: Mary, London, HBO, Google\n",
    "\n",
    "CCONJ: 连词；sample: and, or, but\n",
    "\n",
    "参照：http://universaldependencies.org/u/pos/all.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面的例子，演示如何通过Spacy获取一句话中各个单词的词性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getwordtokenattributes(text):\n",
    "    doc = nlp(text)\n",
    "    result = []\n",
    "    wordlist = []\n",
    "    for token in doc:\n",
    "#         if token.text not in wordlist:\n",
    "        dictinfo = {}\n",
    "        dictinfo['text'] = token.text\n",
    "        dictinfo['lemma_'] = token.lemma_\n",
    "        dictinfo['pos_'] = token.pos_\n",
    "        dictinfo['tag_'] = token.tag_\n",
    "        dictinfo['dep_'] = token.dep_\n",
    "        dictinfo['shape_'] = token.shape_\n",
    "        dictinfo['is_alpha'] = token.is_alpha\n",
    "        dictinfo['is_stop'] = token.is_stop\n",
    "        wordlist.append(token.text)\n",
    "        result.append(dictinfo)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = getwordtokenattributes(r\"It's supposed to be removed.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'text': 'It', 'lemma_': '-PRON-', 'pos_': 'PRON', 'tag_': 'PRP', 'dep_': 'nsubjpass', 'shape_': 'Xx', 'is_alpha': True, 'is_stop': False}, {'text': \"'s\", 'lemma_': 'be', 'pos_': 'VERB', 'tag_': 'VBZ', 'dep_': 'auxpass', 'shape_': \"'x\", 'is_alpha': False, 'is_stop': False}, {'text': 'supposed', 'lemma_': 'suppose', 'pos_': 'VERB', 'tag_': 'VBN', 'dep_': 'ROOT', 'shape_': 'xxxx', 'is_alpha': True, 'is_stop': False}, {'text': 'to', 'lemma_': 'to', 'pos_': 'PART', 'tag_': 'TO', 'dep_': 'aux', 'shape_': 'xx', 'is_alpha': True, 'is_stop': True}, {'text': 'be', 'lemma_': 'be', 'pos_': 'VERB', 'tag_': 'VB', 'dep_': 'auxpass', 'shape_': 'xx', 'is_alpha': True, 'is_stop': True}, {'text': 'removed', 'lemma_': 'remove', 'pos_': 'VERB', 'tag_': 'VBN', 'dep_': 'xcomp', 'shape_': 'xxxx', 'is_alpha': True, 'is_stop': False}, {'text': '.', 'lemma_': '.', 'pos_': 'PUNCT', 'tag_': '.', 'dep_': 'punct', 'shape_': '.', 'is_alpha': False, 'is_stop': False}]\n"
     ]
    }
   ],
   "source": [
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以用Pandas的DataFrame，将结果变得容易阅读："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dep_</th>\n",
       "      <th>is_alpha</th>\n",
       "      <th>is_stop</th>\n",
       "      <th>lemma_</th>\n",
       "      <th>pos_</th>\n",
       "      <th>shape_</th>\n",
       "      <th>tag_</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>nsubjpass</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>-PRON-</td>\n",
       "      <td>PRON</td>\n",
       "      <td>Xx</td>\n",
       "      <td>PRP</td>\n",
       "      <td>It</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>auxpass</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>be</td>\n",
       "      <td>VERB</td>\n",
       "      <td>'x</td>\n",
       "      <td>VBZ</td>\n",
       "      <td>'s</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ROOT</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>suppose</td>\n",
       "      <td>VERB</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>VBN</td>\n",
       "      <td>supposed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>aux</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>to</td>\n",
       "      <td>PART</td>\n",
       "      <td>xx</td>\n",
       "      <td>TO</td>\n",
       "      <td>to</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>auxpass</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>be</td>\n",
       "      <td>VERB</td>\n",
       "      <td>xx</td>\n",
       "      <td>VB</td>\n",
       "      <td>be</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>xcomp</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>remove</td>\n",
       "      <td>VERB</td>\n",
       "      <td>xxxx</td>\n",
       "      <td>VBN</td>\n",
       "      <td>removed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>punct</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>.</td>\n",
       "      <td>PUNCT</td>\n",
       "      <td>.</td>\n",
       "      <td>.</td>\n",
       "      <td>.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        dep_  is_alpha  is_stop   lemma_   pos_ shape_ tag_      text\n",
       "0  nsubjpass      True    False   -PRON-   PRON     Xx  PRP        It\n",
       "1    auxpass     False    False       be   VERB     'x  VBZ        's\n",
       "2       ROOT      True    False  suppose   VERB   xxxx  VBN  supposed\n",
       "3        aux      True     True       to   PART     xx   TO        to\n",
       "4    auxpass      True     True       be   VERB     xx   VB        be\n",
       "5      xcomp      True    False   remove   VERB   xxxx  VBN   removed\n",
       "6      punct     False    False        .  PUNCT      .    .         ."
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.DataFrame(result)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 通过词性还原获得语干"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lemmatization(sentence, allowed_postags=''):\n",
    "    \"\"\"https://spacy.io/api/annotation\"\"\"\n",
    "    doc = nlp(sentence)\n",
    "    # allowed_postags, such as 'NOUN,ADJ,VERB,ADV',\n",
    "    # 但是大多数情况，不能加allow_postags，否则很多词，比如no,  or就没有了\n",
    "    if len(allowed_postags) > 0:\n",
    "        resultlist = [token.lemma_\n",
    "                      for token\n",
    "                      in doc\n",
    "                      if token.pos_\n",
    "                      in [postag.upper().strip() for postag in allowed_postags.split(',')]]\n",
    "    else:\n",
    "        resultlist =  [token.lemma_ for token in doc]\n",
    "    return resultlist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = r'The product is the best than others.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the product be the good than other .\n"
     ]
    }
   ],
   "source": [
    "print(' '.join(lemmatization(text)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 通过词性表达式获得短语"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import textacy\n",
    "def extractverbphrase(text, pattern=r'(<ADV>*<NOUN|PROPN>*<VERB><DET>?<ADV>*<VERB|ADJ>+<ADP>?<DET>?<NUM>*<ADJ>*<NOUN|PROPN>*<ADV>?)|(<VERB>?<NOUN|PROPN>*<ADV>?<VERB><ADP>?<ADJ|VERB>*<ADP>?<DET>?<VERB>?<NOUN|PROPN>*)|(<DET>?<ADJ>+<NOUN|PROPN>+)|(<ADV>*<ADJ><ADP><DET>?<VERB|ADJ>*<NOUN|PROPN>*)|(<DET><NOUN><CCONJ><NOUN>)|(<NOUN|PROPN>*<CCONJ>?<NOUN|PROPN>+<ADP><NOUN|PROPN>+)|(<ADP><DET><NOUN|PROPN>+)'):\n",
    "    # ADV: 副词；sample：very, well, exactly, tomorrow, up, down\n",
    "    # VERB: 动词；sample: run, eat, ate, running, eats\n",
    "    # ADJ: 形容词；sample: big, old, green\n",
    "    # DET: 限定词；sample: a, an, this, this, no\n",
    "    # NOUN: 名词；sample: girl, boy, cat, tree\n",
    "    # ADP: 介词；sample: in, to, during\n",
    "    # PROPN: 专属名词；sample: Mary, London, HBO, Google\n",
    "    # CCONJ: 连词；sample: and, or, but\n",
    "    # 参照：http://universaldependencies.org/u/pos/all.html\n",
    "    doc = nlp(text)\n",
    "    return list(textacy.extract.pos_regex_matches(doc, pattern))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "April\n",
      "Investment Divisions\n",
      "Accumulation Unit\n"
     ]
    }
   ],
   "source": [
    "text = r'Effective April 24, 2017, there are new Investment Divisions for which Accumulation Unit information is not yet available.'\n",
    "phraselist = extractverbphrase(text, pattern=r'(<PROPN>+)')\n",
    "for phrase in phraselist:\n",
    "    print(phrase)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 统一的文字清洗方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将清理逻辑连接起来，构成一个统一的文字清洗方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def removespecialchar(sentence):\n",
    "    result = re.sub('\\W', ' ', sentence)\n",
    "    return re.sub('( ){2,}', ' ', result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clearandlemmasentence(sentence,\n",
    "                          stopword='for a an the and in among'):\n",
    "    stoplist = set(stopword.split())\n",
    "    sentence = removespecialchar(sentence).lower().strip()\n",
    "    sentence = ' '.join([word.strip() for word\n",
    "                         in sentence.lower().strip().split()\n",
    "                         if len(word.strip()) > 0\n",
    "                         and word not in stoplist]).strip()\n",
    "    sentence = re.sub(r'(propn\\s+){2,}', 'propn ', sentence)\n",
    "    if len(sentence) == 0:\n",
    "        sentence = 'only for test'\n",
    "    lemmawordlist = lemmatization(sentence)\n",
    "    return lemmawordlist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def replacevariabletextfromtextblock(textblock):\n",
    "    \"\"\"\n",
    "    Variable Text:\n",
    "    1. PROPN words, such as: Mainstay VP Funds Trust, replace them with propn\n",
    "    2. Date part, such as January 1, 2018, replace them with date\n",
    "    3. Number, such as 1, 2, replace with space\n",
    "    :param textblock:\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    # replace date string with \"date\"\n",
    "    datepattern = r'((January|February|March|April|May|June|July|August|September|October|November|December)[\\s]*[0-9]{1,2}[\\s]*,[\\s]*[0-9]{4})|([0-9]{1,2}/[0-9]{1,2}/[0-9]{4})'\n",
    "    textblock = re.sub(datepattern, 'date', textblock)\n",
    "    datepattern = r'\\d{2}\\/\\d{2}\\/(\\d{4}|\\d{2})'\n",
    "    textblock = re.sub(datepattern, 'date', textblock)\n",
    "    # 应对*CTIVP这种情况，无法识别PROPN\n",
    "    textblock = textblock.replace('*', ' ')\n",
    "    textblock = re.sub(r'( ){2,}', ' ', textblock).strip()\n",
    "    # 因为Money Market Fund前缀与后缀词经常是具体的基金公司，\n",
    "    # 所以去除具体基金公司名称的同时，\n",
    "    # 避免其被作为专属名词替换\n",
    "    textblock = textblock.replace(' of ', ' ')\\\n",
    "        .replace(' Inc.', ' ')\\\n",
    "        .replace('&', '')\\\n",
    "        .replace(' LLC ', ' ')\\\n",
    "        .replace(' BlackRock ', ' ')\\\n",
    "        .replace(' SP ', ' ')\n",
    "    textblock = textblock.replace('-', ' ').\\\n",
    "        replace('–', ' ').\\\n",
    "        replace('Addition', 'addition')\n",
    "    textblock = re.sub(r'\\d', ' ', textblock)\n",
    "    textblock = re.sub(r'( ){2,}', ' ', textblock).strip()\n",
    "    phraselist = extractverbphrase(textblock, '<PROPN>+')\n",
    "    phraselist.sort(key=lambda i: len(i), reverse=True)\n",
    "    if len(phraselist) > 0:\n",
    "        for phrase in phraselist:\n",
    "            phrasetext = phrase.text\n",
    "            # avoid remove important words which are related with category\n",
    "            if 'money market fund' in phrasetext.lower():\n",
    "                textblock = textblock.replace(phrasetext, 'money market fund')\n",
    "            noexcludewordlist = ['date',\n",
    "                                 ' merge ',\n",
    "                                 ' merged ',\n",
    "                                 ' merging ',\n",
    "                                 ' merger ',\n",
    "                                 'acquir',\n",
    "                                 'survive',\n",
    "                                 'surviving',\n",
    "                                 'survived',\n",
    "                                 'liquidat',\n",
    "                                 'transfer',\n",
    "                                 'reorganiz',\n",
    "                                 'expense table',\n",
    "                                 'fee summary',\n",
    "                                 'operating expenses',\n",
    "                                 'annual fund',\n",
    "                                 'the adviser',\n",
    "                                 'benefit payment',\n",
    "                                 'variable account option',\n",
    "                                 ' new ']\n",
    "            shouldignore = False\n",
    "            for word in noexcludewordlist:\n",
    "                if word in phrasetext.lower():\n",
    "                    shouldignore = True\n",
    "                    break\n",
    "            if shouldignore:\n",
    "                continue\n",
    "            if not any([phrasetext.lower() == 'fund',\n",
    "                        len(phrasetext.split()) <= 2]):\n",
    "                textblock = textblock.replace(phrasetext, 'propn')\n",
    "    textblock = textblock.replace('PIMCO', ' ')\n",
    "    textblock = re.sub(r'\\W', ' ', textblock)\n",
    "    textblock = re.sub(r'(propn\\s+){2,}', 'propn ', textblock)\n",
    "    textblock = re.sub(r'( ){2,}', ' ', textblock).strip()\n",
    "    return textblock"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cleardatafordoc2vector(doc):\n",
    "    temp = ' '.join(\n",
    "        clearandlemmasentence(replacevariabletextfromtextblock(doc),\n",
    "                              'for a an of the and or to about after in among as at be been was were is are being b c d e f g h i j k l m n o p q r s t u v w x y z'\n",
    "                              )).strip()\n",
    "    temp = temp.replace('-PRON-', 'pron')\n",
    "    return temp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在可以做一下效果测试：<br>\n",
    "原句：122 66 32 15 14 5 13 17 *CTIVP SM – Eaton Vance Floating Rate Income Fund (Class 2) liquidated on April 27, 2018. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "propn class liquidate on date\n"
     ]
    }
   ],
   "source": [
    "text = r'122 66 32 15 14 5 13 17 *CTIVP SM – Eaton Vance Floating Rate Income Fund (Class 2) liquidated on April 27, 2018. '\n",
    "print(cleardatafordoc2vector(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 找到一个好的数据表示方式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.1 词袋化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bag-of-words模型是信息检索领域常用的文档表示方法。\n",
    "\n",
    "在信息检索中，BOW模型假定对于一个文档，忽略它的单词顺序和语法、句法等要素，将其仅仅看作是若干个词汇的集合，文档中每个单词的出现都是独立的，不依赖于其它单词是否出现。\n",
    "\n",
    "也就是说，文档中任意一个位置出现的任何单词，都不受该文档语意影响而独立选择的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "词袋模型的缺点："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "词袋模型最重要的是构造词表，然后通过文本为词表中的词赋值，但词袋模型严重缺乏相似词之间的表达。 \n",
    "\n",
    "比如“我喜欢北京”“我不喜欢北京”其实这两个文本是严重不相似的。但词袋模型会判为高度相似。 \n",
    "\n",
    "“我喜欢北京”与“我爱北京”其实表达的意思是非常非常的接近的，但词袋模型不能表示“喜欢”和“爱”之间严重的相似关系。（当然词袋模型也能给这两句话很高的相似度，但是注意我想表达的含义）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在Investment名字相似度这个应用中，正是采用了词袋 + TF/IDF模型 + 余弦相似度作为核心。因为单纯的investment并不存在或者很少存在需要语义分析。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面是代码示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\python36\\lib\\site-packages\\gensim\\utils.py:1209: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n",
      "  warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n"
     ]
    }
   ],
   "source": [
    "from gensim import corpora, models, similarities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "namelist = [\n",
    "    'ODDO BHF US Mid Cap CI-EUR H',\n",
    "    'ODDO BHF US Mid Cap CR-USD',\n",
    "    'Credit Suisse Index Fund (CH) - CSIF (CH) Bond Fiscal Strength EUR Blue ZA',\n",
    "    'Winton Diversified Futures Fund (Luxembourg) C GBP Acc',\n",
    "    'Prescient Core Equity Fund B5',\n",
    "    'Robeco QI GTAA Plus DHL $',\n",
    "    'Franklin US Rising Dividends T',\n",
    "    'FT MLP Closed-End Fund & Energy 52 CA',\n",
    "    'FT Richard Bern Adv TS Amer Ind 16-3 CA',\n",
    "    'FT Municipal FT Income Select CE 81 CA',\n",
    "    'Raiffeisen-Pensionsfonds-Österreich 2007 VT',\n",
    "    'Multipartner SICAV - Carthesio Asian Credit Fund B EUR',\n",
    "    'HSBC Wealth Strategic Solutions Fund (1) - Conservative Portfolio Income X',\n",
    "    'American Beacon Flexible Bond Fund A Class',\n",
    "    'Robeco QI GTAA Plus IHL $',\n",
    "    'AXA World Funds - Global Equity Income M Capitalisation EUR']\n",
    "stoplist = set('for a an of the and to in - $ &'.split())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['ODDO', 'BHF', 'US', 'Mid', 'Cap', 'CI-EUR', 'H'], ['ODDO', 'BHF', 'US', 'Mid', 'Cap', 'CR-USD'], ['Credit', 'Suisse', 'Index', 'Fund', '(CH)', 'CSIF', '(CH)', 'Bond', 'Fiscal', 'Strength', 'EUR', 'Blue', 'ZA'], ['Winton', 'Diversified', 'Futures', 'Fund', '(Luxembourg)', 'C', 'GBP', 'Acc'], ['Prescient', 'Core', 'Equity', 'Fund', 'B5'], ['Robeco', 'QI', 'GTAA', 'Plus', 'DHL'], ['Franklin', 'US', 'Rising', 'Dividends', 'T'], ['FT', 'MLP', 'Closed-End', 'Fund', 'Energy', '52', 'CA'], ['FT', 'Richard', 'Bern', 'Adv', 'TS', 'Amer', 'Ind', '16-3', 'CA'], ['FT', 'Municipal', 'FT', 'Income', 'Select', 'CE', '81', 'CA'], ['Raiffeisen-Pensionsfonds-Österreich', '2007', 'VT'], ['Multipartner', 'SICAV', 'Carthesio', 'Asian', 'Credit', 'Fund', 'B', 'EUR'], ['HSBC', 'Wealth', 'Strategic', 'Solutions', 'Fund', '(1)', 'Conservative', 'Portfolio', 'Income', 'X'], ['American', 'Beacon', 'Flexible', 'Bond', 'Fund', 'A', 'Class'], ['Robeco', 'QI', 'GTAA', 'Plus', 'IHL'], ['AXA', 'World', 'Funds', 'Global', 'Equity', 'Income', 'M', 'Capitalisation', 'EUR']]\n"
     ]
    }
   ],
   "source": [
    "data_train = []\n",
    "for name in namelist:\n",
    "    data_train.append([word for word in name.strip().split() \n",
    "                       if word not in stoplist])\n",
    "print(data_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面的代码演示如何生成词袋字典以及词袋模型，并保存为具体的文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "输出每个单词对应的索引编号\n",
      "{'BHF': 0, 'CI-EUR': 1, 'Cap': 2, 'H': 3, 'Mid': 4, 'ODDO': 5, 'US': 6, 'CR-USD': 7, '(CH)': 8, 'Blue': 9, 'Bond': 10, 'CSIF': 11, 'Credit': 12, 'EUR': 13, 'Fiscal': 14, 'Fund': 15, 'Index': 16, 'Strength': 17, 'Suisse': 18, 'ZA': 19, '(Luxembourg)': 20, 'Acc': 21, 'C': 22, 'Diversified': 23, 'Futures': 24, 'GBP': 25, 'Winton': 26, 'B5': 27, 'Core': 28, 'Equity': 29, 'Prescient': 30, 'DHL': 31, 'GTAA': 32, 'Plus': 33, 'QI': 34, 'Robeco': 35, 'Dividends': 36, 'Franklin': 37, 'Rising': 38, 'T': 39, '52': 40, 'CA': 41, 'Closed-End': 42, 'Energy': 43, 'FT': 44, 'MLP': 45, '16-3': 46, 'Adv': 47, 'Amer': 48, 'Bern': 49, 'Ind': 50, 'Richard': 51, 'TS': 52, '81': 53, 'CE': 54, 'Income': 55, 'Municipal': 56, 'Select': 57, '2007': 58, 'Raiffeisen-Pensionsfonds-Österreich': 59, 'VT': 60, 'Asian': 61, 'B': 62, 'Carthesio': 63, 'Multipartner': 64, 'SICAV': 65, '(1)': 66, 'Conservative': 67, 'HSBC': 68, 'Portfolio': 69, 'Solutions': 70, 'Strategic': 71, 'Wealth': 72, 'X': 73, 'A': 74, 'American': 75, 'Beacon': 76, 'Class': 77, 'Flexible': 78, 'IHL': 79, 'AXA': 80, 'Capitalisation': 81, 'Funds': 82, 'Global': 83, 'M': 84, 'World': 85}\n",
      "输出当前句子中各个单词的索引编号以及出现频率\n",
      "[(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1)]\n",
      "[(0, 1), (2, 1), (4, 1), (5, 1), (6, 1), (7, 1)]\n",
      "[(8, 2), (9, 1), (10, 1), (11, 1), (12, 1), (13, 1), (14, 1), (15, 1), (16, 1), (17, 1), (18, 1), (19, 1)]\n",
      "[(15, 1), (20, 1), (21, 1), (22, 1), (23, 1), (24, 1), (25, 1), (26, 1)]\n",
      "[(15, 1), (27, 1), (28, 1), (29, 1), (30, 1)]\n",
      "[(31, 1), (32, 1), (33, 1), (34, 1), (35, 1)]\n",
      "[(6, 1), (36, 1), (37, 1), (38, 1), (39, 1)]\n",
      "[(15, 1), (40, 1), (41, 1), (42, 1), (43, 1), (44, 1), (45, 1)]\n",
      "[(41, 1), (44, 1), (46, 1), (47, 1), (48, 1), (49, 1), (50, 1), (51, 1), (52, 1)]\n",
      "[(41, 1), (44, 2), (53, 1), (54, 1), (55, 1), (56, 1), (57, 1)]\n",
      "[(58, 1), (59, 1), (60, 1)]\n",
      "[(12, 1), (13, 1), (15, 1), (61, 1), (62, 1), (63, 1), (64, 1), (65, 1)]\n",
      "[(15, 1), (55, 1), (66, 1), (67, 1), (68, 1), (69, 1), (70, 1), (71, 1), (72, 1), (73, 1)]\n",
      "[(10, 1), (15, 1), (74, 1), (75, 1), (76, 1), (77, 1), (78, 1)]\n",
      "[(32, 1), (33, 1), (34, 1), (35, 1), (79, 1)]\n",
      "[(13, 1), (29, 1), (55, 1), (80, 1), (81, 1), (82, 1), (83, 1), (84, 1), (85, 1)]\n"
     ]
    }
   ],
   "source": [
    "dictionary = corpora.Dictionary(data_train)\n",
    "print('输出每个单词对应的索引编号')\n",
    "print(dictionary.token2id)\n",
    "dictpath = './nlpmodel/corpus.dict'\n",
    "dictionary.save(dictpath)\n",
    "corpus = [dictionary.doc2bow(text) for text in data_train]\n",
    "print('输出当前句子中各个单词的索引编号以及出现频率')\n",
    "for corpu in corpus:\n",
    "    print(corpu)\n",
    "modelpath = './nlpmodel/corpus.mm'\n",
    "corpora.MmCorpus.serialize(modelpath, corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下文将演示如何通过TF/IDF模型求语句相似度："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 初始化模型\n",
    "corpus = corpora.MmCorpus(modelpath)\n",
    "dictionary = corpora.Dictionary.load(dictpath)\n",
    "tfidf_model = models.TfidfModel(corpus)\n",
    "index = similarities.SparseMatrixSimilarity(\n",
    "    tfidf_model[corpus],\n",
    "    num_features=len(dictionary.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "准备测试语句\n",
      "[(0, 1), (2, 2), (4, 1), (5, 1), (6, 1), (7, 1)]\n"
     ]
    }
   ],
   "source": [
    "print('准备测试语句')\n",
    "testtext = 'CR-USD ODDO Mid Cap Cap BHF US'.split()\n",
    "doc_text_vec = dictionary.doc2bow(testtext)\n",
    "print(doc_text_vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "直接通过TF/IDF模型获取相似度, 返回数值越大，相似度越高\n",
      "[2.0267534  2.8160036  0.         0.         0.         0.\n",
      " 0.28899837 0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.        ]\n"
     ]
    }
   ],
   "source": [
    "print('直接通过TF/IDF模型获取相似度, 返回数值越大，相似度越高')\n",
    "print(index.get_similarities(doc_text_vec))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(1, 0.9541586), (0, 0.6422975)]\n"
     ]
    }
   ],
   "source": [
    "test_simi = index[tfidf_model[doc_text_vec]]\n",
    "test_simi = sorted(enumerate(test_simi), key=lambda item: -item[1])\n",
    "outputlist = [test for test in test_simi if test[1] > 0.2]\n",
    "print(outputlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果想看与哪一句最相似，直接使用索引，从语料包拿就可以"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "raw sentence:  CR-USD ODDO Mid Cap Cap BHF US\n",
      "ODDO BHF US Mid Cap CR-USD ---------similarity:  0.9541586\n",
      "ODDO BHF US Mid Cap CI-EUR H ---------similarity:  0.6422975\n"
     ]
    }
   ],
   "source": [
    "print('raw sentence: ', ' '.join(testtext))\n",
    "for output in outputlist:\n",
    "    print(namelist[output[0]],'---------similarity: ', output[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.2 Doc2Vector中的TaggedDocument"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doc2Vector其实与Word2Vector类似，都有语义分析成分，但是索引单位是句子"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doc2Vector的训练集的组成单元是TaggedDocument对象, 如下是官方说明："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Represents a document along with a tag, input document format for class: `gensim.models.doc2vec.Doc2Vec`.\n",
    "\n",
    "A single document, made up of `words` (a list of unicode string tokens) and `tags` (a list of tokens).\n",
    "\n",
    "Tags may be one or more unicode string tokens, but typical practice (which will also be the most memory-efficient) is for the tags list to include a unique integer id as the only tag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models.doc2vec import TaggedDocument, Doc2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TaggedDocument(['may', 'prize', 'winner', 'teacher', 'bomb'], ['0'])\n",
      "TaggedDocument(['production', 'value', 'use', 'cgi', 'digital', 'ink', 'paint', 'make', 'thing', 'look', 'really', 'slick', 'voice', 'fine', 'well', 'problem', 'thing', 'script'], ['1'])\n",
      "TaggedDocument(['got', 'heart', 'right', 'place', 'also', 'wilt', 'awhile'], ['2'])\n",
      "TaggedDocument(['prof', 'movie', 'goodness', 'thing', 'good', 'movie'], ['3'])\n",
      "TaggedDocument(['well', 'go', 'forever'], ['4'])\n",
      "TaggedDocument(['overproduced', 'generally', 'disappointing', 'effort', 'likely', 'rouse', 'rush', 'hour', 'crowd'], ['5'])\n"
     ]
    }
   ],
   "source": [
    "sentencelist = [\n",
    "    'may prize winner teacher bomb',\n",
    "    'production value use cgi digital ink paint make thing look really slick voice fine well problem thing script',\n",
    "    'got heart right place also wilt awhile',\n",
    "    'prof movie goodness thing good movie',\n",
    "    'well go forever',\n",
    "    'overproduced generally disappointing effort likely rouse rush hour crowd']\n",
    "x_train = []\n",
    "for index, sentence in enumerate(sentencelist):\n",
    "    document = TaggedDocument(sentence.split(), tags=['{0}'.format(index)])\n",
    "    print(document)\n",
    "    x_train.append(document)\n",
    "# model_dm = Doc2Vec(x_train, min_count=1, window=3, size=200, sample=1e-3, negative=5, workers=2)\n",
    "# print(model_dm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.3 Keras中的Tokenizer与pad_sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### text.Tokenizer类\n",
    "\n",
    "这个类用来对文本中的词进行统计计数，生成文档词典，以支持基于词典位序生成文本的向量表示。 \n",
    "init(num_words) 构造函数，传入词典的最大值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 成员函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- fit_on_text(texts) 使用一系列文档来生成token词典，texts为list类，每个元素为一个文档。\n",
    "- texts_to_sequences(texts) 将多个文档转换为word下标的向量形式,shape为`[len(texts)，len(text)]` -- (文档数，每条文档的长度)\n",
    "- texts_to_matrix(texts) 将多个文档转换为矩阵表示,shape为`[len(texts),num_words]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 成员变量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- document_count 处理的文档数量\n",
    "- word_index 一个dict，保存所有word对应的编号id，从<b>1</b>开始\n",
    "- word_counts 一个dict，保存每个word在所有文档中出现的次数\n",
    "- word_docs 一个dict，保存每个word出现的文档的数量\n",
    "- index_docs 一个dict，保存word的id出现的文档的数量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.preprocessing.text import text_to_word_sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentencelist = [\n",
    "    'may prize winner teacher bomb',\n",
    "    'production value use cgi digital ink paint make thing look really slick voice fine well problem thing script',\n",
    "    'got heart right place also wilt awhile',\n",
    "    'prof movie goodness thing good movie',\n",
    "    'well go forever',\n",
    "    'overproduced generally disappointing effort likely rouse rush hour crowd']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text_to_word_sequence的用法与字符串的split用法类似\n",
      "['may', 'prize', 'winner', 'teacher', 'bomb']\n"
     ]
    }
   ],
   "source": [
    "print('text_to_word_sequence的用法与字符串的split用法类似')\n",
    "print(text_to_word_sequence(sentencelist[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_fatures = 2000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokenizer.word_counts\n",
      "OrderedDict([('may', 1), ('prize', 1), ('winner', 1), ('teacher', 1), ('bomb', 1), ('production', 1), ('value', 1), ('use', 1), ('cgi', 1), ('digital', 1), ('ink', 1), ('paint', 1), ('make', 1), ('thing', 3), ('look', 1), ('really', 1), ('slick', 1), ('voice', 1), ('fine', 1), ('well', 2), ('problem', 1), ('script', 1), ('got', 1), ('heart', 1), ('right', 1), ('place', 1), ('also', 1), ('wilt', 1), ('awhile', 1), ('prof', 1), ('movie', 2), ('goodness', 1), ('good', 1), ('go', 1), ('forever', 1), ('overproduced', 1), ('generally', 1), ('disappointing', 1), ('effort', 1), ('likely', 1), ('rouse', 1), ('rush', 1), ('hour', 1), ('crowd', 1)])\n",
      "\n",
      "tokenizer.word_index\n",
      "{'thing': 1, 'well': 2, 'movie': 3, 'may': 4, 'prize': 5, 'winner': 6, 'teacher': 7, 'bomb': 8, 'production': 9, 'value': 10, 'use': 11, 'cgi': 12, 'digital': 13, 'ink': 14, 'paint': 15, 'make': 16, 'look': 17, 'really': 18, 'slick': 19, 'voice': 20, 'fine': 21, 'problem': 22, 'script': 23, 'got': 24, 'heart': 25, 'right': 26, 'place': 27, 'also': 28, 'wilt': 29, 'awhile': 30, 'prof': 31, 'goodness': 32, 'good': 33, 'go': 34, 'forever': 35, 'overproduced': 36, 'generally': 37, 'disappointing': 38, 'effort': 39, 'likely': 40, 'rouse': 41, 'rush': 42, 'hour': 43, 'crowd': 44}\n",
      "\n",
      "tokenizer.word_docs\n",
      "defaultdict(<class 'int'>, {'winner': 1, 'teacher': 1, 'bomb': 1, 'prize': 1, 'may': 1, 'ink': 1, 'well': 2, 'paint': 1, 'voice': 1, 'make': 1, 'thing': 2, 'value': 1, 'use': 1, 'slick': 1, 'really': 1, 'look': 1, 'digital': 1, 'script': 1, 'cgi': 1, 'problem': 1, 'fine': 1, 'production': 1, 'place': 1, 'wilt': 1, 'heart': 1, 'got': 1, 'right': 1, 'awhile': 1, 'also': 1, 'movie': 1, 'goodness': 1, 'good': 1, 'prof': 1, 'forever': 1, 'go': 1, 'rush': 1, 'likely': 1, 'overproduced': 1, 'rouse': 1, 'crowd': 1, 'disappointing': 1, 'hour': 1, 'generally': 1, 'effort': 1})\n",
      "\n",
      "tokenizer.index_docs\n",
      "defaultdict(<class 'int'>, {6: 1, 7: 1, 8: 1, 5: 1, 4: 1, 14: 1, 2: 2, 15: 1, 20: 1, 16: 1, 1: 2, 10: 1, 11: 1, 19: 1, 18: 1, 17: 1, 13: 1, 23: 1, 12: 1, 22: 1, 21: 1, 9: 1, 27: 1, 29: 1, 25: 1, 24: 1, 26: 1, 30: 1, 28: 1, 3: 1, 32: 1, 33: 1, 31: 1, 35: 1, 34: 1, 42: 1, 40: 1, 36: 1, 41: 1, 44: 1, 38: 1, 43: 1, 37: 1, 39: 1})\n"
     ]
    }
   ],
   "source": [
    "tokenizer = Tokenizer(num_words=max_fatures, split=' ')\n",
    "tokenizer.fit_on_texts(sentencelist)\n",
    "print('tokenizer.word_counts')\n",
    "print(tokenizer.word_counts)\n",
    "print()\n",
    "print('tokenizer.word_index')\n",
    "print(tokenizer.word_index)\n",
    "print()\n",
    "print('tokenizer.word_docs')\n",
    "print(tokenizer.word_docs)\n",
    "\n",
    "print()\n",
    "print('tokenizer.index_docs')\n",
    "print(tokenizer.index_docs) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[4, 5, 6, 7, 8], [9, 10, 11, 12, 13, 14, 15, 16, 1, 17, 18, 19, 20, 21, 2, 22, 1, 23], [24, 25, 26, 27, 28, 29, 30], [31, 3, 32, 1, 33, 3], [2, 34, 35], [36, 37, 38, 39, 40, 41, 42, 43, 44]]\n"
     ]
    }
   ],
   "source": [
    "sequences = tokenizer.texts_to_sequences(sentencelist)\n",
    "print(sequences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One_Hot化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 1. 1. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 1. 0. ... 0. 0. 0.]\n",
      " [0. 0. 1. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.texts_to_matrix(sentencelist))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pad_sequences非常重要，目的是将序列填充到maxlen长度，不足maxlenth的句子，用0填充\n",
    "\n",
    "<b><font color='red'>这个非常重要，Keras用于做分类训练的样本，需要通过填充对齐，才能进行之后的训练</font></b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  4  5  6  7  8]\n",
      " [ 0  0  9 10 11 12 13 14 15 16  1 17 18 19 20 21  2 22  1 23]\n",
      " [ 0  0  0  0  0  0  0  0  0  0  0  0  0 24 25 26 27 28 29 30]\n",
      " [ 0  0  0  0  0  0  0  0  0  0  0  0  0  0 31  3 32  1 33  3]\n",
      " [ 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  2 34 35]\n",
      " [ 0  0  0  0  0  0  0  0  0  0  0 36 37 38 39 40 41 42 43 44]]\n"
     ]
    }
   ],
   "source": [
    "X = pad_sequences(sequences, maxlen=20)\n",
    "print(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "文本清洗，词袋化或“向量化”（这里向量化打引号，表示与真正的词向量概率不同，这里仅仅是将词或句建立向量索引）之后，就是建模了。\n",
    "\n",
    "上述部分已经提及了如何创建TF/IDF这种简单模型，那么如何创建词向量模型(word2vec)，句向量(doc2vec)以及通过Keras创建LSTM, biLSTM, GRU乃至biGRU模型呢？\n",
    "\n",
    "我们将在`5. 常用模型`这一章详细了解。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5 使用模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们将在`5. 常用模型`这一章详细了解。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. 常用自然语言处理包"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "工欲善其事，必先利其器。目前为止已经有很多很多用于NLP专项应用的python包。\n",
    "\n",
    "下面将逐一介绍它们。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 Gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/gensimoffice.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gensim是一个用于从文档中自动提取语义主题的Python库，足够智能。\n",
    "\n",
    "Gensim可以处理原生，非结构化的数值化文本(纯文本)。Gensim里面的算法，比如Latent Semantic Analysis(潜在语义分析LSA)，Latent Dirichlet Allocation，Random Projections，通过在语料库的训练下检验词的统计共生模式(statistical co-occurrence patterns)来发现文档的语义结构。\n",
    "\n",
    "这些算法是无监督的，也就是说你只需要一个语料库的文档集。\n",
    "\n",
    "当得到这些统计模式后，任何文本都能够用语义表示(semantic representation)来简洁的表达，并得到一个局部的相似度与其他文本区分开来。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "常用的功能有：语料(Corpus)，TF/IDF、LSA、LDA模型，Word2Vec, Doc2Vec, 以及通过各种模型获得单词、语句间的相似度。这些都是无监督使用方式。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但是，如果Doc2Vec的TaggedDocument使用得当，甚至可以起到有监督的语句分类的功效。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "安装方式：`pip install -U gensim`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以从Github获取源码：[Gensim on Github](https://github.com/RaRe-Technologies/gensim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/gensimongithub.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "官方站点：[Gensim](https://radimrehurek.com/gensim/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/gensimoffice2.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第三方文档教程：[gensim](https://kite.com/python/docs/gensim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/gensimonkite.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很有特色的英文教程：[Gensim Tutorial – A Complete Beginners Guide](https://www.machinelearningplus.com/nlp/gensim-tutorial/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/gensimtutorial.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 Spacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Spacy是由`Cython`编写，因此它是一个非常快的库，可以说是工业级别的NLP组件库。\n",
    "\n",
    "Spacy源自预训练统计模型，词向量，并且支持30+语言。\n",
    "\n",
    "其号称具有最快的语法解析器，通过卷积神经网络CNN（convolutional neural network models）做token标注、解析以及命名实体识别，并且很方便做深度学习整合应用。\n",
    "\n",
    "Spacy是商业开源软件，基于MIT license发布。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Spacy的特点与功能：\n",
    "\n",
    "- Fastest syntactic parser in the world\n",
    "- Named entity recognition\n",
    "- Non-destructive tokenization\n",
    "- Support for 30+ languages\n",
    "- Pre-trained statistical models and word vectors\n",
    "- Easy deep learning integration\n",
    "- Part-of-speech tagging\n",
    "- Labelled dependency parsing\n",
    "- Syntax-driven sentence segmentation\n",
    "- Built in visualizers for syntax and NER\n",
    "- Convenient string-to-hash mapping\n",
    "- Export to numpy data arrays\n",
    "- Efficient binary serialization\n",
    "- Easy model packaging and deployment\n",
    "- State-of-the-art speed\n",
    "- Robust, rigorously evaluated accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "安装方式：`pip install spacy`或者`pip install -U spacy`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在第三章中，我们使用Spacy做了分句，单词原型化，根据词性表达式获取短语等应用，而这一切都是是基于模型应用的，下面介绍如何下载语言模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最简单的下载模型方式，是基于Spacy的download命令："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "# out-of-the-box: download best-matching default model\n",
    "python -m spacy download en\n",
    "python -m spacy download de\n",
    "python -m spacy download es\n",
    "python -m spacy download pt\n",
    "python -m spacy download fr\n",
    "python -m spacy download it\n",
    "python -m spacy download nl\n",
    "python -m spacy download xx\n",
    "\n",
    "# download best-matching version of specific model for your spaCy installation\n",
    "python -m spacy download en_core_web_sm\n",
    "\n",
    "# download exact model version (doesn't create shortcut link)\n",
    "python -m spacy download en_core_web_sm-2.0.0 --direct\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型下载相关的文档位于：[Models Overview](https://spacy.io/models/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果想获得特别全的英文词向量模型，可以下载：`python -m spacy download en_core_web_lg`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "昨天举了很多Spacy有关的例子，这里再举一个命名实体识别的例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'PERSON': ['Mary'], 'ORG': ['Google,', 'Amazon'], 'GPE': ['New York']}\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en')\n",
    "doc = nlp(\"\"\"Mary has a dog, she works for Google, \n",
    "and likes to buy things on Amazon, \n",
    "she is living in New York.\"\"\")\n",
    "result = {}\n",
    "for ent in doc.ents:\n",
    "    if ent.label_ not in result.keys():\n",
    "        result[ent.label_] = []\n",
    "    if ent.text.strip() != \"\":\n",
    "        result[ent.label_].append(ent.text.strip())\n",
    "#remove duplicate entity values\n",
    "for key in result.keys():\n",
    "    l = []\n",
    "    for label in result[key]:\n",
    "        if not label in l:\n",
    "            l.append(label)\n",
    "    result[key] = l\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "命名实体识别可视化的方式：\n",
    "\n",
    "通过如下代码，即可通过访问http://localhost:5000 的网址浏览实体识别的具体信息。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    Serving on port 5000...\n",
      "    Using the 'ent' visualizer\n",
      "\n",
      "\n",
      "    Shutting down server on port 5000.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from spacy import displacy\n",
    "displacy.serve(doc, style='ent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "效果大致如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/spacyentity.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Github地址：[spaCy: Industrial-strength NLP](https://github.com/explosion/spaCy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/spacygithub.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "官方介绍及教程：[Spacy: Industrial-Strength\n",
    "Natural Language\n",
    "Processing](https://spacy.io/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3 textacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "textacy是基于Spacy开发的自然语言任务工具，相关特性如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Provide a convenient entry point and interface to one or many documents, with the core processing delegated to spaCy\n",
    "- Stream text, json, csv, spaCy binary, and other data to and from disk\n",
    "- Download and explore a variety of included datasets with both text content and metadata, from Congressional speeches to historical literature to Reddit comments\n",
    "- Clean and normalize raw text, before analyzing it\n",
    "- Access and filter basic linguistic elements, such as words, ngrams, and noun chunks; extract named entities, acronyms and their definitions, and key terms\n",
    "- Flexibly tokenize and vectorize documents and corpora, then train, interpret, and visualize topic models using LSA, LDA, or NMF methods\n",
    "- Compare strings, sets, and documents by a variety of similarity metrics\n",
    "- Calculate common text statistics, including Flesch-Kincaid Grade Level, SMOG Index, and multilingual Flesch Reading Ease"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "安装方法：`pip install textacy`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之所以用textacy，是因为其可以预处理文本，比如:\n",
    "去除URLs: 统一替换为url\n",
    "Email：统一替换为email\n",
    "Number: 统一替换为number\n",
    "标点符号，\n",
    "重音符号，\n",
    "HTML标记等，如： "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "please visit url then you will get what you want to search there are over number web pages to review please contact me by email\n"
     ]
    }
   ],
   "source": [
    "import textacy\n",
    "rawtext = \"\"\"Please visit http://www.google.com, \n",
    "then you will get what you want to search. \n",
    "There are over 1000 web pages to review.\n",
    "Please contact me by a@gmail.com.\"\"\"\n",
    "text = textacy.preprocess_text(rawtext, \n",
    "                               no_urls=True, \n",
    "                               no_numbers=True, \n",
    "                               no_emails=True,\n",
    "                               lowercase=True, \n",
    "                               no_punct=True)\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function preprocess_text in module textacy.preprocess:\n",
      "\n",
      "preprocess_text(text, fix_unicode=False, lowercase=False, transliterate=False, no_urls=False, no_emails=False, no_phone_numbers=False, no_numbers=False, no_currency_symbols=False, no_punct=False, no_contractions=False, no_accents=False)\n",
      "    Normalize various aspects of a raw text doc before parsing it with Spacy.\n",
      "    A convenience function for applying all other preprocessing functions in one go.\n",
      "    \n",
      "    Args:\n",
      "        text (str): raw text to preprocess\n",
      "        fix_unicode (bool): if True, fix \"broken\" unicode such as\n",
      "            mojibake and garbled HTML entities\n",
      "        lowercase (bool): if True, all text is lower-cased\n",
      "        transliterate (bool): if True, convert non-ascii characters\n",
      "            into their closest ascii equivalents\n",
      "        no_urls (bool): if True, replace all URL strings with '*URL*'\n",
      "        no_emails (bool): if True, replace all email strings with '*EMAIL*'\n",
      "        no_phone_numbers (bool): if True, replace all phone number strings\n",
      "            with '*PHONE*'\n",
      "        no_numbers (bool): if True, replace all number-like strings\n",
      "            with '*NUMBER*'\n",
      "        no_currency_symbols (bool): if True, replace all currency symbols\n",
      "            with their standard 3-letter abbreviations\n",
      "        no_punct (bool): if True, remove all punctuation (replace with\n",
      "            empty string)\n",
      "        no_contractions (bool): if True, replace *English* contractions\n",
      "            with their unshortened forms\n",
      "        no_accents (bool): if True, replace all accented characters\n",
      "            with unaccented versions; NB: if `transliterate` is True, this option\n",
      "            is redundant\n",
      "    \n",
      "    Returns:\n",
      "        str: input ``text`` processed according to function args\n",
      "    \n",
      "    Warning:\n",
      "        These changes may negatively affect subsequent NLP analysis performed\n",
      "        on the text, so choose carefully, and preprocess at your own risk!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(textacy.preprocess_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除此之外，textacy通过`textacy.Doc`的方式，能够<b>自动检测加载文本的语言</b>。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = 'Tom is happily running in the park'\n",
    "doc = textacy.Doc(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# help(textacy.Doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外，还提供词性正则表达式的功能，可以方便获取想要的短语组合："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Tom is happily running']\n"
     ]
    }
   ],
   "source": [
    "pattern = r'(<NOUN|PROPN>+<VERB>+<DET>?<ADV>*<VERB>+)'\n",
    "phraselist = list(textacy.extract.pos_regex_matches(doc, pattern))\n",
    "print([phrase.text for phrase in phraselist])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "textacy的github地址：[textacy: NLP, before and after spaCy](https://github.com/chartbeat-labs/textacy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/textacygithub.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "官方说明文档: [textacy: NLP, before and after spaCy](https://chartbeat-labs.github.io/textacy/index.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/textacytutorial.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4 NLTK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NLTK是一个高效的Python构建的平台，用来处理人类自然语言数据。它提供了易于使用的接口，通过这些接口可以访问超过50个语料库和词汇资源（如WordNet），还有一套用于分类、标记化、词干标记、解析和语义推理的文本处理库，以及工业级NLP库的封装器和一个活跃的讨论论坛。\n",
    "\n",
    "统计语言学话题方面的手动编程指南加上全面的API文档，使得NLTK非常适用于语言学家、工程师、学生、教育家、研究人员以及行业用户等人群。NLTK可以在Windows、Mac OS X以及Linux系统上使用。最好的一点是，NLTK是一个免费、开源的社区驱动的项目。\n",
    "\n",
    "NLTK被称为“一个使用Python开发的用于统计语言学的教学和研究的有利工具”和“一个自然语言处理的高效库”。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "相比Spacy之类的自然语言处理包，NLTK有一些偏学术化。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Github地址：[NLTK](https://github.com/nltk/nltk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "官方地址：[NLTK](https://www.nltk.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.5 JIEBA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jieba（结巴）是一个强大的分词库，完美支持中文分词。\n",
    "\n",
    "一般来说中文文本不会如同拉丁语系一样，词与词之间有明显的空格作为间隔。\n",
    "\n",
    "如果需要对中文文本建模，那么分词是必须的前提条件，那么Jieba就是目前为止最好的中文分词组件包。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其优点有：\n",
    "\n",
    "1 支持三种分词模式：\n",
    "\n",
    "a. 精确模式，试图将句子最精确地切开，适合文本分析；\n",
    "\n",
    "b. 全模式，把句子中所有的可以成词的词语都扫描出来, 速度非常快，但是不能解决歧义；\n",
    "\n",
    "c. 搜索引擎模式，在精确模式的基础上，对长词再次切分，提高召回率，适合用于搜索引擎分词。\n",
    "\n",
    "2 支持自定义词典"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面来看看如何对中文做分词："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 精准模式\n",
    "\n",
    "试图将句子最精确地切开,适合文本分析"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jieba\n",
    "text = r'小张毕业于深圳大学，这座大学位于南山区'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Building prefix dict from the default dictionary ...\n",
      "Dumping model to file cache C:\\Users\\bhe\\AppData\\Local\\Temp\\jieba.cache\n",
      "Loading model cost 1.194 seconds.\n",
      "Prefix dict has been built succesfully.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "小张/毕业/于/深圳大学/，/这座/大学/位于/南山区\n"
     ]
    }
   ],
   "source": [
    "words = jieba.cut(text)\n",
    "print('/'.join(words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 全模式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "把句子中所有的可以成词的词语都扫描出来, 速度非常快,但是不能解决歧义"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "小张/毕业/于/深圳/深圳大学/大学///这/座/大学/学位/位于/南山/南山区/山区\n"
     ]
    }
   ],
   "source": [
    "words = jieba.cut(text, cut_all=True)\n",
    "print('/'.join(words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 搜索引擎模式\n",
    "\n",
    "在精确模式的基础上,对长词再次切分,提高召回率,适合用于搜索引擎分词"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "小张/毕业/于/深圳/大学/深圳大学/，/这座/大学/位于/南山/山区/南山区\n"
     ]
    }
   ],
   "source": [
    "output = jieba.cut_for_search(text)\n",
    "print('/'.join(output))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 词性标注"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "单词：小张, 词性：n\n",
      "单词：毕业, 词性：n\n",
      "单词：于, 词性：p\n",
      "单词：深圳大学, 词性：nt\n",
      "单词：，, 词性：x\n",
      "单词：这, 词性：r\n",
      "单词：座, 词性：q\n",
      "单词：大学, 词性：n\n",
      "单词：位于, 词性：v\n",
      "单词：南山区, 词性：ns\n"
     ]
    }
   ],
   "source": [
    "import jieba.posseg as pseg\n",
    "words = pseg.cut(text)\n",
    "for word, flag in words:\n",
    "    print('单词：{0}, 词性：{1}'.format(word, flag))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 关键词提取\n",
    "\n",
    "Jieba的关键词提取功能，是基于TF-IDF算法的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "竞技  马德里  欧冠  联赛  球队  一役  西蒙尼  18  26  场均  丢球数  引援  四场  威斯特法伦  上赛季  那样  取得  巴萨  似乎  出线\n"
     ]
    }
   ],
   "source": [
    "import jieba.analyse as analyse\n",
    "text = \"\"\"欧冠提前一轮出线，近四场比赛取得3胜1平，距离终结联赛对巴萨的不胜纪录也只有一步之遥，\n",
    "马德里竞技似乎已经完全从惨败威斯特法伦一役的阴霾中走了出来。\n",
    "球队近来的成绩有所提升，但困扰西蒙尼的战术难题并没有得到解决，\n",
    "马德里竞技要取得一场的胜利似乎总是要付出比其他球队更多的努力，双线战场18战仅仅打入26球，\n",
    "场均丢球数却达到了数年来的峰值，联赛中的两大竞争对手均状态不佳，欧冠分组也十分有利，\n",
    "但马德里竞技依然没能如人们预期的那样脱颖而出，更为尴尬的是，\n",
    "他们此次已经不能像上赛季那样以引援不力作为借口了。\"\"\"\n",
    "print(\"  \".join(analyse.extract_tags(text, topK=20, withWeight=False, allowPOS=())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. 常用模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 TF/ IDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1.1 概念"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TF-IDF（term frequency–inverse document frequency）是一种用于资讯检索与资讯探勘的常用加权技术。\n",
    "   \n",
    "TF-IDF是一种统计方法，用以评估一字词对于一个文件集或一个语料库中的其中一份文件的重要程度。\n",
    "   \n",
    "<b>字词的重要性随着它在文件中出现的次数成正比增加，但同时会随着它在语料库中出现的频率成反比下降。</b>\n",
    "   \n",
    "TF-IDF加权的各种形式常被搜寻引擎应用，作为文件与用户查询之间相关程度的度量或评级。\n",
    "   \n",
    "除了TF-IDF以外，因特网上的搜寻引擎还会使用基于连结分析的评级方法，以确定文件在搜寻结果中出现的顺序。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1.2 原理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在一份给定的文件里，<b>词频 (term frequency, TF)</b> 指的是某一个给定的词语在该文件中出现的次数。这个数字通常会被归一化（分子一般小于分母 区别于IDF），以防止它偏向长的文件。（同一个词语在长文件里可能会比短文件有更高的词频，而不管该词语重要与否。）\n",
    "\n",
    "<b>逆向文件频率 (inverse document frequency, IDF)</b> 是一个词语普遍重要性的度量。某一特定词语的IDF，可以由总文件数目除以包含该词语之文件的数目，再将得到的商取对数得到。\n",
    "\n",
    "某一特定文件内的高词语频率，以及该词语在整个文件集合中的低文件频率，可以产生出高权重的TF-IDF。因此，TF-IDF倾向于过滤掉常见的词语，保留重要的词语。\n",
    "\n",
    "<b>TF-IDF的主要思想是：</b>\n",
    "\n",
    "如果某个词或短语在一篇文章中出现的频率TF高，并且在其他文章中很少出现，则认为此词或者短语具有很好的类别区分能力，适合用来分类。\n",
    "\n",
    "TF-IDF实际上是：TF * IDF，TF词频(Term Frequency)，IDF反文档频率(Inverse Document Frequency)。\n",
    "\n",
    "TF表示词条在文档d中出现的频率（另一说：TF词频(Term Frequency)指的是某一个给定的词语在该文件中出现的次数）。\n",
    "\n",
    "IDF的主要思想是：如果包含词条t的文档越少，也就是n越小，IDF越大，则说明词条t具有很好的类别区分能力。\n",
    "\n",
    "如果某一类文档C中包含词条t的文档数为m，而其它类包含t的文档总数为k，显然所有包含t的文档数n=m+k，当m大的时候，n也大，按照IDF公式得到的IDF的值会小，就说明该词条t类别区分能力不强。\n",
    "\n",
    "（另一说：IDF反文档频率(Inverse Document Frequency)是指果包含词条的文档越少，IDF越大，则说明词条具有很好的类别区分能力。）\n",
    "\n",
    "但是实际上，如果一个词条在一个类的文档中频繁出现，则说明该词条能够很好代表这个类的文本的特征，这样的词条应该给它们赋予较高的权重，并选来作为该类文本的特征词以区别与其它类文档。这就是IDF的不足之处."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最简单的直观公式："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/tf1.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/idf.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/tfidf.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "学术一些的公式如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/tfidf_math.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "维基百科的地址:[tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "示例见本文的：3.3.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1.3 与余弦相似度的结合应用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "余弦相似度的维基百科定义：[Cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "比如两个句子：\n",
    "\n",
    "句子A: 我喜欢看电视，不喜欢看电影。\n",
    "\n",
    "句子B: 我不喜欢看电视，也不喜欢看电影。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 经过分词与计算词频：\n",
    "\n",
    "句子A：我 1，喜欢 2，看 2，电视 1，电影 1，不 1，也 0。\n",
    "\n",
    "句子B：我 1，喜欢 2，看 2，电视 1，电影 1，不 2，也 1。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. 写出词频向量\n",
    "\n",
    "句子A：[1, 2, 2, 1, 1, 1, 0]\n",
    "\n",
    "句子B：[1, 2, 2, 1, 1, 2, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "到这里，问题就变成了如何计算这两个向量的相似程度。\n",
    "\n",
    "我们可以把它们想象成空间中的两条线段，都是从原点（[0, 0, ...]）出发，指向不同的方向。\n",
    "\n",
    "两条线段之间形成一个夹角，如果夹角为0度，意味着方向相同、线段重合；如果夹角为90度，意味着形成直角，方向完全不相似；如果夹角为180度，意味着方向正好相反。\n",
    "\n",
    "因此，我们可以通过夹角的大小，来判断向量的相似程度。夹角越小，就代表越相似。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/cos1.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以二维空间为例，上图的a和b是两个向量，我们要计算它们的夹角θ。余弦定理告诉我们，可以用下面的公式求得："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/cos2.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/cos3.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假定a向量是[x1, y1]，b向量是[x2, y2]，那么可以将余弦定理改写成下面的形式："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/cos4.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/cos5.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数学家已经证明，余弦的这种计算方法对n维向量也成立。假定A和B是两个n维向量，A是 [A1, A2, ..., An] ，B是 [B1, B2, ..., Bn] ，则A与B的夹角θ的余弦等于："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/cos6.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用这个公式，我们就可以得到，句子A与句子B的夹角的余弦。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/cos7.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "余弦值越接近1，就表明夹角越接近0度，也就是两个向量越相似，这就叫\"余弦相似性\"。\n",
    "\n",
    "所以，上面的句子A和句子B是很相似的，事实上它们的夹角大约为20.3度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1.4 找出相似文章的简易算法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "（1）使用TF-IDF算法，找出两篇文章的关键词；\n",
    "\n",
    "（2）每篇文章各取出若干个关键词（比如20个），合并成一个集合，计算每篇文章对于这个集合中的词的词频（为了避免文章长度的差异，可以使用相对词频）；\n",
    "\n",
    "（3）生成两篇文章各自的词频向量；\n",
    "\n",
    "（4）计算两个向量的余弦相似度，值越大就表示越相似。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 词向量（Word2Vec）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2.1 概念"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "自然语言处理的词频处理方法即TF-IDF，这种方法往往只是可以找出一篇文章中比较关键的词语，即找出一些主题词汇。\n",
    "\n",
    "但无法给出词汇的语义，比如同义词漂亮和美丽意思差不多应该相近，巴黎之于法国等同于北京之于中国。\n",
    "\n",
    "对于一句话，如何根据上下文推断出中间的词语是什么，或者由某一个词推测出它的上下文一般是什么词语。\n",
    "\n",
    "这两种不同的思考方式正好对应两种Word2vec模型，即CBOW模型和Skip-gram模型。\n",
    "\n",
    "所谓的word vector，就是指将单词向量化，将某个单词用特定的向量来表示。\n",
    "\n",
    "将单词转化成对应的向量以后，就可以将其应用于各种机器学习的算法中去。\n",
    "\n",
    "\n",
    "一般来讲，词向量主要有两种形式，分别是稀疏向量和密集向量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>所谓稀疏向量</b>，又称为one-hot representation，就是用一个很长的向量来表示一个词，向量的长度为词典的大小N，向量的分量只有一个1，其他全为0，1的位置对应该词在词典中的索引[1]。\n",
    "\n",
    "举例来说，如果有一个词典[“面条”,”方便面”,”狮子”]，那么“面条”对应的词向量就是[1,0,0]，“方便面”对应的词向量就是[0,1,0]。这种表示方法不需要繁琐的计算，简单易得，但是缺点也不少，比如长度过长（这会引发维数灾难），以及无法体现出近义词之间的关系，比如“面条”和“方便面”显然有非常紧密的关系，但转化成向量[1,0,0]和[0,1,0]以后，就看不出两者有什么关系了,因为这两个向量相互正交。\n",
    "\n",
    "当然了，用这种稀疏向量求和来表示文档向量效果还不错，清华的长文本分类工具THUCTC使用的就是此种表示方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>密集向量</b>，又称distributed representation，即分布式表示。最早由Hinton提出，可以克服one-hot representation的上述缺点。\n",
    "\n",
    "基本思路是通过训练将每个词映射成一个固定长度的短向量，所有这些向量就构成一个词向量空间，每一个向量可视为该空间上的一个点[1]。\n",
    "\n",
    "此时向量长度可以自由选择，与词典规模无关。这是非常大的优势。\n",
    "\n",
    "还是用之前的例子[“面条”,”方便面”,”狮子”]，经过训练后，“面条”对应的向量可能是[1,0,1,1,0],而“方便面”对应的可能是[1,0,1,0,0]，而“狮子”对应的可能是[0,1,0,0,1]。\n",
    "\n",
    "这样“面条”向量乘“方便面”=2，而“面条”向量乘“狮子”=0 。这样就体现出面条与方便面之间的关系更加紧密，而与狮子就没什么关系了。这种表示方式更精准的表现出近义词之间的关系，比之稀疏向量优势很明显。可以说这是深度学习在NLP领域的第一个运用（虽然我觉得并没深到哪里去）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2.2 Skip-gram和CBOW模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Skip-gram：如果是用一个词语作为输入，来预测它周围的上下文，那这个模型叫做『Skip-gram 模型』"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/skipgram.jpg' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看成是 单个x->单个y 模型的并联，cost function 是单个 cost function 的累加（取log之后）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- CBOW：如果是拿一个词语的上下文作为输入，来预测这个词语本身，则是 『CBOW 模型』"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/cbow.jpg' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "与Skip-gram 的模型并联不同，这里是输入变成了多个单词，所以要对输入处理下（一般是求和然后平均），输出的 cost function不变"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2.3 Skip-gram与CBOW的对比 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CBOW是以周围词作为输入，中心词作为目标的网络，所以假设一篇语料中有V个单词，那么CBOW将会以每一个单词作为中心词进行训练，因此会有V次\n",
    "\n",
    "Skip-gram是以中心词作为输入，周围词作为目标的网络，那么对于一篇语料来说，每一个词都会作为中心词，每个中心词周围大小选择K个，那么将会进行KV次\n",
    "\n",
    "直观上来看，CBOW训练次数要比skip-gram少，也即精确率不如skip-gram,但是效率高，速度快。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2.3 实例演示"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们使用Text8Corpus做演示，这个语料库是英文的，大小不到100M。下载地址为:[Text8Zip](http://mattmahoney.net/dc/text8.zip )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "考虑到文件有些大， 就不传到github了，大家自行下载"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练及加载模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models.keyedvectors import KeyedVectors\n",
    "from gensim.models import word2vec\n",
    "import os\n",
    "folder = r'./nlpmodel/text8'\n",
    "modelfile = os.path.join(folder, 'text8.w2v.model')\n",
    "if os.path.exists(modelfile):\n",
    "    # 将硬盘中的模型载入\n",
    "    model = KeyedVectors.load(modelfile)\n",
    "else:\n",
    "    # 使用一个很小的英文语料\n",
    "    # 下载地址 http://mattmahoney.net/dc/text8.zip \n",
    "    sentences = word2vec.Text8Corpus(os.path.join(folder, 'text8'))\n",
    "    # Gensim的word2vec的训练模式由参数sg决定，0: CBOW，1: skip-gram，默认为CBOW\n",
    "    model = word2vec.Word2Vec(sentences, size=200, window=5, min_count=5)\n",
    "    # 设置向量为200维，窗口大小为5，忽略掉词频低于5的词\n",
    "    # 经过一段时间的等待，就训练完成了。\n",
    "    # 将训练好的模型保存到硬盘，文件名随意\n",
    "    model.save(os.path.join(folder, 'text8.w2v.model'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个例子是非常经典的，根据positive: worman, king, negative: man，推断出最相近的词是queen的例子\n",
    "\n",
    "其意义是计算一个词d（或者词表），使得该词的向量v(d)与v(a=\"woman\")-v(c=\"man\")+v(b=\"king\")最近"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\python36\\lib\\site-packages\\ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n",
      "d:\\python36\\lib\\site-packages\\gensim\\matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int32 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('queen', 0.6738182306289673), ('throne', 0.575992226600647), ('empress', 0.5739238262176514), ('princess', 0.5734347105026245), ('elizabeth', 0.5587899684906006), ('daughter', 0.555698037147522), ('jadwiga', 0.5538528561592102), ('prince', 0.5492069721221924), ('isabella', 0.545307457447052), ('consort', 0.541345477104187)]\n"
     ]
    }
   ],
   "source": [
    "print(model.most_similar(positive=[\"woman\",\"king\"],negative=[\"man\"],topn=10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\python36\\lib\\site-packages\\ipykernel_launcher.py:2: DeprecationWarning: Call to deprecated `similarity` (Method will be removed in 4.0.0, use self.wv.similarity() instead).\n",
      "  \n",
      "d:\\python36\\lib\\site-packages\\gensim\\matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int32 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6278037\n"
     ]
    }
   ],
   "source": [
    "# 计算两个词的相似度\n",
    "print(model.similarity('mobile', 'phone'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\python36\\lib\\site-packages\\ipykernel_launcher.py:2: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).\n",
      "  \n",
      "d:\\python36\\lib\\site-packages\\gensim\\matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int32 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('bad', 0.7257406711578369), ('poor', 0.5354788303375244), ('safe', 0.5291586518287659), ('quick', 0.5234116911888123), ('luck', 0.5217357873916626), ('reasonable', 0.5067963004112244), ('simple', 0.49832555651664734), ('really', 0.492895245552063), ('you', 0.48568713665008545), ('happy', 0.4837035536766052), ('silly', 0.4836675524711609), ('fun', 0.4832077622413635), ('pleasant', 0.4797707796096802), ('my', 0.4790000915527344), ('fast', 0.4784563183784485), ('easy', 0.4768497943878174), ('helpful', 0.47399526834487915), ('practical', 0.4722078740596771), ('your', 0.4720571041107178), ('little', 0.4691823422908783)]\n"
     ]
    }
   ],
   "source": [
    "# 寻找和某个词最相似的词（会输出词和相似度打分，本以为这个如果自己实现的话会很复杂，竟然在包里就提供了相关方法）\n",
    "print(model.most_similar('good', topn=20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\python36\\lib\\site-packages\\ipykernel_launcher.py:2: DeprecationWarning: Call to deprecated `doesnt_match` (Method will be removed in 4.0.0, use self.wv.doesnt_match() instead).\n",
      "  \n",
      "d:\\python36\\lib\\site-packages\\gensim\\matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int32 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cereal\n"
     ]
    }
   ],
   "source": [
    "# 识别不合群的词\n",
    "print(model.doesnt_match('breakfast cereal dinner lunch'.split()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "anarchism\n",
      "originated\n",
      "as\n",
      "a\n",
      "term\n",
      "of\n",
      "abuse\n",
      "first\n",
      "used\n",
      "against\n",
      "early\n",
      "working\n",
      "class\n",
      "radicals\n",
      "including\n",
      "the\n",
      "diggers\n",
      "english\n",
      "revolution\n",
      "and\n",
      "sans\n",
      "culottes\n",
      "french\n",
      "whilst\n",
      "is\n",
      "still\n",
      "in\n",
      "pejorative\n",
      "way\n",
      "to\n",
      "describe\n",
      "any\n",
      "act\n",
      "that\n",
      "violent\n",
      "means\n",
      "destroy\n",
      "organization\n",
      "society\n",
      "it\n",
      "has\n",
      "also\n",
      "been\n",
      "taken\n",
      "up\n",
      "positive\n",
      "label\n",
      "by\n",
      "self\n",
      "defined\n",
      "anarchists\n",
      "word\n",
      "derived\n",
      "from\n",
      "greek\n",
      "without\n",
      "archons\n",
      "ruler\n",
      "chief\n",
      "king\n",
      "political\n",
      "philosophy\n",
      "belief\n",
      "rulers\n",
      "are\n",
      "unnecessary\n",
      "should\n",
      "be\n",
      "abolished\n",
      "although\n",
      "there\n",
      "differing\n",
      "interpretations\n",
      "what\n",
      "this\n",
      "refers\n",
      "related\n",
      "social\n",
      "movements\n",
      "advocate\n",
      "elimination\n",
      "authoritarian\n",
      "institutions\n",
      "particularly\n",
      "state\n",
      "anarchy\n",
      "most\n",
      "use\n",
      "does\n",
      "not\n",
      "imply\n",
      "chaos\n",
      "nihilism\n",
      "or\n",
      "anomie\n",
      "but\n",
      "rather\n",
      "harmonious\n",
      "anti\n",
      "place\n",
      "regarded\n",
      "structures\n",
      "71290\n"
     ]
    }
   ],
   "source": [
    "# 获得词典中的词\n",
    "for index, key in enumerate(model.wv.vocab.keys()):\n",
    "    print(key)\n",
    "    if index > 100:\n",
    "        break\n",
    "print(len(model.wv.vocab.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3 文档向量 （Doc2Vec）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3.1 Doc2Vec原理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doc2Vec 或者叫做 paragraph2vec, sentence embeddings，是一种非监督式算法，可以获得sentences/paragraphs/documents 的向量表达，是 word2vec 的拓展，Doc2Vec的目的是获得文档的一个固定长度的向量表达, 即向量索引是以Doc为单位，而不是以Word为单位。\n",
    "\n",
    "学出来的向量可以通过计算距离来找 sentences/paragraphs/documents 之间的相似性， 或者进一步可以给文档打标签。\n",
    "\n",
    "例如首先是找到一个向量可以代表文档的意思，然后可以将向量投入到监督式机器学习算法中得到文档的标签， 例如在情感分析sentiment analysis 任务中，标签可以是 “negative”, “neutral”,”positive”"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doc2Vec也有两种方式来实现："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>DBOW (distributed bag of words)</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/DBOW.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>DM(distributed memory)</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./image/DM.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实战示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "假定有表示文档分类的样本集，如："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>sentence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2060 Retirement Fund, Vanguard Variable Insura...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Effective May 1, 2018, the following funds wil...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Effective on or after May 1, 2018, the followi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Effective on or after May 1, 2018, the followi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>The following Investment Options will be avail...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>The following portfolio has been added as an a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>This fund is available beginning 06/11/2018)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>This fund will be available on or about May 21...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>This fund will be available on or around May 2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>3</td>\n",
       "      <td>Effective August 13, 2018, the Investment Divi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4</td>\n",
       "      <td>FORMER FUND NAME MERGED FUND NAME</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>4</td>\n",
       "      <td>Emerging Growth Portfolio merged into The Trav...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>4</td>\n",
       "      <td>Effective on or about 05/01/06, Capital Apprec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>4</td>\n",
       "      <td>Effective on or about 05/01/06, The Travelers ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>4</td>\n",
       "      <td>Effective on or about 05/01/06, The Travelers ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>4</td>\n",
       "      <td>Effective on or about 05/01/06, Capital Apprec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>4</td>\n",
       "      <td>AIM V.I. Premier Equity Fund merged into AIM V...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>4</td>\n",
       "      <td>Effective on or about 05/01/06, The Travelers ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>4</td>\n",
       "      <td>All Cap Growth Portfolio merged into Legg Maso...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>4</td>\n",
       "      <td>AIM V.I. Premier Equity Fund merged into AIM V...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>4</td>\n",
       "      <td>The Travelers Series Trust MFS(R) Value Portfo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>4</td>\n",
       "      <td>MERGERSThe following former Underlying Funds m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>4</td>\n",
       "      <td>MERGED UNDERLYING FUND/TRUST SURVIVING UNDERLY...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>4</td>\n",
       "      <td>Travelers Series Trust Managed Allocation Seri...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>4</td>\n",
       "      <td>On March 15, 2018, the Board of Trustees of Vo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>4</td>\n",
       "      <td>HIMCO VIT INDEX FUND CLASS IB (MERGED INTO BLA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>4</td>\n",
       "      <td>As previously supplemented on February 28, 201...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>4</td>\n",
       "      <td>Subject to shareholder approval, effective aft...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>5</td>\n",
       "      <td>This fund will be liquidating on or about June...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>5</td>\n",
       "      <td>any Fund is liquidated then your allocations t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>5</td>\n",
       "      <td>Fund variable investment option (the \"Investme...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>5</td>\n",
       "      <td>LIQUIDATED AND INVESTED IN INVESCO V.I. GOVERN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>5</td>\n",
       "      <td>— — — — — — *CTIVP SM – Eaton Vance Floating R...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>5</td>\n",
       "      <td>3,003 2,601 2,169 2,225 1,859 759 232 55 — — *...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>5</td>\n",
       "      <td>*CTIVP SM – Eaton Vance Floating Rate Income F...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>5</td>\n",
       "      <td>Accumulation unit value at beginning of period...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>2</td>\n",
       "      <td>Effective May 1, 2018, no new premium payment ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>2</td>\n",
       "      <td>We no longer accept new applications for this ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>2</td>\n",
       "      <td>Effective May 1, 2018, the Long/Short Large Ca...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    category                                           sentence\n",
       "0          1  2060 Retirement Fund, Vanguard Variable Insura...\n",
       "1          1  Effective May 1, 2018, the following funds wil...\n",
       "2          1  Effective on or after May 1, 2018, the followi...\n",
       "3          1  Effective on or after May 1, 2018, the followi...\n",
       "4          1  The following Investment Options will be avail...\n",
       "5          1  The following portfolio has been added as an a...\n",
       "6          1       This fund is available beginning 06/11/2018)\n",
       "7          1  This fund will be available on or about May 21...\n",
       "8          1  This fund will be available on or around May 2...\n",
       "9          3  Effective August 13, 2018, the Investment Divi...\n",
       "10         4                  FORMER FUND NAME MERGED FUND NAME\n",
       "11         4  Emerging Growth Portfolio merged into The Trav...\n",
       "12         4  Effective on or about 05/01/06, Capital Apprec...\n",
       "13         4  Effective on or about 05/01/06, The Travelers ...\n",
       "14         4  Effective on or about 05/01/06, The Travelers ...\n",
       "15         4  Effective on or about 05/01/06, Capital Apprec...\n",
       "16         4  AIM V.I. Premier Equity Fund merged into AIM V...\n",
       "17         4  Effective on or about 05/01/06, The Travelers ...\n",
       "18         4  All Cap Growth Portfolio merged into Legg Maso...\n",
       "19         4  AIM V.I. Premier Equity Fund merged into AIM V...\n",
       "20         4  The Travelers Series Trust MFS(R) Value Portfo...\n",
       "21         4  MERGERSThe following former Underlying Funds m...\n",
       "22         4  MERGED UNDERLYING FUND/TRUST SURVIVING UNDERLY...\n",
       "23         4  Travelers Series Trust Managed Allocation Seri...\n",
       "24         4  On March 15, 2018, the Board of Trustees of Vo...\n",
       "25         4  HIMCO VIT INDEX FUND CLASS IB (MERGED INTO BLA...\n",
       "26         4  As previously supplemented on February 28, 201...\n",
       "27         4  Subject to shareholder approval, effective aft...\n",
       "28         5  This fund will be liquidating on or about June...\n",
       "29         5  any Fund is liquidated then your allocations t...\n",
       "30         5  Fund variable investment option (the \"Investme...\n",
       "31         5  LIQUIDATED AND INVESTED IN INVESCO V.I. GOVERN...\n",
       "32         5  — — — — — — *CTIVP SM – Eaton Vance Floating R...\n",
       "33         5  3,003 2,601 2,169 2,225 1,859 759 232 55 — — *...\n",
       "34         5  *CTIVP SM – Eaton Vance Floating Rate Income F...\n",
       "35         5  Accumulation unit value at beginning of period...\n",
       "36         2  Effective May 1, 2018, no new premium payment ...\n",
       "37         2  We no longer accept new applications for this ...\n",
       "38         2  Effective May 1, 2018, the Long/Short Large Ca..."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "doc2vecmodelrawtextpath = './nlpmodel/vacategorymodelrawtext.csv'\n",
    "rawdata = pd.read_csv(doc2vecmodelrawtextpath, encoding='utf-16', sep='\\t')\n",
    "display(rawdata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面来看一下，如何将这个样本集训练为Doc2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "准备工作：\n",
    "- 将数据集进行文本清洗\n",
    "- 将数据集进行Tag处理\n",
    "- 需要注意的是：Tag由下划线分隔，下划线左边为样本的索引，下划线右边为样本的category，如：1: Added, 2: Closed, 3: Reopen, 4: Merged, 5: Liquidation\n",
    "- <b>如果Tag按照这种方式进行处理，即达到利用无监督的方式，实现有监督的目的</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getdatasetfordoc2vector(file):\n",
    "    print('Get doc list begin')\n",
    "    rawdata = pd.read_csv(file, encoding='utf-16', sep='\\t')\n",
    "    rawdoclist = rawdata['sentence']\n",
    "    rawdata['cleantext'] = rawdata['sentence'].apply(lambda x: cleardatafordoc2vector(x))\n",
    "    rawdata['category'] = rawdata['category'].apply(int)\n",
    "    x_train = []\n",
    "    for index, row in rawdata.iterrows():\n",
    "        word_list = rawdata.loc[index, 'cleantext'].lower().strip().split()\n",
    "        if len(word_list) == 0:\n",
    "            word_list = ['only', 'for', 'test']\n",
    "        tagtype = rawdata.loc[index, 'category']\n",
    "        if not tagtype:\n",
    "            tagtype = 0\n",
    "        document = TaggedDocument(word_list, tags=['{0}_{1}'.format(index, tagtype)])\n",
    "        x_train.append(document)\n",
    "    print('Get doc list end')\n",
    "    return x_train, rawdoclist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练数据集的主方法体：\n",
    "\n",
    "- 清洗样本\n",
    "- 对样本进行训练，生成Doc2Vec模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "def traindoc2vec(rawtextfile,\n",
    "                 modelfolder,\n",
    "                 modelfilename,\n",
    "                 vector_size=200,\n",
    "                 epoch_num=10,\n",
    "                 needregenerate=True):\n",
    "    x_train, rawdoclist = getdatasetfordoc2vector(rawtextfile)\n",
    "    model_dm = doc2vectortrain(modelfolder,\n",
    "                               modelfilename,\n",
    "                               x_train,\n",
    "                               vector_size=vector_size,\n",
    "                               epoch_num=epoch_num,\n",
    "                               needregenerate=needregenerate)\n",
    "    return x_train, rawdoclist, model_dm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练样本为Doc2Vec模型的方法体：\n",
    "\n",
    "- 注意默认的训练次数为30\n",
    "- 特征向量维度为200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "def doc2vectortrain(folder, filename, x_train, vector_size=200, epoch_num=30, needregenerate=False):\n",
    "    print('Train Doc2Vector begin')\n",
    "    modelfilepath = os.path.join(folder, filename)\n",
    "    if not os.path.exists(folder):\n",
    "        os.makedirs(folder)\n",
    "    if needregenerate or not os.path.exists(modelfilepath):\n",
    "        model_dm = Doc2Vec(x_train, min_count=1, window=3, vector_size=vector_size, sample=1e-3, negative=5, workers=2)\n",
    "        model_dm.train(x_train, total_examples=model_dm.corpus_count, epochs=epoch_num)\n",
    "        model_dm.save(modelfilepath)\n",
    "    else:\n",
    "        model_dm = Doc2Vec.load(modelfilepath)\n",
    "    print('Train Doc2Vector end')\n",
    "    return model_dm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练Doc2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Get doc list begin\n",
      "Get doc list end\n",
      "Train Doc2Vector begin\n",
      "Train Doc2Vector end\n"
     ]
    }
   ],
   "source": [
    "rawtextfile = './nlpmodel/vacategorymodelrawtext.csv'\n",
    "modelfolder = './nlpmodel'\n",
    "modelfilename = 'vadoccategorydoc2vec.model'\n",
    "x_train, rawdoclist, model_dm = traindoc2vec(rawtextfile,\n",
    "                                             modelfolder,\n",
    "                                             modelfilename,\n",
    "                                             vector_size=100,\n",
    "                                             epoch_num=5000,\n",
    "                                             needregenerate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "初始化样本集以及Doc2Vec model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc2vecmodel = None\n",
    "alltext = None\n",
    "doc2vecmodelrawtextpath = './nlpmodel/vacategorymodelrawtext.csv'\n",
    "doc2vecmodelpath = './nlpmodel/vadoccategorydoc2vec.model'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "def initialdoc2vecmodel(alltextpath=doc2vecmodelrawtextpath, doc2vecmodelpath=doc2vecmodelpath):\n",
    "    print(\"load text begin\")\n",
    "    global alltext\n",
    "    if alltext is None:\n",
    "        rawdata = pd.read_csv(alltextpath, encoding='utf-16', sep='\\t')\n",
    "        alltext = rawdata['sentence']\n",
    "    print(\"load text end\")\n",
    "\n",
    "    print(\"load doc2vec model begin\")\n",
    "    global doc2vecmodel\n",
    "    if doc2vecmodel is None:\n",
    "        doc2vecmodel = Doc2Vec.load(doc2vecmodelpath)\n",
    "    print(\"load doc2vec model end\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根据Model以及样本数据，获得inver_vector列表 (Infer a vector for given post-bulk training document(为给定的批量后培训文档推断一个向量))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getdoc2vec_inferedvectorlist(doc2vecmodelpath, x_train):\n",
    "    infered_vectors_list = []\n",
    "    print(\"load doc2vec model begin\")\n",
    "    model_dm = Doc2Vec.load(doc2vecmodelpath)\n",
    "    print(\"load doc2vec model end\")\n",
    "\n",
    "    print(\"load train vectors begin\")\n",
    "    for text, label in x_train:\n",
    "        vector = model_dm.infer_vector(text)\n",
    "        infered_vectors_list.append(vector)\n",
    "    print(\"load train vectors end\")\n",
    "    return infered_vectors_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "方法体：根据给定的sentence以及Doc2Vec model，获得相似度最高的前10个句子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getmostsimilaritybydoc2vec(sentence):\n",
    "    global alltext\n",
    "    global doc2vecmodel\n",
    "    if alltext is None or doc2vecmodel is None:\n",
    "        initialdoc2vecmodel()\n",
    "    test_cut = cleardatafordoc2vector(sentence).split()\n",
    "    inferred_vector = doc2vecmodel.infer_vector(test_cut)\n",
    "    simsbow = doc2vecmodel.docvecs.most_similar([inferred_vector], topn=10)\n",
    "    return getcontent(sentence, simsbow, alltext)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getcontent(rawsentence, simsbow, doclist):\n",
    "    similarresult = {'rawsentence': rawsentence, 'similarlist': []}\n",
    "    for i in simsbow:\n",
    "        similardict = {}\n",
    "        similar = doclist[int(i[0].split('_')[0])]\n",
    "        similardict['paraid'] = i[0]\n",
    "        similardict['similarity'] = i[1]\n",
    "        similardict['paracontent'] = similar\n",
    "        similarresult['similarlist'].append(similardict)\n",
    "    return similarresult"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "load text begin\n",
      "load text end\n",
      "load doc2vec model begin\n",
      "load doc2vec model end\n"
     ]
    }
   ],
   "source": [
    "initialdoc2vecmodel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "示例：根据给定的sentence以及Doc2Vec model，获得相似度最高的前10个句子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "d:\\python36\\lib\\site-packages\\gensim\\matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int32 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "###############################\n",
      "raw sentence is: \n",
      "CTIVP SM – Eaton Vance Floating Rate Income Fund (Class 2) liquidated on April 27, 2018.\n",
      "###############################\n",
      "\n",
      "paragraph id: 32_5, similarity: 0.9833414554595947\n",
      "paragraph is:\n",
      "— — — — — — *CTIVP SM – Eaton Vance Floating Rate Income Fund (Class 2) liquidated on April 27, 2018.\n",
      "###############################\n",
      "paragraph id: 34_5, similarity: 0.982774555683136\n",
      "paragraph is:\n",
      "*CTIVP SM – Eaton Vance Floating Rate Income Fund (Class 2) liquidated on April 27, 2018.\n",
      "###############################\n",
      "paragraph id: 33_5, similarity: 0.9814426302909851\n",
      "paragraph is:\n",
      "3,003 2,601 2,169 2,225 1,859 759 232 55 — — *CTIVP SM – Eaton Vance Floating Rate Income Fund (Class 2) liquidated on April 27, 2018.\n",
      "###############################\n",
      "paragraph id: 35_5, similarity: 0.7010773420333862\n",
      "paragraph is:\n",
      "Accumulation unit value at beginning of period $1.08 $1.00 $1.03 $1.05 $1.03 $1.00 Accumulation unit value at end of period $1.09 $1.08 $1.00 $1.03 $1.05 $1.03 Number of accumulation units outstanding at end of period (000 omitted) 10 10 55 57 60 — *CTIVP SM – Eaton Vance Floating Rate Income Fund (Class 2) liquidated on April 27, 2018.\n",
      "###############################\n",
      "paragraph id: 31_5, similarity: 0.640414834022522\n",
      "paragraph is:\n",
      "LIQUIDATED AND INVESTED IN INVESCO V.I. GOVERNMENT MONEY MARKET FUND ON 04/27/2018)\n",
      "###############################\n",
      "paragraph id: 25_4, similarity: 0.6275163888931274\n",
      "paragraph is:\n",
      "HIMCO VIT INDEX FUND CLASS IB (MERGED INTO BLACKROCK S&P 500 INDEX V.I. FUND ON 04/20/2018; REORGANIZED FROM HARTFORD INDEX HLS FUND ON 10/20/2014) UNIT VALUE: Beginning of Period $ 12.128 $ 10.981 $ 10.979\n",
      "###############################\n",
      "paragraph id: 28_5, similarity: 0.6186009049415588\n",
      "paragraph is:\n",
      "This fund will be liquidating on or about June 28, 2018.\n",
      "###############################\n",
      "paragraph id: 17_4, similarity: 0.5357693433761597\n",
      "paragraph is:\n",
      "Effective on or about 05/01/06, The Travelers Series Trust Pioneer Strategic Income Portfolio merged into Met Investors Series Trust Pioneer Strategic Income Portfolio and is no longer available as a funding option.\n",
      "###############################\n",
      "paragraph id: 14_4, similarity: 0.5356754064559937\n",
      "paragraph is:\n",
      "Effective on or about 05/01/06, The Travelers Series Trust Mondrian International Stock Portfolio merged into Met Investors Series Trust Harris Oakmark International Portfolio and is no longer available as a funding option.\n",
      "###############################\n",
      "paragraph id: 12_4, similarity: 0.5340012311935425\n",
      "paragraph is:\n",
      "Effective on or about 05/01/06, Capital Appreciation Fund merged into Met Investors Series Trust Janus Capital Appreciation Portfolio and is no longer available as a funding option.\n",
      "###############################\n"
     ]
    }
   ],
   "source": [
    "# sentence = r'Effective May 1, 2018, the following funds will be available as new investment options under your Policy.'\n",
    "# sentence = r'Fund variable investment option (the \"Investment Option\") will be liquidated.'\n",
    "sentence = r'CTIVP SM – Eaton Vance Floating Rate Income Fund (Class 2) liquidated on April 27, 2018.'\n",
    "result = getmostsimilaritybydoc2vec(sentence)\n",
    "print('###############################')\n",
    "print('raw sentence is: ')\n",
    "print(result['rawsentence'])\n",
    "print('###############################')\n",
    "print()\n",
    "for similar in result['similarlist']:\n",
    "    print('paragraph id: {0}, similarity: {1}'.format(\n",
    "        similar['paraid'],\n",
    "        similar['similarity']))\n",
    "    print('paragraph is:')\n",
    "    print(similar['paracontent'])\n",
    "    print('###############################')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}