notebooks/Gensim.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gensim, JUMAN++, KNPで文書類似度算出\n",
    "\n",
    "[DeepAgeさんの「Doc2Vecの仕組みとGensimを使った文書類似度算出チュートリアル」](https://deepage.net/machine_learning/2017/01/08/doc2vec.html#最も似ている記事を取得する)\n",
    "が大変参考になりました。ありがとうございました！\n",
    "\n",
    "注意：\n",
    "- JUMAN++, KNPは[Ansibleタスク](../roles)でインストールできるようにしてあります\n",
    "- KNPのバージョンが4.17に上がりました\n",
    "- ファイルを開くとき、文字コードをutf-8に指定する必要があります\n",
    "\n",
    "## 参考：\n",
    "\n",
    "- [DeepAge Doc2Vecの仕組みとGensimを使った文書類似度算出チュートリアル](https://deepage.net/machine_learning/2017/01/08/doc2vec.html#最も似ている記事を取得する)\n",
    "- [黒橋・河原研究室　自然言語処理のためのリソース](http://nlp.ist.i.kyoto-u.ac.jp/index.php?NLP%E3%83%AA%E3%82%BD%E3%83%BC%E3%82%B9)\n",
    "- [hassaku's blog 学習済みword2vecモデルを調べてみた](http://blog.hassaku-labs.com/post/pretrained-word2vec/)\n",
    "- [白ヤギコーポレーション word2vecの学習済み日本語モデルを公開します](http://aial.shiroyagi.co.jp/2017/02/japanese-word2vec-model-builder/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## コーパス\n",
    "\n",
    "livedoorのニュースコーパスをダウンロード"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!export LDCC=ldcc-20140209 &&\\\n",
    "mkdir -p /workspace/notebooks/corpora/ldcc &&\\\n",
    "cd /workspace/notebooks/corpora/ldcc &&\\\n",
    "wget http://www.rondhuit.com/download/$LDCC.tar.gz &&\\\n",
    "tar xvfz $LDCC.tar.gz &&\\\n",
    "rm $LDCC.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 助走\n",
    "\n",
    "必要ライブラリをimportし、関数をいくつか定義"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "from os import listdir, path\n",
    "from pyknp import Jumanpp\n",
    "from gensim.models import Doc2Vec\n",
    "from gensim.models.doc2vec import LabeledSentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "記事ファイルをダウンロードしたディレクトリから取得する関数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def corpus_files():\n",
    "    dirs = [path.join('corpora/ldcc/text', x)\n",
    "            for x in listdir('corpora/ldcc/text') if not x.endswith('.txt')]\n",
    "    docs = [path.join(x, y)\n",
    "            for x in dirs for y in listdir(x) if not x.startswith('LICENSE')]\n",
    "    return docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "記事コンテンツをパスから取得する関数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def read_document(path):\n",
    "    with open(path, 'r', encoding=\"utf-8\") as f:\n",
    "        return f.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "JUMAN++を使って記事を単語リストに変換する関数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def split_into_words(text):\n",
    "    result = Jumanpp().analysis(text)\n",
    "    return [mrph.midasi for mrph in result.mrph_list()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 形態素解析\n",
    "\n",
    "記事コンテンツを単語に分割して、Doc2Vecの入力に使うLabeledSentenceに変換する関数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def doc_to_sentence(doc, name):\n",
    "    words = split_into_words(doc)\n",
    "    return LabeledSentence(words=words, tags=[name])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "記事のパスリストから、記事コンテンツに変換し、単語分割して、センテンスのジェネレーターを返す関数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def corpus_to_sentences(corpus):\n",
    "    docs   = [read_document(x) for x in corpus]\n",
    "    for idx, (doc, name) in enumerate(zip(docs, corpus)):\n",
    "        sys.stdout.write('\\r前処理中 {}/{}'.format(idx, len(corpus)))\n",
    "        yield doc_to_sentence(doc, name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doc2Vecパラメータを渡して、学習"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus = corpus_files()\n",
    "sentences = corpus_to_sentences(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 学習\n",
    "\n",
    "dmに1を設定するとdmpvで学習されることになる。1以外であれば、DBoWで学習される。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "前処理中 7375/7376"
     ]
    }
   ],
   "source": [
    "model = Doc2Vec(sentences, dm=0, size=300, window=15, alpha=.025,\n",
    "        min_alpha=.025, min_count=1, sample=1e-6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print('\\n訓練開始')\n",
    "for epoch in range(20):\n",
    "    print('Epoch: {}'.format(epoch + 1))\n",
    "    model.train(sentences, total_examples=model.corpus_count, epochs=model.iter)\n",
    "    model.alpha -= (0.025 - 0.0001) / 19\n",
    "    model.min_alpha = model.alpha\n",
    "\n",
    "print('モデル保存\\n')\n",
    "!mkdir -p models\n",
    "model.save('models/doc2vec.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "モデルの保存と読み込み"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# モデル読み込み\n",
    "model = Doc2Vec.load('models/doc2vec.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 類似記事の取得\n",
    "\n",
    "\n",
    "最も似ている記事を取得する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('corpora/ldcc/text/movie-enter/movie-enter-6165056.txt',\n",
       "  0.24260929226875305)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.docvecs.most_similar('corpora/ldcc/text/livedoor-homme/livedoor-homme-5625149.txt', topn=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 類似度の算出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.11023943942407691"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.docvecs.similarity('corpora/ldcc/text/livedoor-homme/livedoor-homme-4700669.txt', \n",
    "                         'corpora/ldcc/text/movie-enter/movie-enter-5947726.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}