gensim_torchText_janome.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "gensim_torchText_janome.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOQQ8EdkNyNWScHHa7OAC0n",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mashyko/NLP_BERT_Transformers/blob/master/gensim_torchText_janome.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Z2gOxtQSVgB",
        "colab_type": "text"
      },
      "source": [
        "#gensim 、Torchtext　+ Janome を用いた単語間のコサイン類似度の計算\n",
        "\n",
        "以下のウェブサイトを参考にしました。\n",
        "https://qiita.com/MonaCat/items/c26fa660a711f010c39a\n",
        "\n",
        "Torchtext　の使用については、『つくりながら学ぶ! PyTorchによる発展ディープラーニング」（小川雄太郎、マイナビ出版）』を参考にしています。\n",
        "\n",
        "google drive のディレクトリ /data/に、学習済み日本語単語辞書/vector_neologd/と/tsv/ファイルが保存されている必要があります。\n",
        "vedtor_neologdは以下からダウンロードできます。　https://drive.google.com/open?id=0ByFQ96A4DgSPUm9wVWRLdm5qbmc\n",
        "\n",
        "tsvファイルは各自で作成ください。tsv ファイルは、train.tsv val.tsv test.tsv の３種類から構成されます。ファイルの中身は同一で構いませんが、単語に、王、女王、姫、王子、自動運転、などが含まれている必要があります。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t5d1A8LEQ2Lj",
        "colab_type": "code",
        "outputId": "6b79ac0d-264e-4740-833b-14a4ab517d05",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e-xUU2H7Q7wb",
        "colab_type": "code",
        "outputId": "89b0cb6e-f676-4fa1-bfe5-4fa3f3cbcbd1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "\n",
        "%cd /content/drive/My Drive/data\n",
        "!ls"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content/drive/My Drive/data\n",
            "entity_vector.model.bin  train-images-idx3-ubyte  vector_neologd\n",
            "t10k-images-idx3-ubyte\t train-labels-idx1-ubyte\n",
            "t10k-labels-idx1-ubyte\t tsv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NF9ifJ4JRcqq",
        "colab_type": "code",
        "outputId": "8213063a-80b1-484e-ed61-e27932bd5e04",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 326
        }
      },
      "source": [
        "! pip install gensim"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: gensim in /usr/local/lib/python3.6/dist-packages (3.6.0)\n",
            "Requirement already satisfied: smart-open>=1.2.1 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.9.0)\n",
            "Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.17.5)\n",
            "Requirement already satisfied: six>=1.5.0 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.12.0)\n",
            "Requirement already satisfied: scipy>=0.18.1 in /usr/local/lib/python3.6/dist-packages (from gensim) (1.4.1)\n",
            "Requirement already satisfied: boto>=2.32 in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim) (2.49.0)\n",
            "Requirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim) (1.11.15)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from smart-open>=1.2.1->gensim) (2.21.0)\n",
            "Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim) (0.3.3)\n",
            "Requirement already satisfied: botocore<1.15.0,>=1.14.15 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim) (1.14.15)\n",
            "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->smart-open>=1.2.1->gensim) (0.9.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.2.1->gensim) (2019.11.28)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.2.1->gensim) (3.0.4)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.2.1->gensim) (1.24.3)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->smart-open>=1.2.1->gensim) (2.8)\n",
            "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.6/dist-packages (from botocore<1.15.0,>=1.14.15->boto3->smart-open>=1.2.1->gensim) (2.6.1)\n",
            "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.15.0,>=1.14.15->boto3->smart-open>=1.2.1->gensim) (0.15.2)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fXlOOW73RVkz",
        "colab_type": "code",
        "outputId": "7abf61d1-0d33-4707-d976-06557b2e4998",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "source": [
        "import gensim\n",
        "\n",
        "model = gensim.models.KeyedVectors.load_word2vec_format('./vector_neologd/model.vec', binary=False)\n",
        "\n",
        "model.most_similar(positive=['自然言語処理'])"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
            "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n",
            "/usr/local/lib/python3.6/dist-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
            "  if np.issubdtype(vec.dtype, np.int):\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('自然言語理解', 0.7600098848342896),\n",
              " ('自然言語', 0.7503659725189209),\n",
              " ('計算言語学', 0.7258570194244385),\n",
              " ('自動プログラミング', 0.6848069429397583),\n",
              " ('テキストマイニング', 0.6811494827270508),\n",
              " ('コンピュータ言語', 0.6618390083312988),\n",
              " ('メタプログラミング', 0.658093273639679),\n",
              " ('Webプログラミング', 0.6488876342773438),\n",
              " ('形態素解析', 0.6479052901268005),\n",
              " ('コーパス言語学', 0.6465639472007751)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XOv7vUasRfzD",
        "colab_type": "code",
        "outputId": "f9ed0987-aaea-4651-fc5d-23e8e57b2155",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        }
      },
      "source": [
        "model.most_similar(positive=['友人'], negative=['友情'])"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
            "  if np.issubdtype(vec.dtype, np.int):\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('知人', 0.4586910605430603),\n",
              " ('自宅', 0.35488438606262207),\n",
              " ('知り合い', 0.329221248626709),\n",
              " ('入り浸り', 0.3212822675704956),\n",
              " ('親戚', 0.31865695118904114),\n",
              " ('知り合っ', 0.3158203959465027),\n",
              " ('宅', 0.31503963470458984),\n",
              " ('誘わ', 0.302945077419281),\n",
              " ('繁く', 0.30250048637390137),\n",
              " ('同僚', 0.29792869091033936)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8u66EgJdR6fj",
        "colab_type": "code",
        "outputId": "500c95d0-2273-4045-a53c-841fd209d707",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "!pip install janome"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting janome\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/79/f0/bd7f90806132d7d9d642d418bdc3e870cfdff5947254ea3cab27480983a7/Janome-0.3.10-py2.py3-none-any.whl (21.5MB)\n",
            "\u001b[K     |████████████████████████████████| 21.5MB 156kB/s \n",
            "\u001b[?25hInstalling collected packages: janome\n",
            "Successfully installed janome-0.3.10\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z884CDCUR58s",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from janome.tokenizer import Tokenizer\n",
        "\n",
        "j_t = Tokenizer()\n",
        "\n",
        "def tokenizer_janome(text):\n",
        "    return [tok for tok in j_t.tokenize(text, wakati=True)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M6UXYZ4SSCj4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import re\n",
        "import unicodedata\n",
        "import string\n",
        "\n",
        "\n",
        "def format_text(text):\n",
        "    text = unicodedata.normalize(\"NFKC\", text)\n",
        "    table = str.maketrans(\"\", \"\", string.punctuation  + \"「」、。・\")\n",
        "    text = text.translate(table)\n",
        "\n",
        "    return text\n",
        "\n",
        "\n",
        "def preprocessing(text):\n",
        "    \"\"\"\n",
        "    前処理をする関数\n",
        "    \"\"\"\n",
        "    text.translate(str.maketrans({chr(0xFF01 + i): chr(0x21 + i) for i in range(94)}))  # 全角→半角\n",
        "\n",
        "    text = text.lower()  # 大文字→小文字\n",
        "\n",
        "    text = re.sub('\\r', '', text)\n",
        "    text = re.sub('\\n', '', text)\n",
        "    text = re.sub(' ', '', text)\n",
        "    text = re.sub('　', '', text)\n",
        "\n",
        "    text = re.sub(r'[0-9 ０-９]', '', text)  # 数字の除去\n",
        "    text = re.sub(r'[!-/:-@[-`{-~]', '', text)  # 半角記号の除去\n",
        "    text = re.sub(r'/[！-／：-＠［-｀｛-～、-〜”’・]', '', text)  # 全角記号の除去\n",
        "    text = format_text(text)  # 全角記号の除去\n",
        "\n",
        "    return  text\n",
        "\n",
        "\n",
        "def tokenizer_with_preprocessing(text):\n",
        "    text = preprocessing(text)\n",
        "    text = tokenizer_janome(text)\n",
        "\n",
        "    return text"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SK2zvaiASJvd",
        "colab_type": "code",
        "outputId": "c14653d6-1c78-40c6-f2c0-494731e59d19",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "text = '今回は　fastTextを　使って 分散表現を獲得したい！！！?'\n",
        "print(tokenizer_with_preprocessing(text))"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['今回', 'は', 'fasttext', 'を', '使っ', 'て', '分散', '表現', 'を', '獲得', 'し', 'たい']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RF-xmttmSONr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torchtext\n",
        "\n",
        "max_length = 25\n",
        "TEXT = torchtext.data.Field(sequential=True, tokenize=tokenizer_with_preprocessing,\n",
        "                            use_vocab=True, lower=True, include_lengths=True,\n",
        "                            batch_first=True, fix_length=max_length)\n",
        "LABEL = torchtext.data.Field(sequential=False, use_vocab=False)\n",
        "\n",
        "train_ds, val_ds, test_ds = torchtext.data.TabularDataset.splits(\n",
        "    path='/content/drive/My Drive/data/tsv', train='train.tsv',\n",
        "    validation='val.tsv', test='test.tsv', format='tsv',\n",
        "    fields=[('Text', TEXT), ('Label', LABEL)])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BmyWKqSfSoTS",
        "colab_type": "code",
        "outputId": "5a4d8eee-fe4f-4274-daa6-4da739685fca",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "from torchtext.vocab import Vectors\n",
        "\n",
        "vectors = Vectors(name='./vector_neologd/model.vec')\n",
        "\n",
        "# ベクトル化したバージョンのボキャブラリーを作成します\n",
        "TEXT.build_vocab(train_ds, vectors=vectors, min_freq=1)\n",
        "\n",
        "# ボキャブラリーのベクトルを確認します\n",
        "print(TEXT.vocab.vectors.shape)  # 単語が300次元のベクトルで表現されている\n",
        "TEXT.vocab.vectors\n",
        "\n",
        "# ボキャブラリーの単語の順番を確認します\n",
        "TEXT.vocab.stoi"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([73, 300])\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "defaultdict(<function torchtext.vocab._default_unk_index>,\n",
              "            {'<pad>': 1,\n",
              "             '<unk>': 0,\n",
              "             'ある': 14,\n",
              "             'い': 11,\n",
              "             'いる': 15,\n",
              "             'か': 16,\n",
              "             'が': 4,\n",
              "             'き': 29,\n",
              "             'この': 17,\n",
              "             'さ': 18,\n",
              "             'し': 12,\n",
              "             'しまう': 30,\n",
              "             'する': 19,\n",
              "             'その': 31,\n",
              "             'た': 13,\n",
              "             'て': 5,\n",
              "             'で': 20,\n",
              "             'でし': 32,\n",
              "             'と': 7,\n",
              "             'とても': 33,\n",
              "             'な': 8,\n",
              "             'に': 9,\n",
              "             'に対して': 34,\n",
              "             'の': 2,\n",
              "             'は': 3,\n",
              "             'まし': 21,\n",
              "             'ます': 22,\n",
              "             'れ': 23,\n",
              "             'を': 6,\n",
              "             'アニメ': 35,\n",
              "             'クラス': 36,\n",
              "             'ネガティブ': 37,\n",
              "             'ポジティブ': 38,\n",
              "             'モデル': 39,\n",
              "             'レビュー': 40,\n",
              "             '人工': 41,\n",
              "             '使用': 42,\n",
              "             '借り': 43,\n",
              "             '値': 44,\n",
              "             '分類': 24,\n",
              "             '化': 45,\n",
              "             '名': 46,\n",
              "             '君': 47,\n",
              "             '商品': 48,\n",
              "             '国': 49,\n",
              "             '基礎': 50,\n",
              "             '夫婦': 25,\n",
              "             '女性': 51,\n",
              "             '女王': 52,\n",
              "             '姫': 53,\n",
              "             '学習': 54,\n",
              "             '実用': 55,\n",
              "             '川': 56,\n",
              "             '技術': 57,\n",
              "             '文章': 10,\n",
              "             '映像': 58,\n",
              "             '映画': 59,\n",
              "             '村': 60,\n",
              "             '構築': 61,\n",
              "             '機械': 62,\n",
              "             '洗濯': 63,\n",
              "             '王': 64,\n",
              "             '王子': 65,\n",
              "             '男性': 66,\n",
              "             '知能': 67,\n",
              "             '短い': 68,\n",
              "             '綺麗': 69,\n",
              "             '自動': 26,\n",
              "             '自動車': 70,\n",
              "             '行き': 71,\n",
              "             '評価': 27,\n",
              "             '近年': 72,\n",
              "             '運転': 28})"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1h7aclhBSu8G",
        "colab_type": "code",
        "outputId": "74824efc-7eea-42a3-b159-2e57cdc6e637",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "import torch.nn.functional as F\n",
        "\n",
        "tensor_calc = TEXT.vocab.vectors[TEXT.vocab.stoi['王子']]\n",
        "\n",
        "# コサイン類似度\n",
        "print(\"王\", F.cosine_similarity(tensor_calc, TEXT.vocab.vectors[TEXT.vocab.stoi['王']], dim=0))\n",
        "print(\"男性\", F.cosine_similarity(tensor_calc, TEXT.vocab.vectors[TEXT.vocab.stoi['男性']], dim=0))\n",
        "print(\"機械学習\", F.cosine_similarity(tensor_calc, TEXT.vocab.vectors[TEXT.vocab.stoi['機械学習']], dim=0))"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "王 tensor(0.5338)\n",
            "男性 tensor(0.2315)\n",
            "機械学習 tensor(0.)\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}