nlp/Word2Vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word2Vec exploration\n",
    "This notebook is going to explore word2vec and is based on [this problem sets(in JP)](http://www.cl.ecei.tohoku.ac.jp/nlp100/).  \n",
    "Considering machine power, we'll use the 1/100 sample of Wikipedia English passages with more than 400 words at the time of Jan.12th, 2015. We'll focus on dealing with country names here.  \n",
    "The contents include the following:  \n",
    "\n",
    "- Normalize data and construct the corpus\n",
    "- Construct words vector from scratch with demension compression\n",
    "- Construct another set of word vectors with google's word2vec\n",
    "- Compare the two vector sets with some Country-related analogy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Country name normalizer\n",
    "\n",
    "To begin with, since we're going to deal with country names, we might want to make sure that we have a normalized country name list, so that multi-words ones will not be splitted.  \n",
    "We are going to use data from [here](https://mledoze.github.io/countries/). (Thanks to mledoze and all the contributors work). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a normalizer that does the follow:\n",
    "# - Extract common data from countries json data\n",
    "# - Convert all country names to lower case\n",
    "# - Replace spaces in country names with _\n",
    "\n",
    "from __future__ import print_function\n",
    "import json\n",
    "import urllib2\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the united_states is in northern america.\n"
     ]
    }
   ],
   "source": [
    "class CountryNameNormalizer:\n",
    "  def __init__(self):\n",
    "    country_json = urllib2.urlopen(\"https://raw.githubusercontent.com/mledoze/countries/master/dist/countries.json\").read()\n",
    "    country_names = [record[\"name\"][\"common\"] for record in json.loads(country_json)]\n",
    "    self._normalizer_dict = {\n",
    "      country.lower() : re.sub(\" \", \"_\", country.lower())\n",
    "      for country in country_names\n",
    "    }\n",
    "\n",
    "  def normalize_line(self, line):\n",
    "    for k, v in self._normalizer_dict.items():\n",
    "      line = re.sub(k, v, line)\n",
    "    return line\n",
    "  \n",
    "# Test the normalizer\n",
    "sample_sentence = \"The United States is in Northern America.\"\n",
    "print(CountryNameNormalizer().normalize_line(sample_sentence.lower()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Corpus generation\n",
    "\n",
    "Now let's load the data and normalize it to generate the corpus. The above normalization for countries would be added in particular."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: data: File exists\n",
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 21.0M  100 21.0M    0     0  5437k      0  0:00:03  0:00:03 --:--:-- 5437k\n"
     ]
    }
   ],
   "source": [
    "# Uncomment and execute on first time\n",
    "!mkdir data\n",
    "!curl http://www.cl.ecei.tohoku.ac.jp/nlp100/data/enwiki-20150112-400-r100-10576.txt.bz2 -o ./data/enwiki-20150112-400-r100-10576.txt.bz2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bz2\n",
    "# We'll save the corpus into a file in-case we're going to start from the beging again.\n",
    "\n",
    "def clean_word(word):\n",
    "  result = word.lower()\n",
    "  # depuncuate\n",
    "  result = re.sub(\"^[ \\$,\\.:;\\?!`\\+<>'\\\"\\[\\]\\(\\)\\*%#&\\n(?:\\\\n)]+\", \"\", result)\n",
    "  result = re.sub(\"[ \\$,\\.:;\\?!`\\+<>'\\\"\\[\\]\\(\\)\\*%#&\\n(?:\\\\n)]+$\", \"\", result)\n",
    "  # rip number\n",
    "  result = re.sub(\"[0-9]+\", \"0\", result)\n",
    "  return result\n",
    "\n",
    "def generate_corpus(infile, outfile):\n",
    "  country_normalizer = CountryNameNormalizer()\n",
    "\n",
    "  i = 0\n",
    "  with bz2.BZ2File(infile, \"r\") as fin, open(outfile, \"w\") as fout:\n",
    "    for line in fin:\n",
    "      i += 1\n",
    "      cleaned = \" \".join([clean_word(word) for word in line.decode(\"utf-8\").split(\" \")])\n",
    "      cleaned = country_normalizer.normalize_line(cleaned)\n",
    "      if cleaned != \"\":\n",
    "        print(cleaned.encode(\"utf-8\"), file=fout)\n",
    "\n",
    "data_file = \"./data/enwiki-20150112-400-r100-10576.txt.bz2\"\n",
    "corpus_file = \"./data/normalized_corpus.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2h 9min 1s, sys: 3min 33s, total: 2h 12min 35s\n",
      "Wall time: 2h 18min 37s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# -*- coding:utf-8 -*-\n",
    "\n",
    "corpus = generate_corpus(data_file, corpus_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the vectors set from scratch\n",
    "\n",
    "With the corpus ready, we want to build the vector sets. To do that we could do the following:\n",
    "\n",
    "- Build the cooccurance matrix calculated by PPMI(Positive Pointwise Mutual Information)\n",
    "- Reduce the dementionality of the matrix to form the final vector sets\n",
    "\n",
    "Each value of the matrix $X_{tx}$ should be a PPMI value:\n",
    "$$ X_{tc} = PPMI(t, c) = max(log\\frac{N f(t,c)}{f(t,*) f(*,c)}, 0)$$\n",
    "Only when $f(t,c) >= 10$, otherwise it would be 0\n",
    "\n",
    "Where:\n",
    "\n",
    "- $f(t,c)$ represents the counts of cooccurance of words t & c\n",
    "- $f(t,*)$ represents how many times t appear\n",
    "- $f(*,c)$ represents how many times the word c appears in any cooccurance\n",
    "- $N$ is the total counts of the cooccurance pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The cooccurance will take 4 words before and after\n",
    "from collections import Counter, OrderedDict\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "from scipy import sparse\n",
    "import numpy as np\n",
    "import pickle\n",
    "\n",
    "t_encode_file = \"./data/t_encode.pickle\"\n",
    "vectors_file = \"./data/vectors.pickle\"\n",
    "\n",
    "def corpus2vectors(corpus_path, t_encode_path, vectors_path, k=4, batch_size=100000):\n",
    "  buffer_tc, buffer_t, buffer_c = [], [], []\n",
    "  batch = 0\n",
    "  counter_tc = Counter()\n",
    "  counter_t = Counter()\n",
    "  counter_c = Counter()\n",
    "\n",
    "  with open(corpus_file, \"r\") as fi:\n",
    "    for line in fi:\n",
    "      splited = line.decode(\"utf-8\").rstrip(\"\\n\").split(\" \")\n",
    "      # Skip single word without pairing\n",
    "      if len(splited) == 1:\n",
    "        continue\n",
    "      for i, token in enumerate(splited):\n",
    "        for j in range(max(i-k,0), min(i+k+1, len(splited))):\n",
    "          if i == j:\n",
    "            continue\n",
    "          # Use counter to record the data\n",
    "          if batch >= batch_size:\n",
    "            counter_tc.update(buffer_tc)\n",
    "            counter_t.update(buffer_t)\n",
    "            counter_c.update(buffer_c)\n",
    "            buffer_tc, buffer_t, buffer_c = [], [], []\n",
    "            batch = 0\n",
    "          else:\n",
    "            buffer_tc.append(\" \".join([token, splited[j]]))\n",
    "            buffer_t.append(token)\n",
    "            buffer_c.append(splited[j])\n",
    "            batch += 1\n",
    "\n",
    "  #   Pre-calculate for the model\n",
    "  t_size = len(counter_t)\n",
    "  c_size = len(counter_c)\n",
    "  t_encoder = OrderedDict({key : i for i, key in enumerate(counter_t.keys())})\n",
    "  c_encoder = OrderedDict({key : i for i, key in enumerate(counter_c.keys())})\n",
    "  n = sum(counter_tc.values())\n",
    "  with open(t_encode_path, \"wb\") as fopen:\n",
    "    pickle.dump(t_encoder, fopen)\n",
    "\n",
    "  # Calculate the matrix and make it a sparse one\n",
    "  sparse_m = sparse.lil_matrix((t_size, c_size))\n",
    "  for k, v in counter_tc.items():\n",
    "    if counter_tc[k] < 10:\n",
    "        continue\n",
    "    [t, c] = k.split(\" \")\n",
    "    ppmi = max(np.log(n) + np.log(counter_tc[k]) - np.log(counter_t[t] * counter_c[c]) , 0)\n",
    "    sparse_m[t_encoder[t], c_encoder[c]] = ppmi\n",
    "\n",
    "  # Use TruncatedSVD to reduce the dimension\n",
    "  decomposer = TruncatedSVD(n_components = 300)\n",
    "  decomposed = decomposer.fit_transform(sparse_m)\n",
    "  with open(vectors_path, \"wb\") as fopen:\n",
    "    pickle.dump(decomposed, fopen)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 12min 39s, sys: 54.2 s, total: 13min 33s\n",
      "Wall time: 12min 53s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "corpus2vectors(corpus_file, t_encode_file, vectors_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Play around with the vector sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 18.9 s, sys: 5.71 s, total: 24.6 s\n",
      "Wall time: 25.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "import pickle\n",
    "\n",
    "with open(vectors_file, \"rb\") as fo:\n",
    "  vectors = pickle.load(fo)\n",
    "with open(t_encode_file, \"rb\") as fo:\n",
    "  t_enc = pickle.load(fo)\n",
    "\n",
    "def acquire_vec(word):\n",
    "  if word not in t_enc:\n",
    "    print(\"can't find the word {}\".format(word))\n",
    "    return\n",
    "  return vectors[t_enc[word]].reshape(1, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 6.16281422, -5.97383436,  1.0581208 , -3.26995636,  2.68860027,\n",
       "         0.3521841 , -1.97630152, -1.72122963,  5.17808816, -2.46909246,\n",
       "        -3.2041226 , -1.42568934,  0.12640769, -3.90115325,  0.04068044,\n",
       "         0.65825148,  2.17531384,  0.89453377,  0.93187425, -1.08951865,\n",
       "         0.20583662, -1.21054511, -1.10067284, -0.51607978,  1.46278278,\n",
       "         0.57653813, -0.26861058, -1.9193653 ,  0.07108861, -0.34940509,\n",
       "         0.63357496, -1.8316893 , -0.69995353,  2.17526527, -1.47570393,\n",
       "         0.03694056, -0.68575864,  3.96423008, -0.39579198,  0.50706906,\n",
       "        -0.04697827,  0.8331905 ,  0.48540395,  1.26561296,  0.06427495,\n",
       "        -0.56282367,  0.01594325, -1.00673919, -0.04219863, -2.04557386,\n",
       "         0.33909519, -1.41376507,  0.21739398,  0.17391872,  0.41068706,\n",
       "        -0.45960995,  1.03457755, -0.68804824,  0.85563211, -0.4170566 ,\n",
       "        -0.53620743, -0.18710836, -0.87377082,  1.05999815,  0.11985704,\n",
       "         1.23148534, -0.12812664, -0.52435822, -0.49274043, -0.22113596,\n",
       "        -2.63079274,  0.54692286, -0.26605324, -0.37795514, -0.28622278,\n",
       "        -2.73004861, -0.00758082,  0.49650617, -1.03897727,  0.14824868,\n",
       "        -1.12722099,  0.18071896, -1.28247461, -0.15504494,  0.02218669,\n",
       "         1.32486178,  0.091124  ,  2.1707405 ,  0.49646693,  0.27768483,\n",
       "         0.58407798, -2.39427551,  0.51216437, -0.17460081,  0.8766422 ,\n",
       "         0.36125888,  0.05009546, -0.43281623, -0.57558307, -0.56580531,\n",
       "         0.62271286, -0.47883894, -0.03939406,  0.18496834, -0.19089233,\n",
       "         0.02770457,  0.53442391,  3.29080379, -0.21109905,  0.46813296,\n",
       "         0.44306374,  0.51859976,  1.65891916, -0.06893467,  0.26393286,\n",
       "         0.22816502,  0.2120375 , -0.98155317, -0.48717049,  0.27093103,\n",
       "         1.12544141, -0.91980155, -0.23098429,  1.05585667,  0.07539244,\n",
       "        -0.26802868, -0.63642452, -0.55202181, -0.75239052,  1.65709212,\n",
       "         0.86160863, -1.2379818 ,  1.60632152,  0.55192939, -0.43242478,\n",
       "        -0.63512398, -1.55853673,  0.86843714,  1.6392255 ,  0.58438187,\n",
       "         0.80405545, -0.41647212, -1.15912695,  0.55990363,  0.56309457,\n",
       "        -0.78069368,  0.65895906, -0.54999716,  0.75457651, -1.28780573,\n",
       "        -0.4190393 , -1.49070476, -0.30232635,  1.11136281, -0.70152285,\n",
       "        -0.55257104,  0.27797559, -0.22779946,  0.08856576, -1.1591186 ,\n",
       "         0.78175625, -1.02735119, -0.79776799,  1.32301998, -2.29952512,\n",
       "        -0.70997763,  1.38893859,  1.01934643,  1.5032573 , -0.03372934,\n",
       "         1.85473863, -0.83916781,  0.40579415,  0.36310991, -0.07079876,\n",
       "         0.51356071, -0.26027389,  1.10357433, -0.37270935, -1.95082847,\n",
       "        -1.0515764 ,  1.22053545, -0.43398924,  1.38331699,  1.10511923,\n",
       "         1.13214854, -1.21678402,  0.42766685,  0.63625749, -1.07234138,\n",
       "        -0.59695502, -0.06210605,  1.20446402, -0.25072838,  0.84651785,\n",
       "         0.28048564, -0.87495255, -0.55668042,  2.58074225,  0.14424237,\n",
       "        -1.31041842, -1.36979884, -0.46256799, -0.80132487, -0.92461294,\n",
       "         0.52045685, -0.7133766 , -0.67862934, -0.95401029,  0.50159365,\n",
       "        -1.20967937, -0.63271175, -0.09398536,  1.1006906 ,  2.17423746,\n",
       "        -0.88285546, -0.55559176, -2.09455347, -1.19325878,  0.97271936,\n",
       "         1.28066995, -1.20783356,  0.86537515, -1.1527686 ,  0.46544823,\n",
       "        -0.7311073 ,  1.22544523, -0.0831546 , -0.47100462,  0.1884025 ,\n",
       "        -1.27018239, -0.05077751, -1.69868547, -0.40232736, -1.11195161,\n",
       "        -0.98892501, -0.96493145, -0.18088101, -0.63311176,  0.02717409,\n",
       "         0.68064834,  1.27120814, -0.53884553,  0.67383312, -0.13889679,\n",
       "         0.75416398,  0.07641073,  0.11782267,  0.53371164,  1.88088896,\n",
       "        -0.68147689,  0.59994828,  1.82176389, -0.51412993,  0.4846979 ,\n",
       "         0.67082076, -0.89082438, -0.45457974, -0.69778226, -0.52174863,\n",
       "        -1.18360285,  0.92087551, -0.05188661, -0.31368118, -0.73141809,\n",
       "         0.02277519,  0.30131379, -0.36553005,  0.29928009,  0.31026518,\n",
       "        -0.01358918,  0.49183677,  0.59106347, -0.02449587, -0.38704995,\n",
       "         1.80049565, -0.379928  , -1.36851815, -0.75522296,  0.2536851 ,\n",
       "        -0.07394603,  0.77797522,  0.42428569,  1.55650724, -0.6509015 ,\n",
       "         0.87415246,  0.53449313,  1.20248493, -0.42882066, -0.1517883 ,\n",
       "         0.49205587,  0.46643169, -0.78317774,  0.93302368,  0.90654527,\n",
       "         0.11435541, -0.60400689, -0.29042459, -0.11773197,  0.30974277]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acquire_vec(\"united_states\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.81341673]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What about the cosine similarity between United States and U.S.\n",
    "\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "cosine_similarity(acquire_vec(\"united_states\"), acquire_vec(\"u.s\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FromScratchSimilarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>united_states &amp; u.s</th>\n",
       "      <td>0.813417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>united_states &amp; usa</th>\n",
       "      <td>0.383201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>united_kingdom &amp; britain</th>\n",
       "      <td>0.830735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>netherlands &amp; holland</th>\n",
       "      <td>0.012669</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          FromScratchSimilarity\n",
       "united_states & u.s                    0.813417\n",
       "united_states & usa                    0.383201\n",
       "united_kingdom & britain               0.830735\n",
       "netherlands & holland                  0.012669"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What about some other pairs?\n",
    "\n",
    "comparison_set = [\n",
    "  (\"united_states\", \"u.s\"),\n",
    "  (\"united_states\", \"usa\"),\n",
    "  (\"united_kingdom\", \"britain\"),\n",
    "  (\"netherlands\", \"holland\"),\n",
    "]\n",
    "countries_similarity_scratch = pd.DataFrame(\n",
    "  {\"FromScratchSimilarity\" : [\n",
    "    cosine_similarity(acquire_vec(pair[0]), acquire_vec(pair[1]))[0][0]\n",
    "    for pair in comparison_set\n",
    "  ]},\n",
    "  index=[\" & \".join(pair) for pair in comparison_set]\n",
    ")\n",
    "countries_similarity_scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmm, though United State vs. U.S. seems to generate quite good result. The one with Netherlands does not seems quite right.  \n",
    "Let's try to see the similar top 10 vectors for some contries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def top_n_similar(vectors, t_enc, x, n=10):\n",
    "  if isinstance(x, str):\n",
    "    if x not in t_enc:\n",
    "      print(\"Can't find word {}\".format(x))\n",
    "      return\n",
    "    vec = acquire_vec(x)\n",
    "  else:\n",
    "    vec = x\n",
    "  sim_vect = cosine_similarity(vec, vectors).reshape(-1,)\n",
    "  top_indices = sim_vect.argsort()[-1-n : -1][::-1]\n",
    "  words_list = {v : k for k, v in t_enc.items() if v in set(top_indices)}\n",
    "  return [(words_list[i].encode(\"utf-8\"), sim_vect[i]) for i in top_indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.97 s, sys: 592 ms, total: 2.56 s\n",
      "Wall time: 2.53 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "result = top_n_similar(vectors, t_enc, \"united_kingdom\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FromScratchSimilarCountry</th>\n",
       "      <th>FromScratchSimilarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>netherlands</td>\n",
       "      <td>0.857399</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>italy</td>\n",
       "      <td>0.838267</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>britain</td>\n",
       "      <td>0.830735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>germany</td>\n",
       "      <td>0.804151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>france</td>\n",
       "      <td>0.798871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>télévisions</td>\n",
       "      <td>0.773780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>belgium</td>\n",
       "      <td>0.759755</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>ireland</td>\n",
       "      <td>0.732287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>spain</td>\n",
       "      <td>0.712897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>australia</td>\n",
       "      <td>0.706132</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FromScratchSimilarCountry  FromScratchSimilarity\n",
       "1                netherlands               0.857399\n",
       "2                      italy               0.838267\n",
       "3                    britain               0.830735\n",
       "4                    germany               0.804151\n",
       "5                     france               0.798871\n",
       "6                télévisions               0.773780\n",
       "7                    belgium               0.759755\n",
       "8                    ireland               0.732287\n",
       "9                      spain               0.712897\n",
       "10                 australia               0.706132"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top10_uk_scratch = pd.DataFrame(\n",
    "  result,\n",
    "  index = range(1, 11),\n",
    "  columns=[\"FromScratchSimilarCountry\", \"FromScratchSimilarity\"]\n",
    ")\n",
    "top10_uk_scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Somehow, Netherlands appears to be the top relative country of UK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.88 s, sys: 571 ms, total: 2.45 s\n",
      "Wall time: 2.41 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "result = top_n_similar(vectors, t_enc, \"united_states\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FromScratchSimilarCountry</th>\n",
       "      <th>FromScratchSimilarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>u.s</td>\n",
       "      <td>0.813417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>us</td>\n",
       "      <td>0.541306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>canada</td>\n",
       "      <td>0.445503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>reserve</td>\n",
       "      <td>0.431682</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>bureau</td>\n",
       "      <td>0.425039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>census</td>\n",
       "      <td>0.423169</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>europe</td>\n",
       "      <td>0.422780</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>meteorology</td>\n",
       "      <td>0.415249</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>navy</td>\n",
       "      <td>0.397493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>marine</td>\n",
       "      <td>0.388942</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FromScratchSimilarCountry  FromScratchSimilarity\n",
       "1                        u.s               0.813417\n",
       "2                         us               0.541306\n",
       "3                     canada               0.445503\n",
       "4                    reserve               0.431682\n",
       "5                     bureau               0.425039\n",
       "6                     census               0.423169\n",
       "7                     europe               0.422780\n",
       "8                meteorology               0.415249\n",
       "9                       navy               0.397493\n",
       "10                    marine               0.388942"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top10_us_scratch = pd.DataFrame(\n",
    "  result,\n",
    "  index = range(1, 11),\n",
    "  columns=[\"FromScratchSimilarCountry\", \"FromScratchSimilarity\"]\n",
    ")\n",
    "top10_us_scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is some how weird, south_korea seems to have high similarity to many cities.  \n",
    "We'll compare this to googles word2vec later.  \n",
    "For now, let's try one more thing: analogy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_vec = acquire_vec(\"spain\") - acquire_vec(\"madrid\") + acquire_vec(\"athens\")\n",
    "result = top_n_similar(vectors, t_enc, target_vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FromScratchSimilarCountry</th>\n",
       "      <th>FromScratchSimilarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>italy</td>\n",
       "      <td>0.852897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>sweden</td>\n",
       "      <td>0.847748</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>germany</td>\n",
       "      <td>0.836689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>belgium</td>\n",
       "      <td>0.824082</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>austria</td>\n",
       "      <td>0.819144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>netherlands</td>\n",
       "      <td>0.818202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>france</td>\n",
       "      <td>0.812697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>télévisions</td>\n",
       "      <td>0.789071</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>denmark</td>\n",
       "      <td>0.787519</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>norway</td>\n",
       "      <td>0.746578</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FromScratchSimilarCountry  FromScratchSimilarity\n",
       "1                      italy               0.852897\n",
       "2                     sweden               0.847748\n",
       "3                    germany               0.836689\n",
       "4                    belgium               0.824082\n",
       "5                    austria               0.819144\n",
       "6                netherlands               0.818202\n",
       "7                     france               0.812697\n",
       "8                télévisions               0.789071\n",
       "9                    denmark               0.787519\n",
       "10                    norway               0.746578"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top10_greece_analogy_scratch = pd.DataFrame(\n",
    "  result,\n",
    "  index = range(1, 11),\n",
    "  columns=[\"FromScratchSimilarCountry\", \"FromScratchSimilarity\"]\n",
    ")\n",
    "top10_greece_analogy_scratch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_vec = acquire_vec(\"canada\") - acquire_vec(\"ottawa\") + acquire_vec(\"washington\")\n",
    "result = top_n_similar(vectors, t_enc, target_vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FromScratchSimilarCountry</th>\n",
       "      <th>FromScratchSimilarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>canada</td>\n",
       "      <td>0.745063</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>australia</td>\n",
       "      <td>0.546372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>usa</td>\n",
       "      <td>0.542821</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>united_kingdom</td>\n",
       "      <td>0.484901</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>america</td>\n",
       "      <td>0.470903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ontario</td>\n",
       "      <td>0.470820</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>europe</td>\n",
       "      <td>0.464094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>united_states</td>\n",
       "      <td>0.462584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>d.c</td>\n",
       "      <td>0.461587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>new_zealand</td>\n",
       "      <td>0.459321</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FromScratchSimilarCountry  FromScratchSimilarity\n",
       "1                     canada               0.745063\n",
       "2                  australia               0.546372\n",
       "3                        usa               0.542821\n",
       "4             united_kingdom               0.484901\n",
       "5                    america               0.470903\n",
       "6                    ontario               0.470820\n",
       "7                     europe               0.464094\n",
       "8              united_states               0.462584\n",
       "9                        d.c               0.461587\n",
       "10               new_zealand               0.459321"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top10_us_analogy_scratch = pd.DataFrame(\n",
    "  result,\n",
    "  index = range(1, 11),\n",
    "  columns=[\"FromScratchSimilarCountry\", \"FromScratchSimilarity\"]\n",
    ")\n",
    "top10_us_analogy_scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Using word2vec library\n",
    "\n",
    "Here comes google's word2vec.  We'll use the pacakge from gensim."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4min 9s, sys: 6.77 s, total: 4min 15s\n",
      "Wall time: 1min 43s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from gensim.models import word2vec\n",
    "\n",
    "with open(corpus_file, \"rt\") as fi:\n",
    "  result = []\n",
    "  for line in fi:\n",
    "    result.append(line.decode(\"utf-8\").rstrip(\"\\n\").split(\" \"))  \n",
    "  w2v_model = word2vec.Word2Vec(result, size=300, min_count=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would like to go through the same set of problems here with the from scratch model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FromScratchSimilarity</th>\n",
       "      <th>Word2VecSimilarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>united_states &amp; u.s</th>\n",
       "      <td>0.813417</td>\n",
       "      <td>0.856122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>united_states &amp; usa</th>\n",
       "      <td>0.383201</td>\n",
       "      <td>0.540755</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>united_kingdom &amp; britain</th>\n",
       "      <td>0.830735</td>\n",
       "      <td>0.575966</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>netherlands &amp; holland</th>\n",
       "      <td>0.012669</td>\n",
       "      <td>0.446076</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          FromScratchSimilarity  Word2VecSimilarity\n",
       "united_states & u.s                    0.813417            0.856122\n",
       "united_states & usa                    0.383201            0.540755\n",
       "united_kingdom & britain               0.830735            0.575966\n",
       "netherlands & holland                  0.012669            0.446076"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "countries_similarity_w2v = pd.DataFrame(\n",
    "  {\"Word2VecSimilarity\" : [\n",
    "    w2v_model.wv.similarity(pair[0], pair[1])\n",
    "    for pair in comparison_set\n",
    "  ]},\n",
    "  index=[\" & \".join(pair) for pair in comparison_set]\n",
    ")\n",
    "pd.concat((countries_similarity_scratch, countries_similarity_w2v), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike the from scratch model, Netherlands now seems to be quite similar with Holland, which is what we expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FromScratchSimilarCountry</th>\n",
       "      <th>FromScratchSimilarity</th>\n",
       "      <th>Word2VecSimilarCountry</th>\n",
       "      <th>Word2VecSimilarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>netherlands</td>\n",
       "      <td>0.857399</td>\n",
       "      <td>uk</td>\n",
       "      <td>0.788296</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>italy</td>\n",
       "      <td>0.838267</td>\n",
       "      <td>canada</td>\n",
       "      <td>0.785179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>britain</td>\n",
       "      <td>0.830735</td>\n",
       "      <td>netherlands</td>\n",
       "      <td>0.715695</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>germany</td>\n",
       "      <td>0.804151</td>\n",
       "      <td>united_states</td>\n",
       "      <td>0.700951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>france</td>\n",
       "      <td>0.798871</td>\n",
       "      <td>new_zealand</td>\n",
       "      <td>0.699565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>télévisions</td>\n",
       "      <td>0.773780</td>\n",
       "      <td>sweden</td>\n",
       "      <td>0.690162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>belgium</td>\n",
       "      <td>0.759755</td>\n",
       "      <td>australia</td>\n",
       "      <td>0.670177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>ireland</td>\n",
       "      <td>0.732287</td>\n",
       "      <td>europe</td>\n",
       "      <td>0.655038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>spain</td>\n",
       "      <td>0.712897</td>\n",
       "      <td>hong_kong</td>\n",
       "      <td>0.651373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>australia</td>\n",
       "      <td>0.706132</td>\n",
       "      <td>philippines</td>\n",
       "      <td>0.651057</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FromScratchSimilarCountry  FromScratchSimilarity Word2VecSimilarCountry  \\\n",
       "1                netherlands               0.857399                     uk   \n",
       "2                      italy               0.838267                 canada   \n",
       "3                    britain               0.830735            netherlands   \n",
       "4                    germany               0.804151          united_states   \n",
       "5                     france               0.798871            new_zealand   \n",
       "6                télévisions               0.773780                 sweden   \n",
       "7                    belgium               0.759755              australia   \n",
       "8                    ireland               0.732287                 europe   \n",
       "9                      spain               0.712897              hong_kong   \n",
       "10                 australia               0.706132            philippines   \n",
       "\n",
       "    Word2VecSimilarity  \n",
       "1             0.788296  \n",
       "2             0.785179  \n",
       "3             0.715695  \n",
       "4             0.700951  \n",
       "5             0.699565  \n",
       "6             0.690162  \n",
       "7             0.670177  \n",
       "8             0.655038  \n",
       "9             0.651373  \n",
       "10            0.651057  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top10_uk_w2v = pd.DataFrame(\n",
    "  w2v_model.wv.most_similar(positive=[\"united_kingdom\"], topn=10),\n",
    "  index = range(1, 11),\n",
    "  columns=[\"Word2VecSimilarCountry\", \"Word2VecSimilarity\"]\n",
    ")\n",
    "pd.concat((top10_uk_scratch, top10_uk_w2v), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FromScratchSimilarCountry</th>\n",
       "      <th>FromScratchSimilarity</th>\n",
       "      <th>Word2VecSimilarCountry</th>\n",
       "      <th>Word2VecSimilarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>u.s</td>\n",
       "      <td>0.813417</td>\n",
       "      <td>u.s</td>\n",
       "      <td>0.856122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>us</td>\n",
       "      <td>0.541306</td>\n",
       "      <td>united_kingdom</td>\n",
       "      <td>0.700951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>canada</td>\n",
       "      <td>0.445503</td>\n",
       "      <td>canada</td>\n",
       "      <td>0.667973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>reserve</td>\n",
       "      <td>0.431682</td>\n",
       "      <td>philippines</td>\n",
       "      <td>0.618144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>bureau</td>\n",
       "      <td>0.425039</td>\n",
       "      <td>uk</td>\n",
       "      <td>0.608410</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>census</td>\n",
       "      <td>0.423169</td>\n",
       "      <td>us</td>\n",
       "      <td>0.605478</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>europe</td>\n",
       "      <td>0.422780</td>\n",
       "      <td>europe</td>\n",
       "      <td>0.595408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>meteorology</td>\n",
       "      <td>0.415249</td>\n",
       "      <td>new_zealand</td>\n",
       "      <td>0.588238</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>navy</td>\n",
       "      <td>0.397493</td>\n",
       "      <td>australia</td>\n",
       "      <td>0.581822</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>marine</td>\n",
       "      <td>0.388942</td>\n",
       "      <td>taiwan</td>\n",
       "      <td>0.579102</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FromScratchSimilarCountry  FromScratchSimilarity Word2VecSimilarCountry  \\\n",
       "1                        u.s               0.813417                    u.s   \n",
       "2                         us               0.541306         united_kingdom   \n",
       "3                     canada               0.445503                 canada   \n",
       "4                    reserve               0.431682            philippines   \n",
       "5                     bureau               0.425039                     uk   \n",
       "6                     census               0.423169                     us   \n",
       "7                     europe               0.422780                 europe   \n",
       "8                meteorology               0.415249            new_zealand   \n",
       "9                       navy               0.397493              australia   \n",
       "10                    marine               0.388942                 taiwan   \n",
       "\n",
       "    Word2VecSimilarity  \n",
       "1             0.856122  \n",
       "2             0.700951  \n",
       "3             0.667973  \n",
       "4             0.618144  \n",
       "5             0.608410  \n",
       "6             0.605478  \n",
       "7             0.595408  \n",
       "8             0.588238  \n",
       "9             0.581822  \n",
       "10            0.579102  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top10_us_w2v = pd.DataFrame(\n",
    "  w2v_model.wv.most_similar(positive=[\"united_states\"], topn=10),\n",
    "  index = range(1, 11),\n",
    "  columns=[\"Word2VecSimilarCountry\", \"Word2VecSimilarity\"]\n",
    ")\n",
    "pd.concat((top10_us_scratch, top10_us_w2v), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far both models seems to general nice results. Let's see what they are going to propose with Netherlands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FromScratchSimilarCountry</th>\n",
       "      <th>FromScratchSimilarity</th>\n",
       "      <th>Word2VecSimilarCountry</th>\n",
       "      <th>Word2VecSimilarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>italy</td>\n",
       "      <td>0.920498</td>\n",
       "      <td>belgium</td>\n",
       "      <td>0.872092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>belgium</td>\n",
       "      <td>0.915938</td>\n",
       "      <td>italy</td>\n",
       "      <td>0.837967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>télévisions</td>\n",
       "      <td>0.888087</td>\n",
       "      <td>sweden</td>\n",
       "      <td>0.828278</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>germany</td>\n",
       "      <td>0.869561</td>\n",
       "      <td>norway</td>\n",
       "      <td>0.812825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>france</td>\n",
       "      <td>0.862236</td>\n",
       "      <td>finland</td>\n",
       "      <td>0.808494</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>spain</td>\n",
       "      <td>0.858098</td>\n",
       "      <td>spain</td>\n",
       "      <td>0.790100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>united_kingdom</td>\n",
       "      <td>0.857399</td>\n",
       "      <td>austria</td>\n",
       "      <td>0.788743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>austria</td>\n",
       "      <td>0.839080</td>\n",
       "      <td>chile</td>\n",
       "      <td>0.782123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>sweden</td>\n",
       "      <td>0.829271</td>\n",
       "      <td>brazil</td>\n",
       "      <td>0.771756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>denmark</td>\n",
       "      <td>0.811705</td>\n",
       "      <td>denmark</td>\n",
       "      <td>0.767548</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FromScratchSimilarCountry  FromScratchSimilarity Word2VecSimilarCountry  \\\n",
       "1                      italy               0.920498                belgium   \n",
       "2                    belgium               0.915938                  italy   \n",
       "3                télévisions               0.888087                 sweden   \n",
       "4                    germany               0.869561                 norway   \n",
       "5                     france               0.862236                finland   \n",
       "6                      spain               0.858098                  spain   \n",
       "7             united_kingdom               0.857399                austria   \n",
       "8                    austria               0.839080                  chile   \n",
       "9                     sweden               0.829271                 brazil   \n",
       "10                   denmark               0.811705                denmark   \n",
       "\n",
       "    Word2VecSimilarity  \n",
       "1             0.872092  \n",
       "2             0.837967  \n",
       "3             0.828278  \n",
       "4             0.812825  \n",
       "5             0.808494  \n",
       "6             0.790100  \n",
       "7             0.788743  \n",
       "8             0.782123  \n",
       "9             0.771756  \n",
       "10            0.767548  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top10_nl_scratch = pd.DataFrame(\n",
    "  top_n_similar(vectors, t_enc, \"netherlands\"),\n",
    "  index = range(1, 11),\n",
    "  columns=[\"FromScratchSimilarCountry\", \"FromScratchSimilarity\"]\n",
    ")\n",
    "top10_nl_w2v = pd.DataFrame(\n",
    "  w2v_model.wv.most_similar(positive=[\"netherlands\"], topn=10),\n",
    "  index = range(1, 11),\n",
    "  columns=[\"Word2VecSimilarCountry\", \"Word2VecSimilarity\"]\n",
    ")\n",
    "pd.concat((top10_nl_scratch, top10_nl_w2v), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sadly, though both models were able to other European countries that are close, neither have Holland in the top 10. Perhaps the two words Netherlands and Holland are just not close enough or to frequency enough in our corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FromScratchSimilarCountry</th>\n",
       "      <th>FromScratchSimilarity</th>\n",
       "      <th>Word2VecSimilarCountry</th>\n",
       "      <th>Word2VecSimilarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>italy</td>\n",
       "      <td>0.852897</td>\n",
       "      <td>greece</td>\n",
       "      <td>0.796760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>sweden</td>\n",
       "      <td>0.847748</td>\n",
       "      <td>egypt</td>\n",
       "      <td>0.762369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>germany</td>\n",
       "      <td>0.836689</td>\n",
       "      <td>italy</td>\n",
       "      <td>0.737770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>belgium</td>\n",
       "      <td>0.824082</td>\n",
       "      <td>portugal</td>\n",
       "      <td>0.730268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>austria</td>\n",
       "      <td>0.819144</td>\n",
       "      <td>russia</td>\n",
       "      <td>0.729629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>netherlands</td>\n",
       "      <td>0.818202</td>\n",
       "      <td>austria</td>\n",
       "      <td>0.720083</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>france</td>\n",
       "      <td>0.812697</td>\n",
       "      <td>syria</td>\n",
       "      <td>0.714533</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>télévisions</td>\n",
       "      <td>0.789071</td>\n",
       "      <td>germany</td>\n",
       "      <td>0.713075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>denmark</td>\n",
       "      <td>0.787519</td>\n",
       "      <td>norway</td>\n",
       "      <td>0.710760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>norway</td>\n",
       "      <td>0.746578</td>\n",
       "      <td>hungary</td>\n",
       "      <td>0.703329</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FromScratchSimilarCountry  FromScratchSimilarity Word2VecSimilarCountry  \\\n",
       "1                      italy               0.852897                 greece   \n",
       "2                     sweden               0.847748                  egypt   \n",
       "3                    germany               0.836689                  italy   \n",
       "4                    belgium               0.824082               portugal   \n",
       "5                    austria               0.819144                 russia   \n",
       "6                netherlands               0.818202                austria   \n",
       "7                     france               0.812697                  syria   \n",
       "8                télévisions               0.789071                germany   \n",
       "9                    denmark               0.787519                 norway   \n",
       "10                    norway               0.746578                hungary   \n",
       "\n",
       "    Word2VecSimilarity  \n",
       "1             0.796760  \n",
       "2             0.762369  \n",
       "3             0.737770  \n",
       "4             0.730268  \n",
       "5             0.729629  \n",
       "6             0.720083  \n",
       "7             0.714533  \n",
       "8             0.713075  \n",
       "9             0.710760  \n",
       "10            0.703329  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top10_greece_analogy_w2v = pd.DataFrame(\n",
    "  w2v_model.wv.most_similar(positive=[\"spain\", \"athens\"], negative=[\"madrid\"], topn=10),\n",
    "  index = range(1, 11),\n",
    "  columns=[\"Word2VecSimilarCountry\", \"Word2VecSimilarity\"]\n",
    ")\n",
    "pd.concat((top10_greece_analogy_scratch, top10_greece_analogy_w2v), axis=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>FromScratchSimilarCountry</th>\n",
       "      <th>FromScratchSimilarity</th>\n",
       "      <th>Word2VecSimilarCountry</th>\n",
       "      <th>Word2VecSimilarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>canada</td>\n",
       "      <td>0.745063</td>\n",
       "      <td>united_states</td>\n",
       "      <td>0.630832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>australia</td>\n",
       "      <td>0.546372</td>\n",
       "      <td>u.s</td>\n",
       "      <td>0.570035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>usa</td>\n",
       "      <td>0.542821</td>\n",
       "      <td>australia</td>\n",
       "      <td>0.542224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>united_kingdom</td>\n",
       "      <td>0.484901</td>\n",
       "      <td>china</td>\n",
       "      <td>0.532506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>america</td>\n",
       "      <td>0.470903</td>\n",
       "      <td>taiwan</td>\n",
       "      <td>0.507008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ontario</td>\n",
       "      <td>0.470820</td>\n",
       "      <td>america</td>\n",
       "      <td>0.504899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>europe</td>\n",
       "      <td>0.464094</td>\n",
       "      <td>us</td>\n",
       "      <td>0.503777</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>united_states</td>\n",
       "      <td>0.462584</td>\n",
       "      <td>europe</td>\n",
       "      <td>0.498085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>d.c</td>\n",
       "      <td>0.461587</td>\n",
       "      <td>united_kingdom</td>\n",
       "      <td>0.494034</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>new_zealand</td>\n",
       "      <td>0.459321</td>\n",
       "      <td>north_korea</td>\n",
       "      <td>0.466008</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   FromScratchSimilarCountry  FromScratchSimilarity Word2VecSimilarCountry  \\\n",
       "1                     canada               0.745063          united_states   \n",
       "2                  australia               0.546372                    u.s   \n",
       "3                        usa               0.542821              australia   \n",
       "4             united_kingdom               0.484901                  china   \n",
       "5                    america               0.470903                 taiwan   \n",
       "6                    ontario               0.470820                america   \n",
       "7                     europe               0.464094                     us   \n",
       "8              united_states               0.462584                 europe   \n",
       "9                        d.c               0.461587         united_kingdom   \n",
       "10               new_zealand               0.459321            north_korea   \n",
       "\n",
       "    Word2VecSimilarity  \n",
       "1             0.630832  \n",
       "2             0.570035  \n",
       "3             0.542224  \n",
       "4             0.532506  \n",
       "5             0.507008  \n",
       "6             0.504899  \n",
       "7             0.503777  \n",
       "8             0.498085  \n",
       "9             0.494034  \n",
       "10            0.466008  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top10_us_analogy_w2v = pd.DataFrame(\n",
    "  w2v_model.wv.most_similar(positive=['canada', 'washington'], negative=['ottawa'], topn=10),\n",
    "  index = range(1, 11),\n",
    "  columns=[\"Word2VecSimilarCountry\", \"Word2VecSimilarity\"]\n",
    ")\n",
    "pd.concat((top10_us_analogy_scratch, top10_us_analogy_w2v), axis=1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "Here we have built a words vector set from scratch and compared it with google's word2vec model. Although both works fine with country similarity and analogy, word2vec obvious has so much shorter model buiding time compared to the from the basic from-scratch one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}