Wine2Vec.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wine2Vec Exploration\n",
    "##### By Zack Thoutt\n",
    "\n",
    "Here is a little data exploration of my new wine review dataset using word2vec. My theory is that the words a sommelier would use to describe a wine (oaky, tannic, acidic, berry, etc.) can be used to predict the type of wine (Pinot Noir, Cabernet Sav., etc.). Let's see if we can extract some interesting relationships from the data and somewhat validate this theory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "import numpy as np\n",
    "import nltk\n",
    "import re\n",
    "import sklearn.manifold\n",
    "import multiprocessing\n",
    "import pandas as pd\n",
    "import gensim.models.word2vec as w2v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### Get the Data\n",
    "The dataset can be found on [Kaggle](https://www.kaggle.com/zynicide/wine-reviews) or you can run my sraper on [Github](https://github.com/zackthoutt/wine-deep-learning)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = pd.read_json('winemag-data_first150k.json', dtype={\n",
    "    'points': np.int32,\n",
    "    'price': np.float32,\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "labels = data['variety']\n",
    "descriptions = data['description']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore the Data\n",
    "There are several hundred fairly common varietals of wine and probably thousands of other niche varietals. It will be difficult to be able to identify them all, but I hypothesize that it should be possible to classify the most common, say, 50 or 100 wine varietals with this wine review dataset. \n",
    "\n",
    "Let's take a look at a few reviews and see if we as humans can tell a difference in the descriptive words used for different types of wine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cabernet Sauvignon   :   This tremendous 100% varietal wine hails from Oakville and was aged over three years in oak. Juicy red-cherry fruit and a compelling hint of caramel greet the palate, framed by elegant, fine tannins and a subtle minty tone in the background. Balanced and rewarding from start to finish, it has years ahead of it to develop further nuance. Enjoy 2022–2030.\n",
      "Sauvignon Blanc   :   Delicious while also young and textured, this wine comes from biodynamically grown grapes. It has a strong sense of minerality as well as intense citrus and green fruits. It's tight at the moment and needs to round out, so drink from 2018.\n",
      "Chardonnay   :   A smoky scent and earthy, crisp-apple flavors make this medium-bodied wine a change of pace from the average butterball Chardonnay. It has welcome acidity and a nicely smooth texture.\n"
     ]
    }
   ],
   "source": [
    "print('{}   :   {}'.format(labels.tolist()[0], descriptions.tolist()[0]))\n",
    "print('{}   :   {}'.format(labels.tolist()[56], descriptions.tolist()[56]))\n",
    "print('{}   :   {}'.format(labels.tolist()[93], descriptions.tolist()[93]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even if you're not someone who knows wine, I think that there is a pretty clear distinction in the descriptions of these different types of wines. The Cabernet Sauvignon (a red wine) was described with words like cherry, tannin and carmel. The next two reviews are white wines, but even they show differences in their description. The sauvignon blanc is described as minerally, citrus, and green fruits while the chardonnay is described as smokey, earthy, crisp-apple, and buttery. This provides us with good motivation to move forward and explore the data more.\n",
    "\n",
    "One of the limitations that I think we will have with this dataset is that there will be a lot more reviews for popular wine varietals than less popular wine varietals. This isn't bad neccissarily, but it means that we will probably only be able to classify the most popular N varietals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chardonnay                       14482\n",
      "Pinot Noir                       14291\n",
      "Cabernet Sauvignon               12800\n",
      "Red Blend                        10062\n",
      "Bordeaux-style Red Blend          7347\n",
      "Sauvignon Blanc                   6320\n",
      "Syrah                             5825\n",
      "Riesling                          5524\n",
      "Merlot                            5070\n",
      "Zinfandel                         3799\n",
      "Sangiovese                        3345\n",
      "Malbec                            3208\n",
      "White Blend                       2824\n",
      "Rosé                              2817\n",
      "Tempranillo                       2556\n",
      "Nebbiolo                          2241\n",
      "Portuguese Red                    2216\n",
      "Sparkling Blend                   2004\n",
      "Shiraz                            1970\n",
      "Corvina, Rondinella, Molinara     1682\n",
      "Rhône-style Red Blend             1505\n",
      "Barbera                           1365\n",
      "Pinot Gris                        1365\n",
      "Cabernet Franc                    1363\n",
      "Sangiovese Grosso                 1346\n",
      "Pinot Grigio                      1305\n",
      "Viognier                          1263\n",
      "Bordeaux-style White Blend        1261\n",
      "Champagne Blend                   1238\n",
      "Port                              1058\n",
      "Grüner Veltliner                  1042\n",
      "Gewürztraminer                     982\n",
      "Portuguese White                   941\n",
      "Petite Sirah                       897\n",
      "Carmenère                          761\n",
      "Tempranillo Blend                  756\n",
      "Chenin Blanc                       746\n",
      "Glera                              622\n",
      "Grenache                           603\n",
      "Prosecco                           594\n",
      "Albariño                           537\n",
      "Gamay                              535\n",
      "Pinot Blanc                        470\n",
      "Nero d'Avola                       458\n",
      "Moscato                            430\n",
      "Rhône-style White Blend            409\n",
      "Garganega                          402\n",
      "Garnacha                           401\n",
      "Torrontés                          367\n",
      "Dolcetto                           358\n",
      "Name: variety, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "varietal_counts = labels.value_counts()\n",
    "print(varietal_counts[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you drink wine regularly you will probably recognize the most reviewed wines listed above. The value counts for different wine varietals does verify my theory that less popular wines might not have enough reviews to classify them. The most popular wine varietals have thousands of reviews, but even towards the bottom end of the top 50 wine varietals there are only a few hundred reviews. This isn't a problem for building a word2vec model like we are going to do next, but it is something to keep in mind as we move forward trying to create a wine classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word2Vec Model\n",
    "##### Formatting the Data\n",
    "In order to train a word2vec model, all of the description data will need to be concatenated into one giant string. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_raw = \"\"\n",
    "for description in descriptions:\n",
    "    corpus_raw += description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to tokenize the wine corpus using NLTK. This process will essentially break the word corpus into an array of sentences and then break each sentence into an array of words stripping out less usefull characters like commas and hyphens in the process. In this way, we are able to train the word2vec model with the context of sentences and relative word placement. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "raw_sentences = tokenizer.tokenize(corpus_raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sentence_to_wordlist(raw):\n",
    "    clean = re.sub(\"[^a-zA-Z]\",\" \", raw)\n",
    "    words = clean.split()\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = []\n",
    "for raw_sentence in raw_sentences:\n",
    "    if len(raw_sentence) > 0:\n",
    "        sentences.append(sentence_to_wordlist(raw_sentence))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tart cherry lingers on the finish.A deeper salmon color with elegantly lacy bubbles and a slight cloudy appearance, this sparkler by Norm Yost offers dessicated watermelon, dried orange blossoms, yeast, citrus rinds and fresher strawberry notes on the nose.\n",
      "['Tart', 'cherry', 'lingers', 'on', 'the', 'finish', 'A', 'deeper', 'salmon', 'color', 'with', 'elegantly', 'lacy', 'bubbles', 'and', 'a', 'slight', 'cloudy', 'appearance', 'this', 'sparkler', 'by', 'Norm', 'Yost', 'offers', 'dessicated', 'watermelon', 'dried', 'orange', 'blossoms', 'yeast', 'citrus', 'rinds', 'and', 'fresher', 'strawberry', 'notes', 'on', 'the', 'nose']\n"
     ]
    }
   ],
   "source": [
    "print(raw_sentences[234])\n",
    "print(sentence_to_wordlist(raw_sentences[234]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The wine corpus contains 7,077,125 tokens\n"
     ]
    }
   ],
   "source": [
    "token_count = sum([len(sentence) for sentence in sentences])\n",
    "print('The wine corpus contains {0:,} tokens'.format(token_count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For some context, all of the GOT books combined make up only ~1,800,000 tokens, so this dataset is nearly 4x as large as the GOT book series.\n",
    "\n",
    "##### Training the Model\n",
    "It took some experimenting to get the model to train well. The main things hyperparameters that I had to tune were `min_word_count` and `context_size`. \n",
    "\n",
    "I usually train word2vec models with a `min_word_count` closer to 3-5, but since this dataset is so large I had to bump it up to 10. When I was training the model on a smaller `min_word_count` I was getting a lot of winery and vinyard noise in my word similarities (ie the words most similar to \"cherry\" were a bunch of foreign vinyards, wineries, regions, etc.). After looking through some of the descriptions I came to the conclusion that most of the wine descriptions don't mention the wine varietal, vinyard, or winery, but some do. So I played with the `min_word_count` until those rare instances had less of an effect on the model.\n",
    "\n",
    "I also had to play with the `context_size` quite a bit. 10 is a pretty large context size, but it makes sense here because really all of the words in a sentence are related to each other in the context of wine descriptions and what were are trying to accomplish. I might even experiment with bumping the `context_size` up higher at some point, but even now most of the words in each sentence will be associated with each other in the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_features = 300\n",
    "min_word_count = 10\n",
    "num_workers = multiprocessing.cpu_count()\n",
    "context_size = 10\n",
    "downsampling = 1e-3\n",
    "seed=1993"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine2vec = w2v.Word2Vec(\n",
    "    sg=1,\n",
    "    seed=seed,\n",
    "    workers=num_workers,\n",
    "    size=num_features,\n",
    "    min_count=min_word_count,\n",
    "    window=context_size,\n",
    "    sample=downsampling\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "wine2vec.build_vocab(sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word2Vec vocabulary length: 11979\n"
     ]
    }
   ],
   "source": [
    "print('Word2Vec vocabulary length:', len(wine2vec.wv.vocab))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "292314\n"
     ]
    }
   ],
   "source": [
    "print(wine2vec.corpus_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26586210"
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wine2vec.train(sentences, total_examples=wine2vec.corpus_count, epochs=wine2vec.iter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Playing with the Model\n",
    "Now that we have a trained model we can get to the fun part and start playing around with the results. As you can tell from the outputs below, there is definitely still some noise in the data that could be worked out by tuning the parameters further, but overall we are getting pretty good results.\n",
    "\n",
    "##### Words closest to a given word\n",
    "\"melon,\" \"berry,\" and \"oak\" are words that someone might use to describe the taste/smell of a wine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('nectarine', 0.7073046565055847),\n",
       " ('peach', 0.6438678503036499),\n",
       " ('honeydew', 0.6428186893463135),\n",
       " ('papaya', 0.6423326730728149),\n",
       " ('cantaloupe', 0.6362674236297607),\n",
       " ('guava', 0.6074447631835938),\n",
       " ('pear', 0.6029884815216064),\n",
       " ('canteloupe', 0.6017950773239136),\n",
       " ('clementine', 0.6005336046218872),\n",
       " ('passion', 0.5996620059013367)]"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wine2vec.most_similar('melon')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('berries', 0.6290175914764404),\n",
       " ('blackberry', 0.6250549554824829),\n",
       " ('raspberry', 0.559834361076355),\n",
       " ('black', 0.5525908470153809),\n",
       " ('incorporate', 0.5403679013252258),\n",
       " ('blueberry', 0.5378624796867371),\n",
       " ('loganberry', 0.530583381652832),\n",
       " ('huckleberry', 0.5194034576416016),\n",
       " ('manly', 0.5146172046661377),\n",
       " ('Blackberry', 0.5041558742523193)]"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wine2vec.most_similar('berry')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('vanillins', 0.5344773530960083),\n",
       " ('woodsap', 0.5198507905006409),\n",
       " ('charry', 0.47783029079437256),\n",
       " ('cloaked', 0.4773528575897217),\n",
       " ('elaborated', 0.4762468934059143),\n",
       " ('regime', 0.4708051383495331),\n",
       " ('jacket', 0.46834731101989746),\n",
       " ('oaky', 0.4653569459915161),\n",
       " ('application', 0.4612783193588257),\n",
       " ('puncheons', 0.45585477352142334)]"
      ]
     },
     "execution_count": 197,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wine2vec.most_similar('oak')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another thing that someone might use to describe a wine is how acidic it is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('raspingly', 0.5183893442153931),\n",
       " ('tartly', 0.4958021938800812),\n",
       " ('acidically', 0.49269402027130127),\n",
       " ('sheering', 0.49023425579071045),\n",
       " ('unforgiving', 0.48199647665023804),\n",
       " ('angular', 0.4760863184928894),\n",
       " ('pinching', 0.47394663095474243),\n",
       " ('pointy', 0.47168493270874023),\n",
       " ('ultracrisp', 0.46784496307373047),\n",
       " ('frisky', 0.4669594466686249)]"
      ]
     },
     "execution_count": 199,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wine2vec.most_similar('acidic')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or what the body is like. \"full-bodied\" would be something that is thick like whole milk while \"light-bodied\" would be something that is thin like skim milk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Full', 0.584251880645752),\n",
       " ('creamily', 0.4710482656955719),\n",
       " ('voluminous', 0.4688924551010132),\n",
       " ('fulfilling', 0.46560823917388916),\n",
       " ('bodied', 0.46444156765937805),\n",
       " ('expectedly', 0.4592348337173462),\n",
       " ('fullish', 0.4530738592147827),\n",
       " ('mouthfilling', 0.44827550649642944),\n",
       " ('explosively', 0.443299263715744),\n",
       " ('robustly', 0.4367332458496094)]"
      ]
     },
     "execution_count": 206,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wine2vec.most_similar('full')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you can also feel in your mouth how much tannin a wine has. Wines with lots of tannis give you a dry, furry feeling on your tounge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Tannins', 0.6397914290428162),\n",
       " ('tannin', 0.5773420333862305),\n",
       " ('resistance', 0.5176475644111633),\n",
       " ('pliant', 0.5133403539657593),\n",
       " ('Wrapped', 0.5123817920684814),\n",
       " ('furry', 0.5082734823226929),\n",
       " ('tannic', 0.5056681632995605),\n",
       " ('unobtrusive', 0.49775996804237366),\n",
       " ('negotiable', 0.4963838756084442),\n",
       " ('vise', 0.48365217447280884)]"
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wine2vec.most_similar('tannins')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Linear relationships between word pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def nearest_similarity_cosmul(start1, end1, end2):\n",
    "    similarities = wine2vec.most_similar_cosmul(\n",
    "        positive=[end2, start1],\n",
    "        negative=[end1]\n",
    "    )\n",
    "    start2 = similarities[0][0]\n",
    "    print(\"{start1} is related to {end1}, as {start2} is related to {end2}\".format(**locals()))\n",
    "    return start2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "oak is related to vanilla, as cedar is related to cherry\n"
     ]
    }
   ],
   "source": [
    "nearest_similarity_cosmul('oak', 'vanilla', 'cherry');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "full is related to berry, as Unoaked is related to light\n"
     ]
    }
   ],
   "source": [
    "nearest_similarity_cosmul('full', 'berry', 'light');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tannins is related to plum, as refreshing is related to fresh\n"
     ]
    }
   ],
   "source": [
    "nearest_similarity_cosmul('tannins', 'plum', 'fresh');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "full is related to bodied, as pinching is related to acidic\n"
     ]
    }
   ],
   "source": [
    "nearest_similarity_cosmul('full', 'bodied', 'acidic');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "I think that exploring this wine2vec model has helped validate the theory that there is a lot of useful data in these wine descriptions that can probably be used to classify wine varietals. I have not yet trained any classifiers, but we saw early on that descriptions of different wines used different words to describe the wine varietals, and based on our wine2vec model there is definitley enough context to link these descriptive words together and come up with something to classify them when they are used in certain combinations.\n",
    "\n",
    "That's all I have for now. As always, let me know if anyone has any questions, comments, insights, ideas, etc. I'll be posting more of my analyses and models soon!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}