Deep Learning with PyTorch/DL-with-pytorch - 6 [CNN].ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DL-with-pytorch - 6 [CNN].ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"metadata":{"id":"bIAYbYajk1w9","colab_type":"text"},"cell_type":"markdown","source":["# Convolutional Neural Networks\n","- IMDB review sentiment classification with CNN\n","  - Up to last time, we have tried to classify images in the CIFAR-10 dataset with CNN. However, CNNs are not only useful in classifying and recognizing images, but also processing data with temporal dependencies, such as text data.\n","  - Here, let's try classifying movie review data with CNN"]},{"metadata":{"id":"GVU5-yp3N89I","colab_type":"code","outputId":"9e427acb-6e7c-46f8-ccbc-61a5b071edd2","executionInfo":{"status":"ok","timestamp":1545932184206,"user_tz":420,"elapsed":66823,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":336}},"cell_type":"code","source":["!pip3 install torch torchvision"],"execution_count":1,"outputs":[{"output_type":"stream","text":["Collecting torch\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n","\u001b[K    100% |████████████████████████████████| 591.8MB 28kB/s \n","tcmalloc: large alloc 1073750016 bytes == 0x61d2e000 @  0x7f29476992a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n","\u001b[?25hCollecting torchvision\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/ca/0d/f00b2885711e08bd71242ebe7b96561e6f6d01fdb4b9dcf4d37e2e13c5e1/torchvision-0.2.1-py2.py3-none-any.whl (54kB)\n","\u001b[K    100% |████████████████████████████████| 61kB 18.0MB/s \n","\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n","Collecting pillow>=4.1.1 (from torchvision)\n","\u001b[?25l  Downloading https://files.pythonhosted.org/packages/62/94/5430ebaa83f91cc7a9f687ff5238e26164a779cca2ef9903232268b0a318/Pillow-5.3.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n","\u001b[K    100% |████████████████████████████████| 2.0MB 4.5MB/s \n","\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n","Installing collected packages: torch, pillow, torchvision\n","  Found existing installation: Pillow 4.0.0\n","    Uninstalling Pillow-4.0.0:\n","      Successfully uninstalled Pillow-4.0.0\n","Successfully installed pillow-5.3.0 torch-1.0.0 torchvision-0.2.1\n"],"name":"stdout"}]},{"metadata":{"id":"8yy37hEYOEiQ","colab_type":"code","outputId":"d695e86a-a98a-4951-d9cf-185ef8130dd9","executionInfo":{"status":"ok","timestamp":1545933208234,"user_tz":420,"elapsed":1304,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["import numpy as np\n","import pandas as pd\n","import torch, torchvision\n","import torch.nn as nn\n","import torch.nn.functional as F\n","torch.__version__"],"execution_count":1,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'1.0.0'"]},"metadata":{"tags":[]},"execution_count":1}]},{"metadata":{"id":"ewrw93tt2BfV","colab_type":"text"},"cell_type":"markdown","source":["## 1. Import & process dataset\n","- IMDB review dataset for sentiment analysis\n","  - [source](http://ai.stanford.edu/~amaas/data/sentiment/)\n","  - Let's cheat a while and use dataset provided by Keras"]},{"metadata":{"id":"SxDHXFEsf5em","colab_type":"code","outputId":"6995c6a3-d576-4267-840a-f5362e03ef28","executionInfo":{"status":"ok","timestamp":1545933216936,"user_tz":420,"elapsed":8829,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":52}},"cell_type":"code","source":["from keras.datasets import imdb\n","from keras.preprocessing import sequence\n","\n","num_words = 10000\n","maxlen = 50\n","\n","(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words = num_words)\n","\n","X_train = sequence.pad_sequences(X_train, maxlen = maxlen, padding = 'pre')\n","X_test = sequence.pad_sequences(X_test, maxlen = maxlen, padding = 'pre')\n","    \n","print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"],"execution_count":2,"outputs":[{"output_type":"stream","text":["Using TensorFlow backend.\n"],"name":"stderr"},{"output_type":"stream","text":["(25000, 50) (25000, 50) (25000,) (25000,)\n"],"name":"stdout"}]},{"metadata":{"id":"9nznVMdo5edZ","colab_type":"text"},"cell_type":"markdown","source":["## 2. Creating CNN model and training\n","\n","- Create and train CNN model for sentence classification, with one convolutional & average pooling layer\n","- Model architecture is adopted from [Kim 2015](https://www.aclweb.org/anthology/D14-1181)\n","\n","![](https://3qeqpr26caki16dnhd19sv6by6v-wpengine.netdna-ssl.com/wp-content/uploads/2017/08/Example-of-a-CNN-Filter-and-Polling-Architecture-for-Natural-Language-Processing.png)"]},{"metadata":{"id":"tdmsy2B3weeK","colab_type":"code","colab":{}},"cell_type":"code","source":["class imdbTrainDataset(torch.utils.data.Dataset):\n","  def __init__(self):\n","    self.X = X_train\n","    self.y = y_train\n","  \n","  def __getitem__(self, idx):\n","    return self.X[idx], self.y[idx]\n","  \n","  def __len__(self):\n","    return len(self.X)\n","  \n","class imdbTestDataset(torch.utils.data.Dataset):\n","  def __init__(self):\n","    self.X = X_test\n","    self.y = y_test\n","  \n","  def __getitem__(self, idx):\n","    return self.X[idx], self.y[idx]\n","  \n","  def __len__(self):\n","    return len(self.X)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"xFBmmDxKw1t2","colab_type":"code","colab":{}},"cell_type":"code","source":["# create dataset & dataloader instances\n","train_dataset = imdbTrainDataset()\n","test_dataset = imdbTestDataset()\n","\n","train_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True)\n","test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 128, shuffle = False)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"AQawpMRPI7jm","colab_type":"code","colab":{}},"cell_type":"code","source":["# create CNN with one convolution/pooling layer\n","class net(nn.Module):\n","  def __init__(self, input_dim, num_words, embedding_dim, num_filters, kernel_size, stride):\n","    super(net, self).__init__()\n","    self.input_dim = input_dim\n","    self.embedding_dim = embedding_dim\n","    \n","    conv_output_size = int((input_dim - kernel_size)/stride) + 1   # first conv layer output size\n","        \n","    self.embedding = nn.Embedding(num_words, self.embedding_dim)\n","    self.conv = nn.Conv2d(1, num_filters, kernel_size = (kernel_size, self.embedding_dim), stride = stride)     \n","    self.pool = nn.MaxPool2d((conv_output_size, 1))                # Max-over-time pooling (FYI: avg pooling also works)\n","    self.relu = nn.ReLU()\n","    self.dense = nn.Linear(num_filters, 2)     \n","    \n","  def forward(self, x):\n","    x = self.embedding(x)                                   # project to word embedding space\n","    x = x.view(-1, 1, self.input_dim, self.embedding_dim)   # resize to fit into convolutional layer\n","    x = self.conv(x)\n","    x = self.relu(x)\n","    x = self.pool(x) \n","    x = x.view(x.size(0), -1)   # resize to fit into final dense layer\n","    x = self.dense(x)\n","    return x"],"execution_count":0,"outputs":[]},{"metadata":{"id":"rP0Gt5E9ajmd","colab_type":"code","colab":{}},"cell_type":"code","source":["# hyperparameters\n","DEVICE = torch.device('cuda')\n","INPUT_DIM = maxlen\n","NUM_FILTERS = 64\n","KERNEL_SIZE = 3\n","STRIDE = 1\n","EMBEDDING_DIM = 50\n","NUM_WORDS = num_words\n","LEARNING_RATE = 1e-3\n","NUM_EPOCHS = 30              "],"execution_count":0,"outputs":[]},{"metadata":{"id":"cPBm8qDrSWsi","colab_type":"code","colab":{}},"cell_type":"code","source":["model = net(INPUT_DIM, NUM_WORDS, EMBEDDING_DIM, NUM_FILTERS, KERNEL_SIZE, STRIDE).to(DEVICE)\n","criterion = nn.CrossEntropyLoss()   # do not need softmax layer when using CEloss criterion\n","optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)"],"execution_count":0,"outputs":[]},{"metadata":{"id":"SEBtAPYCFeic","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":545},"outputId":"d94310e4-194c-455d-9830-a13d3c638039","executionInfo":{"status":"ok","timestamp":1545933874196,"user_tz":420,"elapsed":210646,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"cell_type":"code","source":["# training for NUM_EPOCHS\n","for i in range(NUM_EPOCHS):\n","  temp_loss = []\n","  for (x, y) in train_loader:\n","    x, y = x.long().to(DEVICE), y.to(DEVICE)  # beware that input to embedding should be type 'long'\n","    outputs = model(x)\n","    loss = criterion(outputs, y)\n","    temp_loss.append(loss.item())\n","    \n","    optimizer.zero_grad()\n","    loss.backward()\n","    optimizer.step()\n","    \n","  print(\"Loss at {}th epoch: {}\".format(i, np.mean(temp_loss)))"],"execution_count":14,"outputs":[{"output_type":"stream","text":["Loss at 0th epoch: 0.6467448935216787\n","Loss at 1th epoch: 0.5299239850469998\n","Loss at 2th epoch: 0.4440507980025544\n","Loss at 3th epoch: 0.38025763858946005\n","Loss at 4th epoch: 0.3228620941541633\n","Loss at 5th epoch: 0.2733502811467161\n","Loss at 6th epoch: 0.22932577779402538\n","Loss at 7th epoch: 0.18939032276370088\n","Loss at 8th epoch: 0.1521880773029157\n","Loss at 9th epoch: 0.12011083876904176\n","Loss at 10th epoch: 0.09256724133251273\n","Loss at 11th epoch: 0.07036281743904158\n","Loss at 12th epoch: 0.0526158744469285\n","Loss at 13th epoch: 0.03983275468784327\n","Loss at 14th epoch: 0.029970756969509686\n","Loss at 15th epoch: 0.022237090422410746\n","Loss at 16th epoch: 0.016928160087946727\n","Loss at 17th epoch: 0.013268464244902134\n","Loss at 18th epoch: 0.01047601109627178\n","Loss at 19th epoch: 0.008334757876582444\n","Loss at 20th epoch: 0.00675514917606868\n","Loss at 21th epoch: 0.005520799474039932\n","Loss at 22th epoch: 0.004574792754209163\n","Loss at 23th epoch: 0.003809949959276662\n","Loss at 24th epoch: 0.0031995022270296302\n","Loss at 25th epoch: 0.00271612963501402\n","Loss at 26th epoch: 0.0023028627079578917\n","Loss at 27th epoch: 0.0019556615081121574\n","Loss at 28th epoch: 0.00167927734271091\n","Loss at 29th epoch: 0.0014462826259158627\n"],"name":"stdout"}]},{"metadata":{"id":"qpAJUiHm529m","colab_type":"text"},"cell_type":"markdown","source":["## 3. Evaluation\n","- Evaluate the trained CNN model with accuracy score \n","  - Store probability of each instance to a list and compare it with true y label"]},{"metadata":{"id":"txXH3dknFpSx","colab_type":"code","outputId":"35322a5b-a28c-463f-85c9-7b01b6cb66d8","executionInfo":{"status":"ok","timestamp":1545933445208,"user_tz":420,"elapsed":964,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"cell_type":"code","source":["y_pred, y_true = [], []\n","with torch.no_grad():\n","  for x, y in test_loader:\n","    x, y = x.long().to(DEVICE), y.to(DEVICE)       # beware that input to embedding should be type 'long'\n","    outputs = F.softmax(model(x)).max(1)[-1]       # predicted label\n","    y_true += list(y.cpu().numpy())                # true label\n","    y_pred += list(outputs.cpu().numpy())   "],"execution_count":9,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n","  \"\"\"\n"],"name":"stderr"}]},{"metadata":{"id":"HV1s3xf5Frkl","colab_type":"code","outputId":"90d07e53-a813-40e9-8227-09cbccd2fa7a","executionInfo":{"status":"ok","timestamp":1545933447202,"user_tz":420,"elapsed":986,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"cell_type":"code","source":["# evaluation result\n","from sklearn.metrics import accuracy_score\n","accuracy_score(y_true, y_pred)"],"execution_count":10,"outputs":[{"output_type":"execute_result","data":{"text/plain":["0.77464"]},"metadata":{"tags":[]},"execution_count":10}]}]}