Running_Predicting_Movie_Reviews_with_BERT_on_TF_Hub.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of Predicting Movie Reviews with BERT on TF Hub.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/saahilgupta93/bert/blob/master/Running_Predicting_Movie_Reviews_with_BERT_on_TF_Hub.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "j0a4mTk9o1Qg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Copyright 2019 Google Inc.\n",
        "\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dCpvgG0vwXAZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Predicting Movie Review Sentiment with BERT on TF Hub"
      ]
    },
    {
      "metadata": {
        "id": "xiYrZKaHwV81",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "If you’ve been following Natural Language Processing over the past year, you’ve probably heard of BERT: Bidirectional Encoder Representations from Transformers. It’s a neural network architecture designed by Google researchers that’s totally transformed what’s state-of-the-art for NLP tasks, like text classification, translation, summarization, and question answering.\n",
        "\n",
        "Now that BERT's been added to [TF Hub](https://www.tensorflow.org/hub) as a loadable module, it's easy(ish) to add into existing Tensorflow text pipelines. In an existing pipeline, BERT can replace text embedding layers like ELMO and GloVE. Alternatively, [finetuning](http://wiki.fast.ai/index.php/Fine_tuning) BERT can provide both an accuracy boost and faster training time in many cases.\n",
        "\n",
        "Here, we'll train a model to predict whether an IMDB movie review is positive or negative using BERT in Tensorflow with tf hub. Some code was adapted from [this colab notebook](https://colab.sandbox.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb). Let's get started!"
      ]
    },
    {
      "metadata": {
        "id": "hsZvic2YxnTz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "04ce39a5-6f67-43ee-a0cf-6012a6f59e34"
      },
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from datetime import datetime"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING: Logging before flag parsing goes to stderr.\n",
            "W0416 14:21:39.337527 140513114511232 __init__.py:56] Some hub symbols are not available because TensorFlow version is less than 1.14\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "cp5wfXDx5SPH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In addition to the standard libraries we imported above, we'll need to install BERT's python package."
      ]
    },
    {
      "metadata": {
        "id": "jviywGyWyKsA",
        "colab_type": "code",
        "outputId": "d9858dc4-41a4-480a-9800-c1aa73eeab2e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        }
      },
      "cell_type": "code",
      "source": [
        "!pip install bert-tensorflow"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting bert-tensorflow\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a6/66/7eb4e8b6ea35b7cc54c322c816f976167a43019750279a8473d355800a93/bert_tensorflow-1.0.1-py2.py3-none-any.whl (67kB)\n",
            "\u001b[K    100% |████████████████████████████████| 71kB 3.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from bert-tensorflow) (1.11.0)\n",
            "Installing collected packages: bert-tensorflow\n",
            "Successfully installed bert-tensorflow-1.0.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "hhbGEfwgdEtw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import bert\n",
        "from bert import run_classifier\n",
        "from bert import optimization\n",
        "from bert import tokenization"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KVB3eOcjxxm1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Below, we'll set an output directory location to store our model output and checkpoints. This can be a local directory, in which case you'd set OUTPUT_DIR to the name of the directory you'd like to create. If you're running this code in Google's hosted Colab, the directory won't persist after the Colab session ends.\n",
        "\n",
        "Alternatively, if you're a GCP user, you can store output in a GCP bucket. To do that, set a directory name in OUTPUT_DIR and the name of the GCP bucket in the BUCKET field.\n",
        "\n",
        "Set DO_DELETE to rewrite the OUTPUT_DIR if it exists. Otherwise, Tensorflow will load existing model checkpoints from that directory (if they exist)."
      ]
    },
    {
      "metadata": {
        "id": "US_EAnICvP7f",
        "colab_type": "code",
        "outputId": "4a8b2e08-dbaa-4780-cb39-bdde9800164c",
        "cellView": "form",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Set the output directory for saving model file\n",
        "# Optionally, set a GCP bucket location\n",
        "\n",
        "OUTPUT_DIR = 'OUTPUT_DIR_NAME'#@param {type:\"string\"}\n",
        "#@markdown Whether or not to clear/delete the directory and create a new one\n",
        "DO_DELETE = False #@param {type:\"boolean\"}\n",
        "#@markdown Set USE_BUCKET and BUCKET if you want to (optionally) store model output on GCP bucket.\n",
        "USE_BUCKET = False #@param {type:\"boolean\"}\n",
        "BUCKET = 'BUCKET_NAME' #@param {type:\"string\"}\n",
        "\n",
        "if USE_BUCKET:\n",
        "  OUTPUT_DIR = 'gs://{}/{}'.format(BUCKET, OUTPUT_DIR)\n",
        "  from google.colab import auth\n",
        "  auth.authenticate_user()\n",
        "\n",
        "if DO_DELETE:\n",
        "  try:\n",
        "    tf.gfile.DeleteRecursively(OUTPUT_DIR)\n",
        "  except:\n",
        "    # Doesn't matter if the directory didn't exist\n",
        "    pass\n",
        "tf.gfile.MakeDirs(OUTPUT_DIR)\n",
        "print('***** Model output directory: {} *****'.format(OUTPUT_DIR))\n"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "***** Model output directory: OUTPUT_DIR_NAME *****\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "pmFYvkylMwXn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Data"
      ]
    },
    {
      "metadata": {
        "id": "MC_w8SRqN0fr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "First, let's download the dataset, hosted by Stanford. The code below, which downloads, extracts, and imports the IMDB Large Movie Review Dataset, is borrowed from [this Tensorflow tutorial](https://www.tensorflow.org/hub/tutorials/text_classification_with_tf_hub)."
      ]
    },
    {
      "metadata": {
        "id": "fom_ff20gyy6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from tensorflow import keras\n",
        "import os\n",
        "import re\n",
        "\n",
        "# Load all files from a directory in a DataFrame.\n",
        "def load_directory_data(directory):\n",
        "  data = {}\n",
        "  data[\"sentence\"] = []\n",
        "  data[\"sentiment\"] = []\n",
        "  for file_path in os.listdir(directory):\n",
        "    with tf.gfile.GFile(os.path.join(directory, file_path), \"r\") as f:\n",
        "      data[\"sentence\"].append(f.read())\n",
        "      data[\"sentiment\"].append(re.match(\"\\d+_(\\d+)\\.txt\", file_path).group(1))\n",
        "  return pd.DataFrame.from_dict(data)\n",
        "\n",
        "# Merge positive and negative examples, add a polarity column and shuffle.\n",
        "def load_dataset(directory):\n",
        "  pos_df = load_directory_data(os.path.join(directory, \"pos\"))\n",
        "  neg_df = load_directory_data(os.path.join(directory, \"neg\"))\n",
        "  pos_df[\"polarity\"] = 1\n",
        "  neg_df[\"polarity\"] = 0\n",
        "  return pd.concat([pos_df, neg_df]).sample(frac=1).reset_index(drop=True)\n",
        "\n",
        "# Download and process the dataset files.\n",
        "def download_and_load_datasets(force_download=False):\n",
        "  dataset = tf.keras.utils.get_file(\n",
        "      fname=\"aclImdb.tar.gz\", \n",
        "      origin=\"http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\", \n",
        "      extract=True)\n",
        "  \n",
        "  train_df = load_dataset(os.path.join(os.path.dirname(dataset), \n",
        "                                       \"aclImdb\", \"train\"))\n",
        "  test_df = load_dataset(os.path.join(os.path.dirname(dataset), \n",
        "                                      \"aclImdb\", \"test\"))\n",
        "  \n",
        "  return train_df, test_df\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2abfwdn-g135",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "5621b1d6-7117-4aab-de10-83d1844484b2"
      },
      "cell_type": "code",
      "source": [
        "train, test = download_and_load_datasets()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n",
            "84131840/84125825 [==============================] - 2s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "XA8WHJgzhIZf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To keep training fast, we'll take a sample of 5000 train and test examples, respectively."
      ]
    },
    {
      "metadata": {
        "id": "lw_F488eixTV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "train = train.sample(5000)\n",
        "test = test.sample(5000)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "prRQM8pDi8xI",
        "colab_type": "code",
        "outputId": "517ffeb5-bee7-490c-b5d3-0a6afb702748",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "train.columns"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['sentence', 'sentiment', 'polarity'], dtype='object')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "metadata": {
        "id": "sfRnHSz3iSXz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "For us, our input data is the 'sentence' column and our label is the 'polarity' column (0, 1 for negative and positive, respecitvely)"
      ]
    },
    {
      "metadata": {
        "id": "IuMOGwFui4it",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "DATA_COLUMN = 'sentence'\n",
        "LABEL_COLUMN = 'polarity'\n",
        "# label_list is the list of labels, i.e. True, False or 0, 1 or 'dog', 'cat'\n",
        "label_list = [0, 1]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "V399W0rqNJ-Z",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Data Preprocessing\n",
        "We'll need to transform our data into a format BERT understands. This involves two steps. First, we create  `InputExample`'s using the constructor provided in the BERT library.\n",
        "\n",
        "- `text_a` is the text we want to classify, which in this case, is the `Request` field in our Dataframe. \n",
        "- `text_b` is used if we're training a model to understand the relationship between sentences (i.e. is `text_b` a translation of `text_a`? Is `text_b` an answer to the question asked by `text_a`?). This doesn't apply to our task, so we can leave `text_b` blank.\n",
        "- `label` is the label for our example, i.e. True, False"
      ]
    },
    {
      "metadata": {
        "id": "p9gEt5SmM6i6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Use the InputExample class from BERT's run_classifier code to create examples from the data\n",
        "train_InputExamples = train.apply(lambda x: bert.run_classifier.InputExample(guid=None, # Globally unique ID for bookkeeping, unused in this example\n",
        "                                                                   text_a = x[DATA_COLUMN], \n",
        "                                                                   text_b = None, \n",
        "                                                                   label = x[LABEL_COLUMN]), axis = 1)\n",
        "\n",
        "test_InputExamples = test.apply(lambda x: bert.run_classifier.InputExample(guid=None, \n",
        "                                                                   text_a = x[DATA_COLUMN], \n",
        "                                                                   text_b = None, \n",
        "                                                                   label = x[LABEL_COLUMN]), axis = 1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SCZWZtKxObjh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next, we need to preprocess our data so that it matches the data BERT was trained on. For this, we'll need to do a couple of things (but don't worry--this is also included in the Python library):\n",
        "\n",
        "\n",
        "1. Lowercase our text (if we're using a BERT lowercase model)\n",
        "2. Tokenize it (i.e. \"sally says hi\" -> [\"sally\", \"says\", \"hi\"])\n",
        "3. Break words into WordPieces (i.e. \"calling\" -> [\"call\", \"##ing\"])\n",
        "4. Map our words to indexes using a vocab file that BERT provides\n",
        "5. Add special \"CLS\" and \"SEP\" tokens (see the [readme](https://github.com/google-research/bert))\n",
        "6. Append \"index\" and \"segment\" tokens to each input (see the [BERT paper](https://arxiv.org/pdf/1810.04805.pdf))\n",
        "\n",
        "Happily, we don't have to worry about most of these details.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "qMWiDtpyQSoU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "To start, we'll need to load a vocabulary file and lowercasing information directly from the BERT tf hub module:"
      ]
    },
    {
      "metadata": {
        "id": "IhJSe0QHNG7U",
        "colab_type": "code",
        "outputId": "7396cfef-d7a0-4a37-cf0a-0a0f9f1e916c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        }
      },
      "cell_type": "code",
      "source": [
        "# This is a path to an uncased (all lowercase) version of BERT\n",
        "BERT_MODEL_HUB = \"https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1\"\n",
        "\n",
        "def create_tokenizer_from_hub_module():\n",
        "  \"\"\"Get the vocab file and casing info from the Hub module.\"\"\"\n",
        "  with tf.Graph().as_default():\n",
        "    bert_module = hub.Module(BERT_MODEL_HUB)\n",
        "    tokenization_info = bert_module(signature=\"tokenization_info\", as_dict=True)\n",
        "    with tf.Session() as sess:\n",
        "      vocab_file, do_lower_case = sess.run([tokenization_info[\"vocab_file\"],\n",
        "                                            tokenization_info[\"do_lower_case\"]])\n",
        "      \n",
        "  return bert.tokenization.FullTokenizer(\n",
        "      vocab_file=vocab_file, do_lower_case=do_lower_case)\n",
        "\n",
        "tokenizer = create_tokenizer_from_hub_module()"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/control_flow_ops.py:3632: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "W0416 14:23:54.982178 140513114511232 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/control_flow_ops.py:3632: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Colocations handled automatically by placer.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:23:57.634938 140513114511232 saver.py:1483] Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "z4oFkhpZBDKm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Great--we just learned that the BERT model we're using expects lowercase data (that's what stored in tokenization_info[\"do_lower_case\"]) and we also loaded BERT's vocab file. We also created a tokenizer, which breaks words into word pieces:"
      ]
    },
    {
      "metadata": {
        "id": "dsBo6RCtQmwx",
        "colab_type": "code",
        "outputId": "c4a488b0-5bee-4060-a454-7fd0888451b1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "cell_type": "code",
      "source": [
        "tokenizer.tokenize(\"This here's an example of using the BERT tokenizer\")"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['this',\n",
              " 'here',\n",
              " \"'\",\n",
              " 's',\n",
              " 'an',\n",
              " 'example',\n",
              " 'of',\n",
              " 'using',\n",
              " 'the',\n",
              " 'bert',\n",
              " 'token',\n",
              " '##izer']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "metadata": {
        "id": "0OEzfFIt6GIc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Using our tokenizer, we'll call `run_classifier.convert_examples_to_features` on our InputExamples to convert them into features BERT understands."
      ]
    },
    {
      "metadata": {
        "id": "LL5W8gEGRTAf",
        "colab_type": "code",
        "outputId": "28efd857-b33a-408f-cc32-8226fe48c428",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 2485
        }
      },
      "cell_type": "code",
      "source": [
        "# We'll set sequences to be at most 128 tokens long.\n",
        "MAX_SEQ_LENGTH = 128\n",
        "# Convert our train and test features to InputFeatures that BERT understands.\n",
        "train_features = bert.run_classifier.convert_examples_to_features(train_InputExamples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
        "test_features = bert.run_classifier.convert_examples_to_features(test_InputExamples, label_list, MAX_SEQ_LENGTH, tokenizer)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 0 of 5000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.789496 140513114511232 run_classifier.py:774] Writing example 0 of 5000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.806034 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.808970 140513114511232 run_classifier.py:462] guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] this was the best movie i ' ve ever seen about bu ##lim ##ia . it hit the exact spot of what bu ##lim ##ia is really about and how it ' s not always about being skinny and diet ##ing . it showed how people with bu ##lim ##ia tend to think about things such as their outlook on life , friends and themselves . the best line and the part that really shows what the problem with bu ##lim ##ia is , is when beth says , \" it ' s not about you ! \" that line really showed a lot about the character and others with the same problem . it showed that people with bu ##lim ##ia don ' t have [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.811326 140513114511232 run_classifier.py:464] tokens: [CLS] this was the best movie i ' ve ever seen about bu ##lim ##ia . it hit the exact spot of what bu ##lim ##ia is really about and how it ' s not always about being skinny and diet ##ing . it showed how people with bu ##lim ##ia tend to think about things such as their outlook on life , friends and themselves . the best line and the part that really shows what the problem with bu ##lim ##ia is , is when beth says , \" it ' s not about you ! \" that line really showed a lot about the character and others with the same problem . it showed that people with bu ##lim ##ia don ' t have [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2023 2001 1996 2190 3185 1045 1005 2310 2412 2464 2055 20934 17960 2401 1012 2009 2718 1996 6635 3962 1997 2054 20934 17960 2401 2003 2428 2055 1998 2129 2009 1005 1055 2025 2467 2055 2108 15629 1998 8738 2075 1012 2009 3662 2129 2111 2007 20934 17960 2401 7166 2000 2228 2055 2477 2107 2004 2037 17680 2006 2166 1010 2814 1998 3209 1012 1996 2190 2240 1998 1996 2112 2008 2428 3065 2054 1996 3291 2007 20934 17960 2401 2003 1010 2003 2043 7014 2758 1010 1000 2009 1005 1055 2025 2055 2017 999 1000 2008 2240 2428 3662 1037 2843 2055 1996 2839 1998 2500 2007 1996 2168 3291 1012 2009 3662 2008 2111 2007 20934 17960 2401 2123 1005 1056 2031 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.813735 140513114511232 run_classifier.py:465] input_ids: 101 2023 2001 1996 2190 3185 1045 1005 2310 2412 2464 2055 20934 17960 2401 1012 2009 2718 1996 6635 3962 1997 2054 20934 17960 2401 2003 2428 2055 1998 2129 2009 1005 1055 2025 2467 2055 2108 15629 1998 8738 2075 1012 2009 3662 2129 2111 2007 20934 17960 2401 7166 2000 2228 2055 2477 2107 2004 2037 17680 2006 2166 1010 2814 1998 3209 1012 1996 2190 2240 1998 1996 2112 2008 2428 3065 2054 1996 3291 2007 20934 17960 2401 2003 1010 2003 2043 7014 2758 1010 1000 2009 1005 1055 2025 2055 2017 999 1000 2008 2240 2428 3662 1037 2843 2055 1996 2839 1998 2500 2007 1996 2168 3291 1012 2009 3662 2008 2111 2007 20934 17960 2401 2123 1005 1056 2031 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.818219 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.823014 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.826739 140513114511232 run_classifier.py:468] label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.860718 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.862686 140513114511232 run_classifier.py:462] guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] luckily i did not pay to see this movie . also , i cannot even reveal any spoil ##ers because i willingly walked out after forty minutes of the movie . it was that bad . i laughed once , when the yahoo ! billboard fell on the guy , and the theme song came on . however , that was only because i thought it was making fun of it , but then i realized it was yet another product placement . < br / > < br / > i loved the cartoon . i used to watch it almost religious ##ly . ( although i missed the last episode . i heard that they show dr . claw and it was nothing more [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.866007 140513114511232 run_classifier.py:464] tokens: [CLS] luckily i did not pay to see this movie . also , i cannot even reveal any spoil ##ers because i willingly walked out after forty minutes of the movie . it was that bad . i laughed once , when the yahoo ! billboard fell on the guy , and the theme song came on . however , that was only because i thought it was making fun of it , but then i realized it was yet another product placement . < br / > < br / > i loved the cartoon . i used to watch it almost religious ##ly . ( although i missed the last episode . i heard that they show dr . claw and it was nothing more [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 15798 1045 2106 2025 3477 2000 2156 2023 3185 1012 2036 1010 1045 3685 2130 7487 2151 27594 2545 2138 1045 18110 2939 2041 2044 5659 2781 1997 1996 3185 1012 2009 2001 2008 2919 1012 1045 4191 2320 1010 2043 1996 20643 999 4908 3062 2006 1996 3124 1010 1998 1996 4323 2299 2234 2006 1012 2174 1010 2008 2001 2069 2138 1045 2245 2009 2001 2437 4569 1997 2009 1010 2021 2059 1045 3651 2009 2001 2664 2178 4031 11073 1012 1026 7987 1013 1028 1026 7987 1013 1028 1045 3866 1996 9476 1012 1045 2109 2000 3422 2009 2471 3412 2135 1012 1006 2348 1045 4771 1996 2197 2792 1012 1045 2657 2008 2027 2265 2852 1012 15020 1998 2009 2001 2498 2062 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.869482 140513114511232 run_classifier.py:465] input_ids: 101 15798 1045 2106 2025 3477 2000 2156 2023 3185 1012 2036 1010 1045 3685 2130 7487 2151 27594 2545 2138 1045 18110 2939 2041 2044 5659 2781 1997 1996 3185 1012 2009 2001 2008 2919 1012 1045 4191 2320 1010 2043 1996 20643 999 4908 3062 2006 1996 3124 1010 1998 1996 4323 2299 2234 2006 1012 2174 1010 2008 2001 2069 2138 1045 2245 2009 2001 2437 4569 1997 2009 1010 2021 2059 1045 3651 2009 2001 2664 2178 4031 11073 1012 1026 7987 1013 1028 1026 7987 1013 1028 1045 3866 1996 9476 1012 1045 2109 2000 3422 2009 2471 3412 2135 1012 1006 2348 1045 4771 1996 2197 2792 1012 1045 2657 2008 2027 2265 2852 1012 15020 1998 2009 2001 2498 2062 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.874556 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.878977 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.883281 140513114511232 run_classifier.py:468] label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.895319 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.899334 140513114511232 run_classifier.py:462] guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] a question for you : a family go to a new house and get stalked by demonic forces . which film am i talking about ? every horror film you ` ve seen ? yes that ` s true but that ` s not the answer i ` m looking for . i ` ll narrow it down by saying there ` s a lot of teen ang ##st scenes . doesn ` t help ? well there ` s lots of bits where the characters are stalked by a creature and you see the characters through the creature ` s po ##v . no fu ##ther forward ? okay there ` s a dream sequence involving lots of blood ? could still be any horror [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.903258 140513114511232 run_classifier.py:464] tokens: [CLS] a question for you : a family go to a new house and get stalked by demonic forces . which film am i talking about ? every horror film you ` ve seen ? yes that ` s true but that ` s not the answer i ` m looking for . i ` ll narrow it down by saying there ` s a lot of teen ang ##st scenes . doesn ` t help ? well there ` s lots of bits where the characters are stalked by a creature and you see the characters through the creature ` s po ##v . no fu ##ther forward ? okay there ` s a dream sequence involving lots of blood ? could still be any horror [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1037 3160 2005 2017 1024 1037 2155 2175 2000 1037 2047 2160 1998 2131 15858 2011 23170 2749 1012 2029 2143 2572 1045 3331 2055 1029 2296 5469 2143 2017 1036 2310 2464 1029 2748 2008 1036 1055 2995 2021 2008 1036 1055 2025 1996 3437 1045 1036 1049 2559 2005 1012 1045 1036 2222 4867 2009 2091 2011 3038 2045 1036 1055 1037 2843 1997 9458 17076 3367 5019 1012 2987 1036 1056 2393 1029 2092 2045 1036 1055 7167 1997 9017 2073 1996 3494 2024 15858 2011 1037 6492 1998 2017 2156 1996 3494 2083 1996 6492 1036 1055 13433 2615 1012 2053 11865 12399 2830 1029 3100 2045 1036 1055 1037 3959 5537 5994 7167 1997 2668 1029 2071 2145 2022 2151 5469 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.907185 140513114511232 run_classifier.py:465] input_ids: 101 1037 3160 2005 2017 1024 1037 2155 2175 2000 1037 2047 2160 1998 2131 15858 2011 23170 2749 1012 2029 2143 2572 1045 3331 2055 1029 2296 5469 2143 2017 1036 2310 2464 1029 2748 2008 1036 1055 2995 2021 2008 1036 1055 2025 1996 3437 1045 1036 1049 2559 2005 1012 1045 1036 2222 4867 2009 2091 2011 3038 2045 1036 1055 1037 2843 1997 9458 17076 3367 5019 1012 2987 1036 1056 2393 1029 2092 2045 1036 1055 7167 1997 9017 2073 1996 3494 2024 15858 2011 1037 6492 1998 2017 2156 1996 3494 2083 1996 6492 1036 1055 13433 2615 1012 2053 11865 12399 2830 1029 3100 2045 1036 1055 1037 3959 5537 5994 7167 1997 2668 1029 2071 2145 2022 2151 5469 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.911612 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.915533 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.919293 140513114511232 run_classifier.py:468] label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.937902 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.941994 140513114511232 run_classifier.py:462] guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] another one for the babe ##s & bullets crowd . the story is much ed ##gie ##r than any other musical i have seen : cannons hidden up the missing legs of females , and places each genera ##tively in the other in a way that comes closer to intelligent comment than we might expect for the local ##e . more effective than contemporary ' drama . ' it is hard identify with a woman who keeps a cannon up her pants - - in lu ##e of leg ##ga ##ge . pretty remarkable if you consider the context . < br / > < br / > despite the cannon up the leg thing providing 90 % of the surprises , this film also chronicles [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.946320 140513114511232 run_classifier.py:464] tokens: [CLS] another one for the babe ##s & bullets crowd . the story is much ed ##gie ##r than any other musical i have seen : cannons hidden up the missing legs of females , and places each genera ##tively in the other in a way that comes closer to intelligent comment than we might expect for the local ##e . more effective than contemporary ' drama . ' it is hard identify with a woman who keeps a cannon up her pants - - in lu ##e of leg ##ga ##ge . pretty remarkable if you consider the context . < br / > < br / > despite the cannon up the leg thing providing 90 % of the surprises , this film also chronicles [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2178 2028 2005 1996 11561 2015 1004 10432 4306 1012 1996 2466 2003 2172 3968 11239 2099 2084 2151 2060 3315 1045 2031 2464 1024 17138 5023 2039 1996 4394 3456 1997 3801 1010 1998 3182 2169 11416 25499 1999 1996 2060 1999 1037 2126 2008 3310 3553 2000 9414 7615 2084 2057 2453 5987 2005 1996 2334 2063 1012 2062 4621 2084 3824 1005 3689 1012 1005 2009 2003 2524 6709 2007 1037 2450 2040 7906 1037 8854 2039 2014 6471 1011 1011 1999 11320 2063 1997 4190 3654 3351 1012 3492 9487 2065 2017 5136 1996 6123 1012 1026 7987 1013 1028 1026 7987 1013 1028 2750 1996 8854 2039 1996 4190 2518 4346 3938 1003 1997 1996 20096 1010 2023 2143 2036 11906 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.950269 140513114511232 run_classifier.py:465] input_ids: 101 2178 2028 2005 1996 11561 2015 1004 10432 4306 1012 1996 2466 2003 2172 3968 11239 2099 2084 2151 2060 3315 1045 2031 2464 1024 17138 5023 2039 1996 4394 3456 1997 3801 1010 1998 3182 2169 11416 25499 1999 1996 2060 1999 1037 2126 2008 3310 3553 2000 9414 7615 2084 2057 2453 5987 2005 1996 2334 2063 1012 2062 4621 2084 3824 1005 3689 1012 1005 2009 2003 2524 6709 2007 1037 2450 2040 7906 1037 8854 2039 2014 6471 1011 1011 1999 11320 2063 1997 4190 3654 3351 1012 3492 9487 2065 2017 5136 1996 6123 1012 1026 7987 1013 1028 1026 7987 1013 1028 2750 1996 8854 2039 1996 4190 2518 4346 3938 1003 1997 1996 20096 1010 2023 2143 2036 11906 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.954286 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.958463 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.962697 140513114511232 run_classifier.py:468] label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.992644 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:14.997287 140513114511232 run_classifier.py:462] guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] ' thunder ##birds ' was an immensely popular sixties show that has trans ##cend ##ed the years and generations to the point it is still as popular now , with both adults and children alike , as it was in its hey ##day . so , one would de ##du ##ce the chance to produce a live - action feature film with a million pound hollywood budget was an excellent opportunity to revive the series as has been done with ' spider - man ' and ' the x - men ' . but a terrible storyline and bland acting ob ##lite ##rated this opportunity and it was soon apparent all that was destined for this film was a trip to the bargain bin of the [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:15.001760 140513114511232 run_classifier.py:464] tokens: [CLS] ' thunder ##birds ' was an immensely popular sixties show that has trans ##cend ##ed the years and generations to the point it is still as popular now , with both adults and children alike , as it was in its hey ##day . so , one would de ##du ##ce the chance to produce a live - action feature film with a million pound hollywood budget was an excellent opportunity to revive the series as has been done with ' spider - man ' and ' the x - men ' . but a terrible storyline and bland acting ob ##lite ##rated this opportunity and it was soon apparent all that was destined for this film was a trip to the bargain bin of the [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1005 8505 12887 1005 2001 2019 24256 2759 22651 2265 2008 2038 9099 23865 2098 1996 2086 1998 8213 2000 1996 2391 2009 2003 2145 2004 2759 2085 1010 2007 2119 6001 1998 2336 11455 1010 2004 2009 2001 1999 2049 4931 10259 1012 2061 1010 2028 2052 2139 8566 3401 1996 3382 2000 3965 1037 2444 1011 2895 3444 2143 2007 1037 2454 9044 5365 5166 2001 2019 6581 4495 2000 17995 1996 2186 2004 2038 2042 2589 2007 1005 6804 1011 2158 1005 1998 1005 1996 1060 1011 2273 1005 1012 2021 1037 6659 9994 1998 20857 3772 27885 22779 9250 2023 4495 1998 2009 2001 2574 6835 2035 2008 2001 16036 2005 2023 2143 2001 1037 4440 2000 1996 17113 8026 1997 1996 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:15.006465 140513114511232 run_classifier.py:465] input_ids: 101 1005 8505 12887 1005 2001 2019 24256 2759 22651 2265 2008 2038 9099 23865 2098 1996 2086 1998 8213 2000 1996 2391 2009 2003 2145 2004 2759 2085 1010 2007 2119 6001 1998 2336 11455 1010 2004 2009 2001 1999 2049 4931 10259 1012 2061 1010 2028 2052 2139 8566 3401 1996 3382 2000 3965 1037 2444 1011 2895 3444 2143 2007 1037 2454 9044 5365 5166 2001 2019 6581 4495 2000 17995 1996 2186 2004 2038 2042 2589 2007 1005 6804 1011 2158 1005 1998 1005 1996 1060 1011 2273 1005 1012 2021 1037 6659 9994 1998 20857 3772 27885 22779 9250 2023 4495 1998 2009 2001 2574 6835 2035 2008 2001 16036 2005 2023 2143 2001 1037 4440 2000 1996 17113 8026 1997 1996 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:15.010491 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:15.015728 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:15.020889 140513114511232 run_classifier.py:468] label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 0 of 5000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.810639 140513114511232 run_classifier.py:774] Writing example 0 of 5000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.822636 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.825028 140513114511232 run_classifier.py:462] guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] this movie gives golf a high mark , it was well acted and well directed . giving you a view of history that some non - sports fans will enjoy . the historic factor alone gives it a high rating , the brook ##line golf course was really done well . i am in the northeast and have seen brook ##line as a fan , and as someone who loves the game . the movie was well done on all levels . a must see 5 stars . the acting was superb , disney has another winner in its bag of great movies . if for no other reason watch the film to give hope and encouragement to young people whom may not see the hope [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.830968 140513114511232 run_classifier.py:464] tokens: [CLS] this movie gives golf a high mark , it was well acted and well directed . giving you a view of history that some non - sports fans will enjoy . the historic factor alone gives it a high rating , the brook ##line golf course was really done well . i am in the northeast and have seen brook ##line as a fan , and as someone who loves the game . the movie was well done on all levels . a must see 5 stars . the acting was superb , disney has another winner in its bag of great movies . if for no other reason watch the film to give hope and encouragement to young people whom may not see the hope [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2023 3185 3957 5439 1037 2152 2928 1010 2009 2001 2092 6051 1998 2092 2856 1012 3228 2017 1037 3193 1997 2381 2008 2070 2512 1011 2998 4599 2097 5959 1012 1996 3181 5387 2894 3957 2009 1037 2152 5790 1010 1996 9566 4179 5439 2607 2001 2428 2589 2092 1012 1045 2572 1999 1996 4794 1998 2031 2464 9566 4179 2004 1037 5470 1010 1998 2004 2619 2040 7459 1996 2208 1012 1996 3185 2001 2092 2589 2006 2035 3798 1012 1037 2442 2156 1019 3340 1012 1996 3772 2001 21688 1010 6373 2038 2178 3453 1999 2049 4524 1997 2307 5691 1012 2065 2005 2053 2060 3114 3422 1996 2143 2000 2507 3246 1998 15846 2000 2402 2111 3183 2089 2025 2156 1996 3246 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.836323 140513114511232 run_classifier.py:465] input_ids: 101 2023 3185 3957 5439 1037 2152 2928 1010 2009 2001 2092 6051 1998 2092 2856 1012 3228 2017 1037 3193 1997 2381 2008 2070 2512 1011 2998 4599 2097 5959 1012 1996 3181 5387 2894 3957 2009 1037 2152 5790 1010 1996 9566 4179 5439 2607 2001 2428 2589 2092 1012 1045 2572 1999 1996 4794 1998 2031 2464 9566 4179 2004 1037 5470 1010 1998 2004 2619 2040 7459 1996 2208 1012 1996 3185 2001 2092 2589 2006 2035 3798 1012 1037 2442 2156 1019 3340 1012 1996 3772 2001 21688 1010 6373 2038 2178 3453 1999 2049 4524 1997 2307 5691 1012 2065 2005 2053 2060 3114 3422 1996 2143 2000 2507 3246 1998 15846 2000 2402 2111 3183 2089 2025 2156 1996 3246 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.841347 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.846641 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.853481 140513114511232 run_classifier.py:468] label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.868101 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.871346 140513114511232 run_classifier.py:462] guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] four unhappy women leave dr ##ear ##y london to spend an enchanted april in a castle on the coast of italy . < br / > < br / > elizabeth von ar ##ni ##m ' s novel comes alive in this charming little film which beautifully demonstrates the virtues of a liter ##ate script and ensemble acting . all the elements come together to produce a movie that , although nearly forgotten now , still produces a feeling of appreciation at the story ' s appropriate resolution . < br / > < br / > the actresses each ac ##qui ##t themselves splendid ##ly . ann harding is the free - spirited wife longing for ' wi ##ster ##ia & tran ##quil ##lity ' [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.875272 140513114511232 run_classifier.py:464] tokens: [CLS] four unhappy women leave dr ##ear ##y london to spend an enchanted april in a castle on the coast of italy . < br / > < br / > elizabeth von ar ##ni ##m ' s novel comes alive in this charming little film which beautifully demonstrates the virtues of a liter ##ate script and ensemble acting . all the elements come together to produce a movie that , although nearly forgotten now , still produces a feeling of appreciation at the story ' s appropriate resolution . < br / > < br / > the actresses each ac ##qui ##t themselves splendid ##ly . ann harding is the free - spirited wife longing for ' wi ##ster ##ia & tran ##quil ##lity ' [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2176 12511 2308 2681 2852 14644 2100 2414 2000 5247 2019 22454 2258 1999 1037 3317 2006 1996 3023 1997 3304 1012 1026 7987 1013 1028 1026 7987 1013 1028 3870 3854 12098 3490 2213 1005 1055 3117 3310 4142 1999 2023 11951 2210 2143 2029 17950 16691 1996 21560 1997 1037 23675 3686 5896 1998 7241 3772 1012 2035 1996 3787 2272 2362 2000 3965 1037 3185 2008 1010 2348 3053 6404 2085 1010 2145 7137 1037 3110 1997 12284 2012 1996 2466 1005 1055 6413 5813 1012 1026 7987 1013 1028 1026 7987 1013 1028 1996 19910 2169 9353 15549 2102 3209 21459 2135 1012 5754 15456 2003 1996 2489 1011 24462 2564 15752 2005 1005 15536 6238 2401 1004 25283 26147 18605 1005 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.878376 140513114511232 run_classifier.py:465] input_ids: 101 2176 12511 2308 2681 2852 14644 2100 2414 2000 5247 2019 22454 2258 1999 1037 3317 2006 1996 3023 1997 3304 1012 1026 7987 1013 1028 1026 7987 1013 1028 3870 3854 12098 3490 2213 1005 1055 3117 3310 4142 1999 2023 11951 2210 2143 2029 17950 16691 1996 21560 1997 1037 23675 3686 5896 1998 7241 3772 1012 2035 1996 3787 2272 2362 2000 3965 1037 3185 2008 1010 2348 3053 6404 2085 1010 2145 7137 1037 3110 1997 12284 2012 1996 2466 1005 1055 6413 5813 1012 1026 7987 1013 1028 1026 7987 1013 1028 1996 19910 2169 9353 15549 2102 3209 21459 2135 1012 5754 15456 2003 1996 2489 1011 24462 2564 15752 2005 1005 15536 6238 2401 1004 25283 26147 18605 1005 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.881401 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.884396 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.887216 140513114511232 run_classifier.py:468] label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.895812 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.899048 140513114511232 run_classifier.py:462] guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] this movie was released in all major cinemas in australia . i watched the movie on the weekend of 7th aug and i thought is was absolute nonsense ( and i am using that word extremely lit ##ely ) . how it got released to every major cinema no idea . the plot seemed simple enough about the world being divided into sub ##class ##es and people needed identity cards and the actor tim robbins playing an investigator who goes to shanghai to investigate a employee stealing id cards , which turns out to be the actress samantha morton . < br / > < br / > you think from that summary the rest of the story should inter ##tyn ##e but it doesn ' [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.901796 140513114511232 run_classifier.py:464] tokens: [CLS] this movie was released in all major cinemas in australia . i watched the movie on the weekend of 7th aug and i thought is was absolute nonsense ( and i am using that word extremely lit ##ely ) . how it got released to every major cinema no idea . the plot seemed simple enough about the world being divided into sub ##class ##es and people needed identity cards and the actor tim robbins playing an investigator who goes to shanghai to investigate a employee stealing id cards , which turns out to be the actress samantha morton . < br / > < br / > you think from that summary the rest of the story should inter ##tyn ##e but it doesn ' [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2023 3185 2001 2207 1999 2035 2350 19039 1999 2660 1012 1045 3427 1996 3185 2006 1996 5353 1997 5504 15476 1998 1045 2245 2003 2001 7619 14652 1006 1998 1045 2572 2478 2008 2773 5186 5507 26006 1007 1012 2129 2009 2288 2207 2000 2296 2350 5988 2053 2801 1012 1996 5436 2790 3722 2438 2055 1996 2088 2108 4055 2046 4942 26266 2229 1998 2111 2734 4767 5329 1998 1996 3364 5199 18091 2652 2019 14064 2040 3632 2000 8344 2000 8556 1037 7904 11065 8909 5329 1010 2029 4332 2041 2000 2022 1996 3883 11415 11164 1012 1026 7987 1013 1028 1026 7987 1013 1028 2017 2228 2013 2008 12654 1996 2717 1997 1996 2466 2323 6970 25680 2063 2021 2009 2987 1005 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.904794 140513114511232 run_classifier.py:465] input_ids: 101 2023 3185 2001 2207 1999 2035 2350 19039 1999 2660 1012 1045 3427 1996 3185 2006 1996 5353 1997 5504 15476 1998 1045 2245 2003 2001 7619 14652 1006 1998 1045 2572 2478 2008 2773 5186 5507 26006 1007 1012 2129 2009 2288 2207 2000 2296 2350 5988 2053 2801 1012 1996 5436 2790 3722 2438 2055 1996 2088 2108 4055 2046 4942 26266 2229 1998 2111 2734 4767 5329 1998 1996 3364 5199 18091 2652 2019 14064 2040 3632 2000 8344 2000 8556 1037 7904 11065 8909 5329 1010 2029 4332 2041 2000 2022 1996 3883 11415 11164 1012 1026 7987 1013 1028 1026 7987 1013 1028 2017 2228 2013 2008 12654 1996 2717 1997 1996 2466 2323 6970 25680 2063 2021 2009 2987 1005 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.907952 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.910953 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.913989 140513114511232 run_classifier.py:468] label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.918624 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.921898 140513114511232 run_classifier.py:462] guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] in an industry dominated by men and in lack of products with a female mark on it ; is it always nice to see a film shown from the woman ' s point of view . i would welcome more films from female writers and directors , and i think lots of other women with me . [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.925232 140513114511232 run_classifier.py:464] tokens: [CLS] in an industry dominated by men and in lack of products with a female mark on it ; is it always nice to see a film shown from the woman ' s point of view . i would welcome more films from female writers and directors , and i think lots of other women with me . [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1999 2019 3068 6817 2011 2273 1998 1999 3768 1997 3688 2007 1037 2931 2928 2006 2009 1025 2003 2009 2467 3835 2000 2156 1037 2143 3491 2013 1996 2450 1005 1055 2391 1997 3193 1012 1045 2052 6160 2062 3152 2013 2931 4898 1998 5501 1010 1998 1045 2228 7167 1997 2060 2308 2007 2033 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.928477 140513114511232 run_classifier.py:465] input_ids: 101 1999 2019 3068 6817 2011 2273 1998 1999 3768 1997 3688 2007 1037 2931 2928 2006 2009 1025 2003 2009 2467 3835 2000 2156 1037 2143 3491 2013 1996 2450 1005 1055 2391 1997 3193 1012 1045 2052 6160 2062 3152 2013 2931 4898 1998 5501 1010 1998 1045 2228 7167 1997 2060 2308 2007 2033 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.931792 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.935351 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.938435 140513114511232 run_classifier.py:468] label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.954722 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.958320 140513114511232 run_classifier.py:462] guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] i agree with ms ##ina ##bot ##tle ; this is a great movie . here are some dialogue s ##ni ##ppet ##s : < br / > < br / > rai ##sul ##i ( sean conner ##y ) to eden pe ##de ##car ##is ( candi ##ce bergen ) : \" you see the man at the well , how he draws the water ? when one bucket em ##pt ##ies , the other fills . it is so with the world . at present , you are full of power . but you ' re spilling it , waste ##fully . and islam is lap ##ping up the drops as they spill from your bucket . \" < br / > < br / [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.962207 140513114511232 run_classifier.py:464] tokens: [CLS] i agree with ms ##ina ##bot ##tle ; this is a great movie . here are some dialogue s ##ni ##ppet ##s : < br / > < br / > rai ##sul ##i ( sean conner ##y ) to eden pe ##de ##car ##is ( candi ##ce bergen ) : \" you see the man at the well , how he draws the water ? when one bucket em ##pt ##ies , the other fills . it is so with the world . at present , you are full of power . but you ' re spilling it , waste ##fully . and islam is lap ##ping up the drops as they spill from your bucket . \" < br / > < br / [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1045 5993 2007 5796 3981 18384 9286 1025 2023 2003 1037 2307 3185 1012 2182 2024 2070 7982 1055 3490 29519 2015 1024 1026 7987 1013 1028 1026 7987 1013 1028 15547 23722 2072 1006 5977 17639 2100 1007 2000 10267 21877 3207 10010 2483 1006 27467 3401 12674 1007 1024 1000 2017 2156 1996 2158 2012 1996 2092 1010 2129 2002 9891 1996 2300 1029 2043 2028 13610 7861 13876 3111 1010 1996 2060 17469 1012 2009 2003 2061 2007 1996 2088 1012 2012 2556 1010 2017 2024 2440 1997 2373 1012 2021 2017 1005 2128 18054 2009 1010 5949 7699 1012 1998 7025 2003 5001 4691 2039 1996 9010 2004 2027 14437 2013 2115 13610 1012 1000 1026 7987 1013 1028 1026 7987 1013 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.965691 140513114511232 run_classifier.py:465] input_ids: 101 1045 5993 2007 5796 3981 18384 9286 1025 2023 2003 1037 2307 3185 1012 2182 2024 2070 7982 1055 3490 29519 2015 1024 1026 7987 1013 1028 1026 7987 1013 1028 15547 23722 2072 1006 5977 17639 2100 1007 2000 10267 21877 3207 10010 2483 1006 27467 3401 12674 1007 1024 1000 2017 2156 1996 2158 2012 1996 2092 1010 2129 2002 9891 1996 2300 1029 2043 2028 13610 7861 13876 3111 1010 1996 2060 17469 1012 2009 2003 2061 2007 1996 2088 1012 2012 2556 1010 2017 2024 2440 1997 2373 1012 2021 2017 1005 2128 18054 2009 1010 5949 7699 1012 1998 7025 2003 5001 4691 2039 1996 9010 2004 2027 14437 2013 2115 13610 1012 1000 1026 7987 1013 1028 1026 7987 1013 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.968695 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.971708 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:24:41.976528 140513114511232 run_classifier.py:468] label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "ccp5trMwRtmr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#Creating a model\n",
        "\n",
        "Now that we've prepared our data, let's focus on building a model. `create_model` does just this below. First, it loads the BERT tf hub module again (this time to extract the computation graph). Next, it creates a single new layer that will be trained to adapt BERT to our sentiment task (i.e. classifying whether a movie review is positive or negative). This strategy of using a mostly trained model is called [fine-tuning](http://wiki.fast.ai/index.php/Fine_tuning)."
      ]
    },
    {
      "metadata": {
        "id": "6o2a5ZIvRcJq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def create_model(is_predicting, input_ids, input_mask, segment_ids, labels,\n",
        "                 num_labels):\n",
        "  \"\"\"Creates a classification model.\"\"\"\n",
        "\n",
        "  bert_module = hub.Module(\n",
        "      BERT_MODEL_HUB,\n",
        "      trainable=True)\n",
        "  bert_inputs = dict(\n",
        "      input_ids=input_ids,\n",
        "      input_mask=input_mask,\n",
        "      segment_ids=segment_ids)\n",
        "  bert_outputs = bert_module(\n",
        "      inputs=bert_inputs,\n",
        "      signature=\"tokens\",\n",
        "      as_dict=True)\n",
        "\n",
        "  # Use \"pooled_output\" for classification tasks on an entire sentence.\n",
        "  # Use \"sequence_outputs\" for token-level output.\n",
        "  output_layer = bert_outputs[\"pooled_output\"]\n",
        "\n",
        "  hidden_size = output_layer.shape[-1].value\n",
        "\n",
        "  # Create our own layer to tune for politeness data.\n",
        "  output_weights = tf.get_variable(\n",
        "      \"output_weights\", [num_labels, hidden_size],\n",
        "      initializer=tf.truncated_normal_initializer(stddev=0.02))\n",
        "\n",
        "  output_bias = tf.get_variable(\n",
        "      \"output_bias\", [num_labels], initializer=tf.zeros_initializer())\n",
        "\n",
        "  with tf.variable_scope(\"loss\"):\n",
        "\n",
        "    # Dropout helps prevent overfitting\n",
        "    output_layer = tf.nn.dropout(output_layer, keep_prob=0.9)\n",
        "\n",
        "    logits = tf.matmul(output_layer, output_weights, transpose_b=True)\n",
        "    logits = tf.nn.bias_add(logits, output_bias)\n",
        "    log_probs = tf.nn.log_softmax(logits, axis=-1)\n",
        "\n",
        "    # Convert labels into one-hot encoding\n",
        "    one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)\n",
        "\n",
        "    predicted_labels = tf.squeeze(tf.argmax(log_probs, axis=-1, output_type=tf.int32))\n",
        "    # If we're predicting, we want predicted labels and the probabiltiies.\n",
        "    if is_predicting:\n",
        "      return (predicted_labels, log_probs)\n",
        "\n",
        "    # If we're train/eval, compute loss between predicted and actual label\n",
        "    per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)\n",
        "    loss = tf.reduce_mean(per_example_loss)\n",
        "    return (loss, predicted_labels, log_probs)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qpE0ZIDOCQzE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next we'll wrap our model function in a `model_fn_builder` function that adapts our model to work for training, evaluation, and prediction."
      ]
    },
    {
      "metadata": {
        "id": "FnH-AnOQ9KKW",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# model_fn_builder actually creates our model function\n",
        "# using the passed parameters for num_labels, learning_rate, etc.\n",
        "def model_fn_builder(num_labels, learning_rate, num_train_steps,\n",
        "                     num_warmup_steps):\n",
        "  \"\"\"Returns `model_fn` closure for TPUEstimator.\"\"\"\n",
        "  def model_fn(features, labels, mode, params):  # pylint: disable=unused-argument\n",
        "    \"\"\"The `model_fn` for TPUEstimator.\"\"\"\n",
        "\n",
        "    input_ids = features[\"input_ids\"]\n",
        "    input_mask = features[\"input_mask\"]\n",
        "    segment_ids = features[\"segment_ids\"]\n",
        "    label_ids = features[\"label_ids\"]\n",
        "\n",
        "    is_predicting = (mode == tf.estimator.ModeKeys.PREDICT)\n",
        "    \n",
        "    # TRAIN and EVAL\n",
        "    if not is_predicting:\n",
        "\n",
        "      (loss, predicted_labels, log_probs) = create_model(\n",
        "        is_predicting, input_ids, input_mask, segment_ids, label_ids, num_labels)\n",
        "\n",
        "      train_op = bert.optimization.create_optimizer(\n",
        "          loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu=False)\n",
        "\n",
        "      # Calculate evaluation metrics. \n",
        "      def metric_fn(label_ids, predicted_labels):\n",
        "        accuracy = tf.metrics.accuracy(label_ids, predicted_labels)\n",
        "        f1_score = tf.contrib.metrics.f1_score(\n",
        "            label_ids,\n",
        "            predicted_labels)\n",
        "        auc = tf.metrics.auc(\n",
        "            label_ids,\n",
        "            predicted_labels)\n",
        "        recall = tf.metrics.recall(\n",
        "            label_ids,\n",
        "            predicted_labels)\n",
        "        precision = tf.metrics.precision(\n",
        "            label_ids,\n",
        "            predicted_labels) \n",
        "        true_pos = tf.metrics.true_positives(\n",
        "            label_ids,\n",
        "            predicted_labels)\n",
        "        true_neg = tf.metrics.true_negatives(\n",
        "            label_ids,\n",
        "            predicted_labels)   \n",
        "        false_pos = tf.metrics.false_positives(\n",
        "            label_ids,\n",
        "            predicted_labels)  \n",
        "        false_neg = tf.metrics.false_negatives(\n",
        "            label_ids,\n",
        "            predicted_labels)\n",
        "        return {\n",
        "            \"eval_accuracy\": accuracy,\n",
        "            \"f1_score\": f1_score,\n",
        "            \"auc\": auc,\n",
        "            \"precision\": precision,\n",
        "            \"recall\": recall,\n",
        "            \"true_positives\": true_pos,\n",
        "            \"true_negatives\": true_neg,\n",
        "            \"false_positives\": false_pos,\n",
        "            \"false_negatives\": false_neg\n",
        "        }\n",
        "\n",
        "      eval_metrics = metric_fn(label_ids, predicted_labels)\n",
        "\n",
        "      if mode == tf.estimator.ModeKeys.TRAIN:\n",
        "        return tf.estimator.EstimatorSpec(mode=mode,\n",
        "          loss=loss,\n",
        "          train_op=train_op)\n",
        "      else:\n",
        "          return tf.estimator.EstimatorSpec(mode=mode,\n",
        "            loss=loss,\n",
        "            eval_metric_ops=eval_metrics)\n",
        "    else:\n",
        "      (predicted_labels, log_probs) = create_model(\n",
        "        is_predicting, input_ids, input_mask, segment_ids, label_ids, num_labels)\n",
        "\n",
        "      predictions = {\n",
        "          'probabilities': log_probs,\n",
        "          'labels': predicted_labels\n",
        "      }\n",
        "      return tf.estimator.EstimatorSpec(mode, predictions=predictions)\n",
        "\n",
        "  # Return the actual model function in the closure\n",
        "  return model_fn\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "OjwJ4bTeWXD8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Compute train and warmup steps from batch size\n",
        "# These hyperparameters are copied from this colab notebook (https://colab.sandbox.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb)\n",
        "BATCH_SIZE = 32\n",
        "LEARNING_RATE = 2e-5\n",
        "NUM_TRAIN_EPOCHS = 3.0\n",
        "# Warmup is a period of time where hte learning rate \n",
        "# is small and gradually increases--usually helps training.\n",
        "WARMUP_PROPORTION = 0.1\n",
        "# Model configs\n",
        "SAVE_CHECKPOINTS_STEPS = 500\n",
        "SAVE_SUMMARY_STEPS = 100"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "emHf9GhfWBZ_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Compute # train and warmup steps from batch size\n",
        "num_train_steps = int(len(train_features) / BATCH_SIZE * NUM_TRAIN_EPOCHS)\n",
        "num_warmup_steps = int(num_train_steps * WARMUP_PROPORTION)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "oEJldMr3WYZa",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Specify outpit directory and number of checkpoint steps to save\n",
        "run_config = tf.estimator.RunConfig(\n",
        "    model_dir=OUTPUT_DIR,\n",
        "    save_summary_steps=SAVE_SUMMARY_STEPS,\n",
        "    save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "q_WebpS1X97v",
        "colab_type": "code",
        "outputId": "0a328cc0-fa2b-4fba-f4b4-a87e17ec783a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 275
        }
      },
      "cell_type": "code",
      "source": [
        "model_fn = model_fn_builder(\n",
        "  num_labels=len(label_list),\n",
        "  learning_rate=LEARNING_RATE,\n",
        "  num_train_steps=num_train_steps,\n",
        "  num_warmup_steps=num_warmup_steps)\n",
        "\n",
        "estimator = tf.estimator.Estimator(\n",
        "  model_fn=model_fn,\n",
        "  config=run_config,\n",
        "  params={\"batch_size\": BATCH_SIZE})\n"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': 'OUTPUT_DIR_NAME', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 500, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcb4ac021d0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:25:43.352699 140513114511232 estimator.py:201] Using config: {'_model_dir': 'OUTPUT_DIR_NAME', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 500, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcb4ac021d0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "NOO3RfG1DYLo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next we create an input builder function that takes our training feature set (`train_features`) and produces a generator. This is a pretty standard design pattern for working with Tensorflow [Estimators](https://www.tensorflow.org/guide/estimators)."
      ]
    },
    {
      "metadata": {
        "id": "1Pv2bAlOX_-K",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Create an input function for training. drop_remainder = True for using TPUs.\n",
        "train_input_fn = bert.run_classifier.input_fn_builder(\n",
        "    features=train_features,\n",
        "    seq_length=MAX_SEQ_LENGTH,\n",
        "    is_training=True,\n",
        "    drop_remainder=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "t6Nukby2EB6-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now we train our model! For me, using a Colab notebook running on Google's GPUs, my training time was about 14 minutes."
      ]
    },
    {
      "metadata": {
        "id": "nucD4gluYJmK",
        "colab_type": "code",
        "outputId": "b896ab91-ed53-4f0a-f350-7f944586f36b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 2971
        }
      },
      "cell_type": "code",
      "source": [
        "print(f'Beginning Training!')\n",
        "current_time = datetime.now()\n",
        "estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)\n",
        "print(\"Training took time \", datetime.now() - current_time)"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Beginning Training!\n",
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:25:57.636805 140513114511232 estimator.py:1111] Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:26:01.426409 140513114511232 saver.py:1483] Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From <ipython-input-16-ca03218f28a6>:34: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "W0416 14:26:01.610532 140513114511232 deprecation.py:506] From <ipython-input-16-ca03218f28a6>:34: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/learning_rate_decay_v2.py:321: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Deprecated in favor of operator or tf.math.divide.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "W0416 14:26:01.671662 140513114511232 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/learning_rate_decay_v2.py:321: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Deprecated in favor of operator or tf.math.divide.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "W0416 14:26:01.770556 140513114511232 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n",
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
            "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/metrics_impl.py:455: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "W0416 14:26:12.409970 140513114511232 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/metrics_impl.py:455: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            "WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
            "For more information, please see:\n",
            "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
            "  * https://github.com/tensorflow/addons\n",
            "If you depend on functionality not listed there, please file an issue.\n",
            "\n",
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:26:15.380828 140513114511232 estimator.py:1113] Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Create CheckpointSaverHook.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:26:15.389488 140513114511232 basic_session_run_hooks.py:527] Create CheckpointSaverHook.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:26:19.739669 140513114511232 monitored_session.py:222] Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:26:25.041578 140513114511232 session_manager.py:491] Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:26:25.318210 140513114511232 session_manager.py:493] Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving checkpoints for 0 into OUTPUT_DIR_NAME/model.ckpt.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:26:37.276090 140513114511232 basic_session_run_hooks.py:594] Saving checkpoints for 0 into OUTPUT_DIR_NAME/model.ckpt.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.8031093, step = 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:26:55.187022 140513114511232 basic_session_run_hooks.py:249] loss = 0.8031093, step = 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:global_step/sec: 0.597777\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:29:42.472645 140513114511232 basic_session_run_hooks.py:680] global_step/sec: 0.597777\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:loss = 0.2448835, step = 100 (167.289 sec)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:29:42.475821 140513114511232 basic_session_run_hooks.py:247] loss = 0.2448835, step = 100 (167.289 sec)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-23-6187edf75f66>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf'Beginning Training!'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mcurrent_time\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdatetime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_fn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtrain_input_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_steps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnum_train_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Training took time \"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdatetime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mcurrent_time\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self, input_fn, hooks, steps, max_steps, saving_listeners)\u001b[0m\n\u001b[1;32m    356\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    357\u001b[0m       \u001b[0msaving_listeners\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_check_listeners_type\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msaving_listeners\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 358\u001b[0;31m       \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_train_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msaving_listeners\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    359\u001b[0m       \u001b[0mlogging\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minfo\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Loss for final step: %s.'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    360\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py\u001b[0m in \u001b[0;36m_train_model\u001b[0;34m(self, input_fn, hooks, saving_listeners)\u001b[0m\n\u001b[1;32m   1122\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_train_model_distributed\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msaving_listeners\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1123\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1124\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_train_model_default\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msaving_listeners\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1125\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1126\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_train_model_default\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msaving_listeners\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py\u001b[0m in \u001b[0;36m_train_model_default\u001b[0;34m(self, input_fn, hooks, saving_listeners)\u001b[0m\n\u001b[1;32m   1156\u001b[0m       return self._train_with_estimator_spec(estimator_spec, worker_hooks,\n\u001b[1;32m   1157\u001b[0m                                              \u001b[0mhooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mglobal_step_tensor\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1158\u001b[0;31m                                              saving_listeners)\n\u001b[0m\u001b[1;32m   1159\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1160\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_train_model_distributed\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msaving_listeners\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py\u001b[0m in \u001b[0;36m_train_with_estimator_spec\u001b[0;34m(self, estimator_spec, worker_hooks, hooks, global_step_tensor, saving_listeners)\u001b[0m\n\u001b[1;32m   1405\u001b[0m       \u001b[0many_step_done\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1406\u001b[0m       \u001b[0;32mwhile\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mmon_sess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshould_stop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1407\u001b[0;31m         \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmon_sess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mestimator_spec\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain_op\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mestimator_spec\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1408\u001b[0m         \u001b[0many_step_done\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1409\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0many_step_done\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    674\u001b[0m                           \u001b[0mfeed_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    675\u001b[0m                           \u001b[0moptions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 676\u001b[0;31m                           run_metadata=run_metadata)\n\u001b[0m\u001b[1;32m    677\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    678\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mrun_step_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstep_fn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1169\u001b[0m                               \u001b[0mfeed_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1170\u001b[0m                               \u001b[0moptions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1171\u001b[0;31m                               run_metadata=run_metadata)\n\u001b[0m\u001b[1;32m   1172\u001b[0m       \u001b[0;32mexcept\u001b[0m \u001b[0m_PREEMPTION_ERRORS\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1173\u001b[0m         logging.info('An error was raised. This may be due to a preemption in '\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1253\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1254\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1255\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1256\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0m_PREEMPTION_ERRORS\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1257\u001b[0m       \u001b[0;32mraise\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1325\u001b[0m                                   \u001b[0mfeed_dict\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1326\u001b[0m                                   \u001b[0moptions\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0moptions\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1327\u001b[0;31m                                   run_metadata=run_metadata)\n\u001b[0m\u001b[1;32m   1328\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1329\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mhook\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_hooks\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1089\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1090\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1091\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sess\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1092\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1093\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0mrun_step_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstep_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mraw_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrun_with_hooks\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    927\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    928\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 929\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    930\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    931\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1150\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mfeed_dict_tensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1151\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m-> 1152\u001b[0;31m                              feed_dict_tensor, options, run_metadata)\n\u001b[0m\u001b[1;32m   1153\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1154\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1326\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1327\u001b[0m       return self._do_call(_run_fn, feeds, fetches, targets, options,\n\u001b[0;32m-> 1328\u001b[0;31m                            run_metadata)\n\u001b[0m\u001b[1;32m   1329\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1330\u001b[0m       \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_prun_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeeds\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetches\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1332\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1333\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1334\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1335\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1336\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1317\u001b[0m       \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_extend_graph\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1318\u001b[0m       return self._call_tf_sessionrun(\n\u001b[0;32m-> 1319\u001b[0;31m           options, feed_dict, fetch_list, target_list, run_metadata)\n\u001b[0m\u001b[1;32m   1320\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1321\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_call_tf_sessionrun\u001b[0;34m(self, options, feed_dict, fetch_list, target_list, run_metadata)\u001b[0m\n\u001b[1;32m   1405\u001b[0m     return tf_session.TF_SessionRun_wrapper(\n\u001b[1;32m   1406\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1407\u001b[0;31m         run_metadata)\n\u001b[0m\u001b[1;32m   1408\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1409\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_call_tf_sessionprun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "metadata": {
        "id": "CmbLTVniARy3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now let's use our test data to see how well our model did:"
      ]
    },
    {
      "metadata": {
        "id": "JIhejfpyJ8Bx",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "test_input_fn = run_classifier.input_fn_builder(\n",
        "    features=test_features,\n",
        "    seq_length=MAX_SEQ_LENGTH,\n",
        "    is_training=False,\n",
        "    drop_remainder=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PPVEXhNjYXC-",
        "colab_type": "code",
        "outputId": "3a43a778-a903-4fd6-b988-62562c50f922",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 734
        }
      },
      "cell_type": "code",
      "source": [
        "estimator.evaluate(input_fn=test_input_fn, steps=None)"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:32:25.773646 140513114511232 estimator.py:1111] Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:32:29.449234 140513114511232 saver.py:1483] Saver not created because there are no variables in the graph to restore\n",
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
            "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:32:41.170440 140513114511232 estimator.py:1113] Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Starting evaluation at 2019-04-16T14:32:41Z\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:32:41.205410 140513114511232 evaluation.py:257] Starting evaluation at 2019-04-16T14:32:41Z\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:32:43.119855 140513114511232 monitored_session.py:222] Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to check for files with this prefix.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "W0416 14:32:43.124171 140513114511232 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use standard file APIs to check for files with this prefix.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from OUTPUT_DIR_NAME/model.ckpt-0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:32:43.128459 140513114511232 saver.py:1270] Restoring parameters from OUTPUT_DIR_NAME/model.ckpt-0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:32:45.575964 140513114511232 session_manager.py:491] Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:32:45.833696 140513114511232 session_manager.py:493] Done running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Finished evaluation at 2019-04-16-14:34:07\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:07.940688 140513114511232 evaluation.py:277] Finished evaluation at 2019-04-16-14:34:07\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving dict for global step 0: auc = 0.5000973, eval_accuracy = 0.5064, f1_score = 0.67233133, false_negatives = 19.0, false_positives = 2449.0, global_step = 0, loss = 0.7254833, precision = 0.506449, recall = 0.9924961, true_negatives = 19.0, true_positives = 2513.0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:07.943975 140513114511232 estimator.py:1979] Saving dict for global step 0: auc = 0.5000973, eval_accuracy = 0.5064, f1_score = 0.67233133, false_negatives = 19.0, false_positives = 2449.0, global_step = 0, loss = 0.7254833, precision = 0.506449, recall = 0.9924961, true_negatives = 19.0, true_positives = 2513.0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 0: OUTPUT_DIR_NAME/model.ckpt-0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:10.839275 140513114511232 estimator.py:2039] Saving 'checkpoint_path' summary for global step 0: OUTPUT_DIR_NAME/model.ckpt-0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'auc': 0.5000973,\n",
              " 'eval_accuracy': 0.5064,\n",
              " 'f1_score': 0.67233133,\n",
              " 'false_negatives': 19.0,\n",
              " 'false_positives': 2449.0,\n",
              " 'global_step': 0,\n",
              " 'loss': 0.7254833,\n",
              " 'precision': 0.506449,\n",
              " 'recall': 0.9924961,\n",
              " 'true_negatives': 19.0,\n",
              " 'true_positives': 2513.0}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "metadata": {
        "id": "ueKsULteiz1B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now let's write code to make predictions on new sentences:"
      ]
    },
    {
      "metadata": {
        "id": "OsrbTD2EJTVl",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def getPrediction(in_sentences):\n",
        "  labels = [\"Negative\", \"Positive\"]\n",
        "  input_examples = [run_classifier.InputExample(guid=\"\", text_a = x, text_b = None, label = 0) for x in in_sentences] # here, \"\" is just a dummy label\n",
        "  input_features = run_classifier.convert_examples_to_features(input_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
        "  predict_input_fn = run_classifier.input_fn_builder(features=input_features, seq_length=MAX_SEQ_LENGTH, is_training=False, drop_remainder=False)\n",
        "  predictions = estimator.predict(predict_input_fn)\n",
        "  return [(sentence, prediction['probabilities'], labels[prediction['labels']]) for sentence, prediction in zip(in_sentences, predictions)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-thbodgih_VJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "pred_sentences = [\n",
        "  \"That movie was absolutely awful\",\n",
        "  \"The acting was a bit lacking\",\n",
        "  \"The film was creative and surprising\",\n",
        "  \"Absolutely fantastic!\"\n",
        "]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "QrZmvZySKQTm",
        "colab_type": "code",
        "outputId": "46eaaf7c-1ee8-452a-e110-e424fb3844bf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1261
        }
      },
      "cell_type": "code",
      "source": [
        "predictions = getPrediction(pred_sentences)"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 0 of 4\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.305106 140513114511232 run_classifier.py:774] Writing example 0 of 4\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.314020 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.321030 140513114511232 run_classifier.py:462] guid: \n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] that movie was absolutely awful [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.323260 140513114511232 run_classifier.py:464] tokens: [CLS] that movie was absolutely awful [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2008 3185 2001 7078 9643 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.327465 140513114511232 run_classifier.py:465] input_ids: 101 2008 3185 2001 7078 9643 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.332740 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.337809 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.343080 140513114511232 run_classifier.py:468] label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.348157 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.353101 140513114511232 run_classifier.py:462] guid: \n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] the acting was a bit lacking [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.369657 140513114511232 run_classifier.py:464] tokens: [CLS] the acting was a bit lacking [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1996 3772 2001 1037 2978 11158 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.377041 140513114511232 run_classifier.py:465] input_ids: 101 1996 3772 2001 1037 2978 11158 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.384162 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.391245 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.393522 140513114511232 run_classifier.py:468] label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.397384 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.407195 140513114511232 run_classifier.py:462] guid: \n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] the film was creative and surprising [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.412153 140513114511232 run_classifier.py:464] tokens: [CLS] the film was creative and surprising [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1996 2143 2001 5541 1998 11341 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.415762 140513114511232 run_classifier.py:465] input_ids: 101 1996 2143 2001 5541 1998 11341 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.419824 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.423922 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.429255 140513114511232 run_classifier.py:468] label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.433537 140513114511232 run_classifier.py:461] *** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.438330 140513114511232 run_classifier.py:462] guid: \n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] absolutely fantastic ! [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.442460 140513114511232 run_classifier.py:464] tokens: [CLS] absolutely fantastic ! [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 7078 10392 999 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.447072 140513114511232 run_classifier.py:465] input_ids: 101 7078 10392 999 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.451157 140513114511232 run_classifier.py:466] input_mask: 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.456343 140513114511232 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.460860 140513114511232 run_classifier.py:468] label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:38.522915 140513114511232 estimator.py:1111] Calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:42.691006 140513114511232 saver.py:1483] Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:42.930141 140513114511232 estimator.py:1113] Done calling model_fn.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Graph was finalized.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:44.131840 140513114511232 monitored_session.py:222] Graph was finalized.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from OUTPUT_DIR_NAME/model.ckpt-0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:44.146198 140513114511232 saver.py:1270] Restoring parameters from OUTPUT_DIR_NAME/model.ckpt-0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:44.977670 140513114511232 session_manager.py:491] Running local_init_op.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "I0416 14:34:45.084196 140513114511232 session_manager.py:493] Done running local_init_op.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "metadata": {
        "id": "MXkRiEBUqN3n",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Voila! We have a sentiment classifier!"
      ]
    },
    {
      "metadata": {
        "id": "ERkTE8-7oQLZ",
        "colab_type": "code",
        "outputId": "8af557d1-efe8-43fd-b782-721ebe9ac45a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "cell_type": "code",
      "source": [
        "predictions"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('That movie was absolutely awful',\n",
              "  array([-0.83282685, -0.5706081 ], dtype=float32),\n",
              "  'Positive'),\n",
              " ('The acting was a bit lacking',\n",
              "  array([-0.6835434, -0.7028441], dtype=float32),\n",
              "  'Negative'),\n",
              " ('The film was creative and surprising',\n",
              "  array([-1.0058708, -0.4552743], dtype=float32),\n",
              "  'Positive'),\n",
              " ('Absolutely fantastic!',\n",
              "  array([-0.9006722, -0.5213752], dtype=float32),\n",
              "  'Positive')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    }
  ]
}