refactoring.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Refactoring Python Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "---\n",
    "\n",
    "### Evgeny Demchenko\n",
    "\n",
    "* Twitter/Github: [@littlepea](https://twitter.com/littlepea12)\n",
    "* Email: [evgeny.demchenko@gmail.com](mailto:evgeny.demchenko@gmail.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Scope\n",
    "\n",
    "* What is refactoring?\n",
    "* Why should we refactor?\n",
    "* When to refactor?\n",
    "* Refactoring examples (how to refactor?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What is Refactoring?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "> \"Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior.\" -- © Martin Fowler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Or in other words:\n",
    "\n",
    "> Refactoring is a process of improving your code without writing any new functionality."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Why should we refactor?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Make the code easier to read and maintain\n",
    "* Reduce complexity\n",
    "* Improve performance\n",
    "* Make the code more reusable and flexible\n",
    "* Pay off technical debt\n",
    "* Keep the \"cost of change\" low"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# When to refactor?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* Ideally: as you code...\n",
    "* Otherwise:\n",
    "  * When you find yourself duplicating functionality\n",
    "  * When working with legacy code\n",
    "  * When adding features\n",
    "  * When fixing bugs\n",
    "  \n",
    "### The boy-scout rule:\n",
    "  \n",
    "> \"always leave the code behind in a better state than you found it.\" -- © Uncle Bob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# How to refactor?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Types of Refactoring:\n",
    "\n",
    "* Improve form (names, functions size, nesting)\n",
    "* Improve style (PEP8/PyLint, pythonic code)\n",
    "* Reduce duplication (DRY) and complexity (KISS)\n",
    "* Apply design patterns (such as \"Ports and Adapters\")\n",
    "* Apply SOLID principles (such as \"Single Responsibility Principle\")\n",
    "* Improve code structure (modularity and composability)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Refactoring examples\n",
    "\n",
    "We're going to refactor a sample console movie reviews aggregation utility to demonstrate some examples of all kinds of refactoring.\n",
    "\n",
    "We'll start with a working application that aggregates reviews from Twitter and IMDB using the movie name as input and we'll apply one-by-one different refartoring patters to make it more readable and maintainable python code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Code \"Before Refactoring\" \n",
    "\n",
    "[movie_reviews.py](https://github.com/littlepea/python-refactoring-talk/blob/master/before/movie_reviews.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"Movie Reviews.\n",
    "\n",
    "Usage:  movie_reviews.py <title>\n",
    "        movie_reviews.py (-h | --help)\n",
    "        movie_reviews.py --version\n",
    "\n",
    "Arguments:\n",
    "  <title>         Movie title\n",
    "\n",
    "Options:\n",
    "  -h --help     Show this screen.\n",
    "  --version     Show version.\n",
    "\"\"\"\n",
    "\n",
    "from docopt import docopt\n",
    "from TwitterSearch import *\n",
    "from dateutil import parser\n",
    "from imdbpie import Imdb\n",
    "import logging\n",
    "import os\n",
    "\n",
    "def main(title):\n",
    "    reviews = []\n",
    "\n",
    "    # Search tweets\n",
    "    ts = TwitterSearch(\n",
    "        consumer_key = os.environ.get('TWITTER_CONSUMER_KEY'),\n",
    "        consumer_secret = os.environ.get('TWITTER_CONSUMER_SECRET'),\n",
    "        access_token = os.environ.get('TWITTER_ACCESS_TOKEN'),\n",
    "        access_token_secret = os.environ.get('TWITTER_TOKEN_SECRET')\n",
    "    )\n",
    "    try:\n",
    "        ts.connect()\n",
    "\n",
    "        tso = TwitterSearchOrder() # create a TwitterSearchOrder object\n",
    "        tso.setKeywords(['#' + title + 'Movie']) # let's define all words we would like to have a look for\n",
    "        tso.setLanguage('en') # we want to see German tweets only\n",
    "        tso.setIncludeEntities(False) # and don't give us all those entity information\n",
    "\n",
    "        # add tweets to reviews list\n",
    "        results = ts.getSearchResults(tso)\n",
    "\n",
    "    except TwitterSearchException as e: # take care of all those ugly errors if there are some\n",
    "        logging.exception(str(e))\n",
    "        ts.cleanUp()\n",
    "    else:\n",
    "        for offset in range(results.getSize()):\n",
    "            if offset > 9:\n",
    "                break\n",
    "            tweet = results.getTweetByIndex(offset)\n",
    "            reviews.append({\n",
    "                'author': tweet.getUserName(),\n",
    "                'summary': tweet.getText(),\n",
    "                'text': tweet.getText(),\n",
    "                'date': parser.parse(tweet.getCreatedDate(), ignoretz=True),\n",
    "                'source': 'Twitter'\n",
    "            })\n",
    "    finally:\n",
    "        ts.disconnect()\n",
    "\n",
    "    # Search Imdb\n",
    "    imdb = Imdb()\n",
    "    try:\n",
    "        response = imdb.search_for_title(title)[0]\n",
    "        title_id = response['imdb_id']\n",
    "        response = imdb.get_title_reviews(title_id, max_results=10)\n",
    "    except IndexError as e:\n",
    "        logging.exception(str(e))\n",
    "    else:\n",
    "        for review in response:\n",
    "            reviews.append({\n",
    "                'author': review.username,\n",
    "                'summary': review.summary,\n",
    "                'text': review.text,\n",
    "                'date': parser.parse(review.date, ignoretz=True),\n",
    "                'source': 'IMDB'\n",
    "            })\n",
    "\n",
    "    # Sort reviews by date\n",
    "    reviews.sort(cmp=_cmprev)\n",
    "\n",
    "    # Print reviews\n",
    "    for review in reviews:\n",
    "        print('(%s) @%s: %s [Source: %s]' % ( review['date'].strftime('%Y-%m-%d'), review['author'], review['summary'], review['source'] ) )\n",
    "\n",
    "def _cmprev(r1, r2):\n",
    "    if r1['date']>r2['date']:\n",
    "        return -1\n",
    "    elif r1['date']<r2['date']:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Improve Style\n",
    "\n",
    "## PEP8/PyLint:\n",
    "\n",
    "* Imports rules/order\n",
    "* Whitespace rules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Imports: BEFORE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from docopt import docopt\n",
    "from TwitterSearch import *\n",
    "from dateutil import parser\n",
    "from imdbpie import Imdb\n",
    "import logging\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "### Problems:\n",
    "\n",
    "* 3rd-party packages before the standard library\n",
    "* Wildcard imports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Imports: AFTER\n",
    "\n",
    "[commit 1fa2de3](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/1fa2de3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "\n",
    "from docopt import docopt\n",
    "from dateutil import parser\n",
    "from imdbpie import Imdb\n",
    "from TwitterSearch import TwitterSearch, TwitterSearchOrder, TwitterSearchException"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Improve Names\n",
    "\n",
    "* Explicit better than implicit\n",
    "* Names should reflect the purpose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Sorting function: BEFORE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def _cmprev(r1, r2):\n",
    "    if r1['date']>r2['date']:\n",
    "        return -1\n",
    "    elif r1['date']<r2['date']:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Sorting function: AFTER\n",
    "\n",
    "[commit 80fc3d6](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/80fc3d6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def _sort_by_date_desc(first, second):\n",
    "    first_date = first['date']\n",
    "    second_date = second['date']\n",
    "\n",
    "    if first_date > second_date:\n",
    "        return -1\n",
    "    elif first_date < second_date:\n",
    "        return 1\n",
    "\n",
    "    return 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Create Adapters for 3rd-party code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Benefits:\n",
    "\n",
    "* Decouple the dependencies on the application boundary\n",
    "* Only expose the functionality you need\n",
    "* Expose 3rd-party functionality via a pythonic API\n",
    "* Easy to mock/test\n",
    "* Easy to swap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Twitter Search: BEFORE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def main(title):\n",
    "    reviews = []\n",
    "\n",
    "    # Search tweets\n",
    "    ts = TwitterSearch(\n",
    "        consumer_key = os.environ.get('TWITTER_CONSUMER_KEY'),\n",
    "        consumer_secret = os.environ.get('TWITTER_CONSUMER_SECRET'),\n",
    "        access_token = os.environ.get('TWITTER_ACCESS_TOKEN'),\n",
    "        access_token_secret = os.environ.get('TWITTER_TOKEN_SECRET')\n",
    "    )\n",
    "    try:\n",
    "        ts.connect()\n",
    "\n",
    "        tso = TwitterSearchOrder()  # create a TwitterSearchOrder object\n",
    "        tso.setKeywords(['#' + title + 'Movie'])  # let's define all words we would like to have a look for\n",
    "        tso.setLanguage('en')  # we want to see German tweets only\n",
    "        tso.setIncludeEntities(False)  # and don't give us all those entity information\n",
    "\n",
    "        # add tweets to reviews list\n",
    "        results = ts.getSearchResults(tso)\n",
    "\n",
    "    except TwitterSearchException as e:  # take care of all those ugly errors if there are some\n",
    "        logging.exception(str(e))\n",
    "        ts.cleanUp()\n",
    "    else:\n",
    "        for offset in range(results.getSize()):\n",
    "            if offset > 9:\n",
    "                break\n",
    "            tweet = results.getTweetByIndex(offset)\n",
    "            reviews.append({\n",
    "                'author': tweet.getUserName(),\n",
    "                'summary': tweet.getText(),\n",
    "                'text': tweet.getText(),\n",
    "                'date': parser.parse(tweet.getCreatedDate(), ignoretz=True),\n",
    "                'source': 'Twitter'\n",
    "            })\n",
    "    finally:\n",
    "        ts.disconnect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### TwitterReviews adapter\n",
    "\n",
    "[commit 3156496](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/3156496)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "from dateutil import parser\n",
    "\n",
    "from TwitterSearch import TwitterSearch, TwitterSearchOrder, TwitterSearchException\n",
    "\n",
    "\n",
    "class TwitterReviews(object):\n",
    "    def __init__(self, movie, limit=10, language='en'):\n",
    "        self.movie = movie\n",
    "        self.limit = 10\n",
    "        self.language = language\n",
    "        self.client = TwitterSearch(\n",
    "            consumer_key=os.environ.get('TWITTER_CONSUMER_KEY'),\n",
    "            consumer_secret=os.environ.get('TWITTER_CONSUMER_SECRET'),\n",
    "            access_token=os.environ.get('TWITTER_ACCESS_TOKEN'),\n",
    "            access_token_secret=os.environ.get('TWITTER_TOKEN_SECRET')\n",
    "        )\n",
    "\n",
    "    def __enter__(self):\n",
    "        self.client.connect()\n",
    "        return self\n",
    "\n",
    "    def __exit__(self, exc_type, exc_val, exc_tb):\n",
    "        if exc_type == TwitterSearchException:\n",
    "            logging.exception(str(exc_val))\n",
    "            self.client.cleanUp()\n",
    "        self.client.disconnect()\n",
    "\n",
    "    @property\n",
    "    def reviews(self):\n",
    "        return Reviews(self._get_results(), limit=self.limit)\n",
    "\n",
    "    def _prepare_request(self):\n",
    "        tso = TwitterSearchOrder()\n",
    "        tso.setKeywords(self._get_keywords())\n",
    "        tso.setLanguage(self.language)\n",
    "        tso.setIncludeEntities(False)\n",
    "        return tso\n",
    "\n",
    "    def _get_keywords(self):\n",
    "        return ['#' + self.movie + 'Movie']\n",
    "\n",
    "    def _get_results(self):\n",
    "        request = self._prepare_request()\n",
    "        return self.client.getSearchResults(request)\n",
    "\n",
    "\n",
    "class Reviews(object):\n",
    "    def __init__(self, tweets, limit=10):\n",
    "        self.limit = limit\n",
    "        self.tweets = tweets\n",
    "\n",
    "    def __len__(self):\n",
    "        size = self.tweets.getSize()\n",
    "        return size if size < self.limit else self.limit\n",
    "\n",
    "    def __getitem__(self, item):\n",
    "        if item >= len(self):\n",
    "            raise IndexError\n",
    "        tweet = self.tweets.getTweetByIndex(item)\n",
    "        return Review(tweet)\n",
    "\n",
    "\n",
    "class Review(object):\n",
    "    def __init__(self, review):\n",
    "        self.review = review\n",
    "\n",
    "    @property\n",
    "    def author(self):\n",
    "        return self.review.getUserName()\n",
    "\n",
    "    @property\n",
    "    def summary(self):\n",
    "        return self.review.getText()\n",
    "\n",
    "    @property\n",
    "    def text(self):\n",
    "        return self.review.getText()\n",
    "\n",
    "    @property\n",
    "    def date(self):\n",
    "        return parser.parse(self.review.getCreatedDate(), ignoretz=True)\n",
    "\n",
    "    @property\n",
    "    def source(self):\n",
    "        return 'Twitter'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Twitter Search: AFTER\n",
    "\n",
    "[commit d0a40a4](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/d0a40a4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def main(title, year=None):\n",
    "    reviews = []\n",
    "    \n",
    "    with TwitterReviews(title) as reviews_backend:\n",
    "        for review in reviews_backend.reviews:\n",
    "            reviews.append(review)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Remove Duplication (DRY)\n",
    "\n",
    "* Duplicated code should be extracted into separate functions, methods and classes\n",
    "* But the purpose is reusability, not just \"not allowing any duplication\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### IMDB Reviews: BEFORE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def main(title, year=None):\n",
    "    reviews = []\n",
    "    \n",
    "    with TwitterReviews(title) as reviews_backend:\n",
    "        for review in reviews_backend.reviews:\n",
    "            reviews.append(review)\n",
    "            \n",
    "    # Search Imdb\n",
    "    imdb = Imdb()\n",
    "    try:\n",
    "        response = imdb.search_for_title(title)[0]\n",
    "        title_id = response['imdb_id']\n",
    "        response = imdb.get_title_reviews(title_id, max_results=10)\n",
    "    except IndexError as e:\n",
    "        logging.exception(str(e))\n",
    "    else:\n",
    "        for review in response:\n",
    "            reviews.append({\n",
    "                'author': review.username,\n",
    "                'summary': review.summary,\n",
    "                'text': review.text,\n",
    "                'date': parser.parse(review.date, ignoretz=True),\n",
    "                'source': 'IMDB'\n",
    "            })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### IMDB Reviews adapter\n",
    "\n",
    "[commit aefc54e](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/aefc54e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "from dateutil import parser\n",
    "from imdbpie import Imdb\n",
    "\n",
    "\n",
    "class IMDBReviews(object):\n",
    "    def __init__(self, movie, limit=10, language='en'):\n",
    "        self.movie = movie\n",
    "        self.limit = limit\n",
    "        self.language = language\n",
    "        self.client = Imdb()\n",
    "\n",
    "    def __enter__(self):\n",
    "        return self\n",
    "\n",
    "    def __exit__(self, exc_type, exc_val, exc_tb):\n",
    "        if exc_val:\n",
    "            logging.exception(str(exc_val))\n",
    "\n",
    "    @property\n",
    "    def reviews(self):\n",
    "        return [Review(result) for result in self._get_results()]\n",
    "\n",
    "    def _get_results(self):\n",
    "        try:\n",
    "            response = self.client.search_for_title(self.movie)[0]\n",
    "            title_id = response['imdb_id']\n",
    "            response = self.client.get_title_reviews(title_id, max_results=10)\n",
    "            return response\n",
    "        except IndexError as e:\n",
    "            logging.exception(str(e))\n",
    "            return []\n",
    "\n",
    "\n",
    "class Review(object):\n",
    "    def __init__(self, review):\n",
    "        self.review = review\n",
    "\n",
    "    @property\n",
    "    def author(self):\n",
    "        return self.review.username\n",
    "\n",
    "    @property\n",
    "    def summary(self):\n",
    "        return self.review.summary\n",
    "\n",
    "    @property\n",
    "    def text(self):\n",
    "        return self.review.text\n",
    "\n",
    "    @property\n",
    "    def date(self):\n",
    "        return parser.parse(self.review.date, ignoretz=True)\n",
    "\n",
    "    @property\n",
    "    def source(self):\n",
    "        return 'IMDB'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### IMDB Reviews: AFTER\n",
    "\n",
    "[commit 23888d9](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/23888d9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def main(title):\n",
    "    reviews = []\n",
    "    \n",
    "    for backend_class in (TwitterReviews, IMDBReviews):\n",
    "        with backend_class(title) as reviews_backend:\n",
    "            for review in reviews_backend.reviews:\n",
    "                reviews.append(review)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### BaseReview class\n",
    "\n",
    "[commit 4a314cd](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/4a314cd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "class BaseReview(object):\n",
    "    def __init__(self, review):\n",
    "        self.review = review\n",
    "\n",
    "    @property\n",
    "    def author(self):\n",
    "        raise NotImplementedError\n",
    "\n",
    "    @property\n",
    "    def summary(self):\n",
    "        raise NotImplementedError\n",
    "\n",
    "    @property\n",
    "    def text(self):\n",
    "        raise NotImplementedError\n",
    "\n",
    "    @property\n",
    "    def date(self):\n",
    "        raise NotImplementedError\n",
    "\n",
    "    @property\n",
    "    def source(self):\n",
    "        raise NotImplementedError\n",
    "\n",
    "    def display(self):\n",
    "        print '(%s) @%s: %s [Source: %s]' % (\n",
    "            self.date.strftime('%Y-%m-%d'),\n",
    "            self.author,\n",
    "            self.summary,\n",
    "            self.source)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding more backends\n",
    "\n",
    "* Now that we've refactored using new abstractions we can easily add more review sources\n",
    "* Let's add NY Times movie reviews to the mix\n",
    "\n",
    "[commit 801f26c](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/801f26c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "\n",
    "import requests\n",
    "from dateutil import parser\n",
    "\n",
    "BaseReview = object  # ignore this line, it's just for Jupyter notebook\n",
    "\n",
    "\n",
    "class NYTimesReviews(object):\n",
    "    def __init__(self, movie, limit=10, language='en'):\n",
    "        self.movie = movie\n",
    "        self.limit = 10\n",
    "        self.language = language\n",
    "        self.url = 'https://api.nytimes.com/svc/movies/v2/reviews/search.json'\n",
    "        self.api_key = os.environ.get('NY_TIMES_API_KEY')\n",
    "\n",
    "    def __enter__(self):\n",
    "        return self\n",
    "\n",
    "    def __exit__(self, exc_type, exc_val, exc_tb):\n",
    "        if exc_val:\n",
    "            logging.exception(str(exc_val))\n",
    "\n",
    "    @property\n",
    "    def reviews(self):\n",
    "        return [Review(result) for result in self._get_results()]\n",
    "\n",
    "    def _get_results(self):\n",
    "        response = requests.get(self.url, self._prepare_request_data())\n",
    "        return response.json()['results']\n",
    "\n",
    "    def _prepare_request_data(self):\n",
    "        data = {\n",
    "            'query': self.movie,\n",
    "            'api-key': self.api_key\n",
    "        }\n",
    "        return data\n",
    "\n",
    "\n",
    "class Review(BaseReview):\n",
    "    @property\n",
    "    def author(self):\n",
    "        return self.review['byline']\n",
    "\n",
    "    @property\n",
    "    def summary(self):\n",
    "        return self.review['headline']\n",
    "\n",
    "    @property\n",
    "    def text(self):\n",
    "        return self.review['summary_short']\n",
    "\n",
    "    @property\n",
    "    def date(self):\n",
    "        return parser.parse(self.review['date_updated'], ignoretz=True)\n",
    "\n",
    "    @property\n",
    "    def source(self):\n",
    "        return 'NYTimes'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### NY Times Reviews: BEFORE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def main(title):\n",
    "    reviews = []\n",
    "\n",
    "    for backend_class in (TwitterReviews, IMDBReviews):\n",
    "        with backend_class(title) as reviews_backend:\n",
    "            for review in reviews_backend.reviews:\n",
    "                reviews.append(review)\n",
    "\n",
    "    # Search NYTimes\n",
    "    url = \"https://api.nytimes.com/svc/movies/v2/reviews/search.json\"\n",
    "    data = {\n",
    "        'query': title,\n",
    "        'api-key': os.environ.get('NY_TIMES_API_KEY')\n",
    "    }\n",
    "    response = requests.get(url, data)\n",
    "    count = 0\n",
    "    for review in response.json()['results']:\n",
    "        if count > 9:\n",
    "            break\n",
    "        reviews.append({\n",
    "            'author': review['byline'],\n",
    "            'summary': review['headline'],\n",
    "            'text': review['summary_short'],\n",
    "            'date': parser.parse(review['date_updated'], ignoretz=True),\n",
    "            'source': 'NYTimes'\n",
    "        })\n",
    "        count += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### NY Times Reviews: AFTER\n",
    "\n",
    "[commit 49079b6](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/49079b6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def main(title):\n",
    "    reviews = []\n",
    "\n",
    "    for backend_class in (TwitterReviews, IMDBReviews, NYTimesReviews):\n",
    "        with backend_class(title) as reviews_backend:\n",
    "            for review in reviews_backend.reviews:\n",
    "                reviews.append(review)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Fix sorting\n",
    "\n",
    "* `cmp` functions are deprecated in Python 3 and are hard to read\n",
    "* Let's use `key` function instead\n",
    "* And separate getting sorted reviews into a service function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Sorting: BEFORE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def _cmprev(r1, r2):\n",
    "    if r1['date']>r2['date']:\n",
    "        return -1\n",
    "    elif r1['date']<r2['date']:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Sorting and Services: AFTER\n",
    "\n",
    "[commit fa75a8f](https://github.com/littlepea/python-refactoring-talk/pull/1/commits/fa75a8f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def get_reviews(title):\n",
    "    reviews = []\n",
    "    for backend_class in (TwitterReviews, IMDBReviews, NYTimesReviews):\n",
    "        with backend_class(title) as reviews_backend:\n",
    "            for review in reviews_backend.reviews:\n",
    "                reviews.append(review)\n",
    "\n",
    "    return reversed(sorted(reviews, key=_get_review_date))\n",
    "\n",
    "\n",
    "def _get_review_date(review):\n",
    "    return review.date"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Final: BEFORE\n",
    "\n",
    "[movie_reviews.py](https://github.com/littlepea/python-refactoring-talk/blob/master/before/movie_reviews.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"Movie Reviews.\n",
    "\n",
    "Usage:  movie_reviews.py <title>\n",
    "        movie_reviews.py (-h | --help)\n",
    "        movie_reviews.py --version\n",
    "\n",
    "Arguments:\n",
    "  <title>         Movie title\n",
    "\n",
    "Options:\n",
    "  -h --help     Show this screen.\n",
    "  --version     Show version.\n",
    "\"\"\"\n",
    "\n",
    "from docopt import docopt\n",
    "from TwitterSearch import *\n",
    "from dateutil import parser\n",
    "from imdbpie import Imdb\n",
    "import logging\n",
    "import os\n",
    "\n",
    "def main(title):\n",
    "    reviews = []\n",
    "\n",
    "    # Search tweets\n",
    "    ts = TwitterSearch(\n",
    "        consumer_key = os.environ.get('TWITTER_CONSUMER_KEY'),\n",
    "        consumer_secret = os.environ.get('TWITTER_CONSUMER_SECRET'),\n",
    "        access_token = os.environ.get('TWITTER_ACCESS_TOKEN'),\n",
    "        access_token_secret = os.environ.get('TWITTER_TOKEN_SECRET')\n",
    "    )\n",
    "    try:\n",
    "        ts.connect()\n",
    "\n",
    "        tso = TwitterSearchOrder() # create a TwitterSearchOrder object\n",
    "        tso.setKeywords(['#' + title + 'Movie']) # let's define all words we would like to have a look for\n",
    "        tso.setLanguage('en') # we want to see German tweets only\n",
    "        tso.setIncludeEntities(False) # and don't give us all those entity information\n",
    "\n",
    "        # add tweets to reviews list\n",
    "        results = ts.getSearchResults(tso)\n",
    "\n",
    "    except TwitterSearchException as e: # take care of all those ugly errors if there are some\n",
    "        logging.exception(str(e))\n",
    "        ts.cleanUp()\n",
    "    else:\n",
    "        for offset in range(results.getSize()):\n",
    "            if offset > 9:\n",
    "                break\n",
    "            tweet = results.getTweetByIndex(offset)\n",
    "            reviews.append({\n",
    "                'author': tweet.getUserName(),\n",
    "                'summary': tweet.getText(),\n",
    "                'text': tweet.getText(),\n",
    "                'date': parser.parse(tweet.getCreatedDate(), ignoretz=True),\n",
    "                'source': 'Twitter'\n",
    "            })\n",
    "    finally:\n",
    "        ts.disconnect()\n",
    "\n",
    "    # Search Imdb\n",
    "    imdb = Imdb()\n",
    "    try:\n",
    "        response = imdb.search_for_title(title)[0]\n",
    "        title_id = response['imdb_id']\n",
    "        response = imdb.get_title_reviews(title_id, max_results=10)\n",
    "    except IndexError as e:\n",
    "        logging.exception(str(e))\n",
    "    else:\n",
    "        for review in response:\n",
    "            reviews.append({\n",
    "                'author': review.username,\n",
    "                'summary': review.summary,\n",
    "                'text': review.text,\n",
    "                'date': parser.parse(review.date, ignoretz=True),\n",
    "                'source': 'IMDB'\n",
    "            })\n",
    "\n",
    "    # Search NYTimes\n",
    "    url = \"https://api.nytimes.com/svc/movies/v2/reviews/search.json\"\n",
    "    data = {\n",
    "        'query': title,\n",
    "        'api-key': os.environ.get('NY_TIMES_API_KEY')\n",
    "    }\n",
    "    response = requests.get(url, data)\n",
    "    count = 0\n",
    "    for review in response.json()['results']:\n",
    "        if count > 9:\n",
    "            break\n",
    "        reviews.append({\n",
    "            'author': review['byline'],\n",
    "            'summary': review['headline'],\n",
    "            'text': review['summary_short'],\n",
    "            'date': parser.parse(review['date_updated'], ignoretz=True),\n",
    "            'source': 'NYTimes'\n",
    "        })\n",
    "        count += 1\n",
    "\n",
    "    # Sort reviews by date\n",
    "    reviews.sort(cmp=_cmprev)\n",
    "\n",
    "    # Print reviews\n",
    "    for review in reviews:\n",
    "        print('(%s) @%s: %s [Source: %s]' % ( review['date'].strftime('%Y-%m-%d'), review['author'], review['summary'], review['source'] ) )\n",
    "\n",
    "def _cmprev(r1, r2):\n",
    "    if r1['date']>r2['date']:\n",
    "        return -1\n",
    "    elif r1['date']<r2['date']:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Final `main()` program: AFTER\n",
    "\n",
    "[movie_reviews.py](https://github.com/littlepea/python-refactoring-talk/blob/develop/after/movie_reviews.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def main(title):\n",
    "    for review in get_reviews(title):\n",
    "        review.display()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Final code structure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "* [backends/](https://github.com/littlepea/python-refactoring-talk/tree/develop/after/backends)\n",
    "  * base.py\n",
    "  * imdb.py\n",
    "  * ny_times.py\n",
    "  * twitter.py\n",
    "* [movie_reviews.py](https://github.com/littlepea/python-refactoring-talk/tree/develop/after/movie_reviews.py)\n",
    "* [services.py](https://github.com/littlepea/python-refactoring-talk/tree/develop/after/services.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "---\n",
    "\n",
    "### References:\n",
    "\n",
    "* [Pull Request](https://github.com/littlepea/python-refactoring-talk/pull/1/files?diff=unified)\n",
    "* [Commits](https://github.com/littlepea/python-refactoring-talk/pull/1/commits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Q & A\n",
    "\n",
    "---\n",
    "\n",
    "### Evgeny Demchenko\n",
    "\n",
    "* Twitter/Github: [@littlepea](https://twitter.com/littlepea12)\n",
    "* Email: [evgeny.demchenko@gmail.com](mailto:evgeny.demchenko@gmail.com)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}