anscombe.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:15e4b3a4cd0dc406e050cf372865fff4abfa599f3b8a26281d027471d9b08197"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Summary Statistics and Data Exploration\n",
      "\n",
      "In this quick tutorial, we will explore how to use Pandas to do data exploration and form hypotheses on four different datasets, using the power combo of Pandas + Statsmodels (which is essentially R in Python syntax). "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%matplotlib inline"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pandas as pd\n",
      "import statsmodels.api as sm"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Anscombe's quartet\n",
      "\n",
      "Francis Anscombe, an American statistician, devised a quartet of data sets to show the importance of data visualization as part of data analysis. To illustrate this, consider the following data set.\n",
      "\n",
      "### Loading Data from CSV\n",
      "\n",
      "The first step to start interacting with data in Pandas is to load the data into a `DataFrame` - basically a data structure that can operate on rows and columns of data. Use the `pd.read_csv` function to load the data (it is actually a TSV, tab separated data, hence the delimiter choice). Then print the data out as a table so we can inspect it."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df = pd.read_csv('fixtures/data.csv', delimiter='\\t')\n",
      "df"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Can you make any judgements about the data sets? Ask yourselves the following questions:\n",
      "\n",
      "- What are the most important features of each data set?\n",
      "- How are each of the data sets related?\n",
      "- What can you hypothesize about the data?\n",
      "- What can you do to start digging into this data a bit more meaningfully?\n",
      "\n",
      "It's hard - but give it a go; humans aren't good at this kind of analysis, but it's only 11 rows. What techniques are you using to judge the results of the table?\n",
      "\n",
      "### Summary Statistics\n",
      "\n",
      "One common methodology of data exploration is summary statistics. Hopefully in your head you asked exploratory questions like \"what is the biggest and smallest numbers in each data set?\" or \"how close together are the numbers?\". Summary statistics can give us this information."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df.describe()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The `describe` function prints out the count, mean, standard deviation, minimum, maximum as well as the quartile data for every data set in the `DataFrame`. This extremely useful for quickly getting a feel for your data. \n",
      "\n",
      "- What do you notice about this data set?\n",
      "- What is the variance of each data set? (what is the relationship of variance to standard deviation?)\n",
      "\n",
      "These scatter plots obviously belong together, so let's also look at how X and Y correlate for each data set using the `corr` function. (By default this is the Pearson correlation). Note that you could do this on the entire `DataFrame` which would give you a correlation of every variable to every other variable - but that's tough to see, and we already know that the Xn,Yn pairs go together. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df[[u'X1', u'Y1']].corr()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df[[u'X2', u'Y2']].corr()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df[[u'X3', u'Y3']].corr()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df[[u'X4', u'Y4']].corr()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### Regressions\n",
      "\n",
      "The Pearson correlations are all the same, and this gives us an idea that datasets must be positively (and probably linearly) correlated - so let's run some linear regressions on the data sets. (Note that in this tutorial we are using statsmodels, but you could have just as easily used scikit-learn). "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Setup the X and Y variables\n",
      "D = 1\n",
      "y = df['Y%i' % D]\n",
      "X = df['X%i' % D]\n",
      "\n",
      "# Add column of 1s to get constant term for y-intercept\n",
      "X = sm.add_constant(X) \n",
      "\n",
      "# Create model (Ordinary Least Squares) and fit\n",
      "model   = sm.OLS(y, X)\n",
      "result  = model.fit()\n",
      "\n",
      "# Output the Summary\n",
      "print result.summary()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Output the parameters of the model\n",
      "print result.params"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Pretty print a nice looking formula\n",
      "print \"y = %0.3fx + %0.3f\" % (result.params['X%i' % D], result.params['const'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "All of these numbers and models are incredibly important - but we still don't have a good grasp about what this data is. Granted, this data is generated data meant for a specific purpose. But we still need to ask ourselves crucial questions:\n",
      "\n",
      "1. What can we hypothesize about the relationship of each X and Y and the relationship of each data set?\n",
      "2. These investigative results are promising, but what models can we generate from here?\n",
      "\n",
      "### Visualize\n",
      "\n",
      "This dataset was intended to show the importance of visualizations in the formation of hypotheses. Hopefully the point is not lost on performing the summary statistics that they, however important, have not led us to very good questions quite yet. Let's visualize the data now:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "y = 'Y%i' % D\n",
      "X = 'X%i' % D\n",
      "\n",
      "df.plot(kind='scatter', x=X, y=y)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Conclusion\n",
      "\n",
      "Hopefully we've showed how statistical summarization can be acheived easily in Python with pandas, but also that you should visualize in order to hypothesize - and from this you'll get to build more complex models and be able to make predictive statements. \n",
      "\n",
      "The pandas/statsmodels combo is great for exploring data and modeling it interactively. Another option is Tableau - an interactive visualization tool from Stanford. You can get a student license by visiting [Tableau Academics](http://www.tableausoftware.com/academic/students). Georgetown students must put in their @georgetown.edu email addresses, then upload a receipt for the certificate program as proof of enrollment. You'll then get a 1 year license for Tableau. \n",
      "\n",
      "Finally, as you're exploring hypotheses you can generate from these initial exploration, consider the following blogs for inspiration:\n",
      "\n",
      "- [FlowingData](http://flowingdata.com/)\n",
      "- [FiveThirtyEight](http://fivethirtyeight.com/)\n",
      "- [DataKind](http://www.datakind.org/)\n",
      "\n",
      "For more on linear models, see the following Blog posts:\n",
      "\n",
      "- [Linear Regression with Python](http://connor-johnson.com/2014/02/18/linear-regression-with-python/) by Conner Johnson\n",
      "- [Linear Regression from R to Python](http://davidcoallier.com/blog/linear-regression-from-r-to-python/) by David Collier\n",
      "\n",
      "For more on plotting with Pandas:\n",
      "\n",
      "- [Pandas Plotting](http://pandas.pydata.org/pandas-docs/version/0.15.0/visualization.html)\n",
      "\n",
      "For more on Graphical Models of Linear Relationships:\n",
      "\n",
      "- [Graphing Linear Models with Seaborn](http://nbviewer.ipython.org/github/mwaskom/seaborn/blob/master/examples/linear_models.ipynb)"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}