Week8/notebooks/intro2pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<DIV ALIGN=CENTER>\n",
    "\n",
    "# Introduction to Pandas\n",
    "## Professor Robert J. Brunner\n",
    "  \n",
    "</DIV>  \n",
    "-----\n",
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "One of the early criticisms of many in the data science arena of the\n",
    "Python language was the lack of useful data structures for performing\n",
    "data analysis tasks. This stemmed in part from comparisons between the R\n",
    "language and Python, since R has a built-in _DataFrame_ object that\n",
    "greatly simplified many data analysis tasks. This deficiency was\n",
    "addressed in 2008 by Wes McKinney with the creation of [Pandas][1] (the\n",
    "name was originally an abbreviation of Panel datas). To quote the Pandas\n",
    "documentation:\n",
    "\n",
    ">Python has long been great for data munging and preparation, but less\n",
    ">so for data analysis and modeling. pandas helps fill this gap, enabling\n",
    ">you to carry out your entire data analysis workflow in Python without\n",
    ">having to switch to a more domain specific language like R.\n",
    "\n",
    "Pandas introduces several new data structures like the `Series`,\n",
    "`DataFrame`, and `Panel` that build on top of existing\n",
    "tools like `numpy` to speed-up data analysis tasks. Pandas also provides\n",
    "efficient mechanisms for moving data between in memory representations\n",
    "and different data formats including CSV and text files, JSON files, SQL\n",
    "databases, HDF5 format files, and even Excel spreadsheets. Pandas also\n",
    "provides support for dealing with missing or incomplete data and\n",
    "aggregating or grouping data.\n",
    "\n",
    "-----\n",
    "[1]: http://pandas.pydata.org"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Brief introduction to Pandas\n",
    "\n",
    "Before using Pandas, we must first import the Pandas library:\n",
    "\n",
    "    import pandas as pd\n",
    "\n",
    "Second, we simply create and use the appropriate Pandas data structure.\n",
    "The two most important data structures for typical data science tasks\n",
    "are the `Series` and the `DataFrame`:\n",
    "\n",
    "1. `Series`: a one-dimensional labeled array that can hold any data type\n",
    "such as integers, floating-point numbers, strings, or Python objects. A\n",
    "`Series` has both a data column and a label column called the _index_.\n",
    "\n",
    "2. `DataFrame`: a two-dimensional labeled data structure with columns\n",
    "that can be hold different data types, similar to a spreadsheet or\n",
    "relational database table. \n",
    "\n",
    "Pandas also provides a date/time data structure sometimes refereed to as\n",
    "a `TimeSeries` and a three-dimensional data structure known as a\n",
    "`Panel`. \n",
    "\n",
    "### `Series`\n",
    "\n",
    "A `Series` is useful to hold data that should be accesible by using a\n",
    "specific label. You create a `Series` by passing in an appropriate data\n",
    "set along with an optional index:\n",
    "\n",
    "    values = pd.Series(data, index=idx)\n",
    "\n",
    "The index varies depending on the type of data passed in to create the\n",
    "`Series:\n",
    "\n",
    "- if data is a NumPy array, the index should be the same length as the\n",
    "data array. If no index is provided one will be created that enables\n",
    "integer access that mirrors a traditional NumPy array (i.e., zero\n",
    "indexed). \n",
    "\n",
    "- if data is a Python dictionary, `idx` can contain specific labels to\n",
    "indicate which values in the dictionary should be used to create the\n",
    "`Series`. If no index is specified, an index is created from the sorted\n",
    "dictionary keys. \n",
    "\n",
    "- if data is a scalar value, an inde must be supplied. In this case, the\n",
    "scalar value will be repeated to ensure that each label in the index has\n",
    "a value in the `Series`.\n",
    "\n",
    "These different options are demonstrated in the following code cells.\n",
    "\n",
    "-----\n",
    "[df]: http://pandas.pydata.org/pandas-docs/stable/dsintro.html#dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "q    0.721990\n",
      "w    0.236611\n",
      "e    0.994790\n",
      "r    0.418525\n",
      "t    0.775141\n",
      "y    0.917286\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# We label the random values\n",
    "s1 = pd.Series(np.random.rand(6), index=['q', 'w', 'e', 'r', 't', 'y'])\n",
    "\n",
    "print(s1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "q    11\n",
      "w    21\n",
      "r    41\n",
      "y   NaN\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "d = {'q': 11, 'w': 21, 'e': 31, 'r': 41}\n",
    "\n",
    "# We pick out the q, w, and r keys, but have an undefined y key.\n",
    "s2 = pd.Series(d, index = ['q', 'w', 'r', 'y'])\n",
    "\n",
    "print(s2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "q    42\n",
      "w    42\n",
      "e    42\n",
      "r    42\n",
      "t    42\n",
      "y    42\n",
      "dtype: int64\n",
      "\n",
      "The \"e\" value is  42\n"
     ]
    }
   ],
   "source": [
    "# We create a Series from an integer constant with explicit labels\n",
    "s3 = pd.Series(42, index = ['q', 'w', 'e', 'r', 't', 'y'])\n",
    "\n",
    "print(s3)\n",
    "\n",
    "print('\\nThe \"e\" value is ', s3['e'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "q    0.721990\n",
      "w    0.236611\n",
      "e    0.994790\n",
      "r    0.418525\n",
      "dtype: float64\n",
      "\n",
      "Sum Series:\n",
      "q    42.721990\n",
      "w    42.236611\n",
      "e    42.994790\n",
      "r    42.418525\n",
      "t    42.775141\n",
      "y    42.917286\n",
      "dtype: float64\n",
      "\n",
      "Series operations:\n",
      "q     53.8\n",
      "w    103.8\n",
      "r    203.8\n",
      "y      NaN\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# We can slice like NumPy arrays\n",
    "\n",
    "print(s1[:-2])\n",
    "\n",
    "# We can also perform vectorized operations\n",
    "print('\\nSum Series:')\n",
    "print(s1 + s3)\n",
    "print('\\nSeries operations:')\n",
    "print(s2 * 5 - 1.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "\n",
    "### `DataFrame`\n",
    "\n",
    "The second major data structure that Pandas provdis is he `DataFrame`,\n",
    "which is a two-dimensional array, where each column is effectively a\n",
    "`Series` with a shared index. A DataFrame is a very powerful data\n",
    "structure and provides a nice mapping for a number of different data\n",
    "formats (and storage mechanisms). For example, you can easily read data\n",
    "from a CSV file, a fixed width format text file, a JSOPN file, an HTML\n",
    "file, and HDF file, and a relational database into a Pandas `DataFrame`.\n",
    "This is demonstrated in the next set of code cells, where we extract\n",
    "data from files we created in the [Introduction to Data\n",
    "Formats][df] Notebook.\n",
    "\n",
    "-----\n",
    "[df]: dataformats.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                   airport              city state\n",
      "iata                                              \n",
      "00M               Thigpen        Bay Springs    MS\n",
      "00R   Livingston Municipal        Livingston    TX\n",
      "00V            Meadow Lake  Colorado Springs    CO\n",
      "01G           Perry-Warsaw             Perry    NY\n",
      "01J       Hilliard Airpark          Hilliard    FL\n"
     ]
    }
   ],
   "source": [
    "# Read data from CSV file, and display subset\n",
    "\n",
    "dfa = pd.read_csv('data.csv', delimiter='|', index_col='iata')\n",
    "\n",
    "# We can grab the first five rows, and only extract the 'airport' column\n",
    "print(dfa[['airport', 'city', 'state']].head(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        0                          1            2   3         5         6\n",
      "3372  ZEF            Elkin Municipal        Elkin  NC  36.28002 -80.78607\n",
      "3373  ZER  Schuylkill Cty/Joe Zerbey   Pottsville  PA  40.70645 -76.37315\n",
      "3374  ZPH      Zephyrhills Municipal  Zephyrhills  FL  28.22806 -82.15592\n",
      "3375  ZUN                 Black Rock         Zuni  NM  35.08323 -108.7918\n",
      "3376  ZZV       Zanesville Municipal   Zanesville  OH  39.94446 -81.89211\n"
     ]
    }
   ],
   "source": [
    "# Read data from our JSON file\n",
    "dfb = pd.read_json('data.json')\n",
    "\n",
    "# Grab the last five rows\n",
    "print(dfb[[0, 1, 2, 3, 5, 6]].tail(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "\n",
    "In the previous code cells, we read data first from a delimiter\n",
    "separated value file and second from a JSON file into a Pandas\n",
    "`DataFrame`. In each code cell, we display data contained in the new\n",
    "DataFrame, first by using the `head` method to display the first few\n",
    "rows, and second by using the `tail` method to display the last few\n",
    "lines. For the delimiter separated value file, we explicitly specified\n",
    "the delimiter, which is a vertical bar `|`, the default is to assume a\n",
    "comma as the delimiter. We also explicitly specify the `iata` column\n",
    "should be used as the index column, which is how we can refer to rows in\n",
    "the array. \n",
    "\n",
    "We also explicitly select columns for display in both code cells. In the\n",
    "first code cell, we explicitly name the columns, passing in a list of\n",
    "the names to the DataFrame. Alternatively, in the second code cell, we\n",
    "pass in a list of the column ids, which we must do since we did not\n",
    "create named columns when reading data from the JSON file. The list of\n",
    "integers can be used even if the columns of the array have been assigned\n",
    "names.\n",
    "\n",
    "Pandas includes a tremendous amount of functionality, especially for\n",
    "the `DataFrame`, to learn more, view the [detailed documentation][pdd].\n",
    "Several useful functions are demonstrated below, however, including\n",
    "information summaries, slicing, and column operations on DataFrames.\n",
    "\n",
    "-----\n",
    "\n",
    "[pdd]: http://pandas.pydata.org/pandas-docs/stable/dsintro.html#dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "airport     object\n",
       "city        object\n",
       "state       object\n",
       "country     object\n",
       "lat        float64\n",
       "long       float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Lets look at the datatypes of each column\n",
    "dfa.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lat</th>\n",
       "      <th>long</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>3376.000000</td>\n",
       "      <td>3376.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>40.036524</td>\n",
       "      <td>-98.621205</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>8.329559</td>\n",
       "      <td>22.869458</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>7.367222</td>\n",
       "      <td>-176.646031</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>34.688427</td>\n",
       "      <td>-108.761121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>39.434449</td>\n",
       "      <td>-93.599425</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>43.372612</td>\n",
       "      <td>-84.137519</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>71.285448</td>\n",
       "      <td>145.621384</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               lat         long\n",
       "count  3376.000000  3376.000000\n",
       "mean     40.036524   -98.621205\n",
       "std       8.329559    22.869458\n",
       "min       7.367222  -176.646031\n",
       "25%      34.688427  -108.761121\n",
       "50%      39.434449   -93.599425\n",
       "75%      43.372612   -84.137519\n",
       "max      71.285448   145.621384"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can get a summary of numerical information in the dataframe\n",
    "\n",
    "dfa.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    object\n",
       "1    object\n",
       "2    object\n",
       "3    object\n",
       "4    object\n",
       "5    object\n",
       "6    object\n",
       "dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Notice the JSON data did not automatically specify data types\n",
    "dfb.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>3377</td>\n",
       "      <td>3377</td>\n",
       "      <td>3377</td>\n",
       "      <td>3377</td>\n",
       "      <td>3377</td>\n",
       "      <td>3377.000000</td>\n",
       "      <td>3377.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>3377</td>\n",
       "      <td>3246</td>\n",
       "      <td>2677</td>\n",
       "      <td>58</td>\n",
       "      <td>6</td>\n",
       "      <td>3376.000000</td>\n",
       "      <td>3376.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>GUY</td>\n",
       "      <td>Municipal</td>\n",
       "      <td>NA</td>\n",
       "      <td>AK</td>\n",
       "      <td>USA</td>\n",
       "      <td>41.610333</td>\n",
       "      <td>-88.915616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>12</td>\n",
       "      <td>263</td>\n",
       "      <td>3372</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           0          1     2     3     4            5            6\n",
       "count   3377       3377  3377  3377  3377  3377.000000  3377.000000\n",
       "unique  3377       3246  2677    58     6  3376.000000  3376.000000\n",
       "top      GUY  Municipal    NA    AK   USA    41.610333   -88.915616\n",
       "freq       1          5    12   263  3372     2.000000     2.000000"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This affects the output of the describe method, dfb has no numerical data types.\n",
    "\n",
    "dfb.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                 airport              city state country        lat  \\\n",
      "iata                                                                  \n",
      "00V          Meadow Lake  Colorado Springs    CO     USA  38.945749   \n",
      "11R    Brenham Municipal           Brenham    TX     USA  30.219000   \n",
      "12C   Rochelle Municipal          Rochelle    IL     USA  41.893001   \n",
      "\n",
      "            long  \n",
      "iata              \n",
      "00V  -104.569893  \n",
      "11R   -96.374278  \n",
      "12C   -89.078290  \n"
     ]
    }
   ],
   "source": [
    "# We can slice out rows using the indicated index for dfa\n",
    "\n",
    "print(dfa.loc[['00V', '11R', '12C']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       0                   1         2   3    4         5         6\n",
      "100  11J        Early County   Blakely  GA  USA  31.39699 -84.89526\n",
      "101  11R   Brenham Municipal   Brenham  TX  USA    30.219 -96.37428\n",
      "102  12C  Rochelle Municipal  Rochelle  IL  USA    41.893 -89.07829\n",
      "103  12D     Tower Municipal     Tower  MN  USA  47.81833 -92.29167\n",
      "104  12J   Brewton Municipal   Brewton  AL  USA  31.05126 -87.06797\n"
     ]
    }
   ],
   "source": [
    "# We can slice out rows using the row index for dfb\n",
    "print(dfb[100:105])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       airport      city state country        lat        long\n",
      "iata                                                         \n",
      "ADK       Adak      Adak    AK     USA  51.877964 -176.646031\n",
      "AKA       Atka      Atka    AK     USA  52.220348 -174.206350\n",
      "GAM    Gambell   Gambell    AK     USA  63.766766 -171.732824\n",
      "SNP   St. Paul  St. Paul    AK     USA  57.167333 -170.220444\n",
      "SVA   Savoonga  Savoonga    AK     USA  63.686394 -170.492636\n"
     ]
    }
   ],
   "source": [
    "# We can also select rows based on boolean tests on columns\n",
    "print(dfa[(dfa.lat > 48) & (dfa.long < -170)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "\n",
    "We can also perform numerical operations on a `DataFrame`, just as was the\n",
    "case with NumPy arrays. To demonstrate this, we create a numerical\n",
    "DataFrame, apply different operations, and view the results.\n",
    "\n",
    "-----"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          0         1         2         3         4         5\n",
      "0 -0.081686 -0.778009 -0.640970  0.996453  1.142358 -0.688399\n",
      "1  0.184744  0.482777  1.294283 -0.311165 -0.365347  0.820360\n",
      "2  0.600033 -0.946207 -0.539372  0.097345  0.901086  0.009482\n",
      "3 -0.691284 -1.001558 -0.072856  0.691576 -1.336217 -0.580029\n",
      "4 -0.801537 -0.101565  1.392538  0.532039  1.624052  0.701913\n"
     ]
    }
   ],
   "source": [
    "df = pd.DataFrame(np.random.randn(5, 6))\n",
    "\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.418314</td>\n",
       "      <td>1.721991</td>\n",
       "      <td>1.859030</td>\n",
       "      <td>3.496453</td>\n",
       "      <td>3.642358</td>\n",
       "      <td>1.811601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.684744</td>\n",
       "      <td>2.982777</td>\n",
       "      <td>3.794283</td>\n",
       "      <td>2.188835</td>\n",
       "      <td>2.134653</td>\n",
       "      <td>3.320360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.100033</td>\n",
       "      <td>1.553793</td>\n",
       "      <td>1.960628</td>\n",
       "      <td>2.597345</td>\n",
       "      <td>3.401086</td>\n",
       "      <td>2.509482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.808716</td>\n",
       "      <td>1.498442</td>\n",
       "      <td>2.427144</td>\n",
       "      <td>3.191576</td>\n",
       "      <td>1.163783</td>\n",
       "      <td>1.919971</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.698463</td>\n",
       "      <td>2.398435</td>\n",
       "      <td>3.892538</td>\n",
       "      <td>3.032039</td>\n",
       "      <td>4.124052</td>\n",
       "      <td>3.201913</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0         1         2         3         4         5\n",
       "0  2.418314  1.721991  1.859030  3.496453  3.642358  1.811601\n",
       "1  2.684744  2.982777  3.794283  2.188835  2.134653  3.320360\n",
       "2  3.100033  1.553793  1.960628  2.597345  3.401086  2.509482\n",
       "3  1.808716  1.498442  2.427144  3.191576  1.163783  1.919971\n",
       "4  1.698463  2.398435  3.892538  3.032039  4.124052  3.201913"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can incorporate operate with basic scalar values\n",
    "\n",
    "df + 2.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.581686</td>\n",
       "      <td>4.278009</td>\n",
       "      <td>4.140970</td>\n",
       "      <td>2.503547</td>\n",
       "      <td>2.357642</td>\n",
       "      <td>4.188399</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.315256</td>\n",
       "      <td>3.017223</td>\n",
       "      <td>2.205717</td>\n",
       "      <td>3.811165</td>\n",
       "      <td>3.865347</td>\n",
       "      <td>2.679640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.899967</td>\n",
       "      <td>4.446207</td>\n",
       "      <td>4.039372</td>\n",
       "      <td>3.402655</td>\n",
       "      <td>2.598914</td>\n",
       "      <td>3.490518</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.191284</td>\n",
       "      <td>4.501558</td>\n",
       "      <td>3.572856</td>\n",
       "      <td>2.808424</td>\n",
       "      <td>4.836217</td>\n",
       "      <td>4.080029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.301537</td>\n",
       "      <td>3.601565</td>\n",
       "      <td>2.107462</td>\n",
       "      <td>2.967961</td>\n",
       "      <td>1.875948</td>\n",
       "      <td>2.798087</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0         1         2         3         4         5\n",
       "0  3.581686  4.278009  4.140970  2.503547  2.357642  4.188399\n",
       "1  3.315256  3.017223  2.205717  3.811165  3.865347  2.679640\n",
       "2  2.899967  4.446207  4.039372  3.402655  2.598914  3.490518\n",
       "3  4.191284  4.501558  3.572856  2.808424  4.836217  4.080029\n",
       "4  4.301537  3.601565  2.107462  2.967961  1.875948  2.798087"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# And perform more complex scalar operations\n",
    "\n",
    "-1.0 * df + 3.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.081595</td>\n",
       "      <td>-0.701863</td>\n",
       "      <td>-0.597973</td>\n",
       "      <td>0.839549</td>\n",
       "      <td>0.909616</td>\n",
       "      <td>-0.635301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.183695</td>\n",
       "      <td>0.464241</td>\n",
       "      <td>0.962013</td>\n",
       "      <td>-0.306167</td>\n",
       "      <td>-0.357274</td>\n",
       "      <td>0.731392</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.564669</td>\n",
       "      <td>-0.811203</td>\n",
       "      <td>-0.513597</td>\n",
       "      <td>0.097191</td>\n",
       "      <td>0.784001</td>\n",
       "      <td>0.009482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.637527</td>\n",
       "      <td>-0.842312</td>\n",
       "      <td>-0.072791</td>\n",
       "      <td>0.637752</td>\n",
       "      <td>-0.972612</td>\n",
       "      <td>-0.548049</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.718426</td>\n",
       "      <td>-0.101391</td>\n",
       "      <td>0.984154</td>\n",
       "      <td>0.507292</td>\n",
       "      <td>0.998582</td>\n",
       "      <td>0.645679</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0         1         2         3         4         5\n",
       "0 -0.081595 -0.701863 -0.597973  0.839549  0.909616 -0.635301\n",
       "1  0.183695  0.464241  0.962013 -0.306167 -0.357274  0.731392\n",
       "2  0.564669 -0.811203 -0.513597  0.097191  0.784001  0.009482\n",
       "3 -0.637527 -0.842312 -0.072791  0.637752 -0.972612 -0.548049\n",
       "4 -0.718426 -0.101391  0.984154  0.507292  0.998582  0.645679"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can also apply vectorized functions\n",
    "\n",
    "np.sin(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.081686</td>\n",
       "      <td>0.184744</td>\n",
       "      <td>0.600033</td>\n",
       "      <td>-0.691284</td>\n",
       "      <td>-0.801537</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.778009</td>\n",
       "      <td>0.482777</td>\n",
       "      <td>-0.946207</td>\n",
       "      <td>-1.001558</td>\n",
       "      <td>-0.101565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.640970</td>\n",
       "      <td>1.294283</td>\n",
       "      <td>-0.539372</td>\n",
       "      <td>-0.072856</td>\n",
       "      <td>1.392538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.996453</td>\n",
       "      <td>-0.311165</td>\n",
       "      <td>0.097345</td>\n",
       "      <td>0.691576</td>\n",
       "      <td>0.532039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.142358</td>\n",
       "      <td>-0.365347</td>\n",
       "      <td>0.901086</td>\n",
       "      <td>-1.336217</td>\n",
       "      <td>1.624052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-0.688399</td>\n",
       "      <td>0.820360</td>\n",
       "      <td>0.009482</td>\n",
       "      <td>-0.580029</td>\n",
       "      <td>0.701913</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0         1         2         3         4\n",
       "0 -0.081686  0.184744  0.600033 -0.691284 -0.801537\n",
       "1 -0.778009  0.482777 -0.946207 -1.001558 -0.101565\n",
       "2 -0.640970  1.294283 -0.539372 -0.072856  1.392538\n",
       "3  0.996453 -0.311165  0.097345  0.691576  0.532039\n",
       "4  1.142358 -0.365347  0.901086 -1.336217  1.624052\n",
       "5 -0.688399  0.820360  0.009482 -0.580029  0.701913"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can tranpose the dataframe\n",
    "\n",
    "df.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "\n",
    "The above description merely scratches the surface of what you can do\n",
    "with a Pandas `Series` or a `DataFrame`. The best way to learn how to\n",
    "effectively use these data structures is to just do it!\n",
    "\n",
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional References\n",
    "\n",
    "1. [Pandas Documentation][pdd]\n",
    "2. A slightly dated Pandas [tutorial][pdt]\n",
    "-----\n",
    "\n",
    "[pdd]: http://pandas.pydata.org/pandas-docs/stable/index.html\n",
    "[pdt]: http://www.gregreda.com/2013/10/26/intro-to-pandas-data-structures/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}