Tutorials/PythonNumpyPandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python, Numpy, and Pandas\n",
    "**by [Richard W. Evans](https://sites.google.com/site/rickecon/), May 2017**\n",
    "\n",
    "Python has three main environments in which we will manipulate data. First, Python has its own native data structures that can be effectively used. However, these Python structures are often more general than a researcher might want. `Numpy` provides some extra structure on numerical arrays that is often helpful in scientific computing. For traditional data analysis where the unit of analysis is an obervation, the `pandas` library is a great environment in Python for working with data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Python\n",
    "\n",
    "Python objects consist of the types of the elements within objects (e.g., int, long, float, complex, string) and the types of objects that contain other objects (e.g., list, tuple, set, dictionary) called sequence data types."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Python Element Types: Numerical Types\n",
    "The `type()` built-in function allows the user to check what is the type of an object. The following two examples show the difference between an integer and a float."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "type(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "type(3.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can perform traditional float division."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "15.0 / 4.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "15 / 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can perform integer division, which rounds to the nearest integer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "15 // 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can perform modular division, which gives you the remainder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "7 % 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We won't use complex numbers in this class, but you can create and analyze complex numbers with the `complex()`, `real()`, and `imag()` functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "x = complex(2,3)\n",
    "print(x)\n",
    "print(x.real)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y = 4 + 5j\n",
    "print(y)\n",
    "print(y.imag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Python Element Types: Strings\n",
    "Strings are an important data type. They can be created by enclosing characters in double quotes \"\" or single quotes ''. And you can do different operations on those strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "str1 = \"I love\"\n",
    "str2 = 'the MACSS program'\n",
    "str3 = str1 + ' ' + str2 + '!'\n",
    "print(str3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can pull out particular elements of a string. For example, the 10th element of `str3` is index 9 and should be the \"e\" in \"the\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "str3[9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last element of `str3` should be the exclamation point \"!\" which is index -1, and the second-to-last element should be the \"m\" in \"program\" which is index -2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(str3[-1])\n",
    "print(str3[-2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also pull out slices of strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(str3[2:9])\n",
    "print(str3[:-4])\n",
    "print(str3[-4:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And double colons will give us every nth element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(str3[::2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Python Sequence Types: List\n",
    "A Python `list` is created by enclosing comma-separated values with square brackets []. Entries of a list do not have to be of the same type. Accessing entries of a list uses the same indexing and slicing operations as were demonstrated with strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "my_list = [\"Hello\", 93.8, \"world\", 10]\n",
    "my_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(my_list[0])\n",
    "print(my_list[-1])\n",
    "print(my_list[-2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Common list methods (functions) include `append()`, `insert()`, `remove()`, and `pop()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "next_list = [1,2]\n",
    "print(next_list)\n",
    "next_list.append(4)\n",
    "print(next_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the `.insert(x, y)` function to insert element `y` in position `x` of the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "next_list.insert(2, 3)\n",
    "print(next_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the `.remove(y)` function to remove the first instance of the element `y` from a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "your_list = [1, 'hey', 7, 'hey', 'cool', 36]\n",
    "print(your_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "your_list.remove('hey')\n",
    "print(your_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.pop(x)` function will remove and return the `x`th element of a list. If you leave the argument blank, it gives the last element of the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "num_list = [10, 20, 30, 40, 50]\n",
    "print(num_list)\n",
    "print(num_list.pop(3))\n",
    "print(num_list.pop())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A last note about lists is that they are mutable objects. That is, when you replace, change, add to, or take away from the list, it changes the single instance of that object in the computer's memory. Other objects, such as tuples that we will cover soon (and strings covered previously), are immutable. This distinction is important for functional and object oriented programming. You often want immutable objects as the input to and output of a function. For this reason, tuples are the go-to container object for passing arguments to functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 Python Sequence Types: Set\n",
    "A Python `set` is an unordered collection of distinct objects. Objects can be added\n",
    "to or removed from a set after its creation (mutable). Initialize a set with curly braces { },\n",
    "separating the values by commas, or use set() to create an empty set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gym_members = {'Doe, John', 'Doe, John', 'Smith, Jane', 'Brown, Bob'}\n",
    "gym_members"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like mathematical sets, Python `sets` have operations like `union` and `intersection`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gym_members.intersection({'Brown, Bob', 'Smith, Jane', 'Jones, William'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gym_members.union({'Brown, Bob', 'Smith, Jane', 'Jones, William'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5 Python Sequence Types: Tuple\n",
    "A Python `tuple` (pronounced \"tuh-pul\") is created by enclosing comma-separated values with parenthesis (). Entries of a `tuple` do not have to be of the same type. A `tuple` has fewer built-in operations than a `list`. Also, the tuple is an immutable object in that it cannot be changed after assignment. Any operations that behave like they are changing the `tuple` are actually making copies of the `tuple` with the changes. Accessing entries of a `tuple` uses the same indexing and slicing operations as were demonstrated with `lists` and `strings`.\n",
    "\n",
    "The immutability of the `tuple` makes it the ideal object for passing arguments into functions and returning objects from functions. A tuple can be a collection of any object. It can be a collection of `lists`, `dicts`, `Series`, or `DataFrames`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tup1 = (1, 'three', float(6.2), 'five', int(100))\n",
    "tup1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tup1[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could dig out the fourth element of the string that is the second element of the tuple with some advanced slicing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tup1[1][3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can unpack the contents of a tuple with a comma-separated sequence of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mynumber, sixovertwo, wishheight, siblings, oldage = tup1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "(timetogo, milestoMich) = tup1[3:]\n",
    "print(timetogo)\n",
    "print(milestoMich)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.6 Python Sequence Types: Dictionary\n",
    "Like a `list`, a Python `dict` (dictionary) is an unordered data type. A dictionary stores key-value pairs, called items. The values of a dictionary are indexed by its `keys`. Dictionaries are initialized with curly braces, colons, and commas. Use dict()or {} to create an empty dictionary. Dictionaries are a good way to organize objects that are associated with keywords or names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = {'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada'],\n",
    "        'year': [2000, 2001, 2002, 2001, 2002],\n",
    "        'pop': [1.5, 1.7, 3.6, 2.4, 2.9]}\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can list all the `keys` of a dictionary by using the `.keys()` method. The keys that are returned are sorted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can list all of the values of the `keys` using the `values` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data.values()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can select the values associated with a particular `key` in the `dict.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data['pop']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And you can select particular element values of within the key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data['pop'][-2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Numpy\n",
    "\n",
    "`Numpy` is a powerful Python package for manipulating numerical data structures called arrays. The name `numpy` stands for \"numerical Python\".\n",
    "\n",
    "A vector and a matrix are both examples of arrays. A vector is a one-dimensional array, and a matrix is a two-dimensional array. In numerical methods, one will often want to organize and manipulate data that has many dimensions. Arrays are the ideal Euclidean structure for numerical data.\n",
    "\n",
    "`Numpy` is an essential package for working with numerical data. Even though you can often perform the operations you would like to execute with Python's native data types, `numpy` will often provide the most convenient and efficient functionality.\n",
    "\n",
    "We can create a 1-dimensional `numpy` array by importing numpy and using the `array()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr1 = np.array([8, 4, 6, 0, 2])\n",
    "arr1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A 2-dimensional array looks a little more cumbersome to input manually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr2 = np.array([[1, 2, 3], [4, 5, 6]])\n",
    "arr2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr2.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can slice elements of the array with `numpy`'s indexing. The indexing is such that the first dimension is rows, the second dimension is columns, and every other dimension is not geometrically represented. As with all of Python, the first element's index number is 0. If you want a slice from the `m`th element to the `n`th element, you must use `m: n+1`. Further\n",
    "* a scalar `m` means the `m+1`th element\n",
    "* an empty `:` means the entire dimension\n",
    "* two colons followed by an integer `::p` means every `p`th element\n",
    "* a colon followed by an integer `:n` gives a slice from the first element to the `n`th element.\n",
    "* an integer followd by a colon `m:` gives a slice from the `m+1`th element to the last element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr2[:1, 1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can generate some uniformly distributed random numbers between 0 and 1 to fill a 3-dimensional array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "threeD = np.random.uniform(0, 1, (3, 3, 3))\n",
    "threeD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "threeD.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "threeD[:, :, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Numpy` has a lot of great commands for slicing matrices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.diag(threeD[:, :, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also take noncontiguous and nonlinear slices using Boolean masks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "(threeD[:, :, 0] < 0.3) | (threeD[:, :, 0] > 0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "threeD[:,:,0][(threeD[:, :, 0] < 0.3) | (threeD[:, :, 0] > 0.9)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice here that if you made an identity matrix (=1 on diagonal, =0 otherwise) that had Boolean values (True or False), you could pull out the diagonal elements with that object, exactly as the `np.diag()` function did."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ident_num = np.eye(3)\n",
    "print(ident_num)\n",
    "ident_bool = np.eye(3, dtype=bool)\n",
    "print(ident_bool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "threeD[:, :, 0][ident_bool]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, `numpy` has matrix algebra operations. But `numpy`'s default is elementwise operations. This is what you want, because most numerical operations on arrays are elementwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(arr2)\n",
    "print(arr2.T)\n",
    "newvec = np.array([0.5, 2])\n",
    "print(newvec)\n",
    "np.dot(arr2.T, newvec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(arr2)\n",
    "arr2 + np.ones((2, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr2 + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr2 + np.ones(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following \"broadcasting\" will not work. (More on broadcasting later)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr2 + np.ones(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Pandas\n",
    "\n",
    "`Pandas` is Python library for high-level data structures, created by Wes McKinney. The package name `pandas` is derived from the term \"panel data\". In his book, *Python for Data Analysis*, McKinney (2013) states:\n",
    "> I started building pandas in early 2008 during my tenure at AQR, a quantitative investment management firm. At the time, I had a distinct set of requirements that were not well-addressed by an single tool at my disposal.\n",
    "* Data structures with labeled axes supporting automatic or explicit data alignment. This prevents common errors resulting from misaligned data and working with differently-indexed data coming from different sources.\n",
    "* Integrated time series functionality\n",
    "* The same data structures handle both time series data and non-time series data.\n",
    "* Arithmetic operations and reductions (like summing across an axis) would pass on the metadata (axis labels).\n",
    "* Flexible handling of missing data.\n",
    "* Merge and other relational operations found in popular database databases (SQL-based, for example)\n",
    "\n",
    "> I wanted to be able to do all of these things in one place, preferably in a language well-suited to general purpose software development. Python was a good candidate language for this, but at that time there was not an integrated set of data structures and tools providing this functionality. (p. 111)\n",
    "\n",
    "Pandas two main data structures are the `Series` object and the `DataFrame` object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Pandas: Series\n",
    "A `Series` is a one-dimensional array-like oject containing an array of data (of any `numpy` data type) and an associated array of data labels, called its *index*. [Note: To run many of the pandas operations in the following cells, you will need to execute the `import pandas as pd` command and the `from pandas import Series, DataFrame` command.]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from pandas import Series, DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj = Series([4, 7, -5, 3])\n",
    "obj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can create a `Series` with a customized index, as opposed to the default of simple index numbers, by supplying a list of index labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj2 = Series([4, 7, -5, 3], index=['d', 'b', 'a', 'c'])\n",
    "obj2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj2.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj2.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is these customized indices that set the pandas `Series` object apart from the `numpy` array. With `Series`, you can use values in the index when selecting single vlaues or a set of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj2['a']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj2[['c', 'a', 'd']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Numpy` array operations, such as filtering with a boolean array, scalar multiplication or applying math functions, will preserve the index-value link."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj2 > 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj2[obj2 > 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj2 * 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "np.exp(obj2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.log(obj2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Series` object has many of the same properties as a fixed-length, ordered `dict`. The `Series` is a one-to-one mapping of index values to data values. It can be stubstitutted into many functions that expect a `dict`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "'b' in obj2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "'e' in obj2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data stored as a Python `dict` can be easily transformed into a pandas `Series`. Note in the example below that the object displays with the indices in sorted order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sdata = {'Ohio': 35000, 'Texas': 71000, 'Oregon': 16000, 'Utah': 5000}\n",
    "sdata\n",
    "obj3 = Series(sdata)\n",
    "obj3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we create another object with the `dict` named `sdata`, but we label indices that do not match up exactly with the `keys` of the `dict`, the `Series` object will select the indices that do match up and place the missing value of `NaN` for the indices that do not match up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "states = ['California', 'Ohio', 'Oregon', 'Texas']\n",
    "obj4 = Series(sdata, index=states)\n",
    "obj4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two useful pandas methods (functions) for detecting missing values `NaN`s are the `.isnull()` method and the `.notnull()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pd.isnull(obj4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj4.isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj4.notnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj4[obj4.notnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As was mentioned earlier, one of the main benefits of `pandas` is that its indices are treated as a key associative feature. In contrast to `numpy` arrays, a pandas `Series` will automatically align index numbers in arithmetic operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj3 + obj4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can assign a `.name` attribute to a `Series` object as a whole as well as to the index of the `Series`. This is valuable for labeling data. It is also valuable for efficient manipulation of index values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj4.name = 'population'\n",
    "obj4.index.name = 'state'\n",
    "obj4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can change the index values in place if wanted. You might use this function if your data comes with index values that are not as descriptive as you would like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "obj.index = ['Bob', 'Steve', 'Jeff', 'Ryan']\n",
    "obj"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Pandas: DataFrame\n",
    "McKinney (2013) describes the pandas `DataFrame` object.\n",
    "> A `DataFrame` represents a tabular, spreadsheet-like data structure containing an ordered collection of columns, each of which can be a different value type (numeric, string, boolean, etc.). The `DataFrame` has both row and column index; it can be thought of as a `dict` of `Series` (one for all sharing the same index). Compared with other such `DataFrame`-like structures you may have used before (like R's `data.frame`), row-oriented and column-oriented operations in `DataFrame` are treated roughly symmetrically. (p. 115)\n",
    "\n",
    "The `DataFrame` is the standard data structure that you would think of when using programs like Stata, SAS, or R. As with the univariate `Series` object, the `DataFrame` allows for traditional data analysis facility while interacting with all of Python's other functionality. You will notice that many of the methods available to pandas `DataFrames` are also available in `numpy`. Their methods are usually equivalent, but the advantage of performing operations with the `DataFrame` is its respect for the index values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "data = {'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada'],\n",
    "        'year': [2000, 2001, 2002, 2001, 2002],\n",
    "        'pop': [1.5, 1.7, 3.6, 2.4, 2.9]}\n",
    "frame = DataFrame(data)\n",
    "frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can reorder the columns by passing a list of exactly which columns you would like and in what order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "DataFrame(data, columns=['year', 'state', 'pop'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to change the names of the columns, you can just pass a new list into the `column` attribute of the `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "frame.columns = ['Pop', 'State', 'Year']\n",
    "frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When creating the object, we can pass in a column that is not contained in `data`. This will result in a column with that name filled with missing values `NaN`. Further, we can do the same operations with the `index` labels as we did with the `column` labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(DataFrame(data))\n",
    "frame2 = DataFrame(data, columns=['year', 'state', 'pop', 'debt'],\n",
    "                   index=['one', 'two', 'three', 'four', 'five'])\n",
    "frame2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "frame2.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "frame2.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can retrieve a column from a `DataFrame` as a `Series` object either by using a `dict`-like notation or by attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "frame2['state']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "frame2.state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also create a `Series` from a row from a `DataFrame` by using the `.ix` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "frame2.ix['three']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can fill in the `debt` column values using `numpy` arithmetic operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "frame2.debt = 16.5\n",
    "frame2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "frame2['debt'] = np.arange(5)\n",
    "frame2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following example shows how nicely data can be combined based on index values. Suppose we know some `debt` values that are associated with certain `index` values and we want to incorporate that information into the `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "val = Series([-1.2, -1.5, -1.7], index=['two', 'four', 'five'])\n",
    "val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "frame2.debt = val\n",
    "frame2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create new columns in the `DataFrame` based on other columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "frame2['eastern'] = frame2.state == 'Ohio'\n",
    "frame2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can delete columns using the `del` keyword."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "del frame2['eastern']\n",
    "frame2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can take slices of your data by including a list with the columns you want along with a standard `numpy`-type slicing argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "frame2[['year', 'state', 'pop']][:-2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could also explicitly list the particular observations that you want using a `DataFrame` call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "DataFrame(frame2, columns=['year', 'state', 'pop'], index=['two', 'five'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "* McKinney, Wes, *Python for Data Analysis*, O'Reilly Media, Inc. (2013).\n",
    "* [Python labs](http://www.acme.byu.edu/?page_id=2067), Applied and Computational Mathematics Emphasis (ACME), Brigham Young University."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}