02 Blackbirds.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 02 Blackbirds\n",
    "\n",
    "- [Practice of day 01 techniques](#Practice)\n",
    "- [Setting a column to datetime format](#Setting-a-column-to-datetime-format)\n",
    "- [Introducing groupby](#Introducing-groupby)\n",
    "- [Distribution plots with `distplot`](#Distribution-plots)\n",
    "    -  Includes, using `subplots` to get two graphs in one figure\n",
    "- [Hypothesis testing](#Hypothesis-testing)\n",
    "    -  Using `scipy.stats` to run a t-test\n",
    "- [Box plots](#Boxplots)\n",
    "- [Ordinal data](#Ordinal-data)\n",
    "    -  The age categories happened to make sense in alphabetical order. What if they didn't?\n",
    "- [Time series](#Time-series)\n",
    "    -  Now we've grouped by year we can aggregate by mean to plot a time series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Practice\n",
    "\n",
    "![Blackbird](https://www.rspb.org.uk/globalassets/images/birds-and-wildlife/bird-species-illustrations/blackbird_male_1200x675.jpg?preset=landscape_mobile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "1. Import the `blackbirds.csv` data into a `pandas` dataframe.\n",
    "1. How many rows are there in your dataframe? (Try `len()`)\n",
    "1. Is there a sensible index in the dataframe?\n",
    "1. What do each of the columns represent? What do you think the age values mean?\n",
    "1. Find the mean and standard deviation (`std`) of the wing span and weight columns.\n",
    "1. Use the documentation to check *which* standard deviation you're getting.\n",
    "1. Use the `quantile` function to find the median and the IQR too.\n",
    "1. Is there a relationship between wing span and weight? Visualise it and measure it.\n",
    "1. Use the `hue`, `size`, `style` and `markers` of the `seaborn` [scatterplot function](https://seaborn.pydata.org/generated/seaborn.scatterplot.html) to distinguish between the different kinds of blackbird in your plot.\n",
    "1. Find the mean and standard deviation weight and wing span of adult female and male blackbirds separately.\n",
    "1. What other questions could you ask of this data set?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Q1 and Q2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#blackbirds = pd.read_csv(\"blackbirds.csv\")\n",
    "blackbirds = pd.read_csv(\"https://raw.githubusercontent.com/adaapp/dav-introductionToPandas/master/blackbirds.csv\")\n",
    "len(blackbirds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Q3\n",
    "\n",
    "There isn't anything unique in the columns to index by."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Q4\n",
    "\n",
    "The values in the age column are Juvenile, First year, Adult and Unknown"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Q5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "blackbirds.Weight.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Q6\n",
    "\n",
    "The default `ddof` argument is 1, which means the denominator will be $n-1$, so this is sample standard deviation by default."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Q7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "print(\"Weight: The median is {}, with IQR {}\".format(blackbirds.Weight.quantile(0.5),\n",
    "                                                     blackbirds.Weight.quantile(0.75)-blackbirds.Weight.quantile(0.25)))\n",
    "print(\"Wing: The median is {}, with IQR {}\".format(blackbirds.Wing.quantile(0.5),\n",
    "                                                     blackbirds.Wing.quantile(0.75)-blackbirds.Wing.quantile(0.25)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "But actually,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "blackbirds.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Setting a column to datetime format\n",
    "\n",
    "The `Year` column shouldn't really work like that. If you check `blackbirds.dtypes` you'll see why."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "blackbirds.Year = pd.to_datetime(blackbirds.Year,format=\"%Y\")\n",
    "blackbirds.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Check `blackbirds.dtypes` again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Q8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "blackbirds.plot.scatter(\"Wing\",\"Weight\")\n",
    "blackbirds.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Q9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide_input": false,
    "scrolled": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12,6))\n",
    "\n",
    "sns.scatterplot(data=blackbirds,x=\"Wing\",y=\"Weight\",hue=\"Age\", style=\"Sex\", palette=\"hot\", alpha=0.6);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Q10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "blackbirds.loc[(blackbirds.Sex=='M')&(blackbirds.Age=='A')].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "blackbirds.loc[(blackbirds.Sex=='F')&(blackbirds.Age=='A')].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Introducing groupby\n",
    "\n",
    "But this feels like good opportunity to see the `groupby` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "blackbirds.groupby([\"Sex\",\"Age\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "By itself, `groupby` doesn't do much except make a groupby object. Just like with a pivot table, we need to tell it what to *aggregate* by..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "blackbirds.groupby([\"Age\",\"Sex\"]).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Distribution plots\n",
    "\n",
    "`seaborn` has a `distplot` function the combines a histogram with an estimate of the continuous distribution shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# What error message do you get without the dropna?\n",
    "sns.distplot(blackbirds.Weight.dropna());"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# fig is the whole figure, axs is a list of two sets of axes\n",
    "fig,axs = plt.subplots(1,2)\n",
    "fig.suptitle(\"Distribution of weight and wing span\")\n",
    "# I don't care about the numbers on the y-axis\n",
    "axs[0].get_yaxis().set_visible(False)\n",
    "axs[1].get_yaxis().set_visible(False)\n",
    "# Pass the axes to seaborn to tell it where to plot each graph\n",
    "sns.distplot(blackbirds.Weight.dropna(), bins=10, ax=axs[0])\n",
    "sns.distplot(blackbirds.Wing.dropna(), bins=10, ax=axs[1]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Use `distplot` to compare the distribution of weight and the wing span for female and male blackbirds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(1,2)\n",
    "fig.suptitle(\"Weight and wingspan distribution by sex\")\n",
    "\n",
    "axs[0].get_yaxis().set_visible(False)\n",
    "sns.distplot(blackbirds[blackbirds.Sex=='M'].Wing.dropna(),color=\"goldenrod\", ax=axs[0], label='M', bins=10)\n",
    "sns.distplot(blackbirds[blackbirds.Sex=='F'].Wing.dropna(),color=\"rebeccapurple\", ax=axs[0], label='F', bins=10)\n",
    "\n",
    "axs[1].get_yaxis().set_visible(False)\n",
    "sns.distplot(blackbirds[blackbirds.Sex=='M'].Weight.dropna(),color=\"goldenrod\", ax=axs[1], label='M')\n",
    "sns.distplot(blackbirds[blackbirds.Sex=='F'].Weight.dropna(),color=\"rebeccapurple\", ax=axs[1], label='F')\n",
    "axs[0].legend(loc=\"lower left\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "What does this suggest?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Hypothesis testing\n",
    "\n",
    "It looks like the mean wing span for female blackbirds is different from the mean for males. How should we test that?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The `scipy` package has a function for doing t-tests\n",
    "\n",
    "https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.ttest_ind.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "stats.ttest_ind(blackbirds.loc[blackbirds.Sex == 'M',\"Weight\"].dropna(),\n",
    "                blackbirds.loc[blackbirds.Sex == 'F',\"Weight\"].dropna(),\n",
    "               equal_var=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "What can we conclude? Was this a one or a two-tailed test? Does it matter?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Boxplots\n",
    "\n",
    "We can use grouped boxplots to see how weight and wing span change with age"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Make a figure with two subplots with a shared y-axis\n",
    "fig, axs = plt.subplots(1,2, sharey=True)\n",
    "# axs is a list so we can get the first subplot with ax[0]\n",
    "sns.boxplot(x=\"Wing\",y=\"Age\",data=blackbirds, ax=axs[0], whis=3)\n",
    "# and the second with ax[1]\n",
    "sns.boxplot(x=\"Weight\",y=\"Age\",data=blackbirds, ax=axs[1], whis=2);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Investigate the optional arguments for boxplots. What definition of outlier is used?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Ordinal data\n",
    "\n",
    "It so happened that A, F, J and U worked quite well because they're in alphabetical order. But it would be better to tell `pandas` what order we really mean them to come in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "blackbirds.Age = pd.Categorical(blackbirds.Age, categories=[\"U\",\"J\",\"F\",\"A\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(2,1,sharex=True)\n",
    "sns.boxplot(x=\"Wing\",y=\"Age\",data=blackbirds, ax=axs[0])\n",
    "sns.boxplot(x=\"Weight\",y=\"Age\",data=blackbirds, ax=axs[1]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Time series\n",
    "\n",
    "Let's look at how weight and wing span have varied over time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# A groupby by itself doesn't do very much\n",
    "blackbirds.groupby(by=\"Year\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "blackbirds.groupby(by=\"Year\").mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "blackbirds.groupby(by=\"Year\").mean().plot();"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}