Homework 4 2D Array Analysis and Plotting (actual exercise).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 4: 2D Array Analysis, Plotting, and SciPy\n",
    "\n",
    "In this homework exercise, we'll be analyzing electroencephalography (EEG) data. This data, taken from a study in which subjects listened to auditory tones of two different pitches: 200 Hz and 260 Hz.  There was a bit more to the study, but we'll be focusing only on one aspect of these tones: the 200 Hz tone was played much, much more often than the 260 Hz tone.  The hypothesis here is that we will pick up a different pattern of responses to the **Standard (200 Hz)** tones than the **Deviant (260 Hz)** tones.\n",
    "\n",
    "This data was collected by Berti et al at the Swartz Center at the University of California, San Diego.  I downloaded it from their public repository and processed it a bit for our exercise.  The original dataset, along with more information about the study, can be found at http://headit.ucsd.edu/studies/9d557882-a236-11e2-9420-0050563f2612.  We are working with one electrode of Subject 8's data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Package Imports\n",
    "\n",
    "As you import new packages throughout this exercise, put the import statements here, at the top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the Data\n",
    "\n",
    "This time, all of the data variables are stored in a single file: a **.mat file** at **Homework Sample Data/Homework 4/eeg_data.mat**.  MAT files are standard file format that Matlab saves in, so after this exercise, you'll be able to confidently exchange data with your colleagues who use Matlab! Cool, right?\n",
    "\n",
    "The functions you need to load the data can be found in the **scipy.io** package.  I'll leave it to you to work out how to get the data.  In the end, though, you should have 3 variables:\n",
    "  - **eeg**: This is a 2D NumPy array that has all every single trial, from both conditions (Standard and Deviant). \n",
    "  - **time**: This is a 1D array of stimulus-offset time points.  It tells you how long before or after the stimulus each EEG voltage measurement took place.  \n",
    "  - **labels**: These are the condition labels for each trial.  There are only two values: one for the Standard trials, and one for the Deviant trials.  These labels are in the same order as the trials in the **eeg** matrix.  Unfortunately, you'll need to work out which number represents which trials (Hint: the rarer label is the Deviant condition)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the Mean Evoked Potential\n",
    "\n",
    "First, let's see what the overall data looks like.  Plot the mean EEG voltage over time, regardless of condition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detrend the Data\n",
    "\n",
    "You should see a nice peak around 150 msecs post-stimulus!  But wait, there's also a constant increase in voltage throughout the trial.  Well, This really makes it hard to analyse, and could be because of artifacts in the data.  After long examination, we decide to simply detrend the data to flatten it out and make that peak easier to recognize.\n",
    "\n",
    "Now, NumPy doesn't have many great signal processing functions.  For that, we'll turn to **scipy.signal**!  They have a lot of great functions, and all of them work nicely with NumPy arrays--convenient, right?\n",
    "\n",
    "For full scipy docs: http://docs.scipy.org/doc/scipy/reference/\n",
    "\n",
    "For scipy's **detrend()** function, in its signal processing docs: http://docs.scipy.org/doc/scipy/reference/tutorial/signal.html#detrend\n",
    "\n",
    "Take some time to look around the docs, and come back to them regularly--you never know what nice feature you might have overlooked!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detrend the Mean ERP signal, and plot the detrended version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detrend all of the trials, individually, then plot that mean!\n",
    "\n",
    "Before you do it, though, think: Do you expect the result to be the same or different from removing the trend of the mean?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filter Out Data Beyond certain Time Points\n",
    "Remove the EEG data occuring less than -150 msecs before the stimulus. Also remove all the data and greater than 400 msecs after the stimulus.  (We'll only work with the -150 - 400 msec time period from now on.)  Confirm it worked by plotting the mean response again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Separate the Two Conditions from one another, and plot their mean responses\n",
    "\n",
    "### Which code represents which condition?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create two 2D Matrices, one for each Condition (Standard and Deviant)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot The two conditions' ERPs\n",
    "Make the two lines different colors, and label them in the axis!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot some shaded error bars, to help us see whether where there may be significant differences between the two conditions' responses\n",
    "\n",
    "We'll just plot the **Standard Error of the Mean (SEM)**.  To calculate the SEM:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  1. Calculate the Standard Deviation of the EEG values at each time point.  (hint: you'll end up with two 1D arrays of std values, one for each condition)\n",
    "  2. Divide the Standard Deviations by **the square root of one less the number of trials in the condition**.  \n",
    "  \n",
    "That's it!  Here's how the formula looks, in case my description was confusing.  In the formul below:\n",
    "  - **STD**: Standard Deviation\n",
    "  - **SEM**: Standard Error of the Mean\n",
    "  - **N**: The number of trials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/latex": [
       "\n",
       "$$ SEM = \\frac{STD}{\\sqrt{N - 1}}$$"
      ],
      "text/plain": [
       "<IPython.core.display.Latex object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%latex\n",
    "\n",
    "$$ SEM = \\frac{STD}{\\sqrt{N - 1}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Yes, that's right.  Latex formulas render properly in Jupyter Notebooks!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate the SEM of the EEG Data at each Time Pont for each Condition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Plot the ERP of each Condition again, this time with SEM shaded bars both above and below each condition's mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Select the Peak, and Statistically Compare their Heights Between Conditions\n",
    "\n",
    "Let's simplify this analysis, and say that we're now just interested in finding out whether the EEG response around 140 Hz was significantly different between the two conditions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the Mean EEG Voltage between 130 and 150 msecs post-stimulus for each trial. \n",
    "Let's call this the **N140 Response**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filter out any outlier trials -- trials that are extremely different from the rest."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make a histogram showing the distribution of N140s across all conditions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Remove outlier trials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make a Bar Plot comparing the N140 Response between the two conditions.  \n",
    "\n",
    "Don't forget error bars, also showing the SEM!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Do a Statistical Test comparing N140 Between Responses to See if their means are different.\n",
    "\n",
    "Let's do a **Mann-Whitney U Test**.   It's a test that compares the two conditions without making any assumptions about how each condition is distributed.  \n",
    "\n",
    "Lots of good statistical tests can be found at **scipy.stats**, yet another subpackage of Scipy: http://docs.scipy.org/doc/scipy-0.17.0/reference/stats.html\n",
    "\n",
    "### Are the N140 Responses signficantly different between conditions?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add the statistical values to your bar plot!\n",
    "\n",
    "Even if it isn't a significant difference, it's nice to know what the actual values are when we look at a plot, especially if there are only a few values to be made.  Put the U value and the P value on the bar plot you made earlier, in text!\n",
    "\n",
    "Intro on Text in Matplotlib: http://matplotlib.org/users/text_intro.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sandbox\n",
    "\n",
    "What other analysis steps do you think could/should be done on this data?  Try them out here!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}