1-planout-intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Experimental Design with PlanOut"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Outline\n",
    "- A simple A/B test\n",
    "- How PlanOut works\n",
    "- Recipes:\n",
    "  - Factorial designs\n",
    "  - Cluster-randomized experiments\n",
    "  - Experiments with multiple units"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Loading up PlanOut\n",
    "This imports basic operators for doing random assignment and `SimpleExperiment`, the base class for logging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "from planout.ops.random import *\n",
    "from planout.experiment import SimpleExperiment\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Anatomy of a simple A/B test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Define new experiments by subclassing `SimpleExperiment`\n",
    "- Implement logic in `assign()` method.\n",
    "- Use `params` object to perform random assignments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class VotingExperiment(SimpleExperiment):\n",
    "  def assign(self, params, userid):\n",
    "    params.button_text = UniformChoice(\n",
    "      choices=[\"I'm a voter\", \"I'm voting\"],\n",
    "      unit=userid\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Using PlanOut\n",
    "Get randomized assignments for input units by creating instances of the class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I'm a voter\n"
     ]
    }
   ],
   "source": [
    "e = VotingExperiment(userid=212)\n",
    "print e.get('button_text')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Using PlanOut\n",
    "Here are the assignments for 10 userids."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user 0 sees: I'm voting\n",
      "user 1 sees: I'm voting\n",
      "user 2 sees: I'm voting\n",
      "user 3 sees: I'm a voter\n",
      "user 4 sees: I'm voting\n",
      "user 5 sees: I'm a voter\n",
      "user 6 sees: I'm voting\n",
      "user 7 sees: I'm a voter\n",
      "user 8 sees: I'm voting\n",
      "user 9 sees: I'm voting\n"
     ]
    }
   ],
   "source": [
    "for i in xrange(10):\n",
    "    e = VotingExperiment(userid=i)\n",
    "    print \"user %s sees: %s\" % (i, e.get('button_text'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To check to see that the experiment is doing what we expect it to, we can simulate assignments for many userids and construct a dataframe with all of the assignments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'button_text': \"I'm voting\"}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "button_text\n",
      "I'm a voter    5000\n",
      "I'm voting     5000\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "sim_users = [VotingExperiment(userid=i).get_params() for i in xrange(10000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)\n",
    "print assignments.groupby(['button_text']).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# How random assignment works in PlanOut"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* PlanOut uses deterministic hashing to perform random assignments\n",
    "* Unless otherwise specified, all random assignments are independent.\n",
    "* Ensures that:\n",
    "  * The same units (e.g., user ids) get mapped to different values for different experiments or parameters.\n",
    "  * Assignments are as good as random."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Underneath the hood\n",
    "\n",
    "PlanOut computes a hash that looks like:\n",
    "\n",
    "```\n",
    "  f(SHA1(experiment_name.parameter_name.unit_id))\n",
    "```\n",
    "So that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class RandomExample1(SimpleExperiment):\n",
    "  def assign(self, params, userid):\n",
    "    params.x = UniformChoice(choices=[0, 1], unit=userid)\n",
    "    params.y = UniformChoice(choices=['a','b'], unit=userid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "uses a hash like:\n",
    "```\n",
    "  SHA1(RandomExample1.x.4) % 2\n",
    "```\n",
    "  \n",
    "to select the value for `x` when the given `userid` is 4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Parameter-level salts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x  y\n",
      "0  a    531\n",
      "   b    475\n",
      "1  a    478\n",
      "   b    516\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "class RandomExample1(SimpleExperiment):\n",
    "  def assign(self, params, userid):\n",
    "    params.x = UniformChoice(choices=[0, 1], unit=userid)\n",
    "    params.y = UniformChoice(choices=['a','b'], unit=userid)\n",
    "    \n",
    "sim_users = [RandomExample1(userid=i).get_params() for i in xrange(2000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)\n",
    "print assignments.groupby(['x', 'y']).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Automatic salting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PlanOut implicitly uses parameter name as salt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x  y\n",
      "0  a    531\n",
      "   b    475\n",
      "1  a    478\n",
      "   b    516\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "class RandomExample1(SimpleExperiment):\n",
    "  def assign(self, params, userid):\n",
    "    params.x = UniformChoice(choices=[0, 1], unit=userid, salt='x')\n",
    "    params.y = UniformChoice(choices=['a','b'], unit=userid, salt='y')\n",
    "    \n",
    "sim_users = [RandomExample1(userid=i).get_params() for i in xrange(2000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)\n",
    "print assignments.groupby(['x', 'y']).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Changing the salts change the assignments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x  y\n",
      "0  a    507\n",
      "   b    493\n",
      "1  a    506\n",
      "   b    494\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "class RandomExample1(SimpleExperiment):\n",
    "  def assign(self, params, userid):\n",
    "    params.x = UniformChoice(choices=[0, 1], unit=userid, salt='x2')\n",
    "    params.y = UniformChoice(choices=['a','b'], unit=userid, salt='y2')\n",
    "    \n",
    "sim_users = [RandomExample1(userid=i).get_params() for i in xrange(2000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)\n",
    "print assignments.groupby(['x', 'y']).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Correlated assignments\n",
    "- Parameters with the same salt will have correlated assignments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x  y\n",
      "0  a    1006\n",
      "1  b     994\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "class RandomExample1(SimpleExperiment):\n",
    "  def assign(self, params, userid):\n",
    "    params.x = UniformChoice(choices=[0, 1], unit=userid, salt='x')\n",
    "    params.y = UniformChoice(choices=['a','b'], unit=userid, salt='x')\n",
    "    \n",
    "sim_users = [RandomExample1(userid=i).get_params() for i in xrange(2000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)\n",
    "print assignments.groupby(['x', 'y']).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Experiment-level salts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Using experiment name in salt ensures indepence between experiments\n",
    "- By default, experiment class names are used as experiment-level salts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x  y\n",
      "0  a    1009\n",
      "   b     962\n",
      "1  a    1004\n",
      "   b    1025\n",
      "dtype: int64\n",
      "x  y\n",
      "4  m    1017\n",
      "   n     979\n",
      "8  m    1026\n",
      "   n     978\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "class RandomExample1(SimpleExperiment):\n",
    "  def assign(self, params, userid):\n",
    "    params.x = UniformChoice(choices=[0, 1], unit=userid)\n",
    "    params.y = UniformChoice(choices=['a','b'], unit=userid)\n",
    "    \n",
    "class RandomExample2(SimpleExperiment):\n",
    "  def assign(self, params, userid):\n",
    "    params.x = UniformChoice(choices=[4, 8], unit=userid)\n",
    "    params.y = UniformChoice(choices=['m','n'], unit=userid)\n",
    "\n",
    "sim_users = [RandomExample1(userid=i).get_params() for i in xrange(4000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)\n",
    "print assignments.groupby(['x', 'y']).size()\n",
    "\n",
    "sim_users = [RandomExample2(userid=i).get_params() for i in xrange(4000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)\n",
    "print assignments.groupby(['x', 'y']).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Custom experiment-level salts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x  y\n",
      "4  m    1017\n",
      "   n     979\n",
      "8  m    1026\n",
      "   n     978\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "class RandomExample1(SimpleExperiment):\n",
    "  def setup(self):\n",
    "        self.salt = 'RandomExample2'\n",
    "\n",
    "  def assign(self, params, userid):\n",
    "    params.x = UniformChoice(choices=[0, 1], unit=userid)\n",
    "    params.y = UniformChoice(choices=['a','b'], unit=userid)\n",
    "\n",
    "sim_users = [RandomExample2(userid=i).get_params() for i in xrange(4000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)\n",
    "print assignments.groupby(['x', 'y']).agg(len)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Random assignment with multiple units"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- When multiple units are used, units are concatinated,\n",
    "- e.g., if `userid=4` and `url='http://news.ycombinator.com'`, the hash would be:\n",
    "\n",
    "```\n",
    "    f(SHA1('RandomExperiment1.show_thumbnail.6.http://news.ycombinator.com'))\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class RandomExample1(SimpleExperiment):\n",
    "  def assign(self, params, userid, url):\n",
    "    params.show_thumbnail = BernoulliTrial(p=0.15, unit=[userid, url])\n",
    "\n",
    "RandomExample1(userid=6, url='http://news.ycombinator.com').get('show_thumbnail')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Order matters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class RandomExample1(SimpleExperiment):\n",
    "  def assign(self, params, userid, url):\n",
    "    params.show_thumbnail = BernoulliTrial(p=0.15, unit=[url, userid])\n",
    "\n",
    "RandomExample1(userid=6, url='http://news.ycombinator.com').get('show_thumbnail')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Recipes for experimental designs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Factorial designs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parameter assignments (by default) are performed independently, so a 2x2 design can be constructed simply by setting another parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class VotingExperiment(SimpleExperiment):\n",
    "  def assign(self, params, userid):\n",
    "    params.button_text = UniformChoice(\n",
    "      choices=[\"I'm a voter\", \"I'm voting\"],\n",
    "      unit=userid\n",
    "    )\n",
    "    params.has_social_cues = UniformChoice(\n",
    "      choices=[1, 0],\n",
    "      unit=userid\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### A few examples of assignments produced by the data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   button_text  has_social_cues\n",
      "0   I'm voting                1\n",
      "1   I'm voting                0\n",
      "2   I'm voting                1\n",
      "3  I'm a voter                0\n",
      "4   I'm voting                1\n"
     ]
    }
   ],
   "source": [
    "sim_users = [VotingExperiment(userid=i).get_params() for i in xrange(10000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)\n",
    "print assignments[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Crosstabs: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "button_text  has_social_cues\n",
      "I'm a voter  0                  2485\n",
      "             1                  2515\n",
      "I'm voting   0                  2447\n",
      "             1                  2553\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print assignments.groupby(['button_text', 'has_social_cues']).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Unequal probability assignment with WeightedChoice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `WeightedChoice` operator lets you select choices with different probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class VotingExperiment(SimpleExperiment):\n",
    "  def assign(self, params, userid):\n",
    "    params.button_text = UniformChoice(\n",
    "      choices=[\"I'm a voter\", \"I'm voting\"],\n",
    "      unit=userid\n",
    "    )\n",
    "    params.has_social_cues = WeightedChoice(\n",
    "      choices=[1, 0],\n",
    "      weights=[8, 2],\n",
    "      unit=userid\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Crosstabs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "button_text  has_social_cues\n",
      "I'm a voter  0                  192\n",
      "             1                  797\n",
      "I'm voting   0                  196\n",
      "             1                  815\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "sim_users = [VotingExperiment(userid=i).get_params() for i in xrange(2000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)\n",
    "print assignments.groupby(['button_text', 'has_social_cues']).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Arbitrary cell probabilites in a factorial design"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the case where there are organizational factors that limit the size of your experiment. For example, if in a hypothetical get out the vote experiment at Facebook only 5% of the users could be used as part of an experiment used to evaluate the effects of various encouragements to vote.\n",
    "\n",
    "Because of this, the experimenters decided they wanted to assign users with a matrix of probabilities like:\n",
    "\n",
    "$$ \\left[\n",
    "  \\begin{array}{ l c c }\n",
    "     & \\text{has megaphone} & \\text{no megaphone} \\\\\n",
    "     \\text{has feed} & 0.95 & 0.015 \\\\\n",
    "     \\text{no feed} & 0.02 & 0.015\n",
    "  \\end{array} \\right]$$\n",
    "  \n",
    "There are multiple ways to do this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Method 1: Conditional probabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Method 1 first assigns users to see or not see the megaphone with probability 0.97. Half of those 3% are then assigned to see feed stories about voting, while 98% of those who see megaphones are assigned to the see the stories. This produces the assignment probabilities given about."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class VotingExperiment(SimpleExperiment):\n",
    "  def assign(self, params, userid):\n",
    "    params.has_megaphone = BernoulliTrial(\n",
    "      p=0.97,\n",
    "      unit=userid\n",
    "    )\n",
    "    cond_probs = [0.5, 0.98]\n",
    "    params.has_feed = BernoulliTrial(\n",
    "      p=cond_probs[params.has_megaphone],\n",
    "      unit=userid\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "has_feed  has_megaphone\n",
      "0         0                 147\n",
      "          1                 198\n",
      "1         0                 142\n",
      "          1                9513\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "sim_users = [VotingExperiment(userid=i).get_params() for i in xrange(10000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)\n",
    "print assignments.groupby(['has_feed', 'has_megaphone']).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Method 2: Binary encoding of cell positions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Method 2 selects the four conditions with their respective probabilities from a flat list. It then picks out elements of an array corresponding to the cell positions. For a very small number of subjects, this method will produce slightly more imbalance, but for larger N it will acheive similar levels of balance compared to method 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class VotingExperiment(SimpleExperiment):\n",
    "  def assign(self, params, userid):\n",
    "    params.idx = WeightedChoice(\n",
    "      choices=[0, 1, 2, 3],\n",
    "      weights=[0.95, 0.02, 0.015, 0.015],\n",
    "      unit=userid\n",
    "    )\n",
    "    params.has_megaphone = [1,1,0,0][params.idx]\n",
    "    params.has_feed = [1,0,1,0][params.idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Crosstabs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "has_feed  has_megaphone\n",
      "0         0                 156\n",
      "          1                 209\n",
      "1         0                 147\n",
      "          1                9488\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "sim_users = [VotingExperiment(userid=i).get_params() for i in xrange(10000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)\n",
    "print assignments.groupby(['has_feed', 'has_megaphone']).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## More than two factor levels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Small changes in appearance can have significant effects on individual behavior. Google is infamous for testing [41 different shades of blue](http://www.nytimes.com/2009/03/01/business/01marissa.html) for their link colors. Let's implement that experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class ColorExperiment(SimpleExperiment):\n",
    "  def assign(self, params, userid):\n",
    "    params.blue_value = RandomInteger(\n",
    "        min=215,\n",
    "        max=255,\n",
    "        unit=userid\n",
    "    )\n",
    "    params.link_color = '#0000%s' % format(params.blue_value, '02x')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'blue_value': 247, 'link_color': '#0000f7'}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ColorExperiment(userid=10).get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sample data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "sim_users = [ColorExperiment(userid=i).get_params() for i in xrange(20000)]\n",
    "assignments = pd.DataFrame.from_dict(sim_users)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>blue_value</th>\n",
       "      <th>link_color</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td> 231</td>\n",
       "      <td> #0000e7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 248</td>\n",
       "      <td> #0000f8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> 242</td>\n",
       "      <td> #0000f2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 230</td>\n",
       "      <td> #0000e6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td> 239</td>\n",
       "      <td> #0000ef</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   blue_value link_color\n",
       "0         231    #0000e7\n",
       "1         248    #0000f8\n",
       "2         242    #0000f2\n",
       "3         230    #0000e6\n",
       "4         239    #0000ef"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assignments[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEACAYAAABS29YJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAHCFJREFUeJzt3X+sZHd53/H3g43VOG7ZOJS1vXZznYIFW6kyjthESqrc\n",
       "pOA6aWtbqWonSiS7kP7jFkjatNxN1eKqrWOo1NAf8h8thN2ksZMVNJREAbx29lHbP2oKeMFhvbVN\n",
       "uQ1L2XVCcSAJUtf46R9zrj33+s6Z5947Z57v3Pm8pKs953zPzPl8v3fud848c2bWIgIREdnfXlEd\n",
       "QEREhqfJXkRkCWiyFxFZAprsRUSWgCZ7EZEloMleRGQJpCZ7MztgZh8ysyfN7IyZfa+ZXWlmJ83s\n",
       "KTN72MwOjO1/1MyeNrOzZnbzcPFFRCQje2b/r4Hfjog3AH8ROAusAScj4gbg0W4dMzsM3AkcBm4B\n",
       "HjAzvYIQESk0dRI2s1cBfykifgkgIp6PiD8EbgWOd7sdB27vlm8DHoqIixGxDjwDHJl1cBERycuc\n",
       "cV8P/L6ZfdDMPmNm/8HMvh04GBEXun0uAAe75WuAc2O3PwccmlliERHZscxkfylwE/BARNwE/DFd\n",
       "yWZDjL5zoe97F/SdDCIihS5N7HMOOBcR/6Nb/xBwFDhvZldFxHkzuxp4tmv/MnDd2O2v7ba9yMw0\n",
       "+YuI7EJE2G5uZ5kvQjOz/wL8dEQ8ZWb3Apd3TV+NiPeY2RpwICLWujdoH2RUpz8EPAK8NsYOZGax\n",
       "28DzZGb3RsS91TmmUc7ZWoSci5ARlHPW9jJ3Zs7sAd4O/KqZXQZ8AfhbwCXACTN7G7AO3AEQEWfM\n",
       "7ARwBngeuCcyzyhtWqkOkLRSHSBppTpA0kp1gISV6gBJK9UBklaqAwwtNdlHxGeBN23T9OYJ+98H\n",
       "3LeHXCIiMkO6/r3fseoASceqAyQdqw6QdKw6QMKx6gBJx6oDJB2rDjC0VM1+5gddkJq9iEhL9jJ3\n",
       "6sy+h5mtVmfIUM7ZWoSci5ARlLMlmuxFRJaAyjgiIgtCZRwREemlyb7HotTxlHO2FiHnImQE5WyJ\n",
       "JnsRkSWgmr2IyIJQzV5ERHppsu+xKHU85ZytRci5CBlBOVuiyV5EZAmoZi8isiBUsxcRkV6a7Hss\n",
       "Sh1POWdrEXIuQkZQzpZoshcRWQKq2cu+lvn/jvVYlEUxj/+WUGSB9c33mudlOaiM02NR6njKOVuL\n",
       "kHMRMoJytkSTvYjIElDNXva1Uc2+v4yjx6IsCl1nLyIivTTZ91iUOp5yztYi5FyEjKCcLdHVOCJT\n",
       "6PJN2Q+arNnrj0tmZRY1e9X9pRX79Dp7XRst0gqdgM1G5TiqZt9jUep4yjlbi5CzJmP0/GxvEcYS\n",
       "5p1z5+M4C5rsRUSWQKpmb2brwNeBbwEXI+KImV0J/DrwXcA6cEdEPNftfxR4a7f/OyLi4S33l6jZ\n",
       "q0Yqe7cfavYtlFCqx2C/2Os4zuM6+wBWI+KNEXGk27YGnIyIG4BHu3XM7DBwJ3AYuAV4wMz0CkJk\n",
       "T2pe+s+LmUXmp/WMlfmm2ckkvPXZ5FbgeLd8HLi9W74NeCgiLkbEOvAMcIQFpHrj3i3iH0jL4/kS\n",
       "rw6QsrOx7HtCG/Zhks+5uE+6Ozmzf8TMPmVmf7vbdjAiLnTLF4CD3fI1wLmx254DDu05qbzM2GR5\n",
       "qu2JdOOP4RSL9gcibWn9xKHlfNlLL78/Ir5iZn8WOGlmZ8cbI2JaR8p/CbsREV6dYbpFukR1tTpA\n",
       "ymL83lerA6TMfiyHebzPLue0qa7ubzI12UfEV7p/f9/MfoNRWeaCmV0VEefN7Grg2W73LwPXjd38\n",
       "2m7bJmZ2jNEbuwDPAac3D7jz0gN6Y/Pqi61mtrqx/8ZLsGVb3zxW4+Pjm1qr807OtzqXfHs9/ku3\n",
       "2f72G7cZLn/t8ef1eMvef9Xjfe/5cuvjx++W7+52WGcPpl6NY2aXA5dExDfM7NuBh4F/CrwZ+GpE\n",
       "vMfM1oADEbFmozdoH2T0hHAIeAR4bYwdyBbgapzMS67qqw9eGidn+zO9+iskNv8unZfnnMeVLDt7\n",
       "LI1PnLu9j5dn6LfzvwfnpbGc19/Dzsdg61ju/v5HxxhqXugm1VPT95z2ymLYPkybO/tkzuwPAr9h\n",
       "Zhv7/2pEPGxmnwJOmNnb6C69BIiIM2Z2AjgDPA/cE9OeUQrk6menmPxyefg5tLrGJ7O0SOW23Zn0\n",
       "eO3mjvITj5z9+3tq+Ltxhj2zzxyjjVcXbWecpvpV2iyOP5sz+7rbz8LQj8XqM/tchunHX/Qz+yYt\n",
       "QplFRKQVC/xhp3lck+szup+h+a5uNe0ysdmXkXaXc95e/oZci3xHe8//d727nFUW43e+Nwt7Zr/o\n",
       "2qnH798a5SKZz+Oh/3fdzmNShrCwNftMbWzYYwxdH8xlaL0WXF1vXpSa/d7bM4bOoJq9avYiPaad\n",
       "US7Cey/1Z8XtfpjnxQQNfLdN5fGrLXDNfh68OkCST2yZf422j0/YPvR7Lzuzu/rtvPvgA93vrPnY\n",
       "cgu/50nHT1xiv+DKzuzNLv961bGXi2ryIotkqBOxspo9nJ/Q+jngZhahZj/NPGqUlXXWjPbfG1mM\n",
       "evc8asVtt+fuo/69k9r34foU1uwPTtj+nXNNsTe6umF/aL/eLTn6m5tMNftevsfbz6tG6TO+v6F4\n",
       "dYAkrw6Q4NUBknzOx9vt35wPG6sBmuxFRJZAYc1+0nE/A3wPi1Gzb7++p5r9bD6zUf17UB9ayNBG\n",
       "HxawZi+LTvVRkcWhMk4vrw6Q5IXH7quRbn0u8JkffZjve/GZZhyGVwdI8uoASV4dYHA6s5d9QJ8l\n",
       "EJlmX9fsp9v/9b0h64OL8v0+++H3oD60kKGNPqhmvy2d8YmIgGr2U3h1gCQf7J5nWw/3mecbhlcH\n",
       "SPDqAEleHSDJqwMMbp+f2cve6dWRyH6wz2v2Ldfv1Id8+zSL0If98HtQH1rog2r2so/p1YXIXqlm\n",
       "38urAyR5dYAkrw6Q5NUBErw6QJJXB0jy6gCD02QvIrIEVLNvtr2FDOpDGxnUhzYytNGH3dbsdWYv\n",
       "IrIENNn38uoASV4dIMmrAyR5dYAErw6Q5NUBkrw6wOA02YuILAHV7JttbyGD+tBGBvWhjQxt9EE1\n",
       "exERmSg12ZvZJWb2uJn9Zrd+pZmdNLOnzOxhMzswtu9RM3vazM6a2c1DBZ8Prw6Q5NUBkrw6QJJX\n",
       "B0jw6gBJXh0gyasDDC57Zv9O4Awvvb5YA05GxA3Ao906ZnYYuBM4DNwCPGBmevUgIlJs6kRsZtcC\n",
       "Pwq8n5c+m34rcLxbPg7c3i3fBjwUERcjYh14Bjgyy8DztVodIGm1OkDSanWApNXqAAmr1QGSVqsD\n",
       "JK1WBxhc5qz7F4F/ALwwtu1gRFzoli8AB7vla4BzY/udAw7tNaSIiOxN7xehmdlfA56NiMfNbHW7\n",
       "fSJi2nebT2i7G1jplg8AN7L52dXH1r37N9vubLbb9o1jDHX/s2p/H5vHr5V82+2/uoP9q9a3jufG\n",
       "PpP2r2g/DfxM8vbOZvNs39o27+Nn20/P4fi7WXfgWLe+wp5ExMQf4D7gS8AXga8Afwz8CnAWuKrb\n",
       "52rgbLe8BqyN3f7jwPduc78BMeHn09HfHon2zD6Z9lMD3/+s+jAp59AZd9qH7XJWZ9yufWvO1jPO\n",
       "6+9hN+2nZnT/Q/fh1BzGaDZ96Juz+37S19mb2Q8CPxcRf93M3gt8NSLeY2ZrwIGIWOveoH2QUZ3+\n",
       "EPAI8NrYchBdZ59pbyGD+tBGBvWhjQxt9GFe32e/keJ+4ISZvQ1YB+4AiIgzZnaC0ZU7zwP3bJ3o\n",
       "RURk/vQJ2t72U0x+l76lswBn+5ytnck4L89ZnXG7dmdzztYztnxW7IxytnFWPLndgR8a8P5ndR/6\n",
       "BK2IiPTQmX2z7S1kUB/ayKA+tJGhjT7ozF5ERCbSZN/LqwMkeXWAJK8OkOTVARK8OkCSVwdI8uoA\n",
       "g9NkLyKyBFSzb7a9hQzqQxsZ1Ic2MrTRB9XsRURkIk32vbw6QJJXB0jy6gBJXh0gwasDJHl1gCSv\n",
       "DjA4TfYiIktANftm21vIoD60kUF9aCNDG31QzV5ERCbSZN/LqwMkeXWAJK8OkOTVARK8OkCSVwdI\n",
       "8uoAg9NkLyKyBFSzb7a9hQzqQxsZ1Ic2MrTRB9XsRURkIk32vbw6QJJXB0jy6gBJXh0gwasDJHl1\n",
       "gCSvDjA4TfYiIktANftm21vIoD60kUF9aCNDG31QzV5ERCbSZN/LqwMkeXWAJK8OkOTVARK8OkCS\n",
       "VwdI8uoAg9NkLyKyBFSzb7a9hQzqQxsZ1Ic2MrTRB9XsRURkIk32vbw6QJJXB0jy6gBJXh0gwasD\n",
       "JHl1gCSvDjA4TfYiIktANftm21vIoD60kUF9aCNDG31QzV5ERCbSZN/LqwMkeXWAJK8OkOTVARK8\n",
       "OkCSVwdI8uoAg+ud7M3sT5nZY2Z22szOmNkvdNuvNLOTZvaUmT1sZgfGbnPUzJ42s7NmdvPQHRAR\n",
       "kemm1uzN7PKI+BMzuxT4b8DPAbcCfxAR7zWzdwHfERFrZnYYeBB4E3AIeAS4ISJe2HKfqtmrDwvS\n",
       "3kIG9aGNDG30YbCafUT8Sbd4GXAJ8DVGk/3xbvtx4PZu+TbgoYi4GBHrwDPAkd0EExGR2Zk62ZvZ\n",
       "K8zsNHABOBURnwcORsSFbpcLwMFu+Rrg3NjNzzE6w19QXh0gyasDJHl1gCSvDpDg1QGSvDpAklcH\n",
       "GNyl03boSjA3mtmrgE+Y2Q9taY9RWWbyXWy/+W5gpVs+ANwIrI61+9i6d/9m253N9nv76eLjT2rf\n",
       "6/5V61vHc2OfSftXtJ/ewe2dzdT+8vbTU9pncfzdrDtwrFtfYS92dJ29mf1j4JvATwOrEXHezK5m\n",
       "dMb/ejNbA4iI+7v9Pw68OyIe23I/qtmrDwvS3kIG9aGNDG30YZCavZm9euNKGzP7NuAtwOPAR4G7\n",
       "ut3uAj7SLX8U+HEzu8zMrgdeB3xyN8FERGR2ptXsrwZ+p6vZPwb8ZkQ8CtwPvMXMngJ+uFsnIs4A\n",
       "J4AzwMeAe6LiI7oz49UBkrw6QJJXB0jy6gAJXh0gyasDJHl1gMH11uwj4gngpm22/1/gzRNucx9w\n",
       "30zSiYjITOi7cZptbyGD+tBGBvWhjQxt9EHfjSMiIhNpsu/l1QGSvDpAklcHSPLqAAleHSDJqwMk\n",
       "eXWAwWmyFxFZAqrZN9veQgb1oY0M6kMbGdrog2r2IiIykSb7Xl4dIMmrAyR5dYAkrw6Q4NUBkrw6\n",
       "QJJXBxicJnsRkSWgmn2z7S1kUB/ayKA+tJGhjT6oZi8iIhNpsu/l1QGSvDpAklcHSPLqAAleHSDJ\n",
       "qwMkeXWAwWmyFxFZAqrZN9veQgb1oY0M6kMbGdrog2r2IiIykSb7Xl4dIMmrAyR5dYAkrw6Q4NUB\n",
       "krw6QJJXBxicJnsRkSWgmn2z7S1kUB/ayKA+tJGhjT6oZi8iIhNpsu/l1QGSvDpAklcHSPLqAAle\n",
       "HSDJqwMkeXWAwWmyFxFZAqrZN9veQgb1oY0M6kMbGdrog2r2IiIykSb7Xl4dIMmrAyR5dYAkrw6Q\n",
       "4NUBkrw6QJJXBxicJnsRkSWgmn2z7S1kUB/ayKA+tJGhjT6oZi8iIhNpsu/l1QGSvDpAklcHSPLq\n",
       "AAleHSDJqwMkeXWAwWmyFxFZAlNr9mZ2HfDLwGsYFZP+fUT8GzO7Evh14LuAdeCOiHiuu81R4K3A\n",
       "t4B3RMTDW+5TNXv1YUHaW8igPrSRoY0+DFmzvwj8bET8BeD7gL9jZm8A1oCTEXED8Gi3jpkdBu4E\n",
       "DgO3AA+YmV5BiIgUmjoJR8T5iDjdLf8R8CRwCLgVON7tdhy4vVu+DXgoIi5GxDrwDHBkxrnnxKsD\n",
       "JHl1gCSvDpDk1QESvDpAklcHSPLqAIPb0Rm3ma0AbwQeAw5GxIWu6QJwsFu+Bjg3drNzjJ4cRESk\n",
       "yKXZHc3sCuDDwDsj4htmL5WNIiJGdfiJtmm7G1jplg8ANwKrY+0+tu7dv9l2Z7Pdtg99/7Nq39hW\n",
       "dfxJ7Yu6vrFt0vq0/efVzpT2SfvPs321+Pgtte9m3YFj3foKe5H6UJWZvRL4LeBjEfG+bttZYDUi\n",
       "zpvZ1cCpiHi9ma0BRMT93X4fB94dEY+N3Z/eoFUfFqS9hQzqQxsZ2ujDYG/Q2ugU/gPAmY2JvvNR\n",
       "4K5u+S7gI2Pbf9zMLjOz64HXAZ/cTbh6Xh0gyasDJHl1gCSvDpDg1QGSvDpAklcHGFymjPP9wE8B\n",
       "nzOzx7ttR4H7gRNm9ja6Sy8BIuKMmZ0AzgDPA/dExXcyiIjIi/TdOM22t5BBfWgjg/rQRoY2+qDv\n",
       "xhERkYk02ffy6gBJXh0gyasDJHl1gASvDpDk1QGSvDrA4DTZi4gsAdXsm21vIYP60EYG9aGNDG30\n",
       "QTV7ERGZSJN9L68OkOTVAZK8OkCSVwdI8OoASV4dIMmrAwxOk72IyBJQzb7Z9hYyqA9tZFAf2sjQ\n",
       "Rh9UsxcRkYk02ffy6gBJXh0gyasDJHl1gASvDpDk1QGSvDrA4DTZi4gsAdXsm21vIYP60EYG9aGN\n",
       "DG30QTV7ERGZSJN9L68OkOTVAZK8OkCSVwdI8OoASV4dIMmrAwxOk72IyBJQzb7Z9hYyqA9tZFAf\n",
       "2sjQRh9UsxcRkYk02ffy6gBJXh0gyasDJHl1gASvDpDk1QGSvDrA4DTZi4gsAdXsm21vIYP60EYG\n",
       "9aGNDG30QTV7ERGZSJN9L68OkOTVAZK8OkCSVwdI8OoASV4dIMmrAwxOk72IyBJQzb7Z9hYyqA9t\n",
       "ZFAf2sjQRh9UsxcRkYk02ffy6gBJXh0gyasDJHl1gASvDpDk1QGSvDrA4DTZi4gsAdXsm21vIYP6\n",
       "0EYG9aGNDG30YbCavZn9kpldMLMnxrZdaWYnzewpM3vYzA6MtR01s6fN7KyZ3bybUCIiMluZMs4H\n",
       "gVu2bFsDTkbEDcCj3Tpmdhi4Ezjc3eYBM1vgUpFXB0jy6gBJXh0gyasDJHh1gCSvDpDk1QEGN3Ui\n",
       "joj/Cnxty+ZbgePd8nHg9m75NuChiLgYEevAM8CR2UQVEZHd2u1Z98GIuNAtXwAOdsvXAOfG9jsH\n",
       "HNrlMRqwWh0gabU6QNJqdYCk1eoACavVAZJWqwMkrVYHGNyeSywxeoe37x2F+b8DLCIim1y6y9td\n",
       "MLOrIuK8mV0NPNtt/zJw3dh+13bbtnE3sNItHwBuZPOzq4+te/dvtt3ZbLftG8cY6v5n1f4+No9f\n",
       "K/m22391B/tXrW8dz419Ju1f0X4a+Jnk7Z3N5tm+tW3ex8+2n57D8Xez7sCxbn2FPYmIqT/dUZ4Y\n",
       "W38v8K5ueQ24v1s+zGjULgOuB75Ad3nnlvsLiAk/n47+9ki0Z/bJtJ8a+P5n1YdJOYfOuNM+bJez\n",
       "OuN27Vtztp5xXn8Pu2k/NaP7H7oPp+YwRrPpQ2bO3u5n6nX2ZvYQ8IPAqxnV5/8J8J+BE8CfA9aB\n",
       "OyLiuW7/nwfeCjwPvDMiPrHNfQa6zn5KewsZ1Ic2MqgPbWRoow+7vc5eH6pqtr2FDOpDGxnUhzYy\n",
       "tNEHfRHaILw6QJJXB0jy6gBJXh0gwasDJHl1gCSvDjA4TfYiIktAZZxm21vIoD60kUF9aCNDG31Q\n",
       "GUdERCbSZN/LqwMkeXWAJK8OkOTVARK8OkCSVwdI8uoAg9NkLyKyBFSzb7a9hQzqQxsZ1Ic2MrTR\n",
       "B9XsRURkIk32vbw6QJJXB0jy6gBJXh0gwasDJHl1gCSvDjA4TfYiIktANftm21vIoD60kUF9aCND\n",
       "G31QzV5ERCbSZN/LqwMkeXWAJK8OkOTVARK8OkCSVwdI8uoAg9NkLyKyBFSzb7a9hQzqQxsZ1Ic2\n",
       "MrTRB9XsRURkIk32vbw6QJJXB0jy6gBJXh0gwasDJHl1gCSvDjA4TfYiIktANftm21vIoD60kUF9\n",
       "aCNDG31QzV5ERCbSZN/LqwMkeXWAJK8OkOTVARK8OkCSVwdI8uoAg9NkLyKyBFSzb7a9hQzqQxsZ\n",
       "1Ic2MrTRB9XsRURkIk32vbw6QJJXB0jy6gBJXh0gwasDJHl1gCSvDjA4TfYiIktANftm21vIoD60\n",
       "kUF9aCNDG31QzV5ERCYaZLI3s1vM7KyZPW1m7xriGPPh1QGSvDpAklcHSPLqAAleHSDJqwMkeXWA\n",
       "wc18sjezS4B/B9wCHAZ+wszeMOvjzMfp6gBJyjlbi5BzETKCcrZjiDP7I8AzEbEeEReBXwNuG+A4\n",
       "c/BcdYAk5ZytRci5CBlBOdsxxGR/CPjS2Pq5bpuIiBS5dID7TF7e88N/uP32r18CXDGzNHuyXh0g\n",
       "ab06QNJ6dYCk9eoACevVAZLWqwMkrVcHGNzML700s+8D7o2IW7r1o8ALEfGesX3mf72niMg+sNtL\n",
       "L4eY7C8F/ifwl4H/A3wS+ImIeHKmBxIRkbSZl3Ei4nkz+7vAJ4BLgA9oohcRqVXyCVoREZmvIa6z\n",
       "v87MTpnZ583sd83sHd32v9lt+5aZ3TS2/4qZfdPMHu9+Hph1ph3m/Jdm9qSZfdbM/pOZvWrsNke7\n",
       "D4qdNbObW8zZ4Hj+sy7jaTN71MyuG7tNS+O5bc7WxnOs/e+b2QtmduXYtrmO504ztjaWZnavmZ0b\n",
       "y/MjY7dp6bG5NefG+6E7G8+ImOkPcBVwY7d8BaP6/RuA1wM3AKeAm8b2XwGemHWOPeR8C/CKbvv9\n",
       "wP3d8mFGn7x4ZZf5mY39GsvZ2nj+6bF93g68v9HxnJSzqfHs1q8DPg58Ebiyajx3kbGpsQTeDfy9\n",
       "bfZv7bE5KeeOxnPmZ/YRcT4iTnfLfwQ8CVwTEWcj4qlZH2+3enKejIgXut0eA67tlm8DHoqIixGx\n",
       "zugBcKTBnCV6cn5jbLcrgD/ollsbz0k5S0zK2TX/K+AfbrnJ3MdzFxlLTMi58dmf7a5saemx2Zdz\n",
       "Rwb9IjQzWwHeyGgy6nN99zLEzewHhsy0nZ6cbwV+u1u+htEHxDbM/cNiyZzQ2Hia2b8ws98D7gZ+\n",
       "odutufEcy3kXo1dLG5oZTzO7DTgXEZ/bslvpeCYzQjtj+d+7TW/vyncfMLMD3baWHpt9OWEn4zng\n",
       "S5IrgE8Bt2/ZvrWMcxnwHd3yTcDvMfaSeg4vnSbl/EfAh8fW/y3wk2Pr7wd+rMGcTY5n17YGfLDl\n",
       "8dwmZzPjCVzO6Inpz3RtXwS+s3o8d5CxmbHs1l/D6IzZgH/O6MrB5h6bPTl3NJ5DfevlK4EPA/8x\n",
       "Ij7St29E/L+I+Fq3/BngC8Drhsi11aScZnY38KPAT47t/mVGdcgN13bbBreTnC2O55gHgTd1y82N\n",
       "55gXczY2nn+eUZ32s2b2RUZj9mkzO0jReO4g42saG0si4tnoMJrQN0o1TT02J+Xc8XgO8KxkwC8D\n",
       "vzih/RTwPWPrrwYu6Za/m9FLpgNzePbcNiejb+v8PPDqLds33rS5DLi+G1hrMGdr4/m6seW3A7/S\n",
       "6HhOytnUeG7ZZ7s3aOc2nrvI2NRYAlePLf8s8GCjj81JOXc0nkME/gHghW6wHu9+foTRS7wvAd8E\n",
       "zgMf6/b/G8Dvdvt9GvirQw/qlJxPA/97bNsDY7f5eUZv1pwF/kqLORsczw8BT3TbPwy8ptHx3DYn\n",
       "8GMtjeeWff4X3URaMZ47zdjaWDKaWD8HfBb4CHCw0cfmtjl3Op76UJWIyBLQf0soIrIENNmLiCwB\n",
       "TfYiIktAk72IyBLQZC8isgQ02YuILAFN9iIiS0CTvYjIEvj/pHi7EeB14t0AAAAASUVORK5CYII=\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x107e4e4d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assignments['blue_value'].hist(bins=41);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Within-subjects designs (simple random assignment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In some cases you might want to assign user-item pairs or user-session pairs to parameters. You can do this by simply passing more units into `assign()` and applying multiple units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class SelectiveExposureExperiment(SimpleExperiment):\n",
    "  def assign(self, params, subjectid, url):\n",
    "    params.has_source_cues = BernoulliTrial(p=0.5, unit=subjectid)\n",
    "    params.has_social_cues = BernoulliTrial(p=0.5, unit=subjectid)\n",
    "    \n",
    "    if params.has_source_cues:\n",
    "      sources = ['Fox News','USA Today', 'CNN', 'MSNBC']\n",
    "      params.source = UniformChoice(\n",
    "        choices=sources,\n",
    "        unit=[subjectid, url]\n",
    "      )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Within-subjects designs (balanced random assignment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "If there are a small number of URLs in the case above, it is sometimes possible to not select a particular source, or disproportionately select one source. If a complete list of URLs is known in advance, it is possible to acheive complete balance by permuting a pre-constructed list of sources containing an exactly balanced list. The following code cycles through all sources, so that e.g., if there are 6 URLs, each source appears exactly twice. It also shuffles the URLs (in a deterministic fashion)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from itertools import cycle, islice\n",
    "\n",
    "class SelectiveExposureExperiment(SimpleExperiment):\n",
    "  def assign(self, params, subjectid, urls): \n",
    "    params.shuffled_urls = Sample(choices=urls, unit=subjectid)\n",
    "    \n",
    "    params.has_source_cues = BernoulliTrial(p=0.5, unit=subjectid)\n",
    "    params.has_social_cues = BernoulliTrial(p=0.5, unit=subjectid)\n",
    "    \n",
    "    if params.has_source_cues:\n",
    "      balanced_sources = list(islice(cycle(['Fox News','USA Today', 'CNN', 'MSNBC']), len(urls)))\n",
    "      params.shuffled_sources = Sample(choices=balanced_sources, unit=subjectid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>has_social_cues</th>\n",
       "      <th>has_source_cues</th>\n",
       "      <th>shuffled_urls</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> http://reuters.com/article0.html</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> http://reuters.com/article7.html</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> http://reuters.com/article2.html</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> http://reuters.com/article6.html</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> http://reuters.com/article4.html</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> http://reuters.com/article1.html</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> http://reuters.com/article5.html</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0</td>\n",
       "      <td> http://reuters.com/article3.html</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   has_social_cues  has_source_cues                     shuffled_urls\n",
       "0                0                0  http://reuters.com/article0.html\n",
       "1                0                0  http://reuters.com/article7.html\n",
       "2                0                0  http://reuters.com/article2.html\n",
       "3                0                0  http://reuters.com/article6.html\n",
       "4                0                0  http://reuters.com/article4.html\n",
       "5                0                0  http://reuters.com/article1.html\n",
       "6                0                0  http://reuters.com/article5.html\n",
       "7                0                0  http://reuters.com/article3.html"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex_urls = ['http://reuters.com/article%s.html' % i for i in xrange(8)]\n",
    "\n",
    "subject = 18\n",
    "e = SelectiveExposureExperiment(subjectid=subject, urls=ex_urls)\n",
    "pd.DataFrame.from_dict(e.get_params())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Clustered randomized assignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random assignment can also occur at the cluster level. For example, a course on MOOCs might randomize classes into treatment conditions.  Here, course ID, rather than the student ID, is used for random assignment. Note that while the student ID is not actually used for random assignment, it is still useful to pass it in to the experiment object for logging purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class BadgeExperiment(SimpleExperiment):\n",
    "  def assign(self, params, studentid, courseid):\n",
    "    params.has_faces = BernoulliTrial(p=0.8, unit=courseid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Stepped-wedge design"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stepped wedge designs expose subjects or clusters of subjects to treatments at a certain random point in time. This can improve precision when there are time-correlated exogenous events that may affect subjects' behaviors, or when there are constraints on how rapidly you can roll out treatments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class BadgeExperiment(SimpleExperiment):\n",
    "  def assign(self, params, studentid, courseid, week):\n",
    "    params.rollout_week = RandomInteger(\n",
    "      min=0,\n",
    "      max=8,\n",
    "      unit=courseid\n",
    "    )\n",
    "    if week >= params.rollout_week:\n",
    "      params.has_faces = 1\n",
    "    else:\n",
    "      params.has_faces = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Demo of time-based rollout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The badge rollout for course 17 is 2...\n",
      "week 0: has_faces = 0 \n",
      "week 1: has_faces = 0 \n",
      "week 2: has_faces = 1 \n",
      "week 3: has_faces = 1 \n",
      "week 4: has_faces = 1 \n",
      "week 5: has_faces = 1 \n",
      "week 6: has_faces = 1 \n",
      "week 7: has_faces = 1 \n",
      "week 8: has_faces = 1 \n"
     ]
    }
   ],
   "source": [
    "example_course = 17\n",
    "e = BadgeExperiment(studentid=2, courseid=example_course, week=0)\n",
    "rollout_week = e.get('rollout_week')\n",
    "print 'The badge rollout for course %s is %s...' % (example_course, rollout_week)\n",
    "\n",
    "for week in xrange(9):\n",
    "  exp = BadgeExperiment(studentid=2, courseid=example_course, week=week)\n",
    "  print 'week %s: has_faces = %s ' % (week, exp.get('has_faces'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Social cues experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's consider a social cues experiment where an individual has a certain number of friends that we might show in association with a page, and we want to deterministically randomly sample from some subset of those friends. This is the setup from [Social Influence in Social Advertising: Evidence from Field Experiments](http://arxiv.org/pdf/1206.4327.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class CueExperiment(SimpleExperiment):\n",
    "  def assign(self, params, userid, page, friends):\n",
    "    params.num_friends_shown = RandomInteger(\n",
    "      min=0,\n",
    "      max=min(3, len(friends)),\n",
    "      unit=[userid, page]\n",
    "    )\n",
    "    params.friends_shown = Sample(\n",
    "      choices=friends,\n",
    "      draws=params.num_friends_shown,\n",
    "      unit=[userid, page]\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Dean', 'Solomon']\n",
      "[]\n",
      "['Ronald', 'Annie', 'Moira']\n",
      "['Solomon', 'Ronald', 'Mike']\n",
      "['Mike']\n",
      "['Sean', 'Cat']\n",
      "['Mike']\n",
      "[]\n",
      "['Mike', 'Ronald', 'Solomon']\n",
      "['Solomon', 'Cat', 'Ronald']\n",
      "['Sean', 'Dean', 'Mike']\n",
      "['Moira', 'Mike', 'Solomon']\n"
     ]
    }
   ],
   "source": [
    "some_friends = [\"Sean\", \"Cat\", \"Solomon\", \"Dean\", \"Annie\", \"Moira\", \"Ronald\", \"Mike\"]\n",
    "\n",
    "for pid in xrange(12):\n",
    "    print CueExperiment(userid=6, page=pid, friends=some_friends).get('friends_shown')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## More complex between subjects experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- How does feedback affect content production?\n",
    "- Use experiments to modulate feedback a focal subject receives by varying encouragements for peers to provide feedback."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class ContentExperiment(SimpleExperiment):\n",
    "  def assign(self, params, producerid, storyid, viewerid):\n",
    "    params.p = RandomFloat(\n",
    "        min=0,\n",
    "        max=1,\n",
    "        unit=producerid\n",
    "    )\n",
    "    params.collapse = BernoulliTrial(\n",
    "        p=params.p,\n",
    "        unit=[storyid, viewerid]\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Each user has some overall probability that viewers of their story will see an expanded version of their post. Expanded versions receive more feedback. Subjects are assigned some proportion of impressions of their stories to be expanded. The `expand` parameter is assigned using both the post and the viewer, so that different posts and different viewers may see the story epxnaded some times but not others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "def sim_impressions(producerid, storyid):\n",
    "    viewers = range(5)\n",
    "    sim_users = [ContentExperiment(producerid=producerid, storyid=storyid, viewerid=viewerid).get_params() for viewerid in viewers]\n",
    "    df = pd.DataFrame.from_dict(sim_users)\n",
    "    df['producerid'] = producerid\n",
    "    df['storyid'] = storyid\n",
    "    df['viewerid'] = viewers\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### With a few different producers..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>collapse</th>\n",
       "      <th>p</th>\n",
       "      <th>producerid</th>\n",
       "      <th>storyid</th>\n",
       "      <th>viewerid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.04197</td>\n",
       "      <td> 10</td>\n",
       "      <td> 2</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.04197</td>\n",
       "      <td> 10</td>\n",
       "      <td> 2</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.04197</td>\n",
       "      <td> 10</td>\n",
       "      <td> 2</td>\n",
       "      <td> 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.04197</td>\n",
       "      <td> 10</td>\n",
       "      <td> 2</td>\n",
       "      <td> 3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.04197</td>\n",
       "      <td> 10</td>\n",
       "      <td> 2</td>\n",
       "      <td> 4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   collapse        p  producerid  storyid  viewerid\n",
       "0         0  0.04197          10        2         0\n",
       "1         0  0.04197          10        2         1\n",
       "2         0  0.04197          10        2         2\n",
       "3         0  0.04197          10        2         3\n",
       "4         0  0.04197          10        2         4"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sim_impressions(producerid=10, storyid=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### With a few different producers..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>collapse</th>\n",
       "      <th>p</th>\n",
       "      <th>producerid</th>\n",
       "      <th>storyid</th>\n",
       "      <th>viewerid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 2</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 2</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 2</td>\n",
       "      <td> 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 1</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 2</td>\n",
       "      <td> 3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 2</td>\n",
       "      <td> 4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   collapse         p  producerid  storyid  viewerid\n",
       "0         0  0.155989          17        2         0\n",
       "1         0  0.155989          17        2         1\n",
       "2         0  0.155989          17        2         2\n",
       "3         1  0.155989          17        2         3\n",
       "4         0  0.155989          17        2         4"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sim_impressions(producerid=17, storyid=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### With different stories..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>collapse</th>\n",
       "      <th>p</th>\n",
       "      <th>producerid</th>\n",
       "      <th>storyid</th>\n",
       "      <th>viewerid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 4</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 4</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 4</td>\n",
       "      <td> 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 4</td>\n",
       "      <td> 3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 4</td>\n",
       "      <td> 4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   collapse         p  producerid  storyid  viewerid\n",
       "0         0  0.155989          17        4         0\n",
       "1         0  0.155989          17        4         1\n",
       "2         0  0.155989          17        4         2\n",
       "3         0  0.155989          17        4         3\n",
       "4         0  0.155989          17        4         4"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sim_impressions(producerid=17, storyid=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### With different stories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>collapse</th>\n",
       "      <th>p</th>\n",
       "      <th>producerid</th>\n",
       "      <th>storyid</th>\n",
       "      <th>viewerid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td> 1</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 15</td>\n",
       "      <td> 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td> 1</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 15</td>\n",
       "      <td> 1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 15</td>\n",
       "      <td> 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td> 0</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 15</td>\n",
       "      <td> 3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td> 1</td>\n",
       "      <td> 0.155989</td>\n",
       "      <td> 17</td>\n",
       "      <td> 15</td>\n",
       "      <td> 4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   collapse         p  producerid  storyid  viewerid\n",
       "0         1  0.155989          17       15         0\n",
       "1         1  0.155989          17       15         1\n",
       "2         0  0.155989          17       15         2\n",
       "3         0  0.155989          17       15         3\n",
       "4         1  0.155989          17       15         4"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sim_impressions(producerid=17, storyid=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Different units identify different effects: Like hearding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assigning p based on `storyid` lets you study like hearding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class ContentExperiment(SimpleExperiment):\n",
    "  def assign(self, params, producerid, storyid, viewerid):\n",
    "    params.p = RandomFloat(\n",
    "        min=0,\n",
    "        max=1,\n",
    "        unit=storyid\n",
    "    )\n",
    "    params.expand = BernoulliTrial(\n",
    "        p=params.p,\n",
    "        unit=[storyid, viewerid]\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Different units identify different effects: Reciprocity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Assigning p based on [viewerid, producerid] pairs lets you study reciprocity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class ContentExperiment(SimpleExperiment):\n",
    "  def assign(self, params, producerid, storyid, viewerid):\n",
    "    params.p = RandomFloat(\n",
    "        min=0,\n",
    "        max=1,\n",
    "        unit=[viewerid, producerid]\n",
    "    )\n",
    "    params.expand = BernoulliTrial(\n",
    "        p=params.p,\n",
    "        unit=[storyid, viewerid]\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'p': 0.4080478355600889, 'expand': 0}\n",
      "{'p': 0.2496919900519681, 'expand': 0}\n"
     ]
    }
   ],
   "source": [
    "print(ContentExperiment(producerid=15, viewerid=10, storyid=5).get_params())\n",
    "print(ContentExperiment(producerid=10, viewerid=15, storyid=5).get_params())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Exercise: How can you encourage dyads in both directions direction?\n",
    "\n",
    "- Create an experiment called ContentExperiment2 where the same p is assigned to pairs of users no matter who the viewer is\n",
    "- One possible solution will produce assignments that looks like:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "print(ContentExperiment2(producerid=15, viewerid=10, storyid=5).get_params())\n",
    "print(ContentExperiment2(producerid=10, viewerid=15, storyid=5).get_params())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Exercise: Design the selective exposure experiment in PlanOut\n",
    "- Show 3 stories from a set of 10 stories (assume stories are in a list)\n",
    "- Randomly assign each story to a source cue, Fox News, MSNBC, or CNN\n",
    "- Stories should be able to be any any of the 3 positions, and source cues should appear in any of the 3 positions"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}