02_p-Values.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Null Hypothesis\n",
    "  # Results that come from samples are only mere estimates of population. Because of that, statistics they produce will differ somewhat from their population counterparts.\n",
    "   # Example- Correleation b/w engaement in sample may be r= 0.2 even when correlation b/w those same variables in population is something smaller, such as 0.10 or even 0.\n",
    "    # This can cause apparent relationships & effects to appear in samples when non in fact exist in population.\n",
    "  # Null Hypothesis- that effect/association is ZERO in population\n",
    "   # By implication, any effect/association seen in sample must be entirely due to random chance of \"sampling error\".\n",
    "   # Null hypothesis claims that sample result is a random fluke."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.337790559409466\n",
      "4.8650214885878595\n"
     ]
    }
   ],
   "source": [
    "# Assume two groups (males & females) have identical interest in population, no difference between groups.\n",
    " # we take a sample of males & a sample of females. the error in our estimations will cause a difference to appear.\n",
    "    \n",
    "# Both males & females had an interes level averaging at 5, w/ std dev of 3.\n",
    "\n",
    "import numpy as np\n",
    "import numpy.random as nr\n",
    "nr.seed(515212)\n",
    "\n",
    "# Collect sample of 100 males\n",
    "\n",
    "males = nr.normal(5, 3, 100)\n",
    "\n",
    "# Collect sample of 100 females\n",
    "\n",
    "females = nr.normal(5, 3, 100)\n",
    "\n",
    "print(np.mean(males))\n",
    "print(np.mean(females))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4727690708216068"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Above- The two groups have different sample results.\n",
    "\n",
    "# Below- How large is the difference?\n",
    "\n",
    "np.mean(males) - np.mean(females)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4727690708216068"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Above- Females are almost 1/2 of a point higher than males. You might be tempted to think you'd discovered a female preference for the product.\n",
    " # This different is entirely due to random error in samples, not any real difference in population. We have discovered a fluke in some sample data, nothing more.\n",
    "    \n",
    "# Below- Let's test the null hypothesis between the averages.\n",
    " # H0: mu(male) - mu(female) = 0   (The size of the effect in population is zero)\n",
    " # Testing the difference b/w averages (mu's), stating that the difference b/w them is zero.\n",
    "\n",
    "np.mean(males) - np. mean(females)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Above- This shows that there is a difference in the samples.\n",
    " # Null hypothesis says that whatever effect you are studying is zero in population & your sample results are due to random chance.\n",
    "    \n",
    "## p-Value- If null hypothesis were true, what percentage of the time would I get this result this large?\n",
    "  # p-values represent how often you could get a result as big as you did IF the null were true.\n",
    "  # p-values therefore represent how easy/hard it would be to ger a result by chance.\n",
    "  # p-values do NOT tell you porbablitlity that the results is due to chance; only the probablity of seeing your result if null were true.\n",
    "  # If p-value for result is small, it would be rare to get that result by chance (if null were true).\n",
    "  # If p-value for result is large, it would be common to get that result by chance (if null were true)\n",
    " # Conclusion: p-value is a measure of \"incompatibility\" b/w your result & null.\n",
    "  # If p-value is small, one of the two (data or null) is likely very wrong, we'll opt to trust our data & REJECT the null.\n",
    "    \n",
    "# p-value states the probility of getting your RESULT if the null is true. It is essentially a statement of incompatibility b/w your data & the null.\n",
    " # A small p-value (typically, less than 5% or \"< 0.05\") tell syou that the data * null are highly incompatible.\n",
    " # Since you did in fact observe the data, you conclude the null hypothesis is false. This is the only use for a p-value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.234095719379859\n",
      "2.0725742818363613\n"
     ]
    }
   ],
   "source": [
    "# Every situation is different, but process in so-called \"frequentist\" statistics is always the same.\n",
    " # 1. Observe data & examine result\n",
    " # 2. Compute appropriate \"test statistic\" for that result (e.g. t test, z test, x^2 test, F test, q test, etc.)\n",
    " # 3. Observe how often you could get observed test statistic if null hypothesis was true. This is the p-value\n",
    " # 4. If p-value is less than 0.05, declare result \"significant\" & reject null hypothesis\n",
    "\n",
    "# Example- Use a \"one-sample t-test\" to question whather people have a positive or negative attitude toward training given in an organization.\n",
    " # -5 (very negative) to +5 (very positive) scale w/ 0= neutral option\n",
    " # Imagine they actually have a positive attitude, that population mean= 2.4 w/ std dev of 2.0. (This would really be unknown in real life)\n",
    " # Null hypothesis states the effect is absent H0: mu = 0  w/ an effect being non-zero attitude.\n",
    "    \n",
    "nr.seed(4455)\n",
    "attitude = nr.normal(2.4, 2.0, 100)\n",
    "\n",
    "print(np.mean(attitude))\n",
    "print(np.std(attitude))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results of one-sample two-sided t test\n",
      "Mean         = 2.234\n",
      "t-Statistic  = 10.725\n",
      "p-value      < 2.881e-18\n",
      "On degrees of freedom =   99\n",
      "Confidence Intervals for alpha =0.05\n",
      "Lower =  1.729 Upper = 2.739\n"
     ]
    }
   ],
   "source": [
    "# Above- Null hypothesis is that mean is zero, but our sample result disagree w/ that (sample mu= 2.23).\n",
    " # Does this sample give us enough evidence to reject the null? Need to calculate a test statistic by conducting a one-group t-test for means to answer that.\n",
    "  # We compare size of difference b/w our observed result & null hypothesis, divided by what you would typically expect by chance (i.e. standard error).\n",
    "  # t= (result - null)/ chance ---> t= (xbar - H0)/(SD/sqrt(n)) ---> t= (2.234 - 0)/(2.073/sqrt(100)) = 10.8\n",
    "  # The test assess how much data disagree w/ null (i.e. the effect; top of fraction) compared to what you would typically expect by chance (bottom of fraction).\n",
    "   # \"Our effect was 10.8 times greater than you would expect by chance.\"\n",
    "  # We simply need to know where t-test results tend to be when null is true, & then we can see how rare a scroe of 10.8 would be in that situation, giving us our p-value.\n",
    "\n",
    "# Below- What is our p-value? If null were true (\"under the null\"), how often could we get t-test result as big as 10.8?\n",
    "\n",
    "from scipy import stats\n",
    "def t_one_sample(samp, mu = 0.0, alpha = 0.05):\n",
    "    '''Function for two-sided one-sample t test'''\n",
    "    t_stat = stats.ttest_1samp(samp, mu)\n",
    "    scale = np.std(samp)\n",
    "    loc = np.mean(samp)\n",
    "    ci = stats.t.cdf(alpha/2, len(samp), loc=mu, scale=scale)\n",
    "    print('Results of one-sample two-sided t test')\n",
    "    print('Mean         = %4.3f' % loc)\n",
    "    print('t-Statistic  = %4.3f' % t_stat[0])\n",
    "    print('p-value      < %4.3e' % t_stat[1])\n",
    "    print('On degrees of freedom = %4d' % (len(samp) - 1))\n",
    "    print('Confidence Intervals for alpha =' + str(alpha))\n",
    "    print('Lower =  %4.3f Upper = %4.3f' % (loc - ci, loc + ci))\n",
    "    \n",
    "t_one_sample(attitude)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Above- p-value of 2.9e-18 is clearly less than 0.05, so we can reject null hypothesis & conclude positive attitude observed among particpants was not statistical fluke but likely a real trend in population.\n",
    "\n",
    "# Below- Compute p-value as 1 - cdf, for test statistic, where cdf is cumulative density function & degrees of freedom is n - 1 (100 - 1 = 99).\n",
    "\n",
    "from scipy.stats import t\n",
    "1 - t.cdf(10.8, df = 99, loc=0, scale=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Above- Result is saying there is \"zero\" probability of getting a result this big if null were true, i.e. p= 0.\n",
    " # In reality, p values are never zero but can get infinitely small, in this case the tiny # is rounded to 0.\n",
    "    \n",
    "# Below- For a two-tailed p-value, we would need to double our p-value.\n",
    " # This let's us know the null is false if our result is significantly larger than 0 (positive attitude) OR significantly smaller than 0 (negative attitude).\n",
    " # By testing H0: mu = 0, you are really asking whether mu < 0 or whether mu > 0.\n",
    "\n",
    "2.0 * (1 - t.cdf(10.8, df = 99, loc=0, scale=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "           0\n",
      "0   2.725478\n",
      "1   2.279837\n",
      "2   5.830846\n",
      "3  -0.444898\n",
      "4   0.828241\n",
      "5   1.904276\n",
      "6   0.587809\n",
      "7   2.557436\n",
      "8   2.406236\n",
      "9   4.994005\n",
      "10  1.848412\n",
      "11  4.011133\n",
      "12  3.880843\n",
      "13  3.177942\n",
      "14  2.777390\n",
      "15  2.610310\n",
      "16 -0.303223\n",
      "17  6.352482\n",
      "18  1.403908\n",
      "19  1.790281\n",
      "20  4.352330\n",
      "21  0.637029\n",
      "22  4.577655\n",
      "23  2.272571\n",
      "24  4.267818\n",
      "25  2.065834\n",
      "26  2.283606\n",
      "27  4.514988\n",
      "28 -2.188975\n",
      "29  3.015534\n",
      "..       ...\n",
      "70  5.340033\n",
      "71 -0.323335\n",
      "72  2.560280\n",
      "73  3.668224\n",
      "74  1.437170\n",
      "75  4.037023\n",
      "76  1.634447\n",
      "77  1.308978\n",
      "78  1.615152\n",
      "79  3.424214\n",
      "80  3.370450\n",
      "81  1.733508\n",
      "82  1.824191\n",
      "83  4.509834\n",
      "84  2.162279\n",
      "85  6.646098\n",
      "86  5.606440\n",
      "87  2.652539\n",
      "88 -0.370673\n",
      "89  4.382770\n",
      "90  5.331299\n",
      "91 -0.055439\n",
      "92  0.308916\n",
      "93  2.374227\n",
      "94  0.791616\n",
      "95  1.079155\n",
      "96  4.136870\n",
      "97  4.509649\n",
      "98  3.389122\n",
      "99 -0.326108\n",
      "\n",
      "[100 rows x 1 columns]\n"
     ]
    }
   ],
   "source": [
    "# Save data to a csv file for future work\n",
    "\n",
    "import pandas as pd \n",
    "attitude = pd.DataFrame(attitude)\n",
    "attitude.to_csv(\"Attitude.csv\")\n",
    "\n",
    "print(attitude)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}