smd_handson_testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hands-On: Schätzen und Testen\n",
    "Hier soll eine beispielhafte Analyse die Themen des Maximum-Likelihood-Schätzens und des Likelihood-Ratio-Testens zusammenfassend veranschaulichen.\n",
    "\n",
    "Die benutzten Methoden werden größtenteils zur Veranschaulichung direkt im notebook implementiert.\n",
    "Viele der Methoden hier sind in Statistik-Paketen implementiert und sollten dann eigenen Implementierungen vorgezogen werden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.colors import LogNorm\n",
    "\n",
    "from scipy.stats import norm, chi2\n",
    "from scipy.optimize import minimize\n",
    "\n",
    "from tqdm.auto import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib widget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams['figure.figsize'] = (8, 6)\n",
    "plt.rcParams['figure.dpi'] = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Zunächst wird ein Beispielexperiment erstellt.\n",
    "\n",
    "Das Modell besteht aus der Superposition zweier Normal-Verteilungen mit folgenden Parametern:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_a = norm(0, 1)\n",
    "model_b = norm(3, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Die PDFs sehen wie folgt aus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1d822a91323a4728aac1f2f6c29ae559",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f33aafeb370>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.linspace(-5, 10, 1000)\n",
    "\n",
    "fig = plt.figure(constrained_layout=True)\n",
    "plt.plot(x, model_a.pdf(x), label = 'A')\n",
    "plt.plot(x, model_b.pdf(x), label = 'B')\n",
    "\n",
    "plt.xlabel(r'$x$')\n",
    "plt.ylabel(r'Probability Density Function')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nun soll eine Verteilung `data` aus diesen beiden PDFs erstellt werden, um eine Messung zu simulieren.\n",
    "\n",
    "Dabei wird $20$ mal aus der Verteilung $A$ und $250$ mal aus der Verteilung $B$ gezogen. $A$ stellt das Signal und $B$ den Untergrund dar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "N_A = 20\n",
    "N_B = 250\n",
    "\n",
    "rng = np.random.default_rng(1337)\n",
    "\n",
    "sample_a = model_a.rvs(size=N_A, random_state=rng)\n",
    "sample_b = model_b.rvs(size=N_B, random_state=rng)\n",
    "data = np.append(sample_a, sample_b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dies ergibt die folgende Verteilung:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6635f24664d14ca9990f740ce4f6b47a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Observed Events')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "limits = [data.min(), data.max()]\n",
    "\n",
    "plt.figure(constrained_layout=True)\n",
    "plt.hist(data, histtype = 'step', label='Total', range=limits, bins=25)\n",
    "plt.hist(sample_a, histtype = 'step', label='A', range=limits, bins=25)\n",
    "plt.hist(sample_b, histtype = 'step', label='B', range=limits, bins=25)\n",
    "plt.legend()\n",
    "\n",
    "plt.xlabel(r'$x$')\n",
    "plt.ylabel(r'Observed Events')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Das Ziel dieses Beispiels ist, das kleine Signal $A$ im deutlich größeren Untergrund $B$ zu messen."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Erstellen der Likelihood\n",
    "\n",
    "Die Likelihood (das Produkt über die einzelnen Wahrscheinlichkeiten) ist hier\n",
    "\n",
    "$\\mathcal{L}(f | \\boldsymbol{x}) = \\prod\\limits_i P(f|x_i)$,\n",
    "\n",
    "wobei die kombinierte PDF $P(f|x_i)$ die normierte Superposition von $A$ und $B$ ist\n",
    "\n",
    "$P(f|x) = f\\cdot A(x) + (1-f)\\cdot B(x)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pdf(f):\n",
    "    return f * model_a.pdf(data) + (1 - f) * model_b.pdf(data)\n",
    "\n",
    "\n",
    "def likelihood(f):\n",
    "    return np.prod(pdf(f), axis=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Diese Likelihood ist eine Funktion von $f$, dem Anteil (*fraction*) der Daten, welcher die Signalstärke von $A$ gegenüber $B$ beschreibt.\n",
    "\n",
    "Das Maximum der Funktion sollte dabei bei $f_\\mathrm{max} = \\frac{N_A}{N_A + N_B}$ liegen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0741\n"
     ]
    }
   ],
   "source": [
    "f_true = N_A / (N_A + N_B)\n",
    "print(f\"{f_true:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9169dcc441324b0c823e8828f1b40f54",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "f_space = np.linspace(0, 1, 1000, endpoint=False)\n",
    "\n",
    "fig = plt.figure(constrained_layout=True)\n",
    "\n",
    "lh = likelihood(f_space[:, np.newaxis])\n",
    "f_max = f_space[np.argmax(lh)]\n",
    "\n",
    "plt.plot(f_space, lh, label=r'$\\mathcal{L}(f \\,| \\mathbf{x})$')\n",
    "\n",
    "plt.axvline(f_true, ls=':', color='C1', label='$f_\\mathrm{true}$')\n",
    "plt.axvline(f_max, ls='--', color='C2', label='$f_\\mathrm{max}$')\n",
    "plt.legend()\n",
    "\n",
    "plt.xlabel(r'$f$')\n",
    "plt.ylabel(r'$\\mathcal{L}$')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Der Wertebereich der Likelihood umfasst mehrere Größenordnungen, was die handhabung erschwert.\n",
    "\n",
    "Außerdem stellen die meisten Numerik-Bibliotheken *Minimierer* und nicht *Maximierer* zur Verfügung.\n",
    "\n",
    "Wie üblich, verwenden wir daher die **negative Log-Likelihood**.\n",
    "\n",
    "\\begin{equation}\n",
    "  -\\log\\mathcal{L}(f|\\boldsymbol{x}) = -\\sum\\limits_i \\log\\left(fA(x_i) + (1-f)B(x_i)\\right)\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def negative_log_likelihood(f):\n",
    "    return -np.sum(np.log(pdf(f)), axis=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Da der Logarithmus eine monoton steigende Funktion ist, stimmt das Maximum der *log-likelihood* mit dem Maximum der *Likelihood* überein.\n",
    "\n",
    "Dies kann kurz per Auge überprüft werden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b8da126321c343969e1bee4fe17cdd08",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.053\n"
     ]
    }
   ],
   "source": [
    "f_space = np.linspace(0, 1, 1000, endpoint=False)\n",
    "\n",
    "fig = plt.figure(constrained_layout=True)\n",
    "\n",
    "nll = negative_log_likelihood(f_space[:, np.newaxis])\n",
    "f_min = f_space[np.argmin(nll)]\n",
    "\n",
    "plt.plot(f_space, nll, label=r'$-\\log\\mathcal{L}(f \\,| \\mathbf{x})$')\n",
    "\n",
    "plt.axvline(f_true, ls=':', color='C1', label='$f_\\mathrm{true}$')\n",
    "plt.axvline(f_min, ls='--', color='C2', label='$f_\\mathrm{min}$')\n",
    "plt.legend()\n",
    "\n",
    "plt.xlabel(r'$f$')\n",
    "plt.ylabel(r'$-\\log\\mathcal{L}(f \\,| \\mathbf{x})$')\n",
    "plt.show()\n",
    "\n",
    "print(f_min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bei der Suche nach dem Maximum, wird die _ungebinnte_ Likelihood häufig ausgewertet, da stets über jedes Ereignis iteriert wird.\n",
    "\n",
    "Diese Methode ist leider sehr langsam, vor allem, wenn große Datensätze betrachtet werden.\n",
    "\n",
    "Daher wird die Likelihood in eine gebinnte Verteilung umformuliert."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_linear_bin_edges_and_mids(low, high, n_bins):\n",
    "    bin_edges = np.linspace(low, high, n_bins + 1)\n",
    "    bin_mids = 0.5 * (bin_edges[:-1] + bin_edges[1:])\n",
    "    return bin_edges, bin_mids\n",
    "\n",
    "bin_edges, bin_mids = create_linear_bin_edges_and_mids(-5, 10, 100)\n",
    "binned_data, _ = np.histogram(data, bins=bin_edges)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wir nutzen die kumulative Verteilung (CDF) um die PDF zu diskretisieren."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "binned_a = np.diff(model_a.cdf(bin_edges)) / np.diff(bin_edges)\n",
    "binned_b = np.diff(model_b.cdf(bin_edges)) / np.diff(bin_edges)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sollte keine analytische CDF bekannt sein sondern nur die PDF und eine effiziente Möglichkeit diese zu samplen, kann auch zufällig gesamplet werden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "binned_a_sampled, _ = np.histogram(model_a.rvs(size=1_000_000, random_state=rng), bins=bin_edges, density=True)\n",
    "binned_b_sampled, _ = np.histogram(model_b.rvs(size=1_000_000, random_state=rng), bins=bin_edges, density=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hier der Vergleich der beiden Verteilungen zwischen der gebinnten und der ungebinnten Version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "562ee61a4a8844209463746fc6003922",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(constrained_layout=True)\n",
    "plt.plot(x, model_a.pdf(x), label = 'A')\n",
    "plt.stairs(binned_a, bin_edges, label='A binned', zorder=5)\n",
    "\n",
    "plt.plot(x, model_b.pdf(x), label='B')\n",
    "plt.stairs(binned_b, bin_edges, label='B binned', zorder=5)\n",
    "\n",
    "plt.xlabel(r'$x$')\n",
    "plt.ylabel(r'Probability')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Die gebinnte Likelihood ist ähnlich wie die Ungebinnte definiert, außer dass mehrere Ereignisse, die im selben Bin landen, herausfaktorisiert werden.\n",
    "\n",
    "Somit muss nur über die Bins und nicht über jedes einzelne Ereignis iteriert werden.\n",
    "\n",
    "$\\mathrm{LLH} = \\log\\mathcal{L}(f|\\vec{x}) = \\sum\\limits_\\mathrm{bins} N_\\mathrm{bin} \\log\\left(fA_\\mathrm{bin} + (1-f)B_\\mathrm{bin}\\right)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def binned_negative_log_likelihood(f, binned_sample=binned_data):\n",
    "    mu = f * binned_a + (1 - f) * binned_b\n",
    "    return -np.sum(binned_sample * np.log(mu), axis=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wie zu erwarten, gleicht die gebinnte Likelihood der Ungebinnten."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d4535071cdc14faab84dbe345ebfb826",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "plt.plot(f_space, negative_log_likelihood(f_space[:, np.newaxis]), label='unbinned')\n",
    "plt.plot(f_space, binned_negative_log_likelihood(f_space[:, np.newaxis]), ls=':', label='binned', color='C3')\n",
    "\n",
    "plt.axvline(f_true, color='C1')\n",
    "\n",
    "plt.xlabel(r'$f$')\n",
    "plt.ylabel(r'$-\\log\\mathcal{L}$')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mit einem kurzen Test kann überprüft werden, ob die gebinnte Log-Likelihood die Ungebinnte hinreichend gut beschreibt, sodass ein Minimizer das Maximum findet.\n",
    "\n",
    "Dabei ist der Performanceunterschied deutlich zu sehen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "minimize_kwargs = {\n",
    "    \"x0\": [0.5],\n",
    "    \"bounds\": [(0, 1)]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.8 ms ± 411 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "result = minimize(negative_log_likelihood, **minimize_kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "      fun: 565.1343391188238\n",
       " hess_inv: <1x1 LbfgsInvHessProduct with dtype=float64>\n",
       "      jac: array([0.00011369])\n",
       "  message: 'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'\n",
       "     nfev: 14\n",
       "      nit: 6\n",
       "     njev: 7\n",
       "   status: 0\n",
       "  success: True\n",
       "        x: array([0.05268395])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = minimize(negative_log_likelihood, **minimize_kwargs)\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.41 ms ± 73.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "result = minimize(binned_negative_log_likelihood, **minimize_kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "      fun: 565.371073282461\n",
       " hess_inv: <1x1 LbfgsInvHessProduct with dtype=float64>\n",
       "      jac: array([0.00014779])\n",
       "  message: 'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH'\n",
       "     nfev: 14\n",
       "      nit: 6\n",
       "     njev: 7\n",
       "   status: 0\n",
       "  success: True\n",
       "        x: array([0.05225993])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = minimize(binned_negative_log_likelihood, **minimize_kwargs)\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test der Nullhypothese\n",
    "\n",
    "Da es nun möglich ist den maximal wahrscheinlichen Parameter $f_\\mathrm{best}$ zu finden, können nun im nächsten Schritt Hypothesen getestet werden.\n",
    "\n",
    "Annahme: Wir sind nicht sicher, ob es eine Komponente $A$ in `data` gibt.\n",
    "\n",
    "Dies führt zur Nullhypothese, dass $f = 0$ der wahre Wert ist, die wir nun mit einem Likelihood-Ratio-Test verwerfen wollen.\n",
    "\n",
    "Daraus lässt sich die folgende Test-Statistik erstellen:\n",
    "\n",
    "$\\mathcal{TS} = 2 \\log \\left( \\frac{\\mathcal{L}(f = f_\\mathrm{best})}{\\mathcal{L}(f = 0)}\\right) = 2\\left(\\mathrm{LLH}(f = f_\\mathrm{best}) - \\mathrm{LLH}(f = 0)\\right)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_ts(sample, f_0=0):\n",
    "    binned_sample, _ = np.histogram(sample, bins=bin_edges)\n",
    "\n",
    "    fit_result = minimize(\n",
    "        binned_negative_log_likelihood,\n",
    "        args=(binned_sample, ),\n",
    "        **minimize_kwargs\n",
    "    )\n",
    "    \n",
    "    f_fit = fit_result.x[0]\n",
    "    nllh = fit_result.fun\n",
    "    \n",
    "    return -2 * (nllh - binned_negative_log_likelihood(f_0, binned_sample))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Zum Generieren einer Test-Statistik-Verteilung für die Nullhypothese wird eine große Anzahl an zufällig gezogenen Ereignissen aus $B$ erstellt und für diese der Test-Statistik-Wert berechnet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f3003df073f640de91b9db1073688bcf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/30000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_scrambles = 30000\n",
    "null_ts = np.empty(n_scrambles)\n",
    "\n",
    "for i in tqdm(range(n_scrambles)):\n",
    "    sample = model_b.rvs(size=len(data), random_state=rng)\n",
    "    null_ts[i] = calculate_ts(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Die normierte Nuller-Test-Statistik-Verteilung kann nun mit dem Test-Statistik-Wert der gemessenen Verteilung `data` verglichen werden. Daraus folgt die Signifikanz, mit der die Nullhypothese, dass der Beitrag von $A=0$ ist, verworfen werden kann.\n",
    "\n",
    "Zum Vergleich ist zudem die $\\chi^2$-Verteilung dargestellt. \n",
    "\n",
    "Zur Erinnerung, das Wilks-Theorem sagt:\n",
    "\n",
    "Falls\n",
    "1. sich die Nullhypothese durch eine lineare Parameter-Transformation als ein \n",
    "    Spezialfall der Alternativ-Hypothese darstellen lässt\n",
    "2. die Anzahl der Beobachtungen gegen unendlich geht\n",
    "3. Keiner der Parameter einen Extremwert annimmt\n",
    "\n",
    "ist die Teststatistik $\\chi^2$ verteilt. \n",
    "\n",
    "In unserem Fall ist $f=0$ ein Extremwert, somit gilt Wilks-Theorem hier nicht, auch nicht für einen wesentlich größeren Datensatz, der der gleichen Verteilung folgt.\n",
    "\n",
    "Der Likelihood-Ratio-Test ist trotzdem valide, die Abschätzung der P-Values muss aber durch sampling der Test-Statistik für die Nullhypothese bestimmt werden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a696903d0d4b493ea2ee66b2ca50f0c9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "observed_ts = calculate_ts(data)\n",
    "\n",
    "plt.figure()\n",
    "ts_bin_edges = np.linspace(0, 15, 100)\n",
    "\n",
    "\n",
    "plt.hist(\n",
    "    null_ts,\n",
    "    bins=ts_bin_edges,\n",
    "    density=True,\n",
    "    cumulative=-1,\n",
    "    histtype='step',\n",
    "    label=r'$\\mathcal{TS}$ distribution',\n",
    ")\n",
    "\n",
    "\n",
    "plt.axvline(observed_ts, color='C2', ls='--', label = rf'Observed $\\mathcal{{TS}} = {observed_ts:.2f}$')\n",
    "\n",
    "ts_space = np.linspace(0, 15, 1000)\n",
    "\n",
    "# surfival function = 1 - cdf\n",
    "plt.plot(ts_space, chi2.sf(ts_space, 1), label = r'$\\chi^2$ with $N_\\mathrm{DoF} = 1$')\n",
    "plt.legend()\n",
    "\n",
    "\n",
    "plt.xlabel(r'$\\mathcal{TS}$')\n",
    "plt.ylabel('p-value')\n",
    "\n",
    "plt.yscale('log')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Der p-value bzw. die Signifikanz der Messung gegeben der Hypothese, dass der Beitrag von $A$ zu \"data\" $0$ ist, kann nun aus der Position des gemessenen Test-Statistik-Wertes in der nuller Test-Statistik abgelesen werden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "p-value 0.0356 and significance 2.1 σ of 'data'\n",
      "p-value 0.0709 and significance 1.8 σ according to chi-squared\n"
     ]
    }
   ],
   "source": [
    "def sigma_from_pval(p_value):\n",
    "    return norm.ppf(1 - p_value / 2)\n",
    "\n",
    "p_value = np.count_nonzero(null_ts > observed_ts) / len(null_ts)\n",
    "p_value_chi2 = chi2.sf(observed_ts, 1) \n",
    "\n",
    "print(f\"p-value {p_value:2.4f} and significance {sigma_from_pval(p_value):2.1f} σ of 'data'\")\n",
    "print(f\"p-value {p_value_chi2:2.4f} and significance {sigma_from_pval(p_value_chi2):2.1f} σ according to chi-squared\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Berechnung der Konfidenzintervalle\n",
    "\n",
    "Nun ist es wichtig, neben dem BestFit-Wert auch die entsprechenden Unsicherheitsbereiche mit anzugeben. Wir wissen ja bereits, dass selbst die $2\\sigma$ Fehlergrenze nicht mit Null kompatibel ist.\n",
    "\n",
    "Zunächst wird hierfür die Neyman-Konstruktion erstellt.\n",
    "\n",
    "Dazu wird der Signalstärkeparameter $f$ kleinschrittig abgefahren und für jeden Wert eine Test-Statistik-Verteilung erstellt/gewürfelt und histogrammiert."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_ts_bins = 100\n",
    "ts_bin_edges, ts_bin_mids = create_linear_bin_edges_and_mids(0, 120, n_ts_bins)\n",
    "\n",
    "n_f_bins = 51\n",
    "f_bin_edges, f_bin_mids = create_linear_bin_edges_and_mids(0, 0.3, n_f_bins)\n",
    "\n",
    "neyman_construction = np.empty((n_f_bins, n_ts_bins))\n",
    "\n",
    "def sample_from_combined_pdf(f, n_samples=len(data)):\n",
    "    n_a = int(round(f * n_samples))\n",
    "    n_b = n_samples - n_a\n",
    "    return np.append(\n",
    "        model_a.rvs(size=n_a, random_state=rng),\n",
    "        model_b.rvs(size=n_b, random_state=rng)\n",
    "    )\n",
    "\n",
    "def generate_binned_ts_dist(f, n_scrambles=5000):\n",
    "    ts = np.empty(n_scrambles)\n",
    "    for idx in range(n_scrambles):\n",
    "        sample = sample_from_combined_pdf(f)    \n",
    "        ts[idx] = calculate_ts(sample)\n",
    "        \n",
    "    ts_binned, _ = np.histogram(ts, bins=ts_bin_edges, density=True)\n",
    "    return ts_binned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2dddd3e73962456b9b5a6faa3ffd5250",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/51 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for idx in tqdm(range(n_f_bins)):\n",
    "    neyman_construction[idx] = generate_binned_ts_dist(f_bin_mids[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Das engmaschige Gitter der Neyman-Konstruktion kann gut dargestellt werden.\n",
    "\n",
    "Gut zu erkennen ist der entstehende Gürtel, da sich für ein größeres Signal die Teststatistikverteilung zu höheren Werten verschiebt. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "361efd3b824a4d1292ac5b13f460f49e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "im = ax.pcolormesh(ts_bin_edges, f_bin_edges, neyman_construction, norm=LogNorm())\n",
    "\n",
    "ax.axvline(x=observed_ts, color='C1', ls = '--')\n",
    "ax.set_xlabel(r\"$\\mathcal{TS}$\")\n",
    "ax.set_ylabel(r\"$f$\")\n",
    "fig.colorbar(im)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aus dem Gitter wird nun die normierte Spalte betrachtet, indem der gemessene Test-Statistik-Wert liegt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "efd1672ea1b842c496966c9f613246c2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def get_f_pdf(ts_value):\n",
    "    idx = np.digitize(ts_value, bins=ts_bin_edges)\n",
    "    f_dist = neyman_construction[:, idx]\n",
    "    f_dist /= np.sum(f_dist)\n",
    "    return f_dist\n",
    "\n",
    "f_pdf = get_f_pdf(observed_ts)\n",
    "\n",
    "plt.figure(constrained_layout=True)\n",
    "\n",
    "plt.stairs(f_pdf, f_bin_edges, label=r\"PDF$(f | \\mathcal{TS}_\\mathrm{meas.})$\")\n",
    "\n",
    "plt.axvline(f_true, ls = '--', color='C1', label=r\"$f_\\mathrm{true}$\")\n",
    "plt.axvline(result.x, ls = '--', color='C2', label=r\"$f_\\mathrm{best}$\")\n",
    "plt.xlabel(r\"$f$\")\n",
    "plt.ylabel(r\"Probability\")\n",
    "\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mit einem gewählten Konfidenzlevel $\\alpha=0.9$ können die Grenzen eines Zentralintervalls über die Quantile der Verteilung des Signalstärkeparameters $f$ bestimmt werden.\n",
    "\n",
    "Dabei wird solange das höchste benachbarte bin miteingeschlossen, bis das gewünschte Konfidenzlevel erreicht ist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lower bound: 0.0206\n",
      "upper bound: 0.1088\n"
     ]
    }
   ],
   "source": [
    "def get_upper_and_lower_idx(dist, alpha):\n",
    " \n",
    "    cumulated = np.cumsum(dist)  \n",
    "    outside = (1 - alpha) / 2\n",
    "    \n",
    "    # lower bound \n",
    "    indices, = np.where(cumulated >= outside)\n",
    "    if len(indices) > 0:\n",
    "        f_idx_low = indices[0]\n",
    "    else:\n",
    "        f_idx_low = 0\n",
    "       \n",
    "    # upper bound\n",
    "    indices, = np.where(cumulated >= 1 - outside)\n",
    "    if len(indices) > 0:\n",
    "        f_idx_up = indices[0]\n",
    "    else:\n",
    "        f_idx_up = len(indices) - 1\n",
    "    \n",
    "    return f_idx_low, f_idx_up\n",
    "\n",
    "\n",
    "def get_central_interval(alpha, bin_edges=f_bin_edges):\n",
    "    low_idx, up_idx = get_upper_and_lower_idx(f_pdf, alpha)\n",
    "    bin_mids = 0.5 * (f_bin_edges[:-1] + f_bin_edges[1:])\n",
    "    lim_low = bin_mids[low_idx]\n",
    "    lim_up = bin_mids[up_idx]\n",
    "    \n",
    "    return lim_low, lim_up\n",
    "\n",
    "f_low, f_up = get_central_interval(0.9)\n",
    "print(f\"lower bound: {f_low:.4f}\")\n",
    "print(f\"upper bound: {f_up:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Somit ergibt sich das folgende Ergebnis:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In den Messdaten wurde mit einem p-Value von 0.036 und somit einer Signifikanz von 2.1 sigma ausgeschlossen, dass sie nur aus Untergrundereignissen bestehen.\n",
      "Der Signalstärkeparameter wurde dabei zu 0.052 bestimmt mit dem 90 Prozent Konfidenzintervall (0.0206, 0.1088)\n"
     ]
    }
   ],
   "source": [
    "print(\"In den Messdaten wurde mit einem p-Value von {:.3f} und somit einer Signifikanz von {:.1f} sigma ausgeschlossen, dass sie nur aus Untergrundereignissen bestehen.\".format(p_value, sigma_from_pval(p_value)))\n",
    "print(\"Der Signalstärkeparameter wurde dabei zu {:.3f} bestimmt mit dem 90 Prozent Konfidenzintervall ({:.4f}, {:.4f})\".format(result.x[0], f_low, f_up))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9c85a4fcfb46436e83f0f5f553c9e94f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(constrained_layout=True)\n",
    "\n",
    "plt.stairs(f_pdf, f_bin_edges, label=r\"PDF$(f | \\mathcal{TS}_\\mathrm{meas.})$\")\n",
    "\n",
    "plt.axvline(f_true, ls='--', color='C1', label=r\"$f_\\mathrm{true}$\")\n",
    "plt.axvline(result.x, ls='--', color='C2', label=r\"$f_\\mathrm{best}$\")\n",
    "\n",
    "plt.axvline(f_low, ls='--', color=\"k\", label=r\"$90\\,\\%$ Confidence\")\n",
    "plt.axvline(f_up, ls='--', color=\"k\")\n",
    "\n",
    "plt.xlabel(r\"$f$\")\n",
    "plt.ylabel(r\"Probability\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Inverser Hypothesentest\n",
    "\n",
    "Die bisherige Berechnung ist eine konservative Vorgehensweise, da in der Test-Statistik gegen die Nullhypothese verglichen wird.\n",
    "\n",
    "Nun soll die Test-Statistik jeweils gegen den injezierten Messwert verglichen werden. Hier wird also ein anderer, alternativer Hypothesentest durchgeführt.\n",
    "\n",
    "$\\mathcal{TS}_{f_0} = 2 \\log \\left( \\frac{\\mathcal{L}(f = f_\\mathrm{best})}{\\mathcal{L}(f = f_0)}\\right) = 2\\left(\\mathrm{LLH}(f = f_\\mathrm{best}) - \\mathrm{LLH}(f = f_0)\\right)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Zunächst wird in einem feinen Grid wieder $f$ abgefahren und die Teststatistik erstellt/gewürfelt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3bf5128e369a4227a5cbf5c504b70ad4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_f_bins_inv = 100\n",
    "f_bin_edges_inv, f_bin_mids_inv = create_linear_bin_edges_and_mids(0, 0.25, n_f_bins_inv)\n",
    "n_scrambles_inv = 1000\n",
    "\n",
    "\n",
    "def generate_ts_dist(f_inject, n_scrambles=n_scrambles_inv):\n",
    "    ts = np.empty(n_scrambles)\n",
    "  \n",
    "    for ts_idx in range(n_scrambles):\n",
    "        sample = sample_from_combined_pdf(f_inject)\n",
    "        ts[ts_idx] = calculate_ts(sample, f_inject)\n",
    "\n",
    "    return ts\n",
    "\n",
    "neyman_construction_inv = np.empty((n_f_bins_inv, n_scrambles_inv))\n",
    "for f_idx in tqdm(range(n_f_bins_inv)):\n",
    "    neyman_construction_inv[f_idx] = generate_ts_dist(f_bin_edges_inv[f_idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Anschließend wird das Konfidenzlevel $\\alpha$ festgelegt und die kritischen Werte $\\zeta$ berechnet. $\\zeta$ bezeichnet dabei die Grenze des Quantils der Teststatistik, bei dem der Wert des Konfidenzlevels erreicht wird. Dieser Wert wird für jede vom injezierten $f$ abhängige Teststatistikverteilung berechnet.\n",
    "\n",
    "$\\int\\limits_0^\\zeta -2\\log\\left( \\frac{\\mathcal{L}(\\mu_t)}{\\mathcal{L}(\\mu_b)} \\right) \\mathrm{d}TS = \\alpha$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha = 90\n",
    "critical_values = np.percentile(neyman_construction_inv, alpha, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nach dem Bestimmen der kritischen Werte, können nun die TS-Verteilungen histogrammiert werden, um das äquivalent zur Neyman Konstruktion zu erstellen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_ts_bins_inv = 100\n",
    "ts_bin_edges_inv, ts_bin_mids_inv = create_linear_bin_edges_and_mids(0, 20, n_ts_bins_inv)\n",
    "neyman_construction_binned = np.empty((n_f_bins_inv, n_ts_bins_inv))\n",
    "\n",
    "for idx in range(n_f_bins_inv):\n",
    "    neyman_construction_binned[idx], _ = np.histogram(\n",
    "        neyman_construction_inv[idx],\n",
    "        bins=ts_bin_edges_inv,\n",
    "        density=True,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Diese kleinschrittig in $f$ abgefahrenen Teststatistikverteilungen können gut mit den kritischen Werten dargestellt werden. Diese können des Weiteren mit dem Wilks Theorem verglichen werden.\n",
    "\n",
    "Gut zu erkennen ist, wie für höhere Signalbeiträge, also einer höheren Statistik des Signals in den Messdaten, die kritischen Werte gegen das Wilks Theorem konvergieren."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b381fc3c8a6d4e2e97c806060489be97",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.colorbar.Colorbar at 0x7f33aa5bc1f0>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fig = plt.figure(constrained_layout=True)\n",
    "ax = fig.add_subplot(1, 1, 1)\n",
    "\n",
    "im = ax.pcolormesh(ts_bin_edges_inv, f_bin_edges_inv, neyman_construction_binned, norm=LogNorm())\n",
    "\n",
    "ax.plot(critical_values, f_bin_mids_inv, color = 'r', ls='--', label='critical values')\n",
    "ax.axvline(x = chi2.ppf(0.9, 1) / 2, ls='--', color=\"k\", label='Wilks Theorem')\n",
    "\n",
    "ax.set_xlabel(r\"$\\mathcal{TS}$\")\n",
    "ax.set_ylabel(r\"$f$\")\n",
    "ax.legend()\n",
    "\n",
    "fig.colorbar(im)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Schließlich kann mithilfe eines Likelihood-Scans aus den Schnittpunkten der gemessenen Teststatistikverteilung und den kritischen Werten das $90\\%$ Konfidenz Intervall bestimmt werden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "481db546fbbd4c71b33bb4ed96fe7ad5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f33aa463100>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llh_scan = [calculate_ts(data, f_0=f_bin_mids_inv[idx]) for idx in range(n_f_bins_inv)]\n",
    "\n",
    "fig = plt.figure(constrained_layout=True)\n",
    "ax = fig.add_subplot(1, 1, 1)\n",
    "\n",
    "ax.plot(f_bin_mids_inv, critical_values, color = 'r', ls = '--', label='critical values')\n",
    "ax.plot(f_bin_mids_inv, llh_scan, label='llh scan')\n",
    "\n",
    "\n",
    "ax.axvline(x = f_true, ls = '--', color='C1', label=r\"$f_\\mathrm{true}$\")\n",
    "ax.axvline(x = result.x, ls = '--', color='C2', label=r\"$f_\\mathrm{best}$\")\n",
    "\n",
    "ax.set_xlabel(r\"$f$\") \n",
    "ax.set_ylabel(r\"$TS$\")\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ein Vergleich mit den vorher berechneten Grenzen:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f_low_old: 0.021, f_up_old: 0.109\n",
      "f_low_new: 0.016, f_up_new:  0.096\n"
     ]
    }
   ],
   "source": [
    "f_min = f_bin_mids_inv[llh_scan < critical_values].min()\n",
    "f_max = f_bin_mids_inv[llh_scan < critical_values].max()\n",
    "\n",
    "print(\"f_low_old: {:.3f}, f_up_old: {:.3f}\".format(f_low, f_up))\n",
    "print(\"f_low_new: {:.3f}, f_up_new:  {:.3f}\".format(f_min, f_max))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}