Example_01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTreeReader: Looping on [TTrees](https://root.cern.ch/doc/master/classTTree.html) in Python, fast.\n",
    "<hr style=\"border-top-width: 4px; border-top-color: #34609b;\">\n",
    "The PyTreeReader class solves the problem of looping in a performant way on TTrees in Python. This is achieved just in time compiling a C++ class tailored to the branches the user wants to read and interfacing it conveniently to Python.\n",
    "Usability is a key. The high usability of the old way of looping on trees in Python is preserved, implying for the user as little overhead as possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparation\n",
    "We include ROOT and the PyTreeReader class. Clearly this will have to be provided more conviniently, e.g. as part of the pythonisations PyROOT already provides for TTree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "require(['notebook'],\n",
       "  function() {\n",
       "    IPython.CodeCell.config_defaults.highlight_modes['magic_text/x-c++src'] = {'reg':[/^%%cpp/]};\n",
       "    console.log(\"JupyROOT - %%cpp magic configured\");\n",
       "  }\n",
       ");\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Welcome to JupyROOT 6.07/07\n"
     ]
    }
   ],
   "source": [
    "import ROOT\n",
    "from PyTreeReader import PyTreeReader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is to create an histogram and read a tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "h = ROOT.TH1F(\"h\",\"h\",1024,-256,256)\n",
    "fill = h.Fill\n",
    "\n",
    "f = ROOT.TFile(ROOT.gROOT.GetTutorialsDir()+\"/hsimple.root\")\n",
    "tree = f.ntuple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Traditional looping\n",
    "Now we establish the baseline: banchmark of the old way for looping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 loop, best of 1: 725 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n 1 -r 1\n",
    "for event in tree:\n",
    "    fill(event.px*event.py*event.pz*event.random)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enters the PyTreeReader\n",
    "This is how we use for the first time the PyTreeReader and how we benchmark it. The only difference within the loop body consists in invoking functions called as the branches rather than data members called as the branches themselves. Even for this trivial tree, the difference is impressive.\n",
    "\n",
    "Actually, having little read, decompression and deserialisation is better: the difference in time is really due to the superior performance of the PyTreeReader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 loop, best of 1: 182 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n 1 -r 1\n",
    "for event in PyTreeReader(tree):\n",
    "    fill(event.px()*event.py()*event.pz()*event.random())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Caching in memory the entire tree\n",
    "It is possible to instruct the PyTreeReader to cache in memory the content of the tree. We'll see two benchmarks: the time needed to cache and the time needed to loop.\n",
    "\n",
    "### Caching benchmark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 loop, best of 1: 165 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n 1 -r 1\n",
    "ptr = PyTreeReader(tree, cache=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ptr = PyTreeReader(tree, cache=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loop on the cached values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 loop, best of 1: 128 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n 1 -r 1\n",
    "for event in ptr:\n",
    "    fill(event.px()*event.py()*event.pz()*event.random())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting only some branches\n",
    "It can be inconvenient to construct methods to access all the branches, even more to cache all of their contents by default. This is the reason why PyTreeReader provides the *pattern* constructor argument. For example, in order to consider only branches whose names start with \"p\", one can do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 loop, best of 1: 17.7 ms per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n 1 -r 1\n",
    "ptr = PyTreeReader(tree, cache=True, pattern=\"p[*]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pattern is indeed the pattern to match the desired branches by name (**fnmatch** is used). Its default value is simple, \"*\";"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accessing a vector of cached values\n",
    "Caching values in memory has a price but once this step has been accomplished, more flexibility is available. For example, all the values of a particular branch can be retrieved in bulk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0118254804984\n",
      "0.0378782115877\n",
      "-0.0105458786711\n",
      "0.0484584420919\n",
      "0.509063661098\n",
      "-1.8647428751\n",
      "0.397898256779\n",
      "0.509420633316\n",
      "0.837935447693\n",
      "-1.09520089626\n"
     ]
    }
   ],
   "source": [
    "for py in ptr.py_array()[:10]: print py # Better stopping after a few of them :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wondering about the type returned by *py_array*? A widely adopted C++ data structure contiguous in memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<ROOT.vector<float> object at 0x2e74118>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ptr.py_array()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What can be done more?\n",
    "This is the first implementation of the class. A lot can be achieved with seizable performance improvements with respect to the traditional way of looping. The usability is preserved.\n",
    "\n",
    "However, some interesting features could be added:\n",
    " - If the TTree is a TNtuple, interface to Pandas. For example by getting a panda series.\n",
    " - We want to get rid of all loops. Ok, PyTreeReader speeds them up but can be used as the engine to remove them :) One could think of adding to our histogram classes a method to fill the histogram with a collection of points which is simpler than **TH1::FillN** (no size, just the collection).\n",
    "   - One could even think about doing the same for graphs.\n",
    "   "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}