2-datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Datasets\n",
    "Original Author: Javier Duarte | Edited by Sitong An for CMSDAS2019@Pisa\n",
    "\n",
    "Now we start on the problem of gluon fusion (gg) / vector boson fusion (VV) classificaiton proper. \n",
    "\n",
    "First we need to import the dataset from ROOT files. This notebook uses root2hdf5 utility. Alternativaly, you could also use uproot, described in notebook 2.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert datasets from `ROOT` to `HDF5`\n",
    "Here we convert the datasets using `root2hdf5` utility which comes with `rootpy`: http://www.rootpy.org/commands/root2hdf5.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting tables\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/12/63/007b9cf964a8c6b1a92ddc00b2a6a369e132e69ee71d800cb212847e061e/tables-3.4.4-cp27-cp27mu-manylinux1_x86_64.whl (3.8MB)\n",
      "\u001b[K    100% |████████████████████████████████| 3.8MB 5.6MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: six>=1.9.0 in /cvmfs/sft.cern.ch/lcg/views/LCG_94/x86_64-centos7-gcc7-opt/lib/python2.7/site-packages (from tables) (1.11.0)\n",
      "Requirement already satisfied: numpy>=1.8.0 in /cvmfs/sft.cern.ch/lcg/views/LCG_94/x86_64-centos7-gcc7-opt/lib/python2.7/site-packages (from tables) (1.14.2)\n",
      "Requirement already satisfied: numexpr>=2.5.2 in /cvmfs/sft.cern.ch/lcg/views/LCG_94/x86_64-centos7-gcc7-opt/lib/python2.7/site-packages (from tables) (2.6.6)\n",
      "\u001b[31mtensorflow 1.8.0 requires backports.weakref>=1.0rc1, which is not installed.\u001b[0m\n",
      "\u001b[31mpy2neo 4.0.0 requires colorama, which is not installed.\u001b[0m\n",
      "\u001b[31mpy2neo 4.0.0 requires neo4j-driver>=1.6.0, which is not installed.\u001b[0m\n",
      "\u001b[31mpylint 2.0.0 has requirement astroid>=2.0.0, but you'll have astroid 1.6.5 which is incompatible.\u001b[0m\n",
      "\u001b[31mtensorboard 1.8.0 has requirement bleach==1.5.0, but you'll have bleach 2.1.3 which is incompatible.\u001b[0m\n",
      "\u001b[31mtensorboard 1.8.0 has requirement html5lib==0.9999999, but you'll have html5lib 1.0.1 which is incompatible.\u001b[0m\n",
      "Installing collected packages: tables\n",
      "\u001b[33m  The scripts pt2to3, ptdump, ptrepack and pttree are installed in '/eos/user/q/qnguyen/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
      "Successfully installed tables-3.4.4\n",
      "\u001b[33mYou are using pip version 10.0.1, however version 19.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install tables --user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:rootpy.root2hdf5] Converting data/ntuple_4mu_VV.root ...\n",
      "INFO:rootpy.root2hdf5] Will convert 1 tree in /\n",
      "INFO:rootpy.root2hdf5] Converting tree 'HZZ4LeptonsAnalysisReduced' with 25817 entries ...\n",
      "INFO:rootpy.root2hdf5] Created data/ntuple_4mu_VV.h5\n",
      "INFO:rootpy.root2hdf5] Converting data/ntuple_4mu_bkg.root ...\n",
      "INFO:rootpy.root2hdf5] Will convert 1 tree in /\n",
      "INFO:rootpy.root2hdf5] Converting tree 'HZZ4LeptonsAnalysisReduced' with 58107 entries ...\n",
      "INFO:rootpy.root2hdf5] Created data/ntuple_4mu_bkg.h5\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "root2hdf5 -f data/ntuple_4mu_VV.root data/ntuple_4mu_bkg.root"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load `HDF5` datasets\n",
    "Here we load the converted `HDF5` datasets into structured `NumPy` arrays. Note these structures arrays permit one to manipulate the data by named fields: https://docs.scipy.org/doc/numpy-1.13.0/user/basics.rec.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import h5py\n",
    "\n",
    "treename = 'HZZ4LeptonsAnalysisReduced'\n",
    "filename = {}\n",
    "h5file = {}\n",
    "params = {}\n",
    "\n",
    "filename['bkg'] = 'data/ntuple_4mu_bkg.h5'\n",
    "filename['VV'] = 'data/ntuple_4mu_VV.h5'\n",
    "\n",
    "h5file['bkg'] = h5py.File(filename['bkg'], 'r') # open HDF5 file read-only\n",
    "h5file['VV'] = h5py.File(filename['VV'], 'r') \n",
    "\n",
    "params['bkg'] = h5file['bkg'][treename][()] # returns a structured NumPy array\n",
    "params['VV'] = h5file['VV'][treename][()]\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print all variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('f_run', 'f_lumi', 'f_event', 'f_weight', 'f_int_weight', 'f_pu_weight', 'f_eff_weight', 'f_lept1_pt', 'f_lept1_eta', 'f_lept1_phi', 'f_lept1_charge', 'f_lept1_pfx', 'f_lept1_sip', 'f_lept2_pt', 'f_lept2_eta', 'f_lept2_phi', 'f_lept2_charge', 'f_lept2_pfx', 'f_lept2_sip', 'f_lept3_pt', 'f_lept3_eta', 'f_lept3_phi', 'f_lept3_charge', 'f_lept3_pfx', 'f_lept3_sip', 'f_lept4_pt', 'f_lept4_eta', 'f_lept4_phi', 'f_lept4_charge', 'f_lept4_pfx', 'f_lept4_sip', 'f_iso_max', 'f_sip_max', 'f_Z1mass', 'f_Z2mass', 'f_angle_costhetastar', 'f_angle_costheta1', 'f_angle_costheta2', 'f_angle_phi', 'f_angle_phistar1', 'f_pt4l', 'f_eta4l', 'f_mass4l', 'f_mass4lErr', 'f_njets_pass', 'f_deltajj', 'f_massjj', 'f_D_jet', 'f_jet1_pt', 'f_jet1_eta', 'f_jet1_phi', 'f_jet1_e', 'f_jet2_pt', 'f_jet2_eta', 'f_jet2_phi', 'f_jet2_e', 'f_D_bkg_kin', 'f_D_bkg', 'f_D_gg', 'f_D_g4', 'f_Djet_VAJHU', 'f_pfmet')\n"
     ]
    }
   ],
   "source": [
    "print(params['bkg'].dtype.names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print the shape of the NumPy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(58107,)\n"
     ]
    }
   ],
   "source": [
    "print(params['bkg'].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print the the first entry of the NumPy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 4, 630, 0.00064811, 0., 1.2290536, 1., 32.80312, 0.35433915, -1.4164597, -1., 0., -1.2187678, 23.372433, 0.25018594, 0.9821058, 1., 0.02498744, -1.2894976, 19.231188, -1.2650819, -0.02464505, -1., 0.03294736, -1.8134387, 6.870271, -0.45518333, -0.80094427, 1., 0., 0.54434615, 0., 0., 51.681366, 12.933985, 0.963756, 0.22460742, 0.7629764, 2.597239, 1.9826626, 45.872066, -0.35886624, 91.09813, 0., 1., -999., -999., -999., 35.839058, 0.5195489, 2.5928829, 36.253284, 0., 0., 0., 0., 0.3630876, 0.3630876, -2.2493208e-05, 0.8271159, -1., 18.884806)\n"
     ]
    }
   ],
   "source": [
    "print(params['bkg'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print mass4l value of first entry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "91.09813\n"
     ]
    }
   ],
   "source": [
    "print(params['bkg'][0]['f_mass4l'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print massjj value of first entry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-999.0\n"
     ]
    }
   ],
   "source": [
    "print(params['bkg'][0]['f_massjj'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert `NumPy` arrays to `pandas` DataFrames\n",
    "In my opinion, `pandas` DataFrames are a more convenient/flexible data container in python: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html. \n",
    "So we'll use this instead of structured `NumPy` arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = {}\n",
    "df['bkg'] = pd.DataFrame(params['bkg'])\n",
    "df['VV'] = pd.DataFrame(params['VV'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print first entry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   f_run  f_lumi  f_event  f_weight  f_int_weight  f_pu_weight  f_eff_weight  \\\n",
      "0      1       4      630  0.000648           0.0     1.229054           1.0   \n",
      "\n",
      "   f_lept1_pt  f_lept1_eta  f_lept1_phi    ...      f_jet2_pt  f_jet2_eta  \\\n",
      "0    32.80312     0.354339     -1.41646    ...            0.0         0.0   \n",
      "\n",
      "   f_jet2_phi  f_jet2_e  f_D_bkg_kin   f_D_bkg    f_D_gg    f_D_g4  \\\n",
      "0         0.0       0.0     0.363088  0.363088 -0.000022  0.827116   \n",
      "\n",
      "   f_Djet_VAJHU    f_pfmet  \n",
      "0          -1.0  18.884806  \n",
      "\n",
      "[1 rows x 62 columns]\n"
     ]
    }
   ],
   "source": [
    "print(df['bkg'].iloc[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print shape of DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(58107, 62)\n"
     ]
    }
   ],
   "source": [
    "print(df['bkg'].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print first entry for f_mass4l and f_massjj"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    f_mass4l  f_massjj\n",
      "0  91.098129    -999.0\n"
     ]
    }
   ],
   "source": [
    "print(df['bkg'][['f_mass4l','f_massjj']].iloc[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "convert back into unstructured NumPY array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.00000000e+00  4.00000000e+00  6.30000000e+02 ...  8.27115893e-01\n",
      "  -1.00000000e+00  1.88848057e+01]\n",
      " [ 1.00000000e+00  1.97100000e+03  3.81019000e+05 ...  4.15622257e-02\n",
      "  -1.00000000e+00  2.95897903e+01]\n",
      " [ 1.00000000e+00  7.37000000e+03  1.42459000e+06 ...  8.38690639e-01\n",
      "  -1.00000000e+00  2.04643517e+01]\n",
      " ...\n",
      " [ 1.00000000e+00  3.44060000e+04  6.65084800e+06 ...  8.02676916e-01\n",
      "  -1.00000000e+00  1.25938129e+01]\n",
      " [ 1.00000000e+00  3.44060000e+04  6.65092600e+06 ...  7.27994442e-02\n",
      "  -1.00000000e+00  3.35813141e+01]\n",
      " [ 1.00000000e+00  3.44060000e+04  6.65094300e+06 ...  1.21763824e-02\n",
      "  -1.00000000e+00  1.98157139e+01]]\n",
      "(58107, 62)\n"
     ]
    }
   ],
   "source": [
    "print(df['bkg'].values)\n",
    "print(df['bkg'].values.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "get boolean array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0        False\n",
      "1         True\n",
      "2        False\n",
      "3         True\n",
      "4         True\n",
      "5         True\n",
      "6         True\n",
      "7         True\n",
      "8         True\n",
      "9        False\n",
      "10        True\n",
      "11        True\n",
      "12        True\n",
      "13       False\n",
      "14        True\n",
      "15        True\n",
      "16        True\n",
      "17        True\n",
      "18        True\n",
      "19        True\n",
      "20        True\n",
      "21       False\n",
      "22        True\n",
      "23        True\n",
      "24       False\n",
      "25       False\n",
      "26       False\n",
      "27        True\n",
      "28        True\n",
      "29        True\n",
      "         ...  \n",
      "58077     True\n",
      "58078     True\n",
      "58079     True\n",
      "58080    False\n",
      "58081     True\n",
      "58082     True\n",
      "58083     True\n",
      "58084     True\n",
      "58085    False\n",
      "58086     True\n",
      "58087    False\n",
      "58088     True\n",
      "58089     True\n",
      "58090     True\n",
      "58091     True\n",
      "58092     True\n",
      "58093     True\n",
      "58094    False\n",
      "58095     True\n",
      "58096     True\n",
      "58097     True\n",
      "58098     True\n",
      "58099    False\n",
      "58100    False\n",
      "58101    False\n",
      "58102    False\n",
      "58103     True\n",
      "58104    False\n",
      "58105     True\n",
      "58106     True\n",
      "Name: f_mass4l, Length: 58107, dtype: bool\n"
     ]
    }
   ],
   "source": [
    "print(df['bkg']['f_mass4l'] > 125)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "cut usigg this boolean array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1        201.847610\n",
      "3        586.597412\n",
      "4        135.589798\n",
      "5        734.903442\n",
      "6        341.958466\n",
      "7        254.073425\n",
      "8        209.200073\n",
      "10       249.034500\n",
      "11       152.691574\n",
      "12       217.140778\n",
      "14       381.590332\n",
      "15       264.585419\n",
      "16       193.972916\n",
      "17       310.875885\n",
      "18       178.564941\n",
      "19       222.176819\n",
      "20       301.437988\n",
      "22       473.301605\n",
      "23       199.354355\n",
      "27       126.852921\n",
      "28       280.004089\n",
      "29       231.275574\n",
      "30       256.445831\n",
      "31       248.573166\n",
      "32       572.620361\n",
      "33       593.505493\n",
      "36       220.300735\n",
      "37       178.801941\n",
      "38       229.302322\n",
      "39       277.066864\n",
      "            ...    \n",
      "58066    229.215164\n",
      "58067    154.112381\n",
      "58069    218.869415\n",
      "58070    405.237152\n",
      "58071    216.917175\n",
      "58072    235.896942\n",
      "58073    249.296753\n",
      "58075    151.795853\n",
      "58076    177.322540\n",
      "58077    238.749847\n",
      "58078    248.517776\n",
      "58079    195.696304\n",
      "58081    256.388489\n",
      "58082    221.203857\n",
      "58083    204.776276\n",
      "58084    320.152130\n",
      "58086    186.158600\n",
      "58088    199.261490\n",
      "58089    186.380890\n",
      "58090    235.331329\n",
      "58091    244.582077\n",
      "58092    149.849808\n",
      "58093    235.621552\n",
      "58095    212.593323\n",
      "58096    261.331818\n",
      "58097    225.355103\n",
      "58098    214.074249\n",
      "58103    252.845184\n",
      "58105    250.977417\n",
      "58106    229.470154\n",
      "Name: f_mass4l, Length: 42219, dtype: float32\n"
     ]
    }
   ],
   "source": [
    "print(df['bkg']['f_mass4l'][df['bkg']['f_mass4l'] > 125])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional) Direct conversion to pandas dataframe from ROOT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Supported in ROOT 6.14 release:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "file = ROOT.TFile(filename)\n",
    "tree = file.Get(treename)\n",
    "data, columns = tree.AsMatrix(return_labels=True)\n",
    "df = pandas.DataFrame(data=data, columns=columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Supported in ROOT 6.16 release:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "df = ROOT.RDataFrame(treename, filename)\n",
    "npy = df.AsNumpy()\n",
    "df = pandas.DataFrame(npy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Root.RDataFrame is a utility that aims to provide similar functionalities to pandas Dataframe within ROOT supported since ROOT 6.14. It is developed to provide for a more modern, convenient data analysis interface from within ROOT. If you have extra time, you could click on this link [https://nbviewer.jupyter.org/url/root.cern.ch/doc/master/notebooks/df001_introduction.py.nbconvert.ipynb] for an  introductory demonstration of how it works.\n",
    "For more information, go to [https://root.cern/doc/master/group__tutorial__dataframe.html]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}