4_Numpy_Pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 NumPy and Pandas\n",
    "\n",
    "## Introduction\n",
    "\n",
    "After getting used to how Python works, it is the moment to begin getting our hands dirty with data analysis. We will study two packages: [NumPy](http://www.numpy.org/) is the fundamental numeric computing and linear algebra package in Python, that allows for decent data analysis. We will learn it not only for the data analysis, but more importantly because it will be a package that will be always present in our `import` section as scientists. After NumPy we will go to [Pandas](http://pandas.pydata.org/). Pandas is a dedicated data analysis package with a lot more functionalities than NumPy, making our life much easier in terms of data visualization and manipulation. All the power of Pandas will be completely unleashed in Section 5, where we will see how to visualize information in plots.\n",
    "\n",
    "As usual, we begin importing the necessary packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NumPy: 1.21.0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "print(\"NumPy:\", np.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numeric Python (NumPy)\n",
    "\n",
    "NumPy is an open-source add-on module to Python that provides common mathematical and numerical routines in pre-compiled, fast functions. These are growing into highly mature packages that provide functionality that meets, or perhaps exceeds, that\n",
    "associated with common commercial software like MATLAB. The [NumPy](http://www.numpy.org/) (Numeric Python) package provides basic routines for manipulating large arrays and matrices of numeric data. The main object NumPy works with is a *homogeneous multidimensional array*. Despite its intimidating name, these are nothing but tables of numbers, each labelled by a tuple of indices.\n",
    "\n",
    "We will now explore some capabilities of NumPy that will prove very useful not only for data analyisis, but throughout all our life with Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating Arrays\n",
    "\n",
    "As mentioned before, the main object in NumPy is the *array*. Creating one is as easy as calling the command `array`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([1, 2, 3, 4, 5, 6, 7, 8, 9])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a list\n",
    "mylist = [1, 2, 3, 4, 5, 6, 7, 8, 9]\n",
    "\n",
    "# Create a numpy array from a list\n",
    "my1darr = np.array(mylist)\n",
    "\n",
    "# Data type of a numpy array\n",
    "print(type(my1darr))\n",
    "\n",
    "my1darr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same applies to multidimensional arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3],\n",
       "       [4, 5, 6],\n",
       "       [7, 8, 9]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a numpy array with three elemets, with three elements each.\n",
    "my2darr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])\n",
    "my2darr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[1],\n",
       "        [2],\n",
       "        [3]],\n",
       "\n",
       "       [[4],\n",
       "        [5],\n",
       "        [6]],\n",
       "\n",
       "       [[7],\n",
       "        [8],\n",
       "        [9]]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a numpy array with three elemets, with three elements each, with a single element each\n",
    "my3darr = np.array([[[1], [2], [3]], [[4], [5], [6]], [[7], [8], [9]]])\n",
    "my3darr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A NumPy array has a number of dimensions (or *axes*). To obtain the number of axes and the size of each of them you use the command `shape`. For 2-dimensional arrays (matrices), the order corresponds to (rows, columns)\n",
    "\n",
    "There are two different ways of calling the `shape` command, either with `np.shape(arr)` or `arr.shape`. This is not the only command that works in both formats, and we will be finding some more in our way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(9,)\n",
      "(3, 3)\n",
      "(3, 3, 1)\n",
      "(9,)\n",
      "(3, 3)\n",
      "(3, 3, 1)\n"
     ]
    }
   ],
   "source": [
    "# Shape using `.shape`\n",
    "print(my1darr.shape)\n",
    "print(my2darr.shape)\n",
    "print(my3darr.shape)\n",
    "\n",
    "# Shape using `np.shape()`\n",
    "print(np.shape(my1darr))\n",
    "print(np.shape(my2darr))\n",
    "print(np.shape(my3darr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is one restriction with respect to the use of lists: while you could create lists with data of different type, all the data in an array has to be of the same type, and it will be converted automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "1.0\n",
      "False\n",
      "<class 'float'>\n",
      "<class 'numpy.str_'>\n"
     ]
    }
   ],
   "source": [
    "# Create a list with floats and strings\n",
    "lst = [1.0, \"cat\"]\n",
    "print((lst[0]))\n",
    "\n",
    "# Create a numpy array from a list with mixed data types\n",
    "arr = np.array(lst)\n",
    "print((arr[0]))\n",
    "\n",
    "# The data type of the 1. in the list is the same data type than the 1. in the array?\n",
    "print(type((lst[0])) == type((arr[0])))\n",
    "print(type(lst[0]))\n",
    "print(type(arr[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(We will go deeper into indexing in a while)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Special Arrays\n",
    "\n",
    "Now we review some built-in functions that create matrices commonly used. `ones` and `zeros` return arrays of given shape and type (default is float64), filled with ones or zeros, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0.],\n",
       "       [0., 0.],\n",
       "       [0., 0.]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a zero-filled numpy array of shape 3 x 2\n",
    "np.zeros((3, 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[1, 1, 1, 1],\n",
       "        [1, 1, 1, 1]],\n",
       "\n",
       "       [[1, 1, 1, 1],\n",
       "        [1, 1, 1, 1]],\n",
       "\n",
       "       [[1, 1, 1, 1],\n",
       "        [1, 1, 1, 1]]], dtype=int8)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a one-filled numpy array of shape 3 x 2 x 4\n",
    "np.ones((3, 2, 4), dtype=np.int8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`eye(d)` returns a 2-D, dimension-$d$ array with ones on the diagonal and zeros elsewhere."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 0.],\n",
       "       [0., 1., 0.],\n",
       "       [0., 0., 1.]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create an numpy array of shape 3 x 3 with ones in the diagonal\n",
    "np.eye(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`eye` can also create arrays with ones in upper and lower diagonals. To achieve this, you must call `eye(d, d, k)` where $k$ denotes the diagonal (positive for above the center diagonal, negative for below), or `eye(d, k=num)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0. 0. 1. 0. 0.]\n",
      " [0. 0. 0. 1. 0.]\n",
      " [0. 0. 0. 0. 1.]\n",
      " [0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0.]]\n",
      "\n",
      "\n",
      "[[0. 0. 0. 0.]\n",
      " [1. 0. 0. 0.]\n",
      " [0. 1. 0. 0.]\n",
      " [0. 0. 1. 0.]]\n"
     ]
    }
   ],
   "source": [
    "# Create an numpy array of shape 5 x 5 with ones in the diagonal shifted-up 2 positions\n",
    "print(np.eye(5, 5, 2))\n",
    "print(\"\\n\")\n",
    "\n",
    "# Create an numpy array of shape 4 x 4 with ones in the diagonal shifted-down 1 position\n",
    "print(np.eye(4, k=-1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`diag`, depending on the input, either constructs a diagonal array (if the input is a 1-D array) or extracts a diagonal from a matrix (if the input is a 2-D array)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1, 0, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 2, 0, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 3, 0, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 4, 0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 5, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 6, 0, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 7, 0, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 8, 0],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 9],\n",
       "       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a 2-D array with the array `my1darr` shifted-up 1 position with respect the diagonal\n",
    "np.diag(my1darr, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 5, 9])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The the diagonal from the 2-D array `my2darr`\n",
    "np.diag(my2darr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, 0],\n",
       "       [1, 0, 0, 0],\n",
       "       [0, 5, 0, 0],\n",
       "       [0, 0, 9, 0]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a 2-D array with the diagonal from the 2-D array `my2darr` shifted-down 1 position with respect the diagonal\n",
    "np.diag(np.diag(my2darr), -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`arange(begin, end, step)` returns evenly spaced values within a given interval. Note that the beginning point is included, but not the ending."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  2,  4,  6,  8, 10, 12, 14])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create range with start at 0, stop before 16, counting up by 2\n",
    "myrange = np.arange(0, 16, 2)\n",
    "myrange"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1**: Create an array of the first million of odd numbers, both with `arange` and using loops. Try timing both methods to see which one is faster. For that, use `%timeit`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2 ms ± 26.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n",
      "134 ms ± 1.99 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit np.arange(0, 2000000, 2)\n",
    "\n",
    "%timeit [i for i in range(2000000) if i % 2 == 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, `linspace(begin, end, points)` returns evenly spaced numbers over a specified interval. Here, instead of specifying the step, you specify the amount of points you want. Also with `linspace` you include the ending of the interval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  2.,  4.,  6.,  8., 10., 12., 14.])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create line with start at 0, stop at 14, getting 7 evenly spaced points\n",
    "myline = np.linspace(0, 14, 8)  #\n",
    "myline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "# Print the length of `myrange` and `myline`\n",
    "print(len(myrange))\n",
    "print(len(myline))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`reshape` changes the shape of an array, but not its data. This is another of the commands that can be called before or after the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0  2  4  6]\n",
      " [ 8 10 12 14]]\n",
      "[[ 0  2]\n",
      " [ 4  6]\n",
      " [ 8 10]\n",
      " [12 14]]\n",
      "\n",
      "\n",
      "[[ 0  2  4  6]\n",
      " [ 8 10 12 14]]\n",
      "[[ 0  2]\n",
      " [ 4  6]\n",
      " [ 8 10]\n",
      " [12 14]]\n"
     ]
    }
   ],
   "source": [
    "# Reshape `myrange` using `np.reshape()`\n",
    "print(np.reshape(myrange, (2, 4)))\n",
    "print(np.reshape(myrange, (4, 2)))\n",
    "\n",
    "print(\"\\n\")\n",
    "\n",
    "# Reshape `myrange` using `.reshape()`\n",
    "print(myrange.reshape(2, 4))\n",
    "print(myrange.reshape(4, 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note however that, in order for these changes to be permanent, you should do a reassignment of the variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  2,  4,  6,  8, 10, 12, 14])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# After the reshapings above, the original array stays being the same\n",
    "myrange"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  2,  4,  6],\n",
       "       [ 8, 10, 12, 14]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mynewrange = myrange.reshape(2, 4)\n",
    "# Now that we have reassigned it is when it definitely changes shape\n",
    "mynewrange"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining Arrays\n",
    "\n",
    "The most general command for combining arrays is `concatenate(arrs, d)`. It takes a list of arrays and concatenates them along axis $d$. Remember your 3-D array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[1]\n",
      "  [2]\n",
      "  [3]]\n",
      "\n",
      " [[4]\n",
      "  [5]\n",
      "  [6]]\n",
      "\n",
      " [[7]\n",
      "  [8]\n",
      "  [9]]]\n",
      "(3, 3, 1)\n"
     ]
    }
   ],
   "source": [
    "# Remember the `my3darr` 3-D numpy array\n",
    "print(my3darr)\n",
    "print(my3darr.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[ 1]\n",
      "  [ 2]\n",
      "  [ 3]]\n",
      "\n",
      " [[ 4]\n",
      "  [ 5]\n",
      "  [ 6]]\n",
      "\n",
      " [[ 7]\n",
      "  [ 8]\n",
      "  [ 9]]\n",
      "\n",
      " [[10]\n",
      "  [20]\n",
      "  [30]]\n",
      "\n",
      " [[40]\n",
      "  [50]\n",
      "  [60]]\n",
      "\n",
      " [[70]\n",
      "  [80]\n",
      "  [90]]]\n",
      "(6, 3, 1)\n"
     ]
    }
   ],
   "source": [
    "# Concatenate `my3darr` with itsef multiplied by 10 along dimension 0\n",
    "conc_along0 = np.concatenate([my3darr, 10 * my3darr], 0)\n",
    "print(conc_along0)\n",
    "print(conc_along0.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[ 1]\n",
      "  [ 2]\n",
      "  [ 3]\n",
      "  [10]\n",
      "  [20]\n",
      "  [30]]\n",
      "\n",
      " [[ 4]\n",
      "  [ 5]\n",
      "  [ 6]\n",
      "  [40]\n",
      "  [50]\n",
      "  [60]]\n",
      "\n",
      " [[ 7]\n",
      "  [ 8]\n",
      "  [ 9]\n",
      "  [70]\n",
      "  [80]\n",
      "  [90]]]\n",
      "(3, 6, 1)\n"
     ]
    }
   ],
   "source": [
    "# Concatenate `my3darr` with itsef multiplied by 10 along dimension 1\n",
    "conc_along1 = np.concatenate([my3darr, 10 * my3darr], 1)\n",
    "print(conc_along1)\n",
    "print(conc_along1.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[ 1 10]\n",
      "  [ 2 20]\n",
      "  [ 3 30]]\n",
      "\n",
      " [[ 4 40]\n",
      "  [ 5 50]\n",
      "  [ 6 60]]\n",
      "\n",
      " [[ 7 70]\n",
      "  [ 8 80]\n",
      "  [ 9 90]]]\n",
      "(3, 3, 2)\n"
     ]
    }
   ],
   "source": [
    "# Concatenate `my3darr` with itsef multiplied by 10 along dimension 2\n",
    "conc_along2 = np.concatenate([my3darr, 10 * my3darr], 2)\n",
    "print(conc_along2)\n",
    "print(conc_along2.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, for common combinations there exist special commands. Use `vstack` to stack arrays in sequence vertically (row wise), `hstack` to stack arrays in sequence horizontally (column wise)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 3 4 5 6 7 8 9]\n",
      "(9,)\n"
     ]
    }
   ],
   "source": [
    "# Remember the `my1darr` 1-D numpy array from above\n",
    "print(my1darr)\n",
    "print(my1darr.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1  2  3  4  5  6  7  8  9]\n",
      " [10 20 30 40 50 60 70 80 90]]\n",
      "\n",
      "\n",
      "[ 1  2  3  4  5  6  7  8  9 10 20 30 40 50 60 70 80 90]\n"
     ]
    }
   ],
   "source": [
    "# Stack `my1darr` vertically and horizontally\n",
    "print(np.vstack([my1darr, 10 * my1darr]))\n",
    "print(\"\\n\")\n",
    "print(np.hstack([my1darr, 10 * my1darr]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2 3]\n",
      " [4 5 6]\n",
      " [7 8 9]]\n",
      "(3, 3)\n"
     ]
    }
   ],
   "source": [
    "# Remember the `my2darr` 2-D numpy array from above\n",
    "print(my2darr)\n",
    "print(my2darr.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1  2  3]\n",
      " [ 4  5  6]\n",
      " [ 7  8  9]\n",
      " [10 20 30]\n",
      " [40 50 60]\n",
      " [70 80 90]]\n",
      "\n",
      "\n",
      "[[ 1  2  3 10 20 30]\n",
      " [ 4  5  6 40 50 60]\n",
      " [ 7  8  9 70 80 90]]\n"
     ]
    }
   ],
   "source": [
    "# Stack `my2darr` vertically and horizontally\n",
    "print(np.vstack([my2darr, 10 * my2darr]))\n",
    "print(\"\\n\")\n",
    "print(np.hstack([my2darr, 10 * my2darr]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Operations\n",
    "\n",
    "You can perform easily element-wise operations on arrays of any shape. Use the typical symbols, +, -, \\*, / and \\*\\* to perform element-wise addition, subtraction, multiplication, division and power."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 3]\n",
      "[11 12 13]\n",
      "[3 6 9]\n",
      "[1.         0.5        0.33333333]\n",
      "[1.         0.62996052 0.48074986]\n",
      "[2 4 8]\n"
     ]
    }
   ],
   "source": [
    "x = np.array([1, 2, 3])\n",
    "print(x)\n",
    "print(x + 10)\n",
    "print(3 * x)\n",
    "print(1 / x)\n",
    "print(x ** (-2 / 3))\n",
    "print(2 ** x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also (and obviously) these symbols can be used to operate between two arrays, which must be of the same shape. If this is the case, they also do element-wise operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5 7 9]\n",
      "[ 4 10 18]\n",
      "[0.25 0.4  0.5 ]\n",
      "[  1  32 729]\n"
     ]
    }
   ],
   "source": [
    "y = np.arange(4, 7, 1)\n",
    "print(x + y)  # [1+4, 2+5, 3+6]\n",
    "print(x * y)  # [1*4, 2*5, 3*6]\n",
    "print(x / y)  # [1/4, 2/5, 3/6]\n",
    "print(x ** y)  # [1**4, 2**5, 3**6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For doing vector or matrix multiplication, the command to be used is `dot`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 3]\n",
      "[4 5 6]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(x)\n",
    "print(y)\n",
    "x.dot(y)  # 1*4 + 2*5 + 3*6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With python 3.5 matrix multiplication got it's own operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 3]\n",
      "[4 5 6]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(x)\n",
    "print(y)\n",
    "x @ y  # 1*4 + 2*5 + 3*6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[3 4 5]\n",
      " [4 5 6]\n",
      " [5 6 7]]\n",
      "\n",
      "\n",
      "[[0 1 0]\n",
      " [0 0 1]\n",
      " [1 0 0]]\n"
     ]
    }
   ],
   "source": [
    "X = np.array([[i + j for i in range(3, 6)] for j in range(3)])\n",
    "Y = np.diag([1, 1], 1) + np.diag([1], -2)\n",
    "\n",
    "print(X)\n",
    "print(\"\\n\")\n",
    "print(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The operator `*` returns the elements-wise multiplication of arrays with the same shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 4, 0],\n",
       "       [0, 0, 6],\n",
       "       [5, 0, 0]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X * Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The operator `@` returns the product of arrays compatible shapes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[5, 3, 4],\n",
       "       [6, 4, 5],\n",
       "       [7, 5, 6]])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X @ Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 2**: Take a 10x2 matrix representing $(x1,x2)$ coordinates and transform them into polar coordinates $(r,\\theta)$.\n",
    "\n",
    "*Hint 1: the inverse transformation is given by $x1 = r\\cos\\theta$, $x2 = r\\sin\\theta$*\n",
    "\n",
    "*Hint 2: generate random numbers with the functions in numpy.random*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.36320643 0.7211909  1.1128058  0.47301587 1.11097976 0.73025988\n",
      " 0.62726237 0.67379782 1.0605135  1.05922789]\n",
      "[0.64442544 1.2100256  0.92120452 1.03554393 0.8748161  0.93187241\n",
      " 1.4123739  1.23193151 1.04596646 0.43061869]\n"
     ]
    }
   ],
   "source": [
    "z = np.random.random((10, 2))\n",
    "x1, x2 = z[:, 0], z[:, 1]\n",
    "R = np.sqrt(x1 ** 2 + x2 ** 2)\n",
    "T = np.arctan2(x2, x1)\n",
    "print(R)\n",
    "print(T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transposing\n",
    "\n",
    "Transposition is a very important operation for linear algebra. Although NumPy is capable of correctly doing matrix-vector products correctly regardless of the orientation of the vector, it is not the case for products of matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "Z = np.arange(0, 12, 1).reshape((4, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0  1  2]\n",
      " [ 3  4  5]\n",
      " [ 6  7  8]\n",
      " [ 9 10 11]]\n",
      "[4 5 6]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([ 17,  62, 107, 152])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(Z)\n",
    "print(y)\n",
    "Z @ y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0  1  2]\n",
      " [ 3  4  5]\n",
      " [ 6  7  8]\n",
      " [ 9 10 11]]\n",
      "[4 5 6]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([ 17,  62, 107, 152])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(Z)\n",
    "print(y.T)\n",
    "Z @ y.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0  1  2]\n",
      " [ 3  4  5]\n",
      " [ 6  7  8]\n",
      " [ 9 10 11]]\n",
      "[[3 4 5]\n",
      " [4 5 6]\n",
      " [5 6 7]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[ 14,  17,  20],\n",
       "       [ 50,  62,  74],\n",
       "       [ 86, 107, 128],\n",
       "       [122, 152, 182]])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(Z)\n",
    "print(X)\n",
    "Z @ X  # (4 rows and 3 cols) @ (3 rows and 3 cols) yields (4 rows and 3 cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0  3  6  9]\n",
      " [ 1  4  7 10]\n",
      " [ 2  5  8 11]]\n",
      "[[3 4 5]\n",
      " [4 5 6]\n",
      " [5 6 7]]\n"
     ]
    }
   ],
   "source": [
    "print(Z.T)\n",
    "print(X)\n",
    "# Z.T @ X  # (3 rows and 4 cols) @ (3 rows and 3 cols) yields error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Array Methods\n",
    "\n",
    "To not have to go from NumPy arrays to lists back and forth, NumPy contains some functions to know properties of your arrays. Actually, there are more of these functions than in standard Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n",
      "-4\n",
      "3\n",
      "0.6\n",
      "3.2619012860600183\n"
     ]
    }
   ],
   "source": [
    "a = np.array([-4, -2, 1, 3, 5])\n",
    "print(a.max())\n",
    "print(a.min())\n",
    "print(a.sum())\n",
    "print(a.mean())\n",
    "print(a.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some interesting functions are `argmax` and `argmin`, which return the index of the maximum and minimum values in the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "print(a.argmax())\n",
    "print(a.argmin())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing/Slicing\n",
    "\n",
    "We have already seen briefly that to access individual elements you use the bracket notation: `array[ax_0, ax_1, ...]`, where the `ax_i` denotes the coordinate in the `i`-th axis. You can even use this to assign new values to your elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[198, 5, 6, 7]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = [4, 5, 6, 7]\n",
    "print(r[2])\n",
    "# Reassigning the value stored in index 0\n",
    "r[0] = 198\n",
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To select a range of rows or columns you can use a colon `:`. A second `:` can be used to indicate the step size. `array[start:stop:stepsize]`. If you leave `start` (`stop`) blank, the selection will go from the very beginning (until the very end) of the array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  0   1   4   9  16  25  36  49  64  81 100 121 144]\n",
      "[ 9 16 25 36 49 64 81]\n",
      "[ 4 25 64]\n",
      "[ 64 100 144]\n",
      "[64 36 16  4  0]\n"
     ]
    }
   ],
   "source": [
    "s = np.arange(13) ** 2\n",
    "print(s)\n",
    "\n",
    "# Starting by index 2, until index 10, pick each element\n",
    "print(s[3:10])\n",
    "\n",
    "# Starting by index 2, until index 10, pick every 3 elements\n",
    "print(s[2:10:3])\n",
    "\n",
    "# Starting by index -5, until the end, pick every 2 elements\n",
    "print(s[-5::2])\n",
    "\n",
    "# Starting by index -5, until the end, pick every 2 elements backwards\n",
    "print(s[-5::-2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same applies to matrices or higher-dimensional arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  1,  2,  3,  4,  5],\n",
       "       [ 6,  7,  8,  9, 10, 11],\n",
       "       [12, 13, 14, 15, 16, 17],\n",
       "       [18, 19, 20, 21, 22, 23],\n",
       "       [24, 25, 26, 27, 28, 29],\n",
       "       [30, 31, 32, 33, 34, 35]])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = np.arange(36).reshape((6, 6))\n",
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[13, 14],\n",
       "       [19, 20],\n",
       "       [25, 26]])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Pick rows 2 to 5, and cols 1 to 3 (remember, row 5 and col 3 not included)\n",
    "r[2:5, 1:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also select specific rows and columns, separated by commas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 7,  8],\n",
       "       [19, 20],\n",
       "       [25, 26]])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Pick rows 1, 3 and 4, and cols 1 to 3 (remember, col 3 not included)\n",
    "r[[1, 3, 4], 1:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A very useful tool is *conditional indexing*, where we apply a function, assignment... only to those elements of an array that satisfy some condition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  1,  2,  3,  4,  5],\n",
       "       [ 6,  7,  8,  9, 10, 11],\n",
       "       [12, 13, 14, 15, 16, 17],\n",
       "       [18, 19, 20, 21, 22, 23],\n",
       "       [24, 25, 26, 27, 28, 29],\n",
       "       [30, 30, 30, 30, 30, 30]])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r[r > 30] = 30\n",
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3**: Create a random 1-dimensional array, and find which element is closest to 0.7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6875432133749239\n"
     ]
    }
   ],
   "source": [
    "Z = np.random.uniform(0, 1, 100)\n",
    "z = 0.7\n",
    "m = Z[np.abs(Z - z).argmin()]\n",
    "print(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Copying Data\n",
    "\n",
    "**Be very careful with copying and modifying arrays in NumPy!** You will see the reason right now. Let's begin defining `r2` as a slice of r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  1,  2],\n",
       "       [ 6,  7,  8],\n",
       "       [12, 13, 14]])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r2 = r[:3, :3]\n",
    "r2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now let's set all its elements to zero"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0],\n",
       "       [0, 0, 0],\n",
       "       [0, 0, 0]])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r2[:] = 0\n",
    "r2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When looking at `r`, we see that it has also been changed!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  0,  0,  3,  4,  5],\n",
       "       [ 0,  0,  0,  9, 10, 11],\n",
       "       [ 0,  0,  0, 15, 16, 17],\n",
       "       [18, 19, 20, 21, 22, 23],\n",
       "       [24, 25, 26, 27, 28, 29],\n",
       "       [30, 30, 30, 30, 30, 30]])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The proper way of handling selections without modifying the original arrays is through the `copy` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  0,  0,  3,  4,  5],\n",
       "       [ 0,  0,  0,  9, 10, 11],\n",
       "       [ 0,  0,  0, 15, 16, 17],\n",
       "       [18, 19, 20, 21, 22, 23],\n",
       "       [24, 25, 26, 27, 28, 29],\n",
       "       [30, 30, 30, 30, 30, 30]])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r_copy = r.copy()\n",
    "r_copy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can safely modify `r_copy` without affecting `r`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f{r_copy}\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[ 0,  0,  0,  3,  4,  5],\n",
       "       [ 0,  0,  0,  9, 10, 11],\n",
       "       [ 0,  0,  0, 15, 16, 17],\n",
       "       [18, 19, 20, 21, 22, 23],\n",
       "       [24, 25, 26, 27, 28, 29],\n",
       "       [30, 30, 30, 30, 30, 30]])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r_copy[:] = 10\n",
    "print(\"f{r_copy}\\n\")\n",
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iterating Over Arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you can iterate over arrays in the same way as you iterate over lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 7],\n",
       "       [2, 8, 2],\n",
       "       [2, 6, 6],\n",
       "       [3, 6, 4]])"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = np.random.randint(0, 10, (4, 3))\n",
    "test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can iterate by row:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 7]\n",
      "[2 8 2]\n",
      "[2 6 6]\n",
      "[3 6 4]\n"
     ]
    }
   ],
   "source": [
    "for row in test:\n",
    "    print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or by row index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 7]\n",
      "[2 8 2]\n",
      "[2 6 6]\n",
      "[3 6 4]\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(test)):\n",
    "    print(test[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or by row and index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row 0 is [0 0 7]\n",
      "Row 1 is [2 8 2]\n",
      "Row 2 is [2 6 6]\n",
      "Row 3 is [3 6 4]\n"
     ]
    }
   ],
   "source": [
    "for i, row in enumerate(test):\n",
    "    print(f\"Row {i} is {row}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the same way as with lists, you can use `zip` to iterate over multiple iterables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  0, 49],\n",
       "       [ 4, 64,  4],\n",
       "       [ 4, 36, 36],\n",
       "       [ 9, 36, 16]], dtype=int32)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test2 = test ** 2\n",
    "test2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 7] + [ 0  0 49] = [ 0  0 56]\n",
      "[2 8 2] + [ 4 64  4] = [ 6 72  6]\n",
      "[2 6 6] + [ 4 36 36] = [ 6 42 42]\n",
      "[3 6 4] + [ 9 36 16] = [12 42 20]\n"
     ]
    }
   ],
   "source": [
    "for i, j in zip(test, test2):\n",
    "    print(f\"{i} + {j} = {i+j}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 4**: Create a function that iterates over the columns of a 2-dimensional array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Column 0 is [[0]\n",
      " [2]\n",
      " [2]\n",
      " [3]]\n",
      "Column 1 is [[0]\n",
      " [8]\n",
      " [6]\n",
      " [6]]\n",
      "Column 2 is [[7]\n",
      " [2]\n",
      " [6]\n",
      " [4]]\n"
     ]
    }
   ],
   "source": [
    "def iterate(df):\n",
    "    for i, row in enumerate(df):\n",
    "        shp = row.shape\n",
    "        row.shape = shp + (1,)\n",
    "        print(f\"Column {i} is {row}\")\n",
    "\n",
    "\n",
    "iterate(test.T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading and Saving Data\n",
    "\n",
    "To load and save data NumPy has the `loadtxt` and `savetxt` commands. However, they only work for two-dimensional arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 7.],\n",
       "       [2., 8., 2.],\n",
       "       [2., 6., 6.],\n",
       "       [3., 6., 4.]])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.savetxt(\"numpytest.txt\", test)\n",
    "np.loadtxt(\"numpytest.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas\n",
    "\n",
    "When dealing with numeric matrices and vectors in Python, NumPy makes life a lot easier. For more complex data, however, it leaves a bit to be desired. For those used to working with dedicated languages like R, doing data analysis directly with numpy feels like a step back. Fortunately, some nice folks have written the Python Data Analysis Library (a.k.a. [pandas](http://pandas.pydata.org/)). Pandas provides an R-like DataFrame, produces high quality plots with matplotlib, and integrates nicely with other libraries that expect NumPy arrays.\n",
    "\n",
    "Pandas works with `Series` of data, that then are arranged in `DataFrame`s. A dataframe will be the object closest to an Excel spreadsheet that you will see throughout the course (but of course, given that it is integrated in Python and can be combined with so many different packages, dataframes are much more powerful than Excel spreadsheets). The data in the series can be either qualitative or quantitative data. Creating a series is as easy as creating a NumPy array from a one-dimensional list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pandas: 1.3.1\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "print(\"Pandas:\", pd.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    Tiger\n",
       "1     Bear\n",
       "2    Moose\n",
       "dtype: object"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "animals = [\"Tiger\", \"Bear\", \"Moose\"]\n",
    "pd.Series(animals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    2\n",
       "2    3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numbers = [1, 2, 3]\n",
    "pd.Series(numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the series is indexed by default by integers. You can change this indexing by using a dictionary instead of a list for creating the series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Archery           Bhutan\n",
       "Golf            Scotland\n",
       "Sumo               Japan\n",
       "Taekwondo    South Korea\n",
       "dtype: object"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sports = {\n",
    "    \"Archery\": \"Bhutan\",\n",
    "    \"Golf\": \"Scotland\",\n",
    "    \"Sumo\": \"Japan\",\n",
    "    \"Taekwondo\": \"South Korea\",\n",
    "}\n",
    "s = pd.Series(sports)\n",
    "s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the other hand, `DataFrame`s can be built from two-dimensional arrays, with the ability of labelling columns and indexing the rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "      <th>E</th>\n",
       "      <th>F</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.920657</td>\n",
       "      <td>-0.582968</td>\n",
       "      <td>-0.585235</td>\n",
       "      <td>-0.445596</td>\n",
       "      <td>-0.681712</td>\n",
       "      <td>0.121091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.017103</td>\n",
       "      <td>0.510999</td>\n",
       "      <td>0.298593</td>\n",
       "      <td>-1.279607</td>\n",
       "      <td>-1.660310</td>\n",
       "      <td>-1.427842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-0.467513</td>\n",
       "      <td>-0.211803</td>\n",
       "      <td>-0.178767</td>\n",
       "      <td>0.860208</td>\n",
       "      <td>0.330933</td>\n",
       "      <td>0.128354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.682888</td>\n",
       "      <td>0.578031</td>\n",
       "      <td>0.535269</td>\n",
       "      <td>-0.447142</td>\n",
       "      <td>-1.350380</td>\n",
       "      <td>1.022790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.233067</td>\n",
       "      <td>1.069914</td>\n",
       "      <td>-0.672395</td>\n",
       "      <td>-0.324268</td>\n",
       "      <td>-1.213044</td>\n",
       "      <td>0.753714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2985</th>\n",
       "      <td>0.109807</td>\n",
       "      <td>1.160787</td>\n",
       "      <td>0.454515</td>\n",
       "      <td>0.761941</td>\n",
       "      <td>-0.448961</td>\n",
       "      <td>-0.549234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2988</th>\n",
       "      <td>0.289328</td>\n",
       "      <td>-0.046527</td>\n",
       "      <td>-1.958731</td>\n",
       "      <td>-0.859543</td>\n",
       "      <td>-0.004865</td>\n",
       "      <td>0.865963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2991</th>\n",
       "      <td>-2.171022</td>\n",
       "      <td>-0.993406</td>\n",
       "      <td>-0.547938</td>\n",
       "      <td>0.075995</td>\n",
       "      <td>-0.726624</td>\n",
       "      <td>-1.203640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2994</th>\n",
       "      <td>-1.542033</td>\n",
       "      <td>-1.032567</td>\n",
       "      <td>-0.546570</td>\n",
       "      <td>0.282012</td>\n",
       "      <td>-1.125996</td>\n",
       "      <td>-0.646228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2997</th>\n",
       "      <td>0.393561</td>\n",
       "      <td>-0.340553</td>\n",
       "      <td>1.729834</td>\n",
       "      <td>-0.750995</td>\n",
       "      <td>0.654371</td>\n",
       "      <td>0.932908</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             A         B         C         D         E         F\n",
       "0    -0.920657 -0.582968 -0.585235 -0.445596 -0.681712  0.121091\n",
       "3    -0.017103  0.510999  0.298593 -1.279607 -1.660310 -1.427842\n",
       "6    -0.467513 -0.211803 -0.178767  0.860208  0.330933  0.128354\n",
       "9     0.682888  0.578031  0.535269 -0.447142 -1.350380  1.022790\n",
       "12    0.233067  1.069914 -0.672395 -0.324268 -1.213044  0.753714\n",
       "...        ...       ...       ...       ...       ...       ...\n",
       "2985  0.109807  1.160787  0.454515  0.761941 -0.448961 -0.549234\n",
       "2988  0.289328 -0.046527 -1.958731 -0.859543 -0.004865  0.865963\n",
       "2991 -2.171022 -0.993406 -0.547938  0.075995 -0.726624 -1.203640\n",
       "2994 -1.542033 -1.032567 -0.546570  0.282012 -1.125996 -0.646228\n",
       "2997  0.393561 -0.340553  1.729834 -0.750995  0.654371  0.932908\n",
       "\n",
       "[1000 rows x 6 columns]"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sampling a 1000 rows 6 cols 2D array from the standard normal distribution and creating DataFrame\n",
    "u = pd.DataFrame(\n",
    "    np.random.randn(1000, 6),\n",
    "    index=np.arange(0, 3000, 3),\n",
    "    columns=[\"A\", \"B\", \"C\", \"D\", \"E\", \"F\"],\n",
    ")\n",
    "\n",
    "print(type(u))\n",
    "\n",
    "u"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you might have noticed, it is a bit ugly to deal with large dataframes. There are however some functions that allows to have an idea of the data in a frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "      <th>E</th>\n",
       "      <th>F</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.920657</td>\n",
       "      <td>-0.582968</td>\n",
       "      <td>-0.585235</td>\n",
       "      <td>-0.445596</td>\n",
       "      <td>-0.681712</td>\n",
       "      <td>0.121091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.017103</td>\n",
       "      <td>0.510999</td>\n",
       "      <td>0.298593</td>\n",
       "      <td>-1.279607</td>\n",
       "      <td>-1.660310</td>\n",
       "      <td>-1.427842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-0.467513</td>\n",
       "      <td>-0.211803</td>\n",
       "      <td>-0.178767</td>\n",
       "      <td>0.860208</td>\n",
       "      <td>0.330933</td>\n",
       "      <td>0.128354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.682888</td>\n",
       "      <td>0.578031</td>\n",
       "      <td>0.535269</td>\n",
       "      <td>-0.447142</td>\n",
       "      <td>-1.350380</td>\n",
       "      <td>1.022790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.233067</td>\n",
       "      <td>1.069914</td>\n",
       "      <td>-0.672395</td>\n",
       "      <td>-0.324268</td>\n",
       "      <td>-1.213044</td>\n",
       "      <td>0.753714</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           A         B         C         D         E         F\n",
       "0  -0.920657 -0.582968 -0.585235 -0.445596 -0.681712  0.121091\n",
       "3  -0.017103  0.510999  0.298593 -1.279607 -1.660310 -1.427842\n",
       "6  -0.467513 -0.211803 -0.178767  0.860208  0.330933  0.128354\n",
       "9   0.682888  0.578031  0.535269 -0.447142 -1.350380  1.022790\n",
       "12  0.233067  1.069914 -0.672395 -0.324268 -1.213044  0.753714"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "      <th>E</th>\n",
       "      <th>F</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2985</th>\n",
       "      <td>0.109807</td>\n",
       "      <td>1.160787</td>\n",
       "      <td>0.454515</td>\n",
       "      <td>0.761941</td>\n",
       "      <td>-0.448961</td>\n",
       "      <td>-0.549234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2988</th>\n",
       "      <td>0.289328</td>\n",
       "      <td>-0.046527</td>\n",
       "      <td>-1.958731</td>\n",
       "      <td>-0.859543</td>\n",
       "      <td>-0.004865</td>\n",
       "      <td>0.865963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2991</th>\n",
       "      <td>-2.171022</td>\n",
       "      <td>-0.993406</td>\n",
       "      <td>-0.547938</td>\n",
       "      <td>0.075995</td>\n",
       "      <td>-0.726624</td>\n",
       "      <td>-1.203640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2994</th>\n",
       "      <td>-1.542033</td>\n",
       "      <td>-1.032567</td>\n",
       "      <td>-0.546570</td>\n",
       "      <td>0.282012</td>\n",
       "      <td>-1.125996</td>\n",
       "      <td>-0.646228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2997</th>\n",
       "      <td>0.393561</td>\n",
       "      <td>-0.340553</td>\n",
       "      <td>1.729834</td>\n",
       "      <td>-0.750995</td>\n",
       "      <td>0.654371</td>\n",
       "      <td>0.932908</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             A         B         C         D         E         F\n",
       "2985  0.109807  1.160787  0.454515  0.761941 -0.448961 -0.549234\n",
       "2988  0.289328 -0.046527 -1.958731 -0.859543 -0.004865  0.865963\n",
       "2991 -2.171022 -0.993406 -0.547938  0.075995 -0.726624 -1.203640\n",
       "2994 -1.542033 -1.032567 -0.546570  0.282012 -1.125996 -0.646228\n",
       "2997  0.393561 -0.340553  1.729834 -0.750995  0.654371  0.932908"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 1000 entries, 0 to 2997\n",
      "Data columns (total 6 columns):\n",
      " #   Column  Non-Null Count  Dtype  \n",
      "---  ------  --------------  -----  \n",
      " 0   A       1000 non-null   float64\n",
      " 1   B       1000 non-null   float64\n",
      " 2   C       1000 non-null   float64\n",
      " 3   D       1000 non-null   float64\n",
      " 4   E       1000 non-null   float64\n",
      " 5   F       1000 non-null   float64\n",
      "dtypes: float64(6)\n",
      "memory usage: 54.7 KB\n"
     ]
    }
   ],
   "source": [
    "u.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "      <th>E</th>\n",
       "      <th>F</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.034690</td>\n",
       "      <td>-0.061913</td>\n",
       "      <td>-0.068992</td>\n",
       "      <td>-0.076187</td>\n",
       "      <td>-0.016044</td>\n",
       "      <td>-0.015323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.008428</td>\n",
       "      <td>1.018624</td>\n",
       "      <td>1.002010</td>\n",
       "      <td>1.018882</td>\n",
       "      <td>0.994240</td>\n",
       "      <td>0.957231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-3.873438</td>\n",
       "      <td>-3.581204</td>\n",
       "      <td>-2.851055</td>\n",
       "      <td>-3.334353</td>\n",
       "      <td>-3.375688</td>\n",
       "      <td>-2.631431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-0.627460</td>\n",
       "      <td>-0.648151</td>\n",
       "      <td>-0.720336</td>\n",
       "      <td>-0.767755</td>\n",
       "      <td>-0.639869</td>\n",
       "      <td>-0.711587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.009583</td>\n",
       "      <td>-0.076407</td>\n",
       "      <td>-0.076573</td>\n",
       "      <td>-0.045585</td>\n",
       "      <td>0.017264</td>\n",
       "      <td>-0.016043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.765485</td>\n",
       "      <td>0.619304</td>\n",
       "      <td>0.573383</td>\n",
       "      <td>0.619241</td>\n",
       "      <td>0.671474</td>\n",
       "      <td>0.642756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>2.687426</td>\n",
       "      <td>3.407755</td>\n",
       "      <td>3.116299</td>\n",
       "      <td>3.093830</td>\n",
       "      <td>3.418324</td>\n",
       "      <td>3.516588</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 A            B            C            D            E  \\\n",
       "count  1000.000000  1000.000000  1000.000000  1000.000000  1000.000000   \n",
       "mean      0.034690    -0.061913    -0.068992    -0.076187    -0.016044   \n",
       "std       1.008428     1.018624     1.002010     1.018882     0.994240   \n",
       "min      -3.873438    -3.581204    -2.851055    -3.334353    -3.375688   \n",
       "25%      -0.627460    -0.648151    -0.720336    -0.767755    -0.639869   \n",
       "50%       0.009583    -0.076407    -0.076573    -0.045585     0.017264   \n",
       "75%       0.765485     0.619304     0.573383     0.619241     0.671474   \n",
       "max       2.687426     3.407755     3.116299     3.093830     3.418324   \n",
       "\n",
       "                 F  \n",
       "count  1000.000000  \n",
       "mean     -0.015323  \n",
       "std       0.957231  \n",
       "min      -2.631431  \n",
       "25%      -0.711587  \n",
       "50%      -0.016043  \n",
       "75%       0.642756  \n",
       "max       3.516588  "
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can also change the maximal number of rows that is displayed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "      <th>E</th>\n",
       "      <th>F</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.920657</td>\n",
       "      <td>-0.582968</td>\n",
       "      <td>-0.585235</td>\n",
       "      <td>-0.445596</td>\n",
       "      <td>-0.681712</td>\n",
       "      <td>0.121091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.017103</td>\n",
       "      <td>0.510999</td>\n",
       "      <td>0.298593</td>\n",
       "      <td>-1.279607</td>\n",
       "      <td>-1.660310</td>\n",
       "      <td>-1.427842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-0.467513</td>\n",
       "      <td>-0.211803</td>\n",
       "      <td>-0.178767</td>\n",
       "      <td>0.860208</td>\n",
       "      <td>0.330933</td>\n",
       "      <td>0.128354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.682888</td>\n",
       "      <td>0.578031</td>\n",
       "      <td>0.535269</td>\n",
       "      <td>-0.447142</td>\n",
       "      <td>-1.350380</td>\n",
       "      <td>1.022790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.233067</td>\n",
       "      <td>1.069914</td>\n",
       "      <td>-0.672395</td>\n",
       "      <td>-0.324268</td>\n",
       "      <td>-1.213044</td>\n",
       "      <td>0.753714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2985</th>\n",
       "      <td>0.109807</td>\n",
       "      <td>1.160787</td>\n",
       "      <td>0.454515</td>\n",
       "      <td>0.761941</td>\n",
       "      <td>-0.448961</td>\n",
       "      <td>-0.549234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2988</th>\n",
       "      <td>0.289328</td>\n",
       "      <td>-0.046527</td>\n",
       "      <td>-1.958731</td>\n",
       "      <td>-0.859543</td>\n",
       "      <td>-0.004865</td>\n",
       "      <td>0.865963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2991</th>\n",
       "      <td>-2.171022</td>\n",
       "      <td>-0.993406</td>\n",
       "      <td>-0.547938</td>\n",
       "      <td>0.075995</td>\n",
       "      <td>-0.726624</td>\n",
       "      <td>-1.203640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2994</th>\n",
       "      <td>-1.542033</td>\n",
       "      <td>-1.032567</td>\n",
       "      <td>-0.546570</td>\n",
       "      <td>0.282012</td>\n",
       "      <td>-1.125996</td>\n",
       "      <td>-0.646228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2997</th>\n",
       "      <td>0.393561</td>\n",
       "      <td>-0.340553</td>\n",
       "      <td>1.729834</td>\n",
       "      <td>-0.750995</td>\n",
       "      <td>0.654371</td>\n",
       "      <td>0.932908</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             A         B         C         D         E         F\n",
       "0    -0.920657 -0.582968 -0.585235 -0.445596 -0.681712  0.121091\n",
       "3    -0.017103  0.510999  0.298593 -1.279607 -1.660310 -1.427842\n",
       "6    -0.467513 -0.211803 -0.178767  0.860208  0.330933  0.128354\n",
       "9     0.682888  0.578031  0.535269 -0.447142 -1.350380  1.022790\n",
       "12    0.233067  1.069914 -0.672395 -0.324268 -1.213044  0.753714\n",
       "...        ...       ...       ...       ...       ...       ...\n",
       "2985  0.109807  1.160787  0.454515  0.761941 -0.448961 -0.549234\n",
       "2988  0.289328 -0.046527 -1.958731 -0.859543 -0.004865  0.865963\n",
       "2991 -2.171022 -0.993406 -0.547938  0.075995 -0.726624 -1.203640\n",
       "2994 -1.542033 -1.032567 -0.546570  0.282012 -1.125996 -0.646228\n",
       "2997  0.393561 -0.340553  1.729834 -0.750995  0.654371  0.932908\n",
       "\n",
       "[1000 rows x 6 columns]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option(\"display.max_rows\", 15)\n",
    "\n",
    "u"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing/Slicing in Pandas\n",
    "\n",
    "The easiest way of accessing information in a Pandas dataframe, equivalent to the way used in NumPy, is using the `iloc` command. With this you can also set specific values, do conditional indexing... all that we have seen before in section 2.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>C</th>\n",
       "      <th>F</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>375</th>\n",
       "      <td>0.403273</td>\n",
       "      <td>0.198361</td>\n",
       "      <td>-0.785681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>378</th>\n",
       "      <td>0.147993</td>\n",
       "      <td>1.093704</td>\n",
       "      <td>-0.153657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>381</th>\n",
       "      <td>-1.061079</td>\n",
       "      <td>0.044698</td>\n",
       "      <td>-0.196515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>384</th>\n",
       "      <td>-1.577289</td>\n",
       "      <td>-0.081602</td>\n",
       "      <td>-0.043167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>-1.070241</td>\n",
       "      <td>0.943203</td>\n",
       "      <td>-0.490908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>-0.394896</td>\n",
       "      <td>0.742236</td>\n",
       "      <td>0.992452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>393</th>\n",
       "      <td>-0.294508</td>\n",
       "      <td>-0.255236</td>\n",
       "      <td>1.101463</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            A         C         F\n",
       "375  0.403273  0.198361 -0.785681\n",
       "378  0.147993  1.093704 -0.153657\n",
       "381 -1.061079  0.044698 -0.196515\n",
       "384 -1.577289 -0.081602 -0.043167\n",
       "387 -1.070241  0.943203 -0.490908\n",
       "390 -0.394896  0.742236  0.992452\n",
       "393 -0.294508 -0.255236  1.101463"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Slice-in rows index 125 to 132 (132 included!) from columns index 0, 2 and 5\n",
    "u.iloc[125:132, [0, 2, 5]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also choose specific rows according to their indices with the `loc` command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>C</th>\n",
       "      <th>F</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>375</th>\n",
       "      <td>0.403273</td>\n",
       "      <td>0.198361</td>\n",
       "      <td>-0.785681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>378</th>\n",
       "      <td>0.147993</td>\n",
       "      <td>1.093704</td>\n",
       "      <td>-0.153657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>381</th>\n",
       "      <td>-1.061079</td>\n",
       "      <td>0.044698</td>\n",
       "      <td>-0.196515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>384</th>\n",
       "      <td>-1.577289</td>\n",
       "      <td>-0.081602</td>\n",
       "      <td>-0.043167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>-1.070241</td>\n",
       "      <td>0.943203</td>\n",
       "      <td>-0.490908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>-0.394896</td>\n",
       "      <td>0.742236</td>\n",
       "      <td>0.992452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>393</th>\n",
       "      <td>-0.294508</td>\n",
       "      <td>-0.255236</td>\n",
       "      <td>1.101463</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            A         C         F\n",
       "375  0.403273  0.198361 -0.785681\n",
       "378  0.147993  1.093704 -0.153657\n",
       "381 -1.061079  0.044698 -0.196515\n",
       "384 -1.577289 -0.081602 -0.043167\n",
       "387 -1.070241  0.943203 -0.490908\n",
       "390 -0.394896  0.742236  0.992452\n",
       "393 -0.294508 -0.255236  1.101463"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Slice-in rows 375 to 393 (393 included!) from columns A, C and F\n",
    "u.loc[375:393, [\"A\", \"C\", \"F\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The usual `[]` will select specific rows according to the row number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>C</th>\n",
       "      <th>F</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>375</th>\n",
       "      <td>0.403273</td>\n",
       "      <td>0.198361</td>\n",
       "      <td>-0.785681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>378</th>\n",
       "      <td>0.147993</td>\n",
       "      <td>1.093704</td>\n",
       "      <td>-0.153657</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>381</th>\n",
       "      <td>-1.061079</td>\n",
       "      <td>0.044698</td>\n",
       "      <td>-0.196515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>384</th>\n",
       "      <td>-1.577289</td>\n",
       "      <td>-0.081602</td>\n",
       "      <td>-0.043167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>387</th>\n",
       "      <td>-1.070241</td>\n",
       "      <td>0.943203</td>\n",
       "      <td>-0.490908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>-0.394896</td>\n",
       "      <td>0.742236</td>\n",
       "      <td>0.992452</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>393</th>\n",
       "      <td>-0.294508</td>\n",
       "      <td>-0.255236</td>\n",
       "      <td>1.101463</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            A         C         F\n",
       "375  0.403273  0.198361 -0.785681\n",
       "378  0.147993  1.093704 -0.153657\n",
       "381 -1.061079  0.044698 -0.196515\n",
       "384 -1.577289 -0.081602 -0.043167\n",
       "387 -1.070241  0.943203 -0.490908\n",
       "390 -0.394896  0.742236  0.992452\n",
       "393 -0.294508 -0.255236  1.101463"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Slice-in rows index 125 to 132 (132 included!) from columns A, C and F\n",
    "u[125:132][[\"A\", \"C\", \"F\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, there are a few different ways of accessing the data in a Pandas dataframe, that typically have a more \"direct\" connection with the actual content fo the dataframe. Individual or sets of columns can also be accessed by their column names. Choosing one single column will give a Series, while two or more will produce a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    -0.920657\n",
       "3    -0.017103\n",
       "6    -0.467513\n",
       "9     0.682888\n",
       "12    0.233067\n",
       "Name: A, dtype: float64"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u[\"A\"].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.920657</td>\n",
       "      <td>-0.445596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.017103</td>\n",
       "      <td>-1.279607</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-0.467513</td>\n",
       "      <td>0.860208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.682888</td>\n",
       "      <td>-0.447142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.233067</td>\n",
       "      <td>-0.324268</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           A         D\n",
       "0  -0.920657 -0.445596\n",
       "3  -0.017103 -1.279607\n",
       "6  -0.467513  0.860208\n",
       "9   0.682888 -0.447142\n",
       "12  0.233067 -0.324268"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u[[\"A\", \"D\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not only that, you can access a single column without the need of brackets []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    -0.920657\n",
       "3    -0.017103\n",
       "6    -0.467513\n",
       "9     0.682888\n",
       "12    0.233067\n",
       "Name: A, dtype: float64"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u.A.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, you can access just the elements that satisfy some condition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "      <th>E</th>\n",
       "      <th>F</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>168</th>\n",
       "      <td>0.029182</td>\n",
       "      <td>-0.055373</td>\n",
       "      <td>0.413367</td>\n",
       "      <td>2.159409</td>\n",
       "      <td>-0.107484</td>\n",
       "      <td>-0.568885</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342</th>\n",
       "      <td>0.246142</td>\n",
       "      <td>-0.575288</td>\n",
       "      <td>1.183090</td>\n",
       "      <td>3.093830</td>\n",
       "      <td>-0.048119</td>\n",
       "      <td>-0.228327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>771</th>\n",
       "      <td>0.225022</td>\n",
       "      <td>-0.246722</td>\n",
       "      <td>0.320646</td>\n",
       "      <td>2.989750</td>\n",
       "      <td>-0.153650</td>\n",
       "      <td>1.653815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>807</th>\n",
       "      <td>2.400690</td>\n",
       "      <td>-0.000816</td>\n",
       "      <td>-0.213668</td>\n",
       "      <td>2.044642</td>\n",
       "      <td>-0.211728</td>\n",
       "      <td>-1.184668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1029</th>\n",
       "      <td>-1.521710</td>\n",
       "      <td>-0.463771</td>\n",
       "      <td>0.161482</td>\n",
       "      <td>2.145911</td>\n",
       "      <td>-1.703157</td>\n",
       "      <td>-1.081007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2454</th>\n",
       "      <td>-0.134750</td>\n",
       "      <td>-1.864866</td>\n",
       "      <td>0.036575</td>\n",
       "      <td>2.001025</td>\n",
       "      <td>1.416778</td>\n",
       "      <td>-0.223436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2661</th>\n",
       "      <td>-1.174205</td>\n",
       "      <td>-1.278691</td>\n",
       "      <td>-1.908602</td>\n",
       "      <td>2.478132</td>\n",
       "      <td>-0.696754</td>\n",
       "      <td>0.596047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2694</th>\n",
       "      <td>0.701319</td>\n",
       "      <td>1.700763</td>\n",
       "      <td>0.182507</td>\n",
       "      <td>2.736854</td>\n",
       "      <td>1.095017</td>\n",
       "      <td>0.404257</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2844</th>\n",
       "      <td>1.004502</td>\n",
       "      <td>-1.572246</td>\n",
       "      <td>-0.361376</td>\n",
       "      <td>2.261472</td>\n",
       "      <td>0.130629</td>\n",
       "      <td>0.609477</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2865</th>\n",
       "      <td>-0.627037</td>\n",
       "      <td>-1.778637</td>\n",
       "      <td>-0.357833</td>\n",
       "      <td>2.510799</td>\n",
       "      <td>-1.785096</td>\n",
       "      <td>-0.250410</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>19 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             A         B         C         D         E         F\n",
       "168   0.029182 -0.055373  0.413367  2.159409 -0.107484 -0.568885\n",
       "342   0.246142 -0.575288  1.183090  3.093830 -0.048119 -0.228327\n",
       "771   0.225022 -0.246722  0.320646  2.989750 -0.153650  1.653815\n",
       "807   2.400690 -0.000816 -0.213668  2.044642 -0.211728 -1.184668\n",
       "1029 -1.521710 -0.463771  0.161482  2.145911 -1.703157 -1.081007\n",
       "...        ...       ...       ...       ...       ...       ...\n",
       "2454 -0.134750 -1.864866  0.036575  2.001025  1.416778 -0.223436\n",
       "2661 -1.174205 -1.278691 -1.908602  2.478132 -0.696754  0.596047\n",
       "2694  0.701319  1.700763  0.182507  2.736854  1.095017  0.404257\n",
       "2844  1.004502 -1.572246 -0.361376  2.261472  0.130629  0.609477\n",
       "2865 -0.627037 -1.778637 -0.357833  2.510799 -1.785096 -0.250410\n",
       "\n",
       "[19 rows x 6 columns]"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u[u.D > 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "      <th>E</th>\n",
       "      <th>F</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.920657</td>\n",
       "      <td>-0.582968</td>\n",
       "      <td>-0.585235</td>\n",
       "      <td>-0.445596</td>\n",
       "      <td>-0.681712</td>\n",
       "      <td>0.121091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.017103</td>\n",
       "      <td>0.510999</td>\n",
       "      <td>0.298593</td>\n",
       "      <td>-1.279607</td>\n",
       "      <td>-1.660310</td>\n",
       "      <td>-1.427842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-0.467513</td>\n",
       "      <td>-0.211803</td>\n",
       "      <td>-0.178767</td>\n",
       "      <td>0.860208</td>\n",
       "      <td>0.330933</td>\n",
       "      <td>0.128354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.682888</td>\n",
       "      <td>0.578031</td>\n",
       "      <td>0.535269</td>\n",
       "      <td>-0.447142</td>\n",
       "      <td>-1.350380</td>\n",
       "      <td>1.022790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.233067</td>\n",
       "      <td>1.069914</td>\n",
       "      <td>-0.672395</td>\n",
       "      <td>-0.324268</td>\n",
       "      <td>-1.213044</td>\n",
       "      <td>0.753714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2985</th>\n",
       "      <td>0.109807</td>\n",
       "      <td>1.160787</td>\n",
       "      <td>0.454515</td>\n",
       "      <td>0.761941</td>\n",
       "      <td>-0.448961</td>\n",
       "      <td>-0.549234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2988</th>\n",
       "      <td>0.289328</td>\n",
       "      <td>-0.046527</td>\n",
       "      <td>-1.958731</td>\n",
       "      <td>-0.859543</td>\n",
       "      <td>-0.004865</td>\n",
       "      <td>0.865963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2991</th>\n",
       "      <td>-2.171022</td>\n",
       "      <td>-0.993406</td>\n",
       "      <td>-0.547938</td>\n",
       "      <td>0.075995</td>\n",
       "      <td>-0.726624</td>\n",
       "      <td>-1.203640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2994</th>\n",
       "      <td>-1.542033</td>\n",
       "      <td>-1.032567</td>\n",
       "      <td>-0.546570</td>\n",
       "      <td>0.282012</td>\n",
       "      <td>-1.125996</td>\n",
       "      <td>-0.646228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2997</th>\n",
       "      <td>0.393561</td>\n",
       "      <td>-0.340553</td>\n",
       "      <td>1.729834</td>\n",
       "      <td>-0.750995</td>\n",
       "      <td>0.654371</td>\n",
       "      <td>0.932908</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>981 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             A         B         C         D         E         F\n",
       "0    -0.920657 -0.582968 -0.585235 -0.445596 -0.681712  0.121091\n",
       "3    -0.017103  0.510999  0.298593 -1.279607 -1.660310 -1.427842\n",
       "6    -0.467513 -0.211803 -0.178767  0.860208  0.330933  0.128354\n",
       "9     0.682888  0.578031  0.535269 -0.447142 -1.350380  1.022790\n",
       "12    0.233067  1.069914 -0.672395 -0.324268 -1.213044  0.753714\n",
       "...        ...       ...       ...       ...       ...       ...\n",
       "2985  0.109807  1.160787  0.454515  0.761941 -0.448961 -0.549234\n",
       "2988  0.289328 -0.046527 -1.958731 -0.859543 -0.004865  0.865963\n",
       "2991 -2.171022 -0.993406 -0.547938  0.075995 -0.726624 -1.203640\n",
       "2994 -1.542033 -1.032567 -0.546570  0.282012 -1.125996 -0.646228\n",
       "2997  0.393561 -0.340553  1.729834 -0.750995  0.654371  0.932908\n",
       "\n",
       "[981 rows x 6 columns]"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u[~(u.D > 2)]  # For the inverse of u.D > 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recently `query` has been added to `DataFrame` for the same purpose. While it is less powerful than logical indexing, it is often faster and shorter (when names are longer than just `u`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "      <th>E</th>\n",
       "      <th>F</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>168</th>\n",
       "      <td>0.029182</td>\n",
       "      <td>-0.055373</td>\n",
       "      <td>0.413367</td>\n",
       "      <td>2.159409</td>\n",
       "      <td>-0.107484</td>\n",
       "      <td>-0.568885</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>342</th>\n",
       "      <td>0.246142</td>\n",
       "      <td>-0.575288</td>\n",
       "      <td>1.183090</td>\n",
       "      <td>3.093830</td>\n",
       "      <td>-0.048119</td>\n",
       "      <td>-0.228327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>771</th>\n",
       "      <td>0.225022</td>\n",
       "      <td>-0.246722</td>\n",
       "      <td>0.320646</td>\n",
       "      <td>2.989750</td>\n",
       "      <td>-0.153650</td>\n",
       "      <td>1.653815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>807</th>\n",
       "      <td>2.400690</td>\n",
       "      <td>-0.000816</td>\n",
       "      <td>-0.213668</td>\n",
       "      <td>2.044642</td>\n",
       "      <td>-0.211728</td>\n",
       "      <td>-1.184668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1029</th>\n",
       "      <td>-1.521710</td>\n",
       "      <td>-0.463771</td>\n",
       "      <td>0.161482</td>\n",
       "      <td>2.145911</td>\n",
       "      <td>-1.703157</td>\n",
       "      <td>-1.081007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2454</th>\n",
       "      <td>-0.134750</td>\n",
       "      <td>-1.864866</td>\n",
       "      <td>0.036575</td>\n",
       "      <td>2.001025</td>\n",
       "      <td>1.416778</td>\n",
       "      <td>-0.223436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2661</th>\n",
       "      <td>-1.174205</td>\n",
       "      <td>-1.278691</td>\n",
       "      <td>-1.908602</td>\n",
       "      <td>2.478132</td>\n",
       "      <td>-0.696754</td>\n",
       "      <td>0.596047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2694</th>\n",
       "      <td>0.701319</td>\n",
       "      <td>1.700763</td>\n",
       "      <td>0.182507</td>\n",
       "      <td>2.736854</td>\n",
       "      <td>1.095017</td>\n",
       "      <td>0.404257</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2844</th>\n",
       "      <td>1.004502</td>\n",
       "      <td>-1.572246</td>\n",
       "      <td>-0.361376</td>\n",
       "      <td>2.261472</td>\n",
       "      <td>0.130629</td>\n",
       "      <td>0.609477</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2865</th>\n",
       "      <td>-0.627037</td>\n",
       "      <td>-1.778637</td>\n",
       "      <td>-0.357833</td>\n",
       "      <td>2.510799</td>\n",
       "      <td>-1.785096</td>\n",
       "      <td>-0.250410</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>19 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             A         B         C         D         E         F\n",
       "168   0.029182 -0.055373  0.413367  2.159409 -0.107484 -0.568885\n",
       "342   0.246142 -0.575288  1.183090  3.093830 -0.048119 -0.228327\n",
       "771   0.225022 -0.246722  0.320646  2.989750 -0.153650  1.653815\n",
       "807   2.400690 -0.000816 -0.213668  2.044642 -0.211728 -1.184668\n",
       "1029 -1.521710 -0.463771  0.161482  2.145911 -1.703157 -1.081007\n",
       "...        ...       ...       ...       ...       ...       ...\n",
       "2454 -0.134750 -1.864866  0.036575  2.001025  1.416778 -0.223436\n",
       "2661 -1.174205 -1.278691 -1.908602  2.478132 -0.696754  0.596047\n",
       "2694  0.701319  1.700763  0.182507  2.736854  1.095017  0.404257\n",
       "2844  1.004502 -1.572246 -0.361376  2.261472  0.130629  0.609477\n",
       "2865 -0.627037 -1.778637 -0.357833  2.510799 -1.785096 -0.250410\n",
       "\n",
       "[19 rows x 6 columns]"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "u.query(\"D > 2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reshaping `DataFrame`s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sample</th>\n",
       "      <th>replicate</th>\n",
       "      <th>protein</th>\n",
       "      <th>value1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A</td>\n",
       "      <td>01</td>\n",
       "      <td>P02768</td>\n",
       "      <td>1.175802</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A</td>\n",
       "      <td>02</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.410176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A</td>\n",
       "      <td>03</td>\n",
       "      <td>P02768</td>\n",
       "      <td>1.866996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B</td>\n",
       "      <td>01</td>\n",
       "      <td>P02768</td>\n",
       "      <td>0.035961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B</td>\n",
       "      <td>02</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.414273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>B</td>\n",
       "      <td>03</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.084432</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  sample replicate protein    value1\n",
       "0      A        01  P02768  1.175802\n",
       "1      A        02  P02768 -0.410176\n",
       "2      A        03  P02768  1.866996\n",
       "3      B        01  P02768  0.035961\n",
       "4      B        02  P02768 -0.414273\n",
       "5      B        03  P02768 -0.084432"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = pd.DataFrame()\n",
    "\n",
    "df1[\"sample\"] = [\"A\", \"A\", \"A\", \"B\", \"B\", \"B\"]\n",
    "df1[\"replicate\"] = [\"01\", \"02\", \"03\", \"01\", \"02\", \"03\"]\n",
    "df1[\"protein\"] = \"P02768\"\n",
    "df1[\"value1\"] = np.random.randn(6)\n",
    "\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>sample</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>replicate</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>01</th>\n",
       "      <td>1.175802</td>\n",
       "      <td>0.035961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>02</th>\n",
       "      <td>-0.410176</td>\n",
       "      <td>-0.414273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>03</th>\n",
       "      <td>1.866996</td>\n",
       "      <td>-0.084432</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "sample            A         B\n",
       "replicate                    \n",
       "01         1.175802  0.035961\n",
       "02        -0.410176 -0.414273\n",
       "03         1.866996 -0.084432"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pivot_df1 = df1.pivot(index=\"replicate\", columns=\"sample\", values=\"value1\")\n",
    "\n",
    "pivot_df1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computing With `DataFrames`\n",
    "\n",
    "You can calculate with `DataFrames` or their columns (which are `Series`) the same way you could with `arrays`s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sample</th>\n",
       "      <th>replicate</th>\n",
       "      <th>protein</th>\n",
       "      <th>value1</th>\n",
       "      <th>value2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A</td>\n",
       "      <td>01</td>\n",
       "      <td>P02768</td>\n",
       "      <td>1.175802</td>\n",
       "      <td>0.850483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A</td>\n",
       "      <td>02</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.410176</td>\n",
       "      <td>-2.437978</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A</td>\n",
       "      <td>03</td>\n",
       "      <td>P02768</td>\n",
       "      <td>1.866996</td>\n",
       "      <td>0.535620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B</td>\n",
       "      <td>01</td>\n",
       "      <td>P02768</td>\n",
       "      <td>0.035961</td>\n",
       "      <td>27.807872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B</td>\n",
       "      <td>02</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.414273</td>\n",
       "      <td>-2.413865</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  sample replicate protein    value1     value2\n",
       "0      A        01  P02768  1.175802   0.850483\n",
       "1      A        02  P02768 -0.410176  -2.437978\n",
       "2      A        03  P02768  1.866996   0.535620\n",
       "3      B        01  P02768  0.035961  27.807872\n",
       "4      B        02  P02768 -0.414273  -2.413865"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1[\"value2\"] = 1 / df1[\"value1\"]\n",
    "df1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\qiv\\anaconda3\\lib\\site-packages\\numpy\\core\\fromnumeric.py:3438: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.\n",
      "  return mean(axis=axis, dtype=dtype, out=out, **kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "replicate    1.700502e+09\n",
       "value1       3.616463e-01\n",
       "value2       2.083038e+00\n",
       "dtype: float64"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(df1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can apply functions to the whole dataset or specific columns with the `apply` command. `apply` acts on the whole column at a time (i.e. a Pandas `Series`), so you can compute things that depend on several values of the column, for instance the mean value. To apply functions in a real element-by-element basis the function `applymap` or `Series.apply` should be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "value1    0.361646\n",
       "value2    2.083038\n",
       "dtype: float64"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def mn(col):\n",
    "    return sum(col) / len(col)\n",
    "\n",
    "\n",
    "df1[[\"value1\", \"value2\"]].apply(mn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While most can be directly calculated (including the given example of the mean), `apply` also works on columns with strings or categorical data, where no mathematical operations are defined. The limit is the imagination."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining `DataFrames`\n",
    "\n",
    "Something we will do quite often as scientists is combining data from different sources into one single source. This can be achieved by different commands in Pandas, depending on the actual goal we want.\n",
    "\n",
    "To begin with, appending new rows of data is achieved by the command `append`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sample</th>\n",
       "      <th>replicate</th>\n",
       "      <th>protein</th>\n",
       "      <th>value1</th>\n",
       "      <th>value2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A</td>\n",
       "      <td>01</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-0.847244</td>\n",
       "      <td>-1.180297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A</td>\n",
       "      <td>02</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-1.184034</td>\n",
       "      <td>-0.844570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A</td>\n",
       "      <td>03</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.559044</td>\n",
       "      <td>1.788767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B</td>\n",
       "      <td>01</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.853636</td>\n",
       "      <td>1.171460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B</td>\n",
       "      <td>02</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.371381</td>\n",
       "      <td>2.692653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>B</td>\n",
       "      <td>03</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-0.167774</td>\n",
       "      <td>-5.960408</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  sample replicate protein    value1    value2\n",
       "0      A        01  P69892 -0.847244 -1.180297\n",
       "1      A        02  P69892 -1.184034 -0.844570\n",
       "2      A        03  P69892  0.559044  1.788767\n",
       "3      B        01  P69892  0.853636  1.171460\n",
       "4      B        02  P69892  0.371381  2.692653\n",
       "5      B        03  P69892 -0.167774 -5.960408"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = pd.DataFrame()\n",
    "\n",
    "df2[\"sample\"] = [\"A\", \"A\", \"A\", \"B\", \"B\", \"B\"]\n",
    "df2[\"replicate\"] = [\"01\", \"02\", \"03\", \"01\", \"02\", \"03\"]\n",
    "df2[\"protein\"] = \"P69892\"\n",
    "df2[\"value1\"] = np.random.randn(6)\n",
    "df2[\"value2\"] = 1 / df2[\"value1\"]\n",
    "\n",
    "df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sample</th>\n",
       "      <th>replicate</th>\n",
       "      <th>protein</th>\n",
       "      <th>value1</th>\n",
       "      <th>value2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A</td>\n",
       "      <td>01</td>\n",
       "      <td>P02768</td>\n",
       "      <td>1.175802</td>\n",
       "      <td>0.850483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A</td>\n",
       "      <td>02</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.410176</td>\n",
       "      <td>-2.437978</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A</td>\n",
       "      <td>03</td>\n",
       "      <td>P02768</td>\n",
       "      <td>1.866996</td>\n",
       "      <td>0.535620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B</td>\n",
       "      <td>01</td>\n",
       "      <td>P02768</td>\n",
       "      <td>0.035961</td>\n",
       "      <td>27.807872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B</td>\n",
       "      <td>02</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.414273</td>\n",
       "      <td>-2.413865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>B</td>\n",
       "      <td>03</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.084432</td>\n",
       "      <td>-11.843906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>A</td>\n",
       "      <td>01</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-0.847244</td>\n",
       "      <td>-1.180297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>A</td>\n",
       "      <td>02</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-1.184034</td>\n",
       "      <td>-0.844570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>A</td>\n",
       "      <td>03</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.559044</td>\n",
       "      <td>1.788767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>B</td>\n",
       "      <td>01</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.853636</td>\n",
       "      <td>1.171460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>B</td>\n",
       "      <td>02</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.371381</td>\n",
       "      <td>2.692653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>B</td>\n",
       "      <td>03</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-0.167774</td>\n",
       "      <td>-5.960408</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sample replicate protein    value1     value2\n",
       "0       A        01  P02768  1.175802   0.850483\n",
       "1       A        02  P02768 -0.410176  -2.437978\n",
       "2       A        03  P02768  1.866996   0.535620\n",
       "3       B        01  P02768  0.035961  27.807872\n",
       "4       B        02  P02768 -0.414273  -2.413865\n",
       "5       B        03  P02768 -0.084432 -11.843906\n",
       "6       A        01  P69892 -0.847244  -1.180297\n",
       "7       A        02  P69892 -1.184034  -0.844570\n",
       "8       A        03  P69892  0.559044   1.788767\n",
       "9       B        01  P69892  0.853636   1.171460\n",
       "10      B        02  P69892  0.371381   2.692653\n",
       "11      B        03  P69892 -0.167774  -5.960408"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.append(df2, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same result can be obtained with `concat`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sample</th>\n",
       "      <th>replicate</th>\n",
       "      <th>protein</th>\n",
       "      <th>value1</th>\n",
       "      <th>value2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A</td>\n",
       "      <td>01</td>\n",
       "      <td>P02768</td>\n",
       "      <td>1.175802</td>\n",
       "      <td>0.850483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A</td>\n",
       "      <td>02</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.410176</td>\n",
       "      <td>-2.437978</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A</td>\n",
       "      <td>03</td>\n",
       "      <td>P02768</td>\n",
       "      <td>1.866996</td>\n",
       "      <td>0.535620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B</td>\n",
       "      <td>01</td>\n",
       "      <td>P02768</td>\n",
       "      <td>0.035961</td>\n",
       "      <td>27.807872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B</td>\n",
       "      <td>02</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.414273</td>\n",
       "      <td>-2.413865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>B</td>\n",
       "      <td>03</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.084432</td>\n",
       "      <td>-11.843906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>A</td>\n",
       "      <td>01</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-0.847244</td>\n",
       "      <td>-1.180297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>A</td>\n",
       "      <td>02</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-1.184034</td>\n",
       "      <td>-0.844570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>A</td>\n",
       "      <td>03</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.559044</td>\n",
       "      <td>1.788767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>B</td>\n",
       "      <td>01</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.853636</td>\n",
       "      <td>1.171460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>B</td>\n",
       "      <td>02</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.371381</td>\n",
       "      <td>2.692653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>B</td>\n",
       "      <td>03</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-0.167774</td>\n",
       "      <td>-5.960408</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sample replicate protein    value1     value2\n",
       "0       A        01  P02768  1.175802   0.850483\n",
       "1       A        02  P02768 -0.410176  -2.437978\n",
       "2       A        03  P02768  1.866996   0.535620\n",
       "3       B        01  P02768  0.035961  27.807872\n",
       "4       B        02  P02768 -0.414273  -2.413865\n",
       "5       B        03  P02768 -0.084432 -11.843906\n",
       "6       A        01  P69892 -0.847244  -1.180297\n",
       "7       A        02  P69892 -1.184034  -0.844570\n",
       "8       A        03  P69892  0.559044   1.788767\n",
       "9       B        01  P69892  0.853636   1.171460\n",
       "10      B        02  P69892  0.371381   2.692653\n",
       "11      B        03  P69892 -0.167774  -5.960408"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.concat([df1, df2], ignore_index=True)\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grouping Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>value1</th>\n",
       "      <th>value2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>protein</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>P02768</th>\n",
       "      <td>2.169878</td>\n",
       "      <td>12.498225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P69892</th>\n",
       "      <td>-0.414992</td>\n",
       "      <td>-2.332395</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           value1     value2\n",
       "protein                     \n",
       "P02768   2.169878  12.498225\n",
       "P69892  -0.414992  -2.332395"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(\"protein\").agg(sum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>value1</th>\n",
       "      <th>value2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>protein</th>\n",
       "      <th>sample</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">P02768</th>\n",
       "      <th>A</th>\n",
       "      <td>2.632622</td>\n",
       "      <td>-1.051875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B</th>\n",
       "      <td>-0.462744</td>\n",
       "      <td>13.550101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">P69892</th>\n",
       "      <th>A</th>\n",
       "      <td>-1.472235</td>\n",
       "      <td>-0.236100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B</th>\n",
       "      <td>1.057243</td>\n",
       "      <td>-2.096295</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  value1     value2\n",
       "protein sample                     \n",
       "P02768  A       2.632622  -1.051875\n",
       "        B      -0.462744  13.550101\n",
       "P69892  A      -1.472235  -0.236100\n",
       "        B       1.057243  -2.096295"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby([\"protein\", \"sample\"]).agg(sum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>value1</th>\n",
       "      <th>value2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>protein</th>\n",
       "      <th>sample</th>\n",
       "      <th>replicate</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"6\" valign=\"top\">P02768</th>\n",
       "      <th rowspan=\"3\" valign=\"top\">A</th>\n",
       "      <th>01</th>\n",
       "      <td>1.175802</td>\n",
       "      <td>0.850483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>02</th>\n",
       "      <td>-0.410176</td>\n",
       "      <td>-2.437978</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>03</th>\n",
       "      <td>1.866996</td>\n",
       "      <td>0.535620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">B</th>\n",
       "      <th>01</th>\n",
       "      <td>0.035961</td>\n",
       "      <td>27.807872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>02</th>\n",
       "      <td>-0.414273</td>\n",
       "      <td>-2.413865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>03</th>\n",
       "      <td>-0.084432</td>\n",
       "      <td>-11.843906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"6\" valign=\"top\">P69892</th>\n",
       "      <th rowspan=\"3\" valign=\"top\">A</th>\n",
       "      <th>01</th>\n",
       "      <td>-0.847244</td>\n",
       "      <td>-1.180297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>02</th>\n",
       "      <td>-1.184034</td>\n",
       "      <td>-0.844570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>03</th>\n",
       "      <td>0.559044</td>\n",
       "      <td>1.788767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">B</th>\n",
       "      <th>01</th>\n",
       "      <td>0.853636</td>\n",
       "      <td>1.171460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>02</th>\n",
       "      <td>0.371381</td>\n",
       "      <td>2.692653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>03</th>\n",
       "      <td>-0.167774</td>\n",
       "      <td>-5.960408</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            value1     value2\n",
       "protein sample replicate                     \n",
       "P02768  A      01         1.175802   0.850483\n",
       "               02        -0.410176  -2.437978\n",
       "               03         1.866996   0.535620\n",
       "        B      01         0.035961  27.807872\n",
       "               02        -0.414273  -2.413865\n",
       "               03        -0.084432 -11.843906\n",
       "P69892  A      01        -0.847244  -1.180297\n",
       "               02        -1.184034  -0.844570\n",
       "               03         0.559044   1.788767\n",
       "        B      01         0.853636   1.171460\n",
       "               02         0.371381   2.692653\n",
       "               03        -0.167774  -5.960408"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby([\"protein\", \"sample\", \"replicate\"]).agg(sum)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\qiv\\AppData\\Local\\Temp/ipykernel_50596/96158140.py:1: FutureWarning: Dropping invalid columns in DataFrameGroupBy.transform is deprecated. In a future version, a TypeError will be raised. Before calling .transform, select only columns which should be valid for the transforming function.\n",
      "  df.groupby('protein').transform(np.mean)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>replicate</th>\n",
       "      <th>value1</th>\n",
       "      <th>value2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.700502e+09</td>\n",
       "      <td>0.361646</td>\n",
       "      <td>2.083038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.700502e+09</td>\n",
       "      <td>0.361646</td>\n",
       "      <td>2.083038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.700502e+09</td>\n",
       "      <td>0.361646</td>\n",
       "      <td>2.083038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.700502e+09</td>\n",
       "      <td>0.361646</td>\n",
       "      <td>2.083038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.700502e+09</td>\n",
       "      <td>0.361646</td>\n",
       "      <td>2.083038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.700502e+09</td>\n",
       "      <td>0.361646</td>\n",
       "      <td>2.083038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1.700502e+09</td>\n",
       "      <td>-0.069165</td>\n",
       "      <td>-0.388733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.700502e+09</td>\n",
       "      <td>-0.069165</td>\n",
       "      <td>-0.388733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1.700502e+09</td>\n",
       "      <td>-0.069165</td>\n",
       "      <td>-0.388733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1.700502e+09</td>\n",
       "      <td>-0.069165</td>\n",
       "      <td>-0.388733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1.700502e+09</td>\n",
       "      <td>-0.069165</td>\n",
       "      <td>-0.388733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1.700502e+09</td>\n",
       "      <td>-0.069165</td>\n",
       "      <td>-0.388733</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       replicate    value1    value2\n",
       "0   1.700502e+09  0.361646  2.083038\n",
       "1   1.700502e+09  0.361646  2.083038\n",
       "2   1.700502e+09  0.361646  2.083038\n",
       "3   1.700502e+09  0.361646  2.083038\n",
       "4   1.700502e+09  0.361646  2.083038\n",
       "5   1.700502e+09  0.361646  2.083038\n",
       "6   1.700502e+09 -0.069165 -0.388733\n",
       "7   1.700502e+09 -0.069165 -0.388733\n",
       "8   1.700502e+09 -0.069165 -0.388733\n",
       "9   1.700502e+09 -0.069165 -0.388733\n",
       "10  1.700502e+09 -0.069165 -0.388733\n",
       "11  1.700502e+09 -0.069165 -0.388733"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(\"protein\").transform(np.mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\qiv\\AppData\\Local\\Temp/ipykernel_50596/3558505644.py:1: FutureWarning: Indexing with multiple keys (implicitly converted to a tuple of keys) will be deprecated, use a list instead.\n",
      "  df.groupby('protein')['value1', 'value2'].transform(np.mean)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>value1</th>\n",
       "      <th>value2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.361646</td>\n",
       "      <td>2.083038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.361646</td>\n",
       "      <td>2.083038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.361646</td>\n",
       "      <td>2.083038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.361646</td>\n",
       "      <td>2.083038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.361646</td>\n",
       "      <td>2.083038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.361646</td>\n",
       "      <td>2.083038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-0.069165</td>\n",
       "      <td>-0.388733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-0.069165</td>\n",
       "      <td>-0.388733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-0.069165</td>\n",
       "      <td>-0.388733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>-0.069165</td>\n",
       "      <td>-0.388733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>-0.069165</td>\n",
       "      <td>-0.388733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>-0.069165</td>\n",
       "      <td>-0.388733</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      value1    value2\n",
       "0   0.361646  2.083038\n",
       "1   0.361646  2.083038\n",
       "2   0.361646  2.083038\n",
       "3   0.361646  2.083038\n",
       "4   0.361646  2.083038\n",
       "5   0.361646  2.083038\n",
       "6  -0.069165 -0.388733\n",
       "7  -0.069165 -0.388733\n",
       "8  -0.069165 -0.388733\n",
       "9  -0.069165 -0.388733\n",
       "10 -0.069165 -0.388733\n",
       "11 -0.069165 -0.388733"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(\"protein\")[\"value1\", \"value2\"].transform(np.mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  sample replicate protein    value1    value2\n",
      "0      A        01  P02768  1.175802  0.850483\n",
      "1      A        02  P02768 -0.410176 -2.437978\n",
      "2      A        03  P02768  1.866996  0.535620\n",
      "('P02768', 'A') --> mean value1: 0.8775406031007448\n",
      "      mean value2: -0.35062503903686126\n",
      "\n",
      "  sample replicate protein    value1     value2\n",
      "3      B        01  P02768  0.035961  27.807872\n",
      "4      B        02  P02768 -0.414273  -2.413865\n",
      "5      B        03  P02768 -0.084432 -11.843906\n",
      "('P02768', 'B') --> mean value1: -0.15424800457162488\n",
      "      mean value2: 4.516700189136153\n",
      "\n",
      "  sample replicate protein    value1    value2\n",
      "6      A        01  P69892 -0.847244 -1.180297\n",
      "7      A        02  P69892 -1.184034 -0.844570\n",
      "8      A        03  P69892  0.559044  1.788767\n",
      "('P69892', 'A') --> mean value1: -0.4907448797986365\n",
      "      mean value2: -0.07869991325198093\n",
      "\n",
      "   sample replicate protein    value1    value2\n",
      "9       B        01  P69892  0.853636  1.171460\n",
      "10      B        02  P69892  0.371381  2.692653\n",
      "11      B        03  P69892 -0.167774 -5.960408\n",
      "('P69892', 'B') --> mean value1: 0.35241436993784075\n",
      "      mean value2: -0.6987651120384358\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for g, g_df in df.groupby([\"protein\", \"sample\"]):\n",
    "    print(g_df)\n",
    "    print(f\"{g} --> mean value1: {np.mean(g_df['value1'])}\")\n",
    "    print(f\"      mean value2: {np.mean(g_df['value2'])}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th colspan=\"8\" halign=\"left\">value1</th>\n",
       "      <th colspan=\"8\" halign=\"left\">value2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>protein</th>\n",
       "      <th>sample</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">P02768</th>\n",
       "      <th>A</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.877541</td>\n",
       "      <td>1.167518</td>\n",
       "      <td>-0.410176</td>\n",
       "      <td>0.382813</td>\n",
       "      <td>1.175802</td>\n",
       "      <td>1.521399</td>\n",
       "      <td>1.866996</td>\n",
       "      <td>3.0</td>\n",
       "      <td>-0.350625</td>\n",
       "      <td>1.814543</td>\n",
       "      <td>-2.437978</td>\n",
       "      <td>-0.951179</td>\n",
       "      <td>0.535620</td>\n",
       "      <td>0.693052</td>\n",
       "      <td>0.850483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B</th>\n",
       "      <td>3.0</td>\n",
       "      <td>-0.154248</td>\n",
       "      <td>0.233096</td>\n",
       "      <td>-0.414273</td>\n",
       "      <td>-0.249353</td>\n",
       "      <td>-0.084432</td>\n",
       "      <td>-0.024235</td>\n",
       "      <td>0.035961</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.516700</td>\n",
       "      <td>20.714498</td>\n",
       "      <td>-11.843906</td>\n",
       "      <td>-7.128886</td>\n",
       "      <td>-2.413865</td>\n",
       "      <td>12.697004</td>\n",
       "      <td>27.807872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">P69892</th>\n",
       "      <th>A</th>\n",
       "      <td>3.0</td>\n",
       "      <td>-0.490745</td>\n",
       "      <td>0.924608</td>\n",
       "      <td>-1.184034</td>\n",
       "      <td>-1.015639</td>\n",
       "      <td>-0.847244</td>\n",
       "      <td>-0.144100</td>\n",
       "      <td>0.559044</td>\n",
       "      <td>3.0</td>\n",
       "      <td>-0.078700</td>\n",
       "      <td>1.625962</td>\n",
       "      <td>-1.180297</td>\n",
       "      <td>-1.012433</td>\n",
       "      <td>-0.844570</td>\n",
       "      <td>0.472099</td>\n",
       "      <td>1.788767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.352414</td>\n",
       "      <td>0.510969</td>\n",
       "      <td>-0.167774</td>\n",
       "      <td>0.101804</td>\n",
       "      <td>0.371381</td>\n",
       "      <td>0.612508</td>\n",
       "      <td>0.853636</td>\n",
       "      <td>3.0</td>\n",
       "      <td>-0.698765</td>\n",
       "      <td>4.619759</td>\n",
       "      <td>-5.960408</td>\n",
       "      <td>-2.394474</td>\n",
       "      <td>1.171460</td>\n",
       "      <td>1.932056</td>\n",
       "      <td>2.692653</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               value1                                                    \\\n",
       "                count      mean       std       min       25%       50%   \n",
       "protein sample                                                            \n",
       "P02768  A         3.0  0.877541  1.167518 -0.410176  0.382813  1.175802   \n",
       "        B         3.0 -0.154248  0.233096 -0.414273 -0.249353 -0.084432   \n",
       "P69892  A         3.0 -0.490745  0.924608 -1.184034 -1.015639 -0.847244   \n",
       "        B         3.0  0.352414  0.510969 -0.167774  0.101804  0.371381   \n",
       "\n",
       "                                   value2                                  \\\n",
       "                     75%       max  count      mean        std        min   \n",
       "protein sample                                                              \n",
       "P02768  A       1.521399  1.866996    3.0 -0.350625   1.814543  -2.437978   \n",
       "        B      -0.024235  0.035961    3.0  4.516700  20.714498 -11.843906   \n",
       "P69892  A      -0.144100  0.559044    3.0 -0.078700   1.625962  -1.180297   \n",
       "        B       0.612508  0.853636    3.0 -0.698765   4.619759  -5.960408   \n",
       "\n",
       "                                                          \n",
       "                     25%       50%        75%        max  \n",
       "protein sample                                            \n",
       "P02768  A      -0.951179  0.535620   0.693052   0.850483  \n",
       "        B      -7.128886 -2.413865  12.697004  27.807872  \n",
       "P69892  A      -1.012433 -0.844570   0.472099   1.788767  \n",
       "        B      -2.394474  1.171460   1.932056   2.692653  "
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby([\"protein\", \"sample\"]).describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">value1</th>\n",
       "      <th colspan=\"2\" halign=\"left\">value2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sample</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>protein</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>P02768</th>\n",
       "      <td>0.877541</td>\n",
       "      <td>-0.154248</td>\n",
       "      <td>-0.350625</td>\n",
       "      <td>4.516700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P69892</th>\n",
       "      <td>-0.490745</td>\n",
       "      <td>0.352414</td>\n",
       "      <td>-0.078700</td>\n",
       "      <td>-0.698765</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           value1              value2          \n",
       "sample          A         B         A         B\n",
       "protein                                        \n",
       "P02768   0.877541 -0.154248 -0.350625  4.516700\n",
       "P69892  -0.490745  0.352414 -0.078700 -0.698765"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.pivot_table(index=\"protein\", columns=\"sample\", aggfunc=\"mean\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">value1</th>\n",
       "      <th colspan=\"2\" halign=\"left\">value2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sample</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>protein</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>P02768</th>\n",
       "      <td>-0.410176</td>\n",
       "      <td>-0.414273</td>\n",
       "      <td>0.850483</td>\n",
       "      <td>27.807872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P69892</th>\n",
       "      <td>-1.184034</td>\n",
       "      <td>-0.167774</td>\n",
       "      <td>1.788767</td>\n",
       "      <td>2.692653</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           value1              value2           \n",
       "sample          A         B         A          B\n",
       "protein                                         \n",
       "P02768  -0.410176 -0.414273  0.850483  27.807872\n",
       "P69892  -1.184034 -0.167774  1.788767   2.692653"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.pivot_table(\n",
    "    index=\"protein\", columns=\"sample\", aggfunc={\"value1\": min, \"value2\": max}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading and saving dataframes\n",
    "\n",
    "To load and save Pandas dataframes we will use the `to_csv` and `read_csv` commands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sample</th>\n",
       "      <th>replicate</th>\n",
       "      <th>protein</th>\n",
       "      <th>value1</th>\n",
       "      <th>value2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>P02768</td>\n",
       "      <td>1.175802</td>\n",
       "      <td>0.850483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A</td>\n",
       "      <td>2</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.410176</td>\n",
       "      <td>-2.437978</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A</td>\n",
       "      <td>3</td>\n",
       "      <td>P02768</td>\n",
       "      <td>1.866996</td>\n",
       "      <td>0.535620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B</td>\n",
       "      <td>1</td>\n",
       "      <td>P02768</td>\n",
       "      <td>0.035961</td>\n",
       "      <td>27.807872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B</td>\n",
       "      <td>2</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.414273</td>\n",
       "      <td>-2.413865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>B</td>\n",
       "      <td>3</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.084432</td>\n",
       "      <td>-11.843906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-0.847244</td>\n",
       "      <td>-1.180297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>A</td>\n",
       "      <td>2</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-1.184034</td>\n",
       "      <td>-0.844570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>A</td>\n",
       "      <td>3</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.559044</td>\n",
       "      <td>1.788767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>B</td>\n",
       "      <td>1</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.853636</td>\n",
       "      <td>1.171460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>B</td>\n",
       "      <td>2</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.371381</td>\n",
       "      <td>2.692653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>B</td>\n",
       "      <td>3</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-0.167774</td>\n",
       "      <td>-5.960408</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sample  replicate protein    value1     value2\n",
       "0       A          1  P02768  1.175802   0.850483\n",
       "1       A          2  P02768 -0.410176  -2.437978\n",
       "2       A          3  P02768  1.866996   0.535620\n",
       "3       B          1  P02768  0.035961  27.807872\n",
       "4       B          2  P02768 -0.414273  -2.413865\n",
       "5       B          3  P02768 -0.084432 -11.843906\n",
       "6       A          1  P69892 -0.847244  -1.180297\n",
       "7       A          2  P69892 -1.184034  -0.844570\n",
       "8       A          3  P69892  0.559044   1.788767\n",
       "9       B          1  P69892  0.853636   1.171460\n",
       "10      B          2  P69892  0.371381   2.692653\n",
       "11      B          3  P69892 -0.167774  -5.960408"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.to_csv(\"test.csv\")\n",
    "pd.read_csv(\"test.csv\", index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But, as an addition, Pandas has special commands to load and save Excel spreadsheets (yay!). However, to use it you'll need the `openpyxl` and `xlrd` packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sample</th>\n",
       "      <th>replicate</th>\n",
       "      <th>protein</th>\n",
       "      <th>value1</th>\n",
       "      <th>value2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>P02768</td>\n",
       "      <td>1.175802</td>\n",
       "      <td>0.850483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A</td>\n",
       "      <td>2</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.410176</td>\n",
       "      <td>-2.437978</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A</td>\n",
       "      <td>3</td>\n",
       "      <td>P02768</td>\n",
       "      <td>1.866996</td>\n",
       "      <td>0.535620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B</td>\n",
       "      <td>1</td>\n",
       "      <td>P02768</td>\n",
       "      <td>0.035961</td>\n",
       "      <td>27.807872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B</td>\n",
       "      <td>2</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.414273</td>\n",
       "      <td>-2.413865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>B</td>\n",
       "      <td>3</td>\n",
       "      <td>P02768</td>\n",
       "      <td>-0.084432</td>\n",
       "      <td>-11.843906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>A</td>\n",
       "      <td>1</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-0.847244</td>\n",
       "      <td>-1.180297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>A</td>\n",
       "      <td>2</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-1.184034</td>\n",
       "      <td>-0.844570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>A</td>\n",
       "      <td>3</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.559044</td>\n",
       "      <td>1.788767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>B</td>\n",
       "      <td>1</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.853636</td>\n",
       "      <td>1.171460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>B</td>\n",
       "      <td>2</td>\n",
       "      <td>P69892</td>\n",
       "      <td>0.371381</td>\n",
       "      <td>2.692653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>B</td>\n",
       "      <td>3</td>\n",
       "      <td>P69892</td>\n",
       "      <td>-0.167774</td>\n",
       "      <td>-5.960408</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sample  replicate protein    value1     value2\n",
       "0       A          1  P02768  1.175802   0.850483\n",
       "1       A          2  P02768 -0.410176  -2.437978\n",
       "2       A          3  P02768  1.866996   0.535620\n",
       "3       B          1  P02768  0.035961  27.807872\n",
       "4       B          2  P02768 -0.414273  -2.413865\n",
       "5       B          3  P02768 -0.084432 -11.843906\n",
       "6       A          1  P69892 -0.847244  -1.180297\n",
       "7       A          2  P69892 -1.184034  -0.844570\n",
       "8       A          3  P69892  0.559044   1.788767\n",
       "9       B          1  P69892  0.853636   1.171460\n",
       "10      B          2  P69892  0.371381   2.692653\n",
       "11      B          3  P69892 -0.167774  -5.960408"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.to_excel(\"test.xlsx\", sheet_name=\"My sheet\")\n",
    "pd.read_excel(\"test.xlsx\", \"My sheet\", index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 5**: Download [this dataset](https://raw.githubusercontent.com/ChihChengLiang/pokemongor/master/data-raw/pokemons.csv) and load it, using the first column as the index. Take a look at it, and do the following things:\n",
    "- Choose the columns 'Identifier', 'BaseStamina', 'BaseAttack', 'BaseDefense', 'Type1' and 'Type2' \n",
    "- Create a function that lowercases strings and apply it to 'Type1' and 'Type2' (*Extra: just capitalize the strings, i.e., leave the first letter uppercase and lowercase the rest*)\n",
    "- Create a function that returns a Boolean value (don't be afraif by this, it is a function that returns either True or False) that tells if a Pokémon has high stamina (BaseStamina>170) or not. Store this information in a new column and show the list of Pokémon with high stamina\n",
    "- Show the instructor the last 15 rows of your dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PkMn\n",
      "31      Nidoqueen\n",
      "36       Clefable\n",
      "39     Jigglypuff\n",
      "40     Wigglytuff\n",
      "59       Arcanine\n",
      "          ...    \n",
      "145        Zapdos\n",
      "146       Moltres\n",
      "149     Dragonite\n",
      "150        Mewtwo\n",
      "151           Mew\n",
      "Name: Identifier, Length: 26, dtype: object\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Identifier</th>\n",
       "      <th>BaseStamina</th>\n",
       "      <th>BaseAttack</th>\n",
       "      <th>BaseDefense</th>\n",
       "      <th>Type1</th>\n",
       "      <th>Type2</th>\n",
       "      <th>HighStamina</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PkMn</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>137</th>\n",
       "      <td>Porygon</td>\n",
       "      <td>130</td>\n",
       "      <td>156</td>\n",
       "      <td>158</td>\n",
       "      <td>Normal</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>Omanyte</td>\n",
       "      <td>70</td>\n",
       "      <td>132</td>\n",
       "      <td>160</td>\n",
       "      <td>Rock</td>\n",
       "      <td>Water</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>139</th>\n",
       "      <td>Omastar</td>\n",
       "      <td>140</td>\n",
       "      <td>180</td>\n",
       "      <td>202</td>\n",
       "      <td>Rock</td>\n",
       "      <td>Water</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <td>Kabuto</td>\n",
       "      <td>60</td>\n",
       "      <td>148</td>\n",
       "      <td>142</td>\n",
       "      <td>Rock</td>\n",
       "      <td>Water</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>141</th>\n",
       "      <td>Kabutops</td>\n",
       "      <td>120</td>\n",
       "      <td>190</td>\n",
       "      <td>190</td>\n",
       "      <td>Rock</td>\n",
       "      <td>Water</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>Aerodactyl</td>\n",
       "      <td>160</td>\n",
       "      <td>182</td>\n",
       "      <td>162</td>\n",
       "      <td>Rock</td>\n",
       "      <td>Flying</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>Snorlax</td>\n",
       "      <td>320</td>\n",
       "      <td>180</td>\n",
       "      <td>180</td>\n",
       "      <td>Normal</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>Articuno</td>\n",
       "      <td>180</td>\n",
       "      <td>198</td>\n",
       "      <td>242</td>\n",
       "      <td>Ice</td>\n",
       "      <td>Flying</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>Zapdos</td>\n",
       "      <td>180</td>\n",
       "      <td>232</td>\n",
       "      <td>194</td>\n",
       "      <td>Electric</td>\n",
       "      <td>Flying</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>Moltres</td>\n",
       "      <td>180</td>\n",
       "      <td>242</td>\n",
       "      <td>194</td>\n",
       "      <td>Fire</td>\n",
       "      <td>Flying</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>Dratini</td>\n",
       "      <td>82</td>\n",
       "      <td>128</td>\n",
       "      <td>110</td>\n",
       "      <td>Dragon</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>Dragonair</td>\n",
       "      <td>122</td>\n",
       "      <td>170</td>\n",
       "      <td>152</td>\n",
       "      <td>Dragon</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>Dragonite</td>\n",
       "      <td>182</td>\n",
       "      <td>250</td>\n",
       "      <td>212</td>\n",
       "      <td>Dragon</td>\n",
       "      <td>Flying</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>150</th>\n",
       "      <td>Mewtwo</td>\n",
       "      <td>212</td>\n",
       "      <td>284</td>\n",
       "      <td>202</td>\n",
       "      <td>Psychic</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>151</th>\n",
       "      <td>Mew</td>\n",
       "      <td>200</td>\n",
       "      <td>220</td>\n",
       "      <td>220</td>\n",
       "      <td>Psychic</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Identifier  BaseStamina  BaseAttack  BaseDefense     Type1   Type2  \\\n",
       "PkMn                                                                       \n",
       "137      Porygon          130         156          158    Normal    None   \n",
       "138      Omanyte           70         132          160      Rock   Water   \n",
       "139      Omastar          140         180          202      Rock   Water   \n",
       "140       Kabuto           60         148          142      Rock   Water   \n",
       "141     Kabutops          120         190          190      Rock   Water   \n",
       "142   Aerodactyl          160         182          162      Rock  Flying   \n",
       "143      Snorlax          320         180          180    Normal    None   \n",
       "144     Articuno          180         198          242       Ice  Flying   \n",
       "145       Zapdos          180         232          194  Electric  Flying   \n",
       "146      Moltres          180         242          194      Fire  Flying   \n",
       "147      Dratini           82         128          110    Dragon    None   \n",
       "148    Dragonair          122         170          152    Dragon    None   \n",
       "149    Dragonite          182         250          212    Dragon  Flying   \n",
       "150       Mewtwo          212         284          202   Psychic    None   \n",
       "151          Mew          200         220          220   Psychic    None   \n",
       "\n",
       "      HighStamina  \n",
       "PkMn               \n",
       "137         False  \n",
       "138         False  \n",
       "139         False  \n",
       "140         False  \n",
       "141         False  \n",
       "142         False  \n",
       "143          True  \n",
       "144          True  \n",
       "145          True  \n",
       "146          True  \n",
       "147         False  \n",
       "148         False  \n",
       "149          True  \n",
       "150          True  \n",
       "151          True  "
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\n",
    "    \"https://raw.githubusercontent.com/ChihChengLiang/pokemongor/master/data-raw/pokemons.csv\",\n",
    "    index_col=0,\n",
    ")\n",
    "\n",
    "df = df[[\"Identifier\", \"BaseStamina\", \"BaseAttack\", \"BaseDefense\", \"Type1\", \"Type2\"]]\n",
    "\n",
    "capitalize = lambda st: st.capitalize()\n",
    "\n",
    "for col in [\"Type1\", \"Type2\"]:\n",
    "    df[col] = df[col].apply(capitalize)\n",
    "\n",
    "\n",
    "def highstamina(x):\n",
    "    return True if x > 170 else False\n",
    "\n",
    "\n",
    "df[\"HighStamina\"] = df.BaseStamina.apply(highstamina)\n",
    "\n",
    "print(df[df[\"HighStamina\"] == True].Identifier)\n",
    "\n",
    "df.tail(15)"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "comment_magics": true,
   "text_representation": {
    "extension": ".py",
    "format_name": "percent",
    "format_version": "1.2",
    "jupytext_version": "0.8.6rc1"
   }
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "299px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "oldHeight": 331.518518,
   "position": {
    "height": "353px",
    "left": "595.965px",
    "right": "20px",
    "top": "2.91667px",
    "width": "629px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "varInspector_section_display": "block",
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}