BasicsOfNumpyPandasMatplotlib.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Welcome to Drishti's workshop on Machine Learning/ Deep Learning /Computer vision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:2px solid gray\"> </hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are not familiar with Python Programming language don't worry. Python is one of the easiest computer languages you can learn today and it won't take you long to start writing it yourself! (you would catch up even quicker if you already know a different language like C or javascript or java)\n",
    "\n",
    "These are excellent python tutorials made by Corey Schafer : \n",
    "\n",
    "https://www.youtube.com/watch?v=k9TUPpGqYTo&list=PL-osiE80TeTt2d9bfVyTiXJA-UTHn6WwU&index=2\n",
    "\n",
    "Watch videos: 2,3,4,5,6,7,8,9,15 to 21."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Make sure you watch these before proceeding further**.\n",
    "\n",
    "Feel free to explore python on your own by reading articles about it or watching videos make by other creators. Also don't worry if you don't understand anything at any point. We will be always available to help you. :-)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have watched these videos lets start with our first library:\n",
    "\n",
    "PS: feel free to change the values/shapes of any matrix and try to understand what is going on by experimenting! Have fun :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:2px solid gray\"> </hr>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#this cell is to ignore all warnings that python throws out\n",
    "#Simply run this.\n",
    "#No need to understand or worry about this right now.\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:2px solid gray\"> </hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NumPy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In machine learning/deep learning/computer vision you will deal a lot with matrices. Large matrices. We are talking about matrices containing millions of elements and multiple axes. And you will have to perform various operations on these, like matrix multiplication, addition, etc. But don't worry Python has you cover here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy # pronounced \"num pie\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you ran the above code, you told python to import a library called numpy. Now, NumPy stands for Numerical Python, it is a library made by some brilliant people for the purposes of doing calculations on large matrices effectively. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, here is something that I want you to remember about python. Python is all about doing things that actually matter. In Python we don't believe in reinventing the wheel every time we have to do something. We simply focus on building the best car possible from parts that someone else has already manufactured and optimized. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So many-a-times in python you will see people \"import\" various libraries to use the tools created by other people instead of re-writing the code themselves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy, for example has been written in faster programming languages than python, has been optimized to perfection, is capable of utilizing the multiple cores on you CPU and has an enormous community of people who constantly check for bugs and maintain it. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let me give you a quick demo of the power and simplicity of NumPy. Say, you want a matix with 12 rows and 12 columns that contains all zeros. Well using numpy it takes mearly one line!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numpy.zeros((12,12))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see from running the above code you have a matrix of size (12,12) containing all zeros."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That was fun wasn't it. Alright lets try something new. We are programmers, we are lazy people. We aren't going to count till 12 to check if the above output is actually (12,12). We want numpy to do that too!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So lets run the above code once more but this time we will store the output into a variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_matrix = numpy.zeros((12,12))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how we don't get any output this time. We can see the matrix by printing it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(my_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, now lets find the shape of `my_matrix`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numpy.shape(my_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yes! It is infact (12,12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, writing `numpy` again and again is tiresome (remember we programmers are lazy people!) so lets rename numpy to something else, something short, like `np`. In python you can do this by making a small change in the import statement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voilà! Now you can simply write `np` in the code instead on `numpy` and everything will work just fine! Lets try it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.zeros((12,12))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now look, you can name any library anything you want, but there are certain customs we must follow. Kinda like how you can legally name your son anything you want but still you won't name him dummy due to social pressure or something, although it would make a good name, but I digress."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly in python community we have certain customs like, `numpy` should always be imported as `np`. This makes it easier for someone who is looking at your code to understand what is going on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright now that that is out of the way lets have some more fun with numpy and see what it has to offer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting Python lists to NumPy's Ndarrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_list = [[1 , 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]] # my list is a python list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we have created a list in python. Now whenever we wish to use numpy on python lists we have to convert them into something called an `Ndarray`. We can do this by doing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ndarray_of_my_list = np.array(my_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "ndarray_of_my_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hm... It looks the same doesn't it? So what did we actually do? Well lets use the in-build `type` function to check the types of `my_list` and `ndarray_of_my_list`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(my_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(ndarray_of_my_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see `ndarray_of_my_list` is in fact different from `my_list` . `my_list` is a simple python `list` whereas `ndarray_of_my_list` is an `ndarray`. Now that we have an ndarray in our hands we can do cool operations on it with the power of NumPy!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ndim and Arange"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets find the dimension of that array!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([[1 , 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([[1 , 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.ndim(a) # np.ndim() gives the dimensions of the array."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`a` is a 2-D array."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets see some more dimensions. I am going to use `np.arange` function for this. `np.arange` takes the starting value and the ending value and forms an array for us using the values in between."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = np.arange(1,10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b # we get an array from 1 to 9. np.array will give values including the first element (here 1) \n",
    "  # but excluding the last element (here 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.ndim(b) # as you can see b is a 1-D array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What? You say you want an array with values starting from 1 and going till 100 with span of 3?  Well `arange` got you covered!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "np.arange(1,100,3) # the third argument tells arange to take steps of 3 instead of the default 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets see some more cool examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "np.arange(100,1,-1) # go reverse man! take negitive steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "np.arange(1,3,0.01) # baby steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "np.arange(2,1,-0.01) # negitive baby steps!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I think you get the point. This is why I love python. You get so much done in just one line !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reshape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Awesome! Now lets try to make a 3-D array! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First lets get a long 1-D array and then lets reshape it to out desired shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "one_d_array = np.arange(0,40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "one_d_array # 0 to 39. (remember 40 is not included!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets reshape this. We want there to be "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "three_d_array = np.reshape(one_d_array,(10,2,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "three_d_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.shape(three_d_array) # and as you can see reshape reshaped out array into the desired shape!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.ndim(three_d_array) # also because our shape has a third value, the output is a 3-D array!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well arranging values in an order is all well and good but it is kinda boring to be honest.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we want a matrix with random values. Lets get a little random shall we! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.rand(4,5) # np.random.rand will give you a matrix of random values with the shape you asked for(here (4,5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that `np.random.rand` gives random numbers between 0 and 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can do the following to get values between 0 and 100:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.rand(4,5)*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another cool `random` feature (pun unintended):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.randint(5,size=(2,3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2x3 array with random integers between 0–4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sorting a matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = np.random.rand(4,5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix.sort() # sort is inbuild in python. It can sort lists and ndarrays. Notice how it doesn't return anything..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sort` will sort a matrix. Notice that in this case the variable `matrix` changes to a sorted form."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sort` sorts the matrix about the axis -1 by default (here, sorted along rows...). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can explicitly tell it sort about 0 (here, along the columns) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix.sort(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Copying a matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = np.random.rand(3,3)*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you noticed earlier, when we did `.sort` on a matrix, the values in that matrix got affected. Sometimes we don't want this to happen, we wish to create a copy of the original matrix and work on that instead of affecting the original."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now there is a catch here, simply doing something as shown below won't work (you might thing you are simply assigning the values in matrix to new_matrix but wait and see):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_matrix = matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why won't this work? Well lets first check what elements new_matrix has:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets try modifying `new_matrix`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_matrix.sort(axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, `new_matrix` is modified, well no surprise there..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But lets check `matrix` once shall we.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait what!? `matrix` got sorted too!?? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yes. You see when you do `new_matrix = matrix` you are more or less telling python to refer to `matrix` as `new_matrix`. Thats all. It is like renaming your son from dummy to tummy. He is still gonna be your son!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For people who know about some different programming language this is basically `call by reference`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright so what should we do now? How should we copy the contents of `matrix` into `new_matrix`. (basically `call by value`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`np.copy` to the rescue!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = np.random.rand(3,3)*100 # lets get a fresh new matrix to work with\n",
    "matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_matrix = np.copy(matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_matrix.sort() # sort new_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_matrix # new_matrix sorted along the row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look! `matrix` isn't affected. It is like you cloned your son dummy into a completely new being tummy. (You did that to your own son! omg) And now you can safely experiment on your new son tummy. (that analogy went too dark too quickly)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright alright I sort of lied to you. You can sort a matrix using `np.sort` instead of python's inbuilt `sort` function. That way you won't actually affect the array you are sorting and will get a new array automatically. (I just wanted to teach you about `call by reference` and `call by value`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = np.random.rand(3,3)*100 # lets get a fresh new matrix to work with\n",
    "matrix "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_matrix = np.sort(matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how using `np.sort` instead of `sort` didn't affect the original `matrix` and simply returns a sorted `new_matrix`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding and removing elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_arr = np.arange(1,10)\n",
    "my_arr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can add elements to this array by using the `np.append` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_arr = np.append(my_arr,90) # append one to my_arr and store the output in my_arr\n",
    "my_arr = np.append(my_arr,100)# then append 100 to that\n",
    "my_arr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets say you have 2 arrays. `array1` and `array2`. And you some reason you want to make a bigger array by joining these two arrays together.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "array1 = np.random.randint(low = 0,high = 10,size = (10,))# make an array using random integers from [low,high) of size (10,)\n",
    "array1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "array2 = np.random.randint(low = 0,high = 10,size = (10,))\n",
    "array2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `np.append` you can concatenate the two arrays into one large array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.append(array1,array2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets try this with 2-D arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "array1 = np.random.randint(low = 0,high = 10,size = (10,10))# make an array using random integers from [low,high) of size (10,10)\n",
    "array2 = np.random.randint(low = 0,high = 10,size = (10,12))# make an array using random integers from [low,high) of size (10,12)\n",
    "print('array1\\n',array1)\n",
    "print('array2\\n',array2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.append(array1,array2) # np.append has axis = None by default. So we get this weird output which is 1-D."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.append(array1,array2,axis=-1) #instead if we give it axis=-1 it concatenates the 2 matrixes along the \"column axis\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can remove elements using `np.delete`. `np.delete` take the position of the element to be deleted  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "array1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.delete(array1, (0,0)) # remove element (0,0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slicing and indexing ndarrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before reading this make sure you have watched [Corey Schafer's video on python list slicing and indexing](https://www.youtube.com/watch?v=ajrtAuDg3yw&list=PL-osiE80TeTt2d9bfVyTiXJA-UTHn6WwU&index=20&t=0s). Else you won't understand what is going on.\n",
    "\n",
    "Also you can read [this StackOverflow answer](https://stackoverflow.com/a/509295/11573842) \n",
    "\n",
    "(StackOverflow is a great site meant for developers/coders to ask questions and share problems with other developers/coders and get possible solutions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can slice and index ndarrays just like you slice python lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([[1 , 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a[0] # get row 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a[0][0] # first get row 0 and then get 0th element in that, here 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a[:,2] # all rows, 3rd column "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the result is an ndarray in itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, enough NumPy for now. NumPy is a great and extremely useful library which is capable of doing a lot more than what I have shown you. So in you free time try to learn more about it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic operations on matrices and arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note all of the following examples have been taken from the [official documentation](https://numpy.org/doc/1.18/user/quickstart.html#basic-operations) with some added explanation wherever necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([20,30,40,50])\n",
    "b = np.arange(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a,b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c = a-b\n",
    "c #fairly obvious"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b**2 # square each element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b<3 # do logical operations on each element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array( [[1,1],\n",
    "             [0,1]] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "B = np.array( [[2,0],\n",
    "             [3,4]] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " A * B  # elementwise product"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.multiply(A,B) # elementwise product"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " A @ B   # matrix multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "A.dot(B) # another matrix product"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A.T # transpose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.transpose(A)  # transpose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "a = np.ones((2,3), dtype=int)\n",
    "print(\"before:\\n\",a)\n",
    "a *= 3 # equivalent to a = a*3. This is a short notation. Because, you guessed it, python programmers are lazy people\n",
    "print(\"after:\\n\",a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "a = np.random.random((2,3))\n",
    "print(\"matrix:\\n\",a,\"\\n\")\n",
    "sum_ = a.sum() #sum is a python keyword and can't be used as a variable so I added a _ after it\n",
    "print(\"Sum\",sum_) \n",
    "min_ = a.min() #min is a python keyword and can't be used as a variable so I added a _ after it\n",
    "print(\"Min value\",min_)\n",
    "max_ = a.max() #max is a python keyword and can't be used as a variable so I added a _ after it\n",
    "print(\"Max value\",max_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.argmax(a) # Returns the indices of the maximum values along an axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.argmin(a) # Returns the indices of the minimum values along an axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.arange(-5,5,0.2)\n",
    "a # numpy gives scientific output like this sometimes..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.ceil(a) # Return the ceiling of the input, element-wise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "np.floor(a) #Return the floor of the input, element-wise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:2px solid gray\"> </hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A quick reminder**: *make sure you have watched the [5th video]( https://www.youtube.com/watch?v=daefaLgNkw0&list=PL-osiE80TeTt2d9bfVyTiXJA-UTHn6WwU&index=6&t=0s) and [19th video](https://www.youtube.com/watch?v=ajrtAuDg3yw&list=PL-osiE80TeTt2d9bfVyTiXJA-UTHn6WwU&index=19) of the playlist shared above.*\n",
    "\n",
    "5th video explains about python dictionaries. Pandas dataframes are similar to dictionaries. 19th video explains list slicing which will be useful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets start with Pandas now. Here is what the official Pandas site has to say about it:\n",
    "> Pandas is a fast, powerful, flexible and easy to use open source **data analysis and manipulation** tool,\n",
    "built on top of the Python programming language. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are getting into one of the core aspects of data science. Data analysis and data manipulation. \n",
    "\n",
    "We are going to use this tool to get valuable insight into a developer survey's result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First lets get our data. Go to this [url](https://insights.stackoverflow.com/survey). Then click on `Download Full Data Set (CSV)` for the **year 2019**. Download the entire folder into the `assets` folder by clicking this button in the left top corner of the screen: ![download button](assets\\Images\\download.png)This should have downloaded a zip file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**If you have the appropriate tool to unzip zipped folders then unzip all its contents it into a folder named `developer_survey_2019` inside the `assets` folder. Else run the cell below, it will automatically unzip the zipped folder and save the contents in `developer_survey_2019`.**\n",
    "\n",
    "\n",
    "**If you have any difficulty in this step contact us immediately, as without this you can't proceed any further.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Code to unzip the zipped folder.\n",
    "\n",
    "#Make sure that you saved `developer_survey_2019.zip` folder inside the `assets` folder, only then this code will work.\n",
    "\n",
    "#There is no need to worry about this code.\n",
    "#If you are interested then you can learn about the libraries involved by googling them and reading about them.\n",
    "#But it is ok if you don't understand this part. You will eventually learn python \n",
    "#and its different modules once you start using the language more often...\n",
    "import os\n",
    "import errno\n",
    "import zipfile\n",
    "\n",
    "path_to_zip_file = 'assets/developer_survey_2019.zip'\n",
    "\n",
    "if os.path.exists(path_to_zip_file):\n",
    "    with zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:\n",
    "        zip_ref.extractall('assets/developer_survey_2019/')\n",
    "    assert os.path.exists('assets/developer_survey_2019/')\n",
    "    print('Unzipped successfully into assets/developer_survey_2019')\n",
    "else:\n",
    "    print(\"Make sure that the zip file has been downloaded to the assets folder other wise this code won't work\")\n",
    "    print(\"Contact us if you need any help.\")\n",
    "    raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), 'developer_survey_2019.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `developer_survey_2019` folder contains the results of a survey conducted by [StackOverflow](https://en.wikipedia.org/wiki/Stack_Overflow) in the year 2019. \n",
    "\n",
    "(If you are interested to know more about this survey read the pdf : `so_survey_2019.pdf` inside `assets/developer_survey_2019`)\n",
    "\n",
    "Open the folder `developer_survey_2019` and find a file named `survey_results_public.csv`. This is a `csv` file. `csv` stands for `comma separated values`.\n",
    "\n",
    ">A CSV is a file which allows data to be saved in a tabular format. CSV files can be used with most spreadsheet programs, such as Microsoft Excel or Google Spreadsheets. They differ from other spreadsheet file types because you can only have a single sheet in a file, they can not save cell, column, or row. Also, you cannot not save formulas in this format.\n",
    "\n",
    "**Why are .CSV files used?**\n",
    "\n",
    ">These files serve a number of different business purposes. They help companies export a high volume of data to a more concentrated database."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In short: `csv` files are basically excel files with commas instead of cells.\n",
    "\n",
    "Alright, now try opening the file in excel. You should see something like this:\n",
    "\n",
    "![csv file](assets\\Images\\csv.png)\n",
    "\n",
    "Rows upon rows of tasty data!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets try getting the data from this file this using the pandas library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"assets/developer_survey_2019/survey_results_public.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above line we use `pd.read_csv` to well, read the csv file pointed to by the path : `assets/developer_survey_2019/survey_results_public.csv`. Then we store this into `df`. Now `read_csv` return a pandas dataframe object. \n",
    "\n",
    "Lets see what we have inside `df`..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Immediately you can start seeing valuable information about your data. \n",
    "\n",
    "Look at the first row...\n",
    "\n",
    "![first row](assets\\Images\\first_row.png)\n",
    "\n",
    "This row contains the `features` of you dataframe. While the rest of the rows contain the data corresponding to these features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets use pandas to get more information about this dataset:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now here is some information we can get from this:\n",
    "- So df has `88883 rows` and `85 features`. (You can also use `df.shape` to get this exact information)\n",
    "- We can see the names of all 85 features with info about the datatype of the data within. (Under the `Dtype` column)(You can also use `df.dtypes` to get this exact information)\n",
    "    - `int64` indicates that the feature has integer values in it.\n",
    "    - `float64` indicates that the feature has floating values in it (basically numbers with decimals).\n",
    "    - `object` indicates that the feature has strings in it. (like : \"Not employed, and not looking for work\",\"20s\",etc)\n",
    "- also this line `dtypes: float64(5), int64(1), object(79)` clearly indicates that we have only `5 features with dtype float64`,`1 with dtype int64` and the rest `79 are strings`\n",
    "- also note the `Non-Null Count` column. It shows how many of the 88883 entries have `NaN` values (`NaN` stands for `Not A Number`. It basically means that this data is missing or is `Null`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go back to the `assets\\developer_survey_2019` folder. This time open the file named `survey_results_schema.csv` in excel. You should see this:\n",
    "\n",
    "![csv file](assets\\Images\\csv2.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly this file contains the meaning of all the 85 features. You can go thought them if you are interested."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets get back to our `df`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of printing out the entire `df` everytime we wish to take a peak at it we will use `df.head` function. This allows us to see the \"head\" of the dataframe. That is the top few items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head(7) # show us the first 7 rows. If no aurgument is passed then default = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we are interested in seeing the last few rows we use `df.tail`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.tail() #default = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To see a list of available columns do:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic column and row handling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "=> **Column**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns # and just like that you are blessed with 85 feature names!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets try to see only one column of this `dataframe`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Country'] # just like dictionaries!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doing `df['Country']` shows us only the `Country` column. You can see multiple columns as well by passing them as `list`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df[['Country','Age']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "=> **Row**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to access a particular row we will use `df.iloc`. `iloc` takes in the index of the row and the index of the column. If the index of the column isn't specified then all columns are included."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[0] # get the first row of the df. 0 here is the index of that row"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like columns you can get multiple rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[[0,1]] # first 2 rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get a particular column in these 2 rows too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[[0,1],1] # first 2 rows and column 1 (MainBranch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** You can also use **`.loc`** in this case. `.iloc` and `.loc` quit are similar. `iloc` is based on `index based slicing` whilst `loc` is based on `label based slicing`. Now beware, `iloc` being `index based` follows python's indexing rules. So when you try something like this: `df.iloc[5:9]` you are selecting rows `[5,9)`, where the last value (`9` in this case) isn't included in the slice. Same example as above but with `.loc`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[[0,1],\"MainBranch\"] # notice the label : MainBranch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**value_counts**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets try to explore the data a bit by using the `value_counts` method/function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Country'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And just like that we know that this site is popular amongst people from United States,India,Germany,United Kingdom and Canada."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I want to show you something.. run the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Gender']=='Man'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how this results in a long column of True/False values? Alright run the following cells too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask = df['Hobbyist']==\"Yes\"\n",
    "mask # notice how the mask is True only when 'Hobbyist'==\"Yes\" otherwise it is False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df[mask].head()  #you can also do df.loc[mask].head() to get the exact same result. Try it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we use this mask inside `df` we notice that we only get the rows that had `True` in them essentially we are filtering out the cases where the person is a hobbyist."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can go into a lot more depth with this: Say you are interested in `hobbyists` who are also `male`. We will use `df.query` for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.query('Gender ==\"Man\" and Hobbyist ==\"Yes\"').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are only interested in seeing the `Respondent` feature of the people who are both hobbyists and male. Then do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.query('Gender ==\"Man\" and Hobbyist ==\"Yes\"') ['Respondent'] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember how `df` slicing returned mini dataframes..? This is what allowed us to write `['Respondent']` after `df.query('Gender ==\"Man\" and Hobbyist ==\"Yes\"')` because the output of `df.query('Gender ==\"Man\" and Hobbyist ==\"Yes\"')` is a dataframe in itself.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(df.query('Gender ==\"Man\" and Hobbyist ==\"Yes\"')) # the output is a dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use `&` operator to combine two masks together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask = ((df['Student']==\"No\") & (df['Country']==\"India\")& (df['LanguageWorkedWith']==\"Python\"))\n",
    "df.loc[mask,\"Age1stCode\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above two lines find the `age at which Indian-non-students-pythonistas started coding`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A few more examples:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask = ((df['Age']>14.0) & (df['Age']<20.0))\n",
    "gender = df.loc[mask,\"Gender\"] # get the gender of people between the age group (14,20).\n",
    "print(gender)\n",
    "print(\"\\n---Value counts---\") # get the value counts\n",
    "print(gender.value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ages = df.Age1stCode.value_counts() # you can do df.Age1stCode too instead of df['Age1stCode']\n",
    "ages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(ages[0:10]) # we can slice it to get a list of top 10 ages at which people wrote their first code..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ages_internal_link'></a> We can plot this using the `plot` function. We are using a `bar plot` here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age1stCode'].value_counts()[0:20].plot(kind='bar') # note [0:20] is plotting the top 20 values only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can clearly compare and contrast..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try plotting other things like `Age` and get some insights..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## String methods, inserting/replacing/removing data\n",
    "[//]: # \".... . .-.. .-.. --- / - .... . .-. . -.-.-- / -.-- --- ..- / ..-. --- ..- -. -.. / -- . -.-.-- / .. / .- -- / -. --- - / ... ..- .-. . / .. ..-. / -.-- --- ..- / -.. .. -.. / .. - / .- -.-. -.-. .. -.. . -. - .- .-.. .-.. -.-- / --- .-. / .. -. - . -. - .. --- -. .- .-.. .-.. -.-- / -... ..- - / --. --- --- -.. / .--- --- -... -.-.-- / .. / -.- -. --- .-- / -- --- .-. ... . / -.-. --- -.. . / .-- .- ... / .--. .-. --- -... .- -... .-.. -.-- / .-. . .- .-.. .-.. -.-- / . .- ... -.-- / - --- / .. -.. . -. - .. ..-. -.-- / .- -. -.. / -.. . -.-. --- -.. . / -.--. .... --- .--. . ..-. ..- .-.. .-.. -.-- / ..- ... .. -. --. / ... --- -- . / --- -. .-.. .. -. . / - --- --- .-.. -.--.- .-.-.- / ..- -. .-.. . ... ... / -.-- --- ..- / -.-. .- -. / .- .-.. .-. . .- -.. -.-- / - . .-.. .-.. / - .... .. ... / .. ... / .- -. / . .- ... - . .-. / . --. --. .-.-.- / .. / .- -- / -.-- .- - .. -. .-.-.- / -. --- .-- / - .... .- - / -.-- --- ..- / .... .- ...- . / ..-. --- ..- -. -.. / - .... .. ... / -- . ... ... .- --. . --..-- / .-.. . - / -- . / -.- -. --- .-- .-.-.- / .. / .-- .. .-.. .-.. / --. .. ...- . / -.-- --- ..- / .- / -- .. -. .. / - .-. . .- - / .. ..-. / .. / .- -- / ... - .. .-.. .-.. / .- - / ... ...- -. .. -\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### String methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the `LanguageWorkedWith` the column. Languages are separated using `;`s. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['LanguageWorkedWith']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To deal with this type of data we will use `string methods`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['LanguageWorkedWith'].str.contains(\"Python\",na=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`df['LanguageWorkedWith'].str.contains` checks if the strings within contain `Python`. The `na=False` part is to deal with `NaN` values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Later on we will see how to deal with `NaN` once and for all"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again we can use `df['LanguageWorkedWith'].str.contains(\"Python\",na=False)` as a filter.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask = df['LanguageWorkedWith'].str.contains(\"Python\",na=False)\n",
    "df.loc[mask,[\"Respondent\",\"LanguageWorkedWith\"]] # we can see the Respondent and LanguageWorkedWith"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### inserting/replacing/removing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say during our data analysis we found that a few columns don't really provide us with essential information. We can remove them safely without losing valuable information. We can get rid of these columns by using `df.drop` method.\n",
    "\n",
    "Say we find columns: `OrgSize` and `FizzBuzz` to be of very less interest.\n",
    ">`OrgSize`: Approximately how many people are employed by the company or organization you work for?\n",
    "\n",
    ">`FizzBuzz`: Have you ever been asked to solve FizzBuzz in an interview?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_dataframe = df.drop(columns=['OrgSize', 'FizzBuzz']) \n",
    "new_dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way of removing columns is to use `drop` with `inplace=True`. `inplace = True` tells `drop` to remove the columns directly from `df` instead of just returning a new dataframe. But before doing that lets save 'OrgSize' and  'FizzBuzz' separately first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_be_removed = df[['OrgSize', 'FizzBuzz']]\n",
    "print(\"to_be_removed:\\n\",to_be_removed)\n",
    "df.drop(columns=['OrgSize', 'FizzBuzz'],inplace=True) #notice how this time it doesn't return anything"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now if you check the columns in `df` you will notice that `OrgSize` and `FizzBuzz` aren't amongst them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('OrgSize' in df.columns)\n",
    "print('FizzBuzz' in df.columns)\n",
    "print('Age' in df.columns) # as a check.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice all those `NaN` values in there? These annoying little bastard are hard to deal with. You will have to make tough decisions about what to do with them. If a row is filled with them then you have no other option than to remove it. Otherwise if say only a few rows have missing values then we can do something smart and fill in these `NaN` values with something we believe can fit in (like if all people seem to have an `age` around 20yrs old (the mean of their ages is 20) then for people who's `age` is `NaN` we can safely fill in 20 as their age won't (in most cases) be very far off from it)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `dropna` (drop not a number) function to get rid of them ... Or we can use `fillna` function to fill them with the desired value (desired value can be 0 or mean() or median() etc )\n",
    "\n",
    "To demonstrate these I will copy the dataframe..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_copy1 = df.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_copy1.dropna() #notice how this returns a result. inplace = False by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_copy1.dropna(inplace=True) \n",
    "df_copy1 #df_copy1 got affected as inplace=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_copy1.isnull() # you can use .isnull to check if a given value is null.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "add `any()` after that and it will show you the column name and whether that column has any missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_copy1.isnull().any() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "again add `any()` to see if any columns have True (NaN values) in them..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(df_copy1.isnull().any().any()) # df_copy1 has no NaN values whatsoever\n",
    "print(df.isnull().any().any()) # df has NaN values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright now lets try `fillna` with 0, mean and median. Note: fillna takes all lot of time to run on the entire dataset so we will just focus on the tail of the dataset..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_copy2 = df.tail().copy()# we will only be seeing the changes on the last 5 elements else it will take too long\n",
    "df_copy3 = df.tail().copy()# we will only be seeing the changes on the last 5 elements else it will take too long\n",
    "df_copy4 = df.tail().copy()# we will only be seeing the changes on the last 5 elements else it will take too long"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.tail() #let us see how the tail looks before fillna. In particular notice the NaN in the Age column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_copy2.fillna(0,inplace = True)\n",
    "df_copy2.tail() \n",
    "# fillna(0) replaces all values with 0. Even things like Country are set to 0 (which doesn't make sense...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_copy3.fillna(df_copy3.mean(),inplace = True)\n",
    "df_copy3.tail()\n",
    "# fillna(df_copy3.mean()) replaces only numeric values with 0. (notice Age... rest are unaffected)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_copy4.fillna(df_copy4.median(),inplace = True)\n",
    "df_copy4.tail()\n",
    "# similarly fillna(df_copy4.median()) also affects only numeric values (because how exactly will you find the meadian of string!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can clearly see you can't just blindly use `fillna` or `dropna`. You can make more detailed changes by selecting the column you wish to change..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_copy4.Country.fillna(\"India\",inplace = True) #lets fill NaN countries with \"India\"\n",
    "df_copy4.EdLevel.fillna(\"Unspecified\",inplace = True) #lets fill NaN EdLevel with \"Unspecified\"\n",
    "df_copy4.UndergradMajor.fillna(\"Unspecified\",inplace = True) #lets fill NaN UndergradMajor with \"Unspecified\"\n",
    "#and so on...\n",
    "\n",
    "df_copy4.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember we removed the columns `OrgSize` and `FizzBuzz` from `df` using `df.drop(['OrgSize','FizzBuzz'],inplace = True)`. Well now as an exercise lets try inserting these columns back into `df`..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.insert(loc = 0,column='OrgSize',value=to_be_removed.OrgSize) #column is the column name. loc = 0 means location = first (add column at first)\n",
    "df.insert(loc = 0,column='FizzBuzz',value=to_be_removed.FizzBuzz) #value is actual column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:1px dashed  gray\"> </hr>\n",
    "\n",
    "The `MainBranch` column has been described as following in the `so_survey_2019.pdf` file :\n",
    "\n",
    "![MainBranch](assets\\Images\\ques.png)\n",
    "\n",
    "Notice how these description are too long and can simply be `encoded` as :\n",
    "- `0` for `I am a developer by profession`\n",
    "- `1` for `I am not primarily a developer, but I write code sometimes as part of my work`\n",
    "- `2` for `I used to be a developer by profession, but no longer am`\n",
    "- `3` for `I am a student who is learning to code`\n",
    "- `4` for `I code primarily as a hobby`\n",
    "- `5` for `None of these`\n",
    "- `-1` for `NaN`\n",
    "\n",
    "So essentially what we are interested in doing is to replace these long strings with a single digit. Note: for `NaN` we will still have to use `fillna`. Doing `df_replace['MainBranch'].replace(to_replace = None, value = -1 ,inplace = True)` won't actually work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_replace = df.copy() #to keep df unaffected we shall copy it again.\n",
    "df_replace['MainBranch'].replace(to_replace = \"I am a developer by profession\", value = 0 ,inplace = True)\n",
    "df_replace['MainBranch'].replace(to_replace = \"I am not primarily a developer, but I write code sometimes as part of my work\", value = 1 ,inplace = True)\n",
    "df_replace['MainBranch'].replace(to_replace = \"I used to be a developer by profession, but no longer am\", value = 2 ,inplace = True)\n",
    "df_replace['MainBranch'].replace(to_replace = \"I am a student who is learning to code\", value = 3 ,inplace = True)\n",
    "df_replace['MainBranch'].replace(to_replace = \"I code primarily as a hobby\", value = 4 ,inplace = True)\n",
    "df_replace['MainBranch'].replace(to_replace = \"None of these\", value = 5 ,inplace = True)\n",
    "df_replace['MainBranch'].fillna(-1,inplace = True)\n",
    "df_replace['MainBranch']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_replace['MainBranch']= df_replace['MainBranch'].astype(int) # make the column int instead of float64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_replace['MainBranch']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright. Now lets move on the `Matplotlib`. Again just like `NumPy`, `Pandas` is an awesome library with a ton of interesting and useful features. And just like `NumPy` I can't do justice to it in just one notebook. So you will have to learn more about it on your own."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:1px solid gray\"> </hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The official documentation defines `matplotlib` as :\n",
    ">Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python.\n",
    "\n",
    "We will use `matplotlib` to create different types of graphs and use them to visualize our data. (it can display images too!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt #this is how matplotlib is usually imported.\n",
    "import numpy as np # we will need numpy too..\n",
    "%matplotlib inline "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`%matplotlib inline` is usually added when using `matplotlib` in `jupyter notebook`. It is a magic function that renders the figure in a notebook. No need to worry about it. It isn't really necessary for you to understand... (You can read this [StackOverflow question](https://stackoverflow.com/questions/43027980/purpose-of-matplotlib-inline) about it if you are interested)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plt.plot([1, 2, 3, 4], [1, 4, 2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can probably tell from the above example `[1, 2, 3, 4]` contains the `x coordinates` of the points and `[1, 4, 2, 3]` contain the `y coordinates`.\n",
    "\n",
    "So we are plotting :\n",
    "- (1,1)\n",
    "- (2,4)\n",
    "- (3,2)\n",
    "- (4,3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets try plotting a `sine` wave. We will first create an array that will represent theta. And then we will create another array that will have the corresponding value of `sine` at that time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "theta = np.arange(-2 * np.pi, 2 * np.pi,1)\n",
    "theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sine = np.sin(theta)\n",
    "sine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(theta,sine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What the!?? This doesn't look like a sine wave does it?? What happened?\n",
    "\n",
    "Well we are using only 13 point (check time.shape) between -2$\\pi$ to 2$\\pi$ so we are getting such an ugly graph... Lets increase the number of points by changing the time.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "theta = np.arange(-2 * np.pi, 2 * np.pi,0.01) \n",
    "print(theta,theta.shape) # we are using 1257 points this time..\n",
    "sine = np.sin(theta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(theta,sine)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aaah yes! Beautiful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well my favorite color is orange. So..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(theta,sine,color='orange') # and just like that we have an orange graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well why stop here... lets get some more trigonometric graphs..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cos = np.cos(theta)\n",
    "plt.plot(theta,cos,color='red') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tan = np.tan(theta)\n",
    "plt.plot(theta,tan)\n",
    "plt.ylim(-5, 5) # this is to limit the y axis of the graph from -5 to 5...\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.arange(0.4, 1.1, 0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All this is well and good but I want to see all graphs overlayed on top of one-another..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize = (10,5)) # specify the size of the figure (width,height).. always add this before adding .plot..\n",
    "\n",
    "plt.plot(theta, sine, label='sine')  \n",
    "plt.plot(theta, cos, label='cos')  \n",
    "plt.plot(theta, tan, label='tan')\n",
    "plt.ylim(-5, 5) # try commenting this line and running the cell again..\n",
    "\n",
    "plt.xlabel('theta') # add label for x axis\n",
    "plt.ylabel('Magnitude') # add label for y axis\n",
    "plt.title(\"Trig Functions\") # add title\n",
    "plt.legend() # add legend\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## bar graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = [1,3,5,7,9]\n",
    "y = [5,2,7,8,2]\n",
    "\n",
    "x1 = [5,4,4,8,10]\n",
    "y1 = [8,6,2,5,6]\n",
    "\n",
    "plt.bar(x,y, label=\"bar one\")\n",
    "plt.bar(x1,y1, label=\"bar two\", color='g')\n",
    "plt.xlabel('bar number')\n",
    "plt.ylabel('bar height')\n",
    "plt.title('Sample bar graph')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember how we found the [the ages at which people wrote their first code](#ages_internal_link) and tried plotting it... Well I didn't go deeper into it because you need `matplotlib` along side `pandas` to improve and modify the graph... Let me demonstrate..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age1stCode'].value_counts().plot(kind='bar') #this time lets try plotting all the ages instead of just 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh lord! Clearly we need a larger figure..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20,5))\n",
    "df['Age1stCode'].value_counts().plot(kind='bar') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aaah! much better. Notice how `matplotlib` and `pandas's plot` work together here...\n",
    "\n",
    "Lets try `Age` too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20,5))\n",
    "df['Age'].value_counts().plot(kind='bar') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aaaa... Bigger graph won't do it this time.. lets use `horizontal bar graph`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20,25))\n",
    "df['Age'].value_counts().plot(kind='barh') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The graph looks ok. But let me ask you something. How does the age group `10 to 20 years` compare to other age groups at using the site? \n",
    "\n",
    "Clearly it is hard to tell. We will need a different type of graph to get this information. We shall use a `histogram`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "bins = np.arange(0,100,10)\n",
    "df['Age'].plot(kind='hist',bins = bins,grid=True) \n",
    "plt.locator_params(axis='x', nbins=len(bins)) # this line is to increase the number of x axis tics... Try commenting it \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above code I first specified how to create the bins: `bins = np.arange(0,100,10)`\n",
    "\n",
    "bins : `array([ 0, 10, 20, 30, 40, 50, 60, 70, 80, 90])`\n",
    "\n",
    "So the first bin goes from 0 to 10.\n",
    "\n",
    "The next bin goes from 10 to 20.\n",
    "\n",
    "The next from 20 to 30.\n",
    "\n",
    "And so on..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the line `df['Age'].plot(kind='hist',bins = bins,grid=True)` we are specifying the graph to be `hist`, the bins to be `[ 0, 10, 20, 30, 40, 50, 60, 70, 80, 90]` and to show the grid."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `plt.locator_params(axis='x', nbins=len(bins))` line is used to add x axis tics at 0,10,20...90. Try commenting it out and seeing the results.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subplots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `matplotlib` you can draw several subplots in one main plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.arange(0,10,0.01)\n",
    "sq = x**2 # this is how you raise a number to some power in python\n",
    "cube = x**3\n",
    "log = np.log(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10,10))\n",
    "plt.subplot(2,2,1)\n",
    "plt.plot(x,x)\n",
    "plt.title('Linear')\n",
    "\n",
    "plt.subplot(2,2,2)\n",
    "plt.plot(x,sq)\n",
    "plt.title('Quadratic')\n",
    "\n",
    "plt.subplot(2,2,3)\n",
    "plt.plot(x,cube)\n",
    "plt.title('Cubic')\n",
    "\n",
    "plt.subplot(2,2,4)\n",
    "plt.plot(x,log)\n",
    "plt.title('Logarithmic')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets try to understand what `plt.subplot(2,2,1)` means. The first `2` means 2 rows. The next `2` means 2 columns.\n",
    "\n",
    "So we want to display `4 subplot` (2$*$2 = 4) in a grid of `2 rows` and `2 columns`\n",
    "\n",
    "The third number in `plt.subplot(2,2,1)`, here `1` tells `matplotlib` which `subplots` we are plotting. In this case it can be 1,2,3 or 4 signifying the respective `subplots`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So to display 2 subplots side by side we will do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.subplot(1,2,1) # 1 row , 2 columns, we are drawing subplot = 1 \n",
    "plt.plot(x,x)\n",
    "plt.title('Linear')\n",
    "\n",
    "plt.subplot(1,2,2) # 1 row , 2 columns, we are drawing subplot = 2\n",
    "plt.plot(x,sq)\n",
    "plt.title('Quadratic')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly to display 2 subplots on top of one another we will do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(5,10))\n",
    "\n",
    "plt.subplot(2,1,1) # 2 row , 1 columns, we are drawing subplot = 1\n",
    "plt.plot(x,cube,'g--') # green and made of ---\n",
    "plt.title('Cubic')\n",
    "\n",
    "plt.subplot(2,1,2) # 2 row , 1 columns, we are drawing subplot = 2\n",
    "plt.plot(x,log,'r*') # red and made of *\n",
    "plt.title('Logarithmic')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also read and display images using `matplotlib`. Lets try reading a test image stored in the `assets/Images` directory (in programming we refer to folders as directories... well it sounds cooler)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plt.imread('assets/Images/test.jpg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That doesn't look like a image! Well it actually is an image. Well to be exact these are the pixel values that make up the image given to us in the form of an `ndarray`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can display this image by using `plt.imshow`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_image = plt.imread('assets/Images/test.jpg')\n",
    "plt.imshow(test_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also save images using `imsave`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imsave(fname=\"my_image.jpg\",arr=test_image) #this will save the above image as \"my_image.jpg\"  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: the above line will save the image right next to where you have this `jupyter notebook` saved. We call this place the `working directory` in programming."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"margin: auto;\n",
    "           height: 40px;\n",
    "           background: linear-gradient(135deg, #ECEDDC 25%, transparent 25%) -20px 0, linear-gradient(225deg, #ECEDDC 25%, transparent 25%) -20px 0, linear-gradient(315deg, #ECEDDC 25%, transparent 25%), linear-gradient(45deg, #ECEDDC 25%, transparent 25%);\n",
    "           background-size: 40px 40px;\n",
    "           background-color: #EC173A;\"></hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations! You have completed the first notebook of Drishti's workshop on Machine Learning/ Deep Learning /Computer vision. I hope you learned a lot. Although I have said this twice already I would still like to emphasize that this is just the tip of the iceberg, each of these 3 libraries is incredibly powerful and filled with features. The only way you can possibly learn more about them is to use them in your projects. Our aim was just to get you informed and excited about what you can do with these libraries.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following is a list of resources that you can follow to learn more about these libraries:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NumPy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Official documentation](https://numpy.org/doc/1.18/user/quickstart.html)\n",
    "- [NumPy Tutorials by Amulya's Academy on youtube](https://www.youtube.com/playlist?list=PLzgPDYo_3xukqLLjNeuCxj4CwvkJin03Z)\n",
    "- [NumPy Cheat Sheet — Python for Data Science](https://www.dataquest.io/blog/numpy-cheat-sheet/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Official documentation](https://pandas.pydata.org/docs/getting_started/intro_tutorials/01_table_oriented.html#min-tut-01-tableoriented)\n",
    "- [Pandas tutorials by Corey Schafer on youtube](https://www.youtube.com/playlist?list=PL-osiE80TeTsWmV9i9c58mdDCSskIFdDS)\n",
    "- [Data analysis in Python with pandas by Data School on youtube](https://www.youtube.com/playlist?list=PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9y) -  **recommended**\n",
    "- [The Pandas DataFrame – loading, editing, and viewing data in Python]()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Official documentation](https://matplotlib.org/2.1.1/tutorials/index.html)\n",
    "- [Sentdex's tutorial on Matplotlib on youtube: Matplotlib Tutorial Series - Graphing in Python](https://www.youtube.com/playlist?list=PLQVvvaa0QuDfefDfXb9Yf0la1fPDKluPF)\n",
    "- [Sentdex's website:Introduction to Matplotlib and basic line](https://pythonprogramming.net/matplotlib-intro-tutorial/) - you can get the code he uses in his videos here.\n",
    "- [Matplotlib - Simple Plot by tutorials point](https://www.tutorialspoint.com/matplotlib/matplotlib_simple_plot.htm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have a nice day!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}