Kaggle Taxi XGBoost Challenge.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Problem\n",
    "\n",
    "![alt text](https://kaggle2.blob.core.windows.net/competitions/kaggle/4378/media/portugal_map4.png \"Logo Title Text 1\")\n",
    "\n",
    "# Can we predict how long will a taxi ride last? \n",
    "\n",
    "#### To improve the efficiency of electronic taxi dispatching systems it is important to be able to predict how long a driver will have his taxi occupied. If a dispatcher knew approximately when a taxi driver would be ending their current ride, they would be better able to identify which driver to assign to each pickup request. \n",
    "\n",
    "\n",
    "## The Tools\n",
    "\n",
    "We need data, compute, and algorithms!\n",
    "\n",
    "#### (1) Data https://www.kaggle.com/c/nyc-taxi-trip-duration/data \n",
    "\n",
    "##### Our Features\n",
    "- id - a unique identifier for each trip\n",
    "- vendor_id - a code indicating the provider associated with the trip record\n",
    "- pickup_datetime - date and time when the meter was engaged\n",
    "- dropoff_datetime - date and time when the meter was disengaged\n",
    "- passenger_count - the number of passengers in the vehicle (driver entered value)\n",
    "- pickup_longitude - the longitude where the meter was engaged\n",
    "- pickup_latitude - the latitude where the meter was engaged\n",
    "- dropoff_longitude - the longitude where the meter was disengaged\n",
    "- dropoff_latitude - the latitude where the meter was disengaged\n",
    "- store_and_fwd_flag - This flag indicates whether the trip record was held in vehicle memory - before sending to the vendor because the vehicle did not have a connection to the server \n",
    "- trip_duration - duration of the trip in seconds\n",
    "\n",
    "\n",
    "#### (2) Compute - Google Colab (GPU + Python Environment in the cloud)\n",
    "\n",
    "#### (3) Algorithms - Pandas for data preprocessing, XGBoost for learning, matplotlib for visualization\n",
    "\n",
    "\n",
    "## The Steps\n",
    "\n",
    "- Split into training and testing data\n",
    "- Visualize the data (What do the features look like? how long is a trip? How much overlap between the training and testing data)\n",
    "- Use XGBoost to train a model on the data\n",
    "- Save the trained model \n",
    "\n",
    "## XGBoost\n",
    "\n",
    "\n",
    "\n",
    "### First, lets talk Ensembles\n",
    "\n",
    "![alt text](http://images.slideplayer.com/37/10747814/slides/slide_2.jpg \"Logo Title Text 1\")\n",
    "\n",
    "- An ensemble is just a collection of predictors which come together (e.g. mean of all predictions) to give a final prediction. \n",
    "- The reason we use ensembles is that many different predictors trying to predict same target variable will perform a better job than any single predictor alone. \n",
    "- Ensembling techniques are further classified into Bagging and Boosting.\n",
    "\n",
    "### Wait, whats bagging?\n",
    "\n",
    "- Bagging is a simple ensembling technique in which we build many independent predictors/models/learners and combine them using some model averaging techniques. (e.g. weighted average, majority vote or normal average)\n",
    "- We typically take random sub-sample/bootstrap of data for each model, so that all the models are little different from each other. \n",
    "- Each observation is chosen with replacement to be used as input for each of the model. So, each model will have different observations based on the bootstrap process. \n",
    "- Because this technique takes many uncorrelated learners to make a final model, it reduces error by reducing variance. Example of bagging ensemble is Random Forest models.\n",
    "\n",
    "\n",
    "![alt text](https://cdn-images-1.medium.com/max/592/1*i0o8mjFfCn-uD79-F1Cqkw.png \"Logo Title Text 1\")\n",
    "\n",
    "\n",
    "![alt text](https://www.researchgate.net/profile/Nazar_Zaki/publication/269359645/figure/fig2/AS:295037326905349@1447353788598/Random-Forest-algorithm.png \"Logo Title Text 1\")\n",
    "\n",
    "\n",
    "### Ok, then whats boosting?\n",
    "\n",
    "- Boosting is an ensemble technique in which the predictors are not made independently, but sequentially.\n",
    "- subsequent predictors learn from the mistakes of the previous predictors. \n",
    "- Therefore, the observations have an unequal probability of appearing in subsequent models and ones with the highest error appear most. \n",
    "- So the observations are not chosen based on the bootstrap process, but based on the error - \n",
    "- The predictors can be chosen from a range of models like decision trees, regressors, classifiers etc. \n",
    "- Because new predictors are learning from mistakes committed by previous predictors, it takes less time/iterations to reach close to actual predictions. \n",
    "- But we have to choose the stopping criteria carefully or it could lead to overfitting on training data. \n",
    "\n",
    "#### Gradient Boosting is an type of boosting algorithm!\n",
    "\n",
    "![alt text](https://cdn-images-1.medium.com/max/1600/1*8T4HEjzHto_V8PrEFLkd9A.png \"Logo Title Text 1\")\n",
    "\n",
    "![alt text](https://cdn-images-1.medium.com/max/1600/1*PaXJ8HCYE9r2MgiZ32TQ2A.png \"Logo Title Text 1\")\n",
    "\n",
    "\n",
    "The objective of any supervised learning algorithm is to define a loss function and minimize it. Let’s see how maths work out for Gradient Boosting algorithm. Say we have mean squared error (MSE) as loss defined as:\n",
    "\n",
    "![alt text](https://cdn-images-1.medium.com/max/1600/1*fHenn7NVqcWvw25D3-zRiQ.png \"Logo Title Text 1\")\n",
    "\n",
    "We want our predictions, such that our loss function (MSE) is minimum. By using gradient descent and updating our predictions based on a learning rate, we can find the values where MSE is minimum.\n",
    "\n",
    "![alt text](https://cdn-images-1.medium.com/max/1600/1*LLbC4TstqzXQ3hzA8wCmeg.png \"Logo Title Text 1\")\n",
    "\n",
    "So, we are basically updating the predictions such that the sum of our residuals is close to 0 (or minimum) and predicted values are sufficiently close to actual values.\n",
    "\n",
    "\n",
    "#### The intuition behind gradient boosting algorithm is to repetitively leverage the patterns in residuals and strengthen a model with weak predictions and make it better. Once we reach a stage that residuals do not have any pattern that could be modeled, we can stop modeling residuals (otherwise it might lead to overfitting). Algorithmically, we are minimizing our loss function, such that test loss reach its minima.\n",
    "\n",
    "\n",
    "![alt text](https://qph.fs.quoracdn.net/main-qimg-dc06543fbfbcd10c58659a42cac16dc9 \"Logo Title Text 1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}