notebooks/nyc_taxi_years_correlation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using cuSpatial to Correlate Taxi Data after a Format Change\n",
    "In 2017, the NYC Taxi data switched from giving their pickup and drop off locations in `lat/lon` to one of 262 `LocationID`s.  While these `LocationID`s made it easier to determine some regions and borough information that was lacking in the previous datasets, it made it difficult to compare datasets before and after this transition.  \n",
    "\n",
    "\n",
    "\n",
    "By using cuSpatial `Points in Polygon` (PIP), we can quickly and easily map the latitude and longitude of the pre-2017 taxi dataset to the `LocationID`s of the 2017+ dataset.  In this notebook, we will show you how to do so.  cuSpatial 0.14 PIP only works on 31 polygons per call, so we will show how to process this larger 263 polygon shapefile with minimal memory impact.  cuSpatial 0.15 will eliminate the 31 polygon limitation and provide substantial additional speedup.\n",
    "\n",
    "You may need a 16GB card or larger."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cuspatial\n",
    "import geopandas as gpd\n",
    "import cudf\n",
    "from numba import cuda\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download the Data\n",
    "We're going to download the January NYC Taxi datasets for 2016 and 2017.  We also need the NYC Taxi Zones "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tzones_lonlat.json found\n",
      "taxi2016.csv found\n",
      "taxi2017.csv found\n"
     ]
    }
   ],
   "source": [
    "!if [ ! -f \"tzones_lonlat.json\" ]; then curl \"https://data.cityofnewyork.us/api/geospatial/d3c5-ddgc?method=export&format=GeoJSON\" -o tzones_lonlat.json; else echo \"tzones_lonlat.json found\"; fi\n",
    "!if [ ! -f \"taxi2016.csv\" ]; then curl https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2016-01.csv -o taxi2016.csv; else echo \"taxi2016.csv found\"; fi   \n",
    "!if [ ! -f \"taxi2017.csv\" ]; then curl https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2017-01.csv -o taxi2017.csv; else echo \"taxi2017.csv found\"; fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in the Taxi Data with cuDF\n",
    "Let's read in the pickups and dropoffs for 2016 and 2017."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "taxi2016 = cudf.read_csv(\"taxi2016.csv\")\n",
    "taxi2017 = cudf.read_csv(\"taxi2017.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the columns in `taxi2016` and `taxi2017` to verify the difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'DOLocationID', 'PULocationID'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set(taxi2017.columns).difference(set(taxi2016.columns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in the Spatial Data with cuSpatial\n",
    "\n",
    "cuSpatial loads polygons into a `cudf.Series` of _Feature offsets_, a `cudf.Series` of _ring offsets_, and a `cudf.DataFrame` of `x` and `y` coordinates (which can be used for lon/lat as in this case) with the `read_polygon_shapefile` function. We're working on more advanced I/O integrations and nested `Columns` this year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "tzones = gpd.GeoDataFrame.from_file('tzones_lonlat.json')\n",
    "tzones.to_file('cu_taxi_zones.shp')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "taxi_zones = cuspatial.read_polygon_shapefile('cu_taxi_zones.shp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting lon/lat coordinates to LocationIDs with cuSpatial\n",
    "Looking at the taxi zones and the taxi2016 data, you can see that\n",
    "- 10.09 million pickup locations\n",
    "- 10.09 million dropoff locations\n",
    "- 263 LocationID features\n",
    "- 354 LocationID rings\n",
    "- 98,192 LocationID coordinates\n",
    "\n",
    "Now that we've collected the set of pickup locations and dropoff locations, we can use `cuspatial.point_in_polygon` to quickly determine which pickups and dropoffs occur in each borough.\n",
    "\n",
    "To do this in a memory efficient way, instead of creating two massive 10.09 million x 263 arrays, we're going to use the 31 polygon limit to our advantage and map the resulting true values in the array a new `PULocationID` and `DOLocationID`, matching the 2017 schema.  Locations outside of the `LocationID` areas are `264` and `265`.  We'll be using 264 to indicate our out-of-bounds zones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 31, 62, 93, 124, 155, 186, 217, 248, 263]\n"
     ]
    }
   ],
   "source": [
    "pip_iterations = list(np.arange(0, 263, 31))\n",
    "pip_iterations.append(263)\n",
    "print(pip_iterations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 11.2 s, sys: 6.27 s, total: 17.5 s\n",
      "Wall time: 17.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "taxi2016['PULocationID'] = 264\n",
    "taxi2016['DOLocationID'] = 264\n",
    "for i in range(len(pip_iterations)-1):\n",
    "    start = pip_iterations[i]\n",
    "    end = pip_iterations[i+1]\n",
    "    pickups = cuspatial.point_in_polygon(taxi2016['pickup_longitude'] , taxi2016['pickup_latitude'], taxi_zones[0][start:end], taxi_zones[1], taxi_zones[2]['x'], taxi_zones[2]['y'])\n",
    "    dropoffs = cuspatial.point_in_polygon(taxi2016['dropoff_longitude'] , taxi2016['dropoff_latitude'], taxi_zones[0][start:end], taxi_zones[1], taxi_zones[2]['x'], taxi_zones[2]['y'])\n",
    "    for j in pickups.columns:\n",
    "        taxi2016['PULocationID'].loc[pickups[j]] = j\n",
    "    for j in dropoffs.columns:\n",
    "        taxi2016['DOLocationID'].loc[dropoffs[j]] = j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "del pickups\n",
    "del dropoffs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I wonder how many taxi rides in 2016 started and ended in the same location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.10632302994054163\n"
     ]
    }
   ],
   "source": [
    "print(taxi2016['DOLocationID'].corr(taxi2016['PULocationID']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not nearly as many as I thought. How many exactly?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.07 %\n"
     ]
    }
   ],
   "source": [
    "print(format((taxi2016['DOLocationID'] == taxi2016['PULocationID']).sum()/taxi2016.shape[0], '.2f'), '%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Something with perhaps a higher correlation: It seems likely that pickups and dropoffs by zone are not likely to change much from year to year, especially within a given month. Let's see how similar the pickup and dropoff patterns are in January 2016 and 2017:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6013696841885884\n",
      "0.5822576814239405\n"
     ]
    }
   ],
   "source": [
    "print(taxi2016['DOLocationID'].value_counts().corr(taxi2017['DOLocationID'].value_counts()))\n",
    "print(taxi2016['PULocationID'].value_counts().corr(taxi2017['PULocationID'].value_counts()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bringing Them All Together\n",
    "If you wanted to include this as part of a larger clean up of Taxi data, you'd then concatenate this dataframe into a `dask_cudf` dataframe and delete its `cuDF` version, or convert it into arrow memory format and process it similar to how we did in the mortgage notebook.  For now, as we are only working on a couple of GBs, we'll concatenate in cuDF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = cudf.concat([taxi2017, taxi2016])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final Check\n",
    "Now to test to see if both years are present as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         VendorID tpep_pickup_datetime tpep_dropoff_datetime  passenger_count  \\\n",
      "542479          1  2017-01-15 17:33:37   2017-01-15 17:35:32                1   \n",
      "159122          1  2016-01-01 04:19:02   2016-01-01 04:25:09                3   \n",
      "180769          2  2016-01-01 06:40:21   2016-01-01 06:41:09                2   \n",
      "4871707         1  2016-01-15 01:07:34   2016-01-15 01:11:50                1   \n",
      "7766271         1  2016-01-31 04:38:06   2016-01-31 04:39:05                1   \n",
      "9039663         1  2016-01-22 17:49:06   2016-01-22 17:49:34                2   \n",
      "9259375         1  2016-01-23 10:58:21   2016-01-23 11:20:08                1   \n",
      "\n",
      "         trip_distance  RatecodeID store_and_fwd_flag  PULocationID  \\\n",
      "542479             0.0           5                  N           204   \n",
      "159122             1.5           1                  N           204   \n",
      "180769             0.0           5                  N           204   \n",
      "4871707            1.2           1                  N           204   \n",
      "7766271            0.0           5                  N           204   \n",
      "9039663            0.0           5                  N           204   \n",
      "9259375            4.8           1                  N           204   \n",
      "\n",
      "         DOLocationID  payment_type  ...  extra  mta_tax  tip_amount  \\\n",
      "542479            204             1  ...    0.0      0.0        10.0   \n",
      "159122            204             2  ...    0.5      0.5         0.0   \n",
      "180769            204             1  ...    0.0      0.5         0.0   \n",
      "4871707           117             1  ...    0.5      0.5         0.0   \n",
      "7766271           204             1  ...    0.0      0.0         0.0   \n",
      "9039663           204             1  ...    0.0      0.0        10.0   \n",
      "9259375            77             2  ...    0.0      0.5         0.0   \n",
      "\n",
      "         tolls_amount  improvement_surcharge  total_amount  pickup_longitude  \\\n",
      "542479            0.0                    0.3        103.22              null   \n",
      "159122            0.0                    0.3          8.30      -73.83747864   \n",
      "180769            0.0                    0.3         70.80      -73.83963776   \n",
      "4871707           0.0                    0.3          7.30      -73.82751465   \n",
      "7766271           0.0                    0.3         91.10      -73.83338165   \n",
      "9039663           0.0                    0.3         94.14       -73.8551712   \n",
      "9259375           0.0                    0.3         19.30      -73.84205627   \n",
      "\n",
      "        pickup_latitude dropoff_longitude dropoff_latitude  \n",
      "542479             null              null             null  \n",
      "159122      40.57971573      -73.85549164      40.57531738  \n",
      "180769      40.57685089      -73.83963776      40.57685089  \n",
      "4871707     40.58262253       -73.8052063      40.58834839  \n",
      "7766271     40.58089828      -73.83340454      40.58086777  \n",
      "9039663     40.57473373      -73.85518646      40.57474899  \n",
      "9259375     40.57816315      -73.76178741      40.59595871  \n",
      "\n",
      "[7 rows x 21 columns]\n"
     ]
    }
   ],
   "source": [
    "print(df.query('PULocationID == 204'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, 2017 values lack longitude and latitude. It is trivial and fast using our existing `DataFrame` to compute the mean of each `PULocationID` and `DOLocationID`s. We could inject those into the missing values to easily see how well the new LocationIDs map to pickup and dropoff locations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Back To Your Workflow\n",
    "So now you've seen how to use cuSpatial to clean and correlate your spatial data using the NYC taxi data. You can now perform multi year analytics across the entire range of taxi datasets using your favorite RAPIDS libraries,"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}