Skip to content

Commit

Permalink
new updates to notebook
Browse files Browse the repository at this point in the history
  • Loading branch information
github-actions committed Jun 7, 2024
1 parent c4de154 commit 9b6c32c
Showing 1 changed file with 52 additions and 21 deletions.
73 changes: 52 additions & 21 deletions nyc_buildings/nyc_buildings.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Many plotting libraries can handle collections of polygons, e.g. [Bokeh](https://docs.bokeh.org/en/latest/docs/examples/topics/geo/texas_hover_map.html) or [HoloViews+Bokeh](https://holoviews.org/gallery/demos/bokeh/choropleth_data_link.html). However, because browser-based libraries like Bokeh and Plotly send all the polygon data to the browser, they can struggle when either the collections or the polygons themselves get large. Even natively in Python, typical formats like Shapely for representing polygons scale poorly to large polygon collections, because each polygon is wrapped up as a separate Python object, leading to a lot of duplicated storage overhead when many polygons of the same type are defined.\n",
"Many plotting libraries can handle collections of polygons, including [Bokeh](https://docs.bokeh.org/en/latest/docs/examples/topics/geo/texas_hover_map.html) and [HoloViews](https://holoviews.org/gallery/demos/bokeh/choropleth_data_link.html). However, because browser-based libraries like Bokeh and Plotly send all the polygon data to Javascript running in the browser, they can struggle when either the collections or the individual polygons themselves get large. Even natively in Python, typical formats like Shapely for representing polygons scale poorly to large polygon collections, because each polygon is wrapped up as a full, separate Python object, leading to a lot of duplicated storage overhead when many polygons of the same type are defined.\n",
"\n",
"If you want to work with lots of polygons, here you can see how to use [SpatialPandas](https://github.com/holoviz/spatialpandas) and Dask to represent polygons efficiently in memory, fastparquet to represent them efficiently on disk, and [Datashader](https://datashader.org) to render them quickly in a web browser. This notebook also demonstrates how to support hovering for datashaded polygons, with Bokeh overlaying a single vector-based representation of a polygon where the mouse cursor is, while all the rest are sent to the browser only as rendered pixels. That way hover and other interactive features can be supported fully without ever needing to transfer large amounts of data or store them in the limited memory of the web browser tab. \n",
"If you want to work with lots of polygons, here you can see how to use [GeoPandas](https://github.com/geopandas/geopandas) and Dask to represent polygons efficiently in memory, and [Datashader](https://datashader.org) to render them quickly in a web browser. This notebook also demonstrates how to support hovering for datashaded polygons, with Holoviews setting up Bokeh to overlay a single vector-based representation of a polygon where the mouse cursor is, while all the rest are sent to the browser only as rendered pixels. That way hover and other interactive features can be supported fully without ever needing to transfer large amounts of data or store them in the limited memory of the web browser tab. \n",
"\n",
"This example plots the outlines of all the buildings in New York City. See\n",
"This example plots the outlines of all one million+ buildings in New York City. See\n",
"[nyc.gov](https://nyc.maps.arcgis.com/home/item.html?id=870bf69e8a8044aea4488e564c0b4010#overview) for the original data and its description."
]
},
Expand All @@ -28,11 +28,21 @@
"import hvplot.dask # noqa\n",
"import hvplot.pandas # noqa\n",
"import geopandas as gpd\n",
"import colorcet as cc"
"import colorcet as cc\n",
"from holoviews import opts"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"opts.defaults(\n",
" opts.Polygons(width=600, height=500, xaxis=None, yaxis=None))"
]
},
{
"attachments": {},
"cell_type": "raw",
"metadata": {},
"source": [
Expand Down Expand Up @@ -61,15 +71,17 @@
"outputs": [],
"source": [
"gdf = gpd.read_parquet('new_nyc_buildings.parq')\n",
"\n",
"print(len(gdf))\n",
"gdf.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we compute the top categories and drop everything else:"
"Here you can see that we have 1.1 million \"MultiPolygons\", some of which have a `type` and `name` declared.\n",
"\n",
"To get a look at this data, let's plot all the polygons, overlaid on a tiled map of the region:"
]
},
{
Expand All @@ -78,17 +90,16 @@
"metadata": {},
"outputs": [],
"source": [
"cats = list(gdf['type'].value_counts().iloc[:10].index) + ['unknown']\n",
"gdf['type'] = gdf['type'].fillna('unknown')\n",
"gdf = gdf[gdf['type'].isin(cats)]\n",
"gdf['type'] = gdf['type'].astype('category')"
"gdf.hvplot.polygons(tiles='CartoLight', rasterize=True, aggregator='any')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next we build a legend for the categories and declare a tile source as backdrop:"
"At this scale, the plot looks like a bunch of dots or large colored areas, because each building is smaller than a pixel in the plot. But if you have a live Python server running, you can use the Bokeh tools to zoom in and have the plot dynamically redrawn, showing you the full outline of each polygon. You should see more detail whenever you zoom in, possibly after a short delay after Datashader re-renders the new scene.\n",
"\n",
"Now let's make use of the category information. To get a manageable number of types, we'll compute the top 10 most common categories and drop everything else:"
]
},
{
Expand All @@ -97,15 +108,20 @@
"metadata": {},
"outputs": [],
"source": [
"colors = cc.glasbey_bw_minc_20_maxl_70\n",
"color_key = {cat: tuple(int(e*255.) for e in colors[i]) for i, cat in enumerate(cats)}"
"cats = list(gdf['type'].value_counts().iloc[:10].index) + ['unknown']\n",
"\n",
"gdf = (gdf.assign(type=gdf['type'].fillna('unknown'))\n",
" .loc[lambda df: df['type'].isin(cats)]\n",
" .astype({'type': 'category'}))\n",
"\n",
"gdf.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we put it all together, declaring a `Polygons` element from our data, datashade them and use the `inspect_polygons` operation to allow us to hover on the data:"
"Now we can view each category separately with a selector widget:"
]
},
{
Expand All @@ -114,26 +130,41 @@
"metadata": {},
"outputs": [],
"source": [
"gdf.head(1000).hvplot.polygons(tiles='CartoLight', rasterize=True)"
"gdf.hvplot.polygons(rasterize=True, tiles='CartoLight', groupby='type', aggregator='any')"
]
},
{
"cell_type": "code",
"execution_count": null,
"cell_type": "markdown",
"metadata": {},
"outputs": [],
"source": [
"gdf.hvplot.polygons(tiles='CartoLight', rasterize=True)"
"If you look at each one, you can see that unfortunately most of the categories are unknown, but there are interesting patterns (e.g. almost no garages in Manhattan, and apparently all the sheds are in New Jersey).\n",
"\n",
"Since these buildings don't normally overlap, we can actually combine them all into a single plot using color to show all of the categories (though we have to construct a color key manually):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"colors = cc.glasbey_bw_minc_20_maxl_70\n",
"color_key = {cat: tuple(int(e*255.) for e in colors[i]) for i, cat in enumerate(cats)}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we put it all together, declaring a `Polygons` element from our data, datashade them and use the `inspect_polygons` operation to allow us to hover on the data:"
]
},
{
"cell_type": "raw",
"metadata": {},
"source": [
"# This is the hold-up point\n",
"# gdf.head(1000).hvplot.polygons(tiles='CartoLight', rasterize=True, c='type', cmap=color_key)"
"gdf.hvplot.polygons(tiles='CartoLight', rasterize=True, aggregator='any', c='type', cmap=color_key)"
]
},
{
Expand Down

0 comments on commit 9b6c32c

Please sign in to comment.