Kepler.gl 2019 Roadmap

Author: Shan He

Date: 01/28/2019

Table of Contents

• Background
• Vision Statement
• Milestones
• Features
  • Milestone 1. Performance and Ubiquitous
    • Performance Improvement
      • GPU data filtering integration
      • GPU aggregation layer integration
    • Easy Sharing and Embedding
      • Better image export
      • Save / share map via URL
      • Load data directly from an URL
      • Publish kepler.gl to cdn
      • kepler.gl for jupyter notebook
      • Embed an interactive map as HTML
      • Share directly to social media
      • Export map as a video
      • Export map as a gif
    • Documentation
      • Contribution Guidelines for GitHub community
      • Publish public roadmap
      • Add Q&A and best practices
      • Add API Documentations
  • Milestone 2. Advancing Geo-analytics
    • Geo Operations
      • Allow drawing on map to create paths and polygons
      • Allow filtering by geo boundaries
      • Able to highlight an area
      • Allow connecting points based on sequence
      • Aggregate points by geo-boundaries
    • Advanced Layers
      • H3 aggregation in hexbin layer
      • Add edge bundling layer
      • Add Arrow layer
      • Add traffic animation layer
  • Milestone 3. Improving Usability
    • Layer
      • Allow layer annotations
      • Customize color scale and palette
      • Allow opacity by value
      • Allow duplicating a layer
    • Filter
      • Allow playback on other filter types
      • Allow filter per layer
    • Interaction
      • More tooltip settings
      • More Brushing settings
    • Base Map
      • Allow loading base map tiles from a custom tile server
    • Framework
      • Export data util functions
      • Allow UI customization
      • Provide backend connector
  • Milestone 4. Data Integration
    • File Upload
      • Increase upload dataset limit
      • Support shapefile
      • Support vector tile
      • Support KML
    • Data Manipulation
      • Allow spatial joining
      • Allow editing data types
    • Location Data Integration
      • Load spatial data using 3rd party API
      • Read Movement data
• Backlogs

Background

Kepler.gl has attracted great interests following its launch 8 months ago. It has 100k unique users to this day and used by engineers from Airbnb, Google sidewalk lab, UNICEF etc. People spent an average of 8 minutes on kepler.gl. The majority of our users are data scientist/analysts, software engineers, GIS specialist, and academic researchers. They are impressed by its ease of use, speed, and the beautiful visualizations. Many people also mentioned that the arc layer, 3d hexagon, and time animations are unique features that they love the most.

Among the potential areas that we can improve on, performance came as the top issue. Many mentioned UI lagging when handling large dataset. Another universal requested function is the ability to save and share maps or embed an interactive map on a web page. For usability, people often compare kepler.gl to Carto, ArcGIS, they requested many useful features on top of the existing layer, filter and interaction options. They also wish to integrate other data types into kepler.gl or it connects to different databases.

There are many opportunities ahead, and we want to plan our 2019 roadmap based on all the feedback we got from people using kepler.gl. This roadmap includes all the items we want to work on for 2019. We are excited to release it to the public so that everyone can contribute to kepler.gl and help us achieve our goals.

Vision Statement

Moving on to 2019, we want to push the development of kepler.gl along 4 main themes: Performance & Ubiquitous, Advancing geo-analytics, Improving Usability and Data Integration. We will utilize GPU to accelerate filtering and aggregation, make it easy to save/share/embed kepler.gl and create kepler.gl plugins for tableau and jypyter notebook. To further establish kepler.gl as an advanced geo-analytics tool, we will implement essential geo operations such as drawing, filtering/aggregation by geo-boundaries, add new layers including edge bundling and trip animation layer for advanced visualization. We will also address many enhancements to the usability of layers, filters, and interactions. Finally, we plan to support data manipulation such as spatial joining and integrate location services for geocoding and routing.

To encourage external contribution, we also plan to add contribution guidelines and publish our roadmap. We will add more documentation and framework API to help people use kepler.gl as a framework.

Milestones

Milestone 1. Performance and Ubiquitous

Jan 1 2019 - Feb 28 2019

Milestone 2. Advancing geo-analytics

Mar 1 2019 - Apr 30 2019

Milestone 3. Improving Usability

Mar 1 2019 - Jun 30 2019

Milestone 4. Data Integration

Jul 1 2019 - Aug 30 2019

Features

Priority
Scope
Status

Milestone 1. Performance and Ubiquitous

Performance Improvement

One of kepler.gl’s strong competitive edge is its ability to process and visualize a large amount of data, fast, inside the browser. That’s not to say we are doing everything right. We see complaints about the app crashing, lags during filtering, aggregation and domain calculation. Most of it is because we are still processing large array of data in CPU. In this milestone, we will focus on improving performance in two areas: dynamic filtering and grid aggregation.

To resolve rendering lags caused by expensive CPU computation during dynamic filtering, we will move some of the filtering logic to GPU, using deck.gl’s customized filtering shader module. To improve aggregation speed, we will also integrate with deck.gl’s GPU grid aggregation layer.

• GPU data filtering integration | RFC |

  CPU filtering causes lagging on large datasets, especially during time playback. We will move dynamic data filtering logic into the GPU side, using customized shader modules by deck.gl (data-filter-rfc). This will not apply to all filters, only the filters that require constant updates: time filter and range slider.

• GPU aggregation layer integration | RFC |

  

  • Work with the deck.gl team to add GPU accelerated aggregation [rfc: GPU Aggregation enhancements] in grid layer.

  • GPU Aggregation can only perform MIN, MAX, SUM and MEAN calculation. For other aggregation modes - Median and Mode we will fall back to CPU aggregation.

  • We also need to fall back to CPU aggregation for color domain calculation, because it is not possible to access GPU aggregation result from javascript.

  • We propose to use web worker to execute CPU aggregation in the background, avoid UI lagging, and notify the UI of its progress.

  • We also need a switch between GPU aggregation and CPU aggregation based on layer configuration.

Easy Sharing and Embedding

The top category of requests from our users is around the ability to easy share and embed their map. People also want to export the animation as a gif or video. Most of our users are data scientist and analyst, they want to be able to user kepler.gl inside their most familiar set of toolings such as tableau and jupyter notebook.

For this milestone, we aim to add more functions around exporting and sharing through a range of channels. We will also branch out to other popular data science tools - tableau and juypter, and provide kepler.gl plugins.

• Better image export | RFC |

  • Fix image export silently fail

  • Allow exporting bigger resolution images

• Save / share map via URL | RFC |

  One of the top requests from our users is the ability to share their maps with a simple URL. Sharing should a simple feature if we have a backend to store people’s data. However, kepler.gl is an open source project, and its demo app is a client-side only application where data stays in the browser. In fact, many users think it is crucial that kepler.gl doesn’t store any data for privacy reasons. In order to allow users to share their map / data with others, while at the same time staying away from using our own data storage, we will use Dropbox as a 3rd-party data storage, and store map data / config on users private Dropbox account. Users who opt-in to share their maps with others will need a Dropbox account and manage who can access their maps by sharing the URL with others.

  • Load map and config from 3rd party data storage (dropbox)

  • Save config and data to a 3rd party data storage (dropbox)

  • Save map to a URL with params of data / config location

  • Share map via URL

• Load data directly from an URL | RFC |

  This feature will enable a user to add data to kepler.gl by entering a remote URL linking to raw data, bypassing the 250mb file upload limit. The URL can either contain data in json or csv format or contain the exported kepler.gl config json. A user can also add to kepler.gl/#/demo a?mapUrl= query string, kepler.gl will automatically load data upon loading. This will allow user to share their map with others by uploading a kepler.gl config json to gist or any other domain that CORS policy is defined.

  • Allow adding data from a provided URL in the UI.

  • Allow automatically load data with ?mapUrl= query string.

• Publish kepler.gl to cdn | RFC |

  Publish a kepler.gl bundle to cdn and automatically upgrade with every deploy. This will allow creating examples in codepen, exporting static map without inlining all source code.

  • Add build script to create a umd bundle

• kepler.gl for jupyter notebook | RFC |

  

  • Publish a kepler.gl for Jupyter python package to load kepler.gl inside Jupyter notebook.

• Embed an interactive map as HTML | RFC |

  • In the demo app, we will allow export the current map as a single HTML with links to js in CDN. User has options to either hydrate data and config into the sample or export data/config as a separate JSON file.

• Share directly to social media |

  • Save map as a URL

  • Create a snapshot of the map

  • Compose a tweet using twitter API

• Export map as a video |

  • While time playback is enabled, allow exporting the animation as video sequences. They can be imported into AfterEffects or use FFmpeg to output videos.

• Export map as a gif |

  • While time playback is enabled, allow exporting the animation as a gif

Documentation

We will publish our roadmap and contribution guidelines, add API documentation and a build system to automatically generate the documentation website.

• Contribution Guidelines for GitHub community |

  • Publish contribution.md on Github

• Publish public roadmap |

• Add Q&A and best practices |

• Add API Documentations | RFC |

  • Write API Documentations on how to use the framework

  • Use Documentation.js to automatically generate

Milestone 2. Advancing Geo-analytics

Geo Operations

Many users compare kepler.gl to Carto or ArcGIS. Both apps offer extensive geospatial data operations such as buffering, clipping, aggregating by Geo. For this milestone, we will start adding basic geo operations to kepler.gl including free draw, filter by geometry, connect points to create path and aggregation by geo polygon.

• Allow drawing on map to create paths and polygons |

  

  • Use nebula.gl to allow a user to freely draw on the map and create polygons and paths. New geometries will be added to a geometry filter.

  • Add UI to guide drawing process (Need UI/UX design). Confirm closing polygon, and edit polygon waypoints.

  • In the drawing UI, allow a user to select layers to be filtered by the new geometry

• Allow filtering by geo boundaries |

  • Allow an option to ‘filter points by polygon’ in point layer, select polygons inside a Geojson layer or a polygon drew by the user to filter by.

  • Use turf.js to filter points inside or outside of the selected polygon.

  • Allow an option to filter arcs/lines by whether it begins or end inside the polygon.

• Able to highlight an area |

  • After a user draws a polygon on the map, allow selection of layers to highlight features contained by that polygon.

• Allow connecting points based on sequence |

  • Allow selection of a unique identifier (uuid, name, etc.) and a sequence indicator (timestamp, index) to join points into a collection of paths.

  • Create a new layer with the created paths

• Aggregate points by geo-boundaries |

  

  Besides grid and hexbin, another most used aggregation style is by geo-boundaries such as country, state and county boundary. Adding to our existing aggregation layer, we will create a new polygon aggregation layer that aggregates points by polygon, it will provide a similar set of aggregation options the grid/hex layer provides.

  • Allow selection of point columns and polygon dataset to aggregating points by polygons.

  • The aggregation count will be appended to the polygon datasets

Advanced Layers

Besides layers with basic primitives, we are invested in bringing more advanced geo-visualization layer types into kepler.gl including edge bundling, arrows, and traffic animation. Each of which is proven to be effective in visualizing origin-destination tuples, movement flows and usually takes days of coding by an expert visualization engineer.

• H3 aggregation in hexbin layer |

  • Allow h3 aggregation in hexbin layer

  • Add hex_id to each hexagon

  • Allow download aggregation results

• Add edge bundling layer |

  Arcs and lines are important visual primitives to encode origin-destination visualizations. However, when data gets too large, these visualizations can become unreadable, hiding away underlying patterns. Edge-bundling has been used to reduce visual clutter. To enhance the arc and line layer, we will use the Multilevel Agglomerative Edge Bundling method presented in this paper. Based on the javascript implementation in mingle.

  

  • Add an edge bundling layer to bundle lines together

  • Add curve and delta slider to control bundle results

• Add Arrow layer |

  Add arrows to indicate directions. Arrow can be added to arc and line layer or as a separate layer type.

  

  • Option to display arrow at the target of the arc and line, with a list of arrow shape for selection and slider to adjust arrow size.

  • Add a arow layer which takes origin lat, origin lng and azimuth as inputs for its primitives.

• Add traffic animation layer |

  Deck.gl trip layer is probably the most popular layer so far. To support trip animation in kepler.gl, we will add a trip layer into it and allow customize animation.

  

  • Add trip animation layer to animate paths from a sequence of points

  • Need to specify latitude, longitude, group by (id, uuid) and the timestamp column

  • Add a slider to adjust the tail length, add color by option

  • Automatically display a time playback slider when a trip layer is created

Milestone 3. Improving Usability

We will improve the usability of layers, filters, and interactions based on frequently requested features by our user, such as customize color scale, filter per layer and more freedom in brushing interaction. We will also allow adding customize tile sources to the base map.

Layer

• Allow layer annotations |

  • Allow adding annotations to polygon layer, at the centroid

  • Allow marking freely on a layer to add annotations

• Customize color scale and palette |

  

  • Allow inputting value & range for each assigned color

  • Add log and Jenkins natural break scale

  • Allow inputting custom color palette

• Allow opacity by value |

• Allow duplicating a layer |

Filter

• Allow playback on other filter types |

  • Add playback on range slider

• Allow filter per layer |

  • Allow applying filters to all layers or selected layers

  • Allow applying filters to each split map panel

Interaction

• More tooltip settings |

  • Allow customize tooltip order

  • Allow customize tooltip based on layer

• More Brushing settings |

  • Allow select brush of origin or / target

  • Allow selection of layers for brushing

Base Map

• Allow loading base map tiles from a custom tile server |

  • Allow a user to upload a Mapbox style J (with links to custom tile server)

  • Dynamically request tile-based data

Framework

• Export data util functions |

• Allow UI customization |

  • Allow config ui-state to customize the UI (e.g. hiding certain panels)

• Provide backend connector |

Milestone 4. Data Integration

Kepler.gl’s drag and drop CSV/JSON upload is easy and intuitive, however, users often have to struggle with the 250 MB upload limit while process and convert their data into the 2 supported format. A commonly performed data processing is to join a Geojson file containing geometry data with a CSV file containing metric data for each geometry. City planners and GIS specialists work with Shapefiles most of the time instead of Geojson. They often need to write scripts to run location services such as Google API or OSRM to perform geocoding or routing. As a result, in order to further eliminate data processing steps before uploading to kepler.gl, we will add functions to perform spatial joining, geocoding and routing. More file format - shapefile, KML, and vector tiles will be allowed to upload to kepler.gl, and a user can edit inferred data types after uploading data to kepler.gl.

File Upload

• Increase upload dataset limit |

  • Upload file larger than 250 MB using the binary buffer

  • Re-construct string or JSON from the binary buffer

• Support shapefile |

  • Allow drag & drop shapefiles (multiple file types)

  • Config shapefile into kepler.gl data format

• Support vector tile |

  • Allow drag & drop vector tile

  • Config vector tile into kepler.gl data format

• Support KML |

  • Allow drag & drop KML

  • Config KML into kepler.gl data format

Data Manipulation

• Allow spatial joining |

  • Add joining algorithm based on different joining types (left, right, inner, outer)

  • Add UI to select features and unique columns to be joined on

• Allow editing data types |

  • Add UI to the data table to allow re-assign data type

  • Process data based on the new type, display error when failed

Location Data Integration

• Load spatial data using 3rd party API |

  • Geocoding with Google API

  • Routing with OSRM

• Read Movement data |

Backlogs

File upload

• Upload a template then upload data

• Automatically assign visual channel field to reserved column names

Advanced Layers

• H3 aggregation in hexbin layer

• Contour layer

Filters

• Time filter playback based on the time span