Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dashboard fixes #100

Merged
merged 23 commits into from
Mar 25, 2016
Merged

Dashboard fixes #100

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
f078a28
Added support for pure counts (no corresponding field). Fixed outdate…
jbednar Mar 3, 2016
749d600
Added support for multiple filetypes
jbednar Mar 4, 2016
5cea597
Added data file path to startup message
jbednar Mar 4, 2016
4a39b97
Initial version of census.yml; works for counts
jbednar Mar 4, 2016
f24e96e
moved location of colormap to nest under summary_field
Mar 10, 2016
26bbfaf
added checks for categorical fields and refactored aggregate type menu
Mar 10, 2016
5b5db6f
removed pdb
Mar 10, 2016
5b24f06
added hover support
Mar 11, 2016
ee1bc4b
added mean aggregate downsample for hover layer
Mar 17, 2016
a0f3cbc
moved slider start value back to 4
Mar 17, 2016
75cbd86
Added description of census data
jbednar Mar 19, 2016
70c8260
Added outofcore option and port option
jbednar Mar 19, 2016
a7a77f3
fixed merge conflicts
Mar 21, 2016
740edd6
added cmap back after merge
Mar 21, 2016
3d1f9a0
added colorbar for numeric data
Mar 21, 2016
d82a076
added eq_hist, spread slider, and colorbars
Mar 24, 2016
3a9bc99
resized plot to better fit 15" monitor
Mar 24, 2016
c72e2a0
added log x_axis_type for non-linear transfer_functions
Mar 24, 2016
7ccdab3
adjusted padding on ordinal colorbar, added older-style categorical l…
Mar 25, 2016
ec16992
fixed logscale colorbar
Mar 25, 2016
92a6841
switched out diverging colorramps for sequential
Mar 25, 2016
602b10e
removed pdb
Mar 25, 2016
0ef5ec3
small import cleanup
Mar 25, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions examples/dashboard/census.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---

file: ../data/census.castra

initial_extent:
is_geo: true
xmin: -13914936
ymin: 2632019
xmax: -7235767
ymax: 6446276

axes:
- name: US Census Synthetic people
xaxis: meterswest
yaxis: metersnorth

summary_fields:

- name: Race
field: race
cat_colors:
w: blue
b: green
a: red
h: orange
o: saddlebrown

- name: Counts
field: None
188 changes: 158 additions & 30 deletions examples/dashboard/dashboard.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,30 +5,32 @@
import yaml
import webbrowser
import uuid
import math

from collections import OrderedDict

import datashader as ds
import datashader.transfer_functions as tf
import pandas as pd
import numpy as np
import pdb

from bokeh.server.server import Server
from bokeh.application import Application
from bokeh.application.handlers import FunctionHandler

from bokeh.plotting import Figure
from bokeh.plotting import Figure
from bokeh.models import (Range1d, ImageSource, WMTSTileSource, TileRenderer,
DynamicImageRenderer, HBox, VBox)

from bokeh.models import Select, Slider, CheckboxGroup
from bokeh.models import Select, Slider, CheckboxGroup, CustomJS, ColumnDataSource, Square, HoverTool

from tornado.ioloop import IOLoop
from tornado.web import RequestHandler

from webargs import fields
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How should we handle the dashboard's dependence on webargs? Should we just state that in the README? I don't think we'd necessarily want to make webargs a dependency of datashader, at least not until conda supports optional dependencies.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just an example, so my opinion is that the dependencies don't matter as long as they're listed explicitly somewhere.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do need to make it easy for people to run the example, though. After getting an error message about it, I briefly looked for webargs on conda, tried some bogus versions from non-main channels that didn't work, and eventually pip-installed webargs (which worked). Most people probably aren't that dedicated.

from webargs.tornadoparser import use_args


# http request arguments for datashing HTTP request
ds_args = {
'width': fields.Int(missing=800),
Expand All @@ -51,19 +53,66 @@ def get(self, args):
plot_height=args['height'],
x_range=(xmin, xmax),
y_range=(ymin, ymax))
agg = cvs.points(self.model.df,
self.model.active_axes[1],
self.model.active_axes[2],
self.model.aggregate_function(self.model.field))
pix = tf.interpolate(agg, (255, 204, 204), 'red',
how=self.model.transfer_function)

hover_cvs = ds.Canvas(plot_width=math.ceil(args['width'] / self.model.hover_size),
plot_height=math.ceil(args['height'] / self.model.hover_size),
x_range=(xmin, xmax),
y_range=(ymin, ymax))

# handle categorical field
if self.model.field in self.model.categorical_fields:
agg = cvs.points(self.model.df,
self.model.active_axes[1],
self.model.active_axes[2],
ds.count_cat(self.model.field))

hover_agg = hover_cvs.points(self.model.df,
self.model.active_axes[1],
self.model.active_axes[2],
ds.count_cat(self.model.field))

pix = tf.colorize(agg, self.model.colormap,
how=self.model.transfer_function)

# handle ordinal field
elif self.model.field in self.model.ordinal_fields:
agg = cvs.points(self.model.df,
self.model.active_axes[1],
self.model.active_axes[2],
self.model.aggregate_function(self.model.field))

hover_agg = hover_cvs.points(self.model.df,
self.model.active_axes[1],
self.model.active_axes[2],
self.model.aggregate_function(self.model.field))

pix = tf.interpolate(agg, (255, 204, 204), 'red',
how=self.model.transfer_function)
# handle no field
else:
agg = cvs.points(self.model.df,
self.model.active_axes[1],
self.model.active_axes[2])

hover_agg = hover_cvs.points(self.model.df,
self.model.active_axes[1],
self.model.active_axes[2])
pix = tf.interpolate(agg, (255, 204, 204), 'red',
how=self.model.transfer_function)

def hover_callback():
xs, ys = np.meshgrid(hover_agg.x_axis.values,
hover_agg.y_axis.values)
self.model.hover_source.data['x'] = xs.flatten()
self.model.hover_source.data['y'] = ys.flatten()
self.model.hover_source.data['value'] = hover_agg.values.flatten()

server.get_sessions('/')[0].with_document_locked(hover_callback)
# serialize to image
img_io = pix.to_bytesio()
self.write(img_io.getvalue())
self.set_header("Content-type", "image/png")


class AppState(object):
"""Simple value object to hold app state"""

Expand Down Expand Up @@ -117,6 +166,10 @@ def __init__(self, config_file, app_port=5000):
# set defaults
self.load_datasets()

# hover
self.hover_source = ColumnDataSource(data=dict(x=[], y=[], val=[]))
self.hover_size = 8

def load_config_file(self, config_path):
'''load and parse yaml config file'''

Expand All @@ -141,28 +194,55 @@ def load_config_file(self, config_path):

# parse summary field
self.fields = OrderedDict()
self.colormaps = OrderedDict()
self.ordinal_fields = []
self.categorical_fields = []
for f in self.config['summary_fields']:
self.fields[f['name']] = f['field']
self.fields[f['name']] = None if f['field'] == 'None' else f['field']

if 'cat_colors' in f.keys():
self.colormaps[f['name']] = f['cat_colors']
self.categorical_fields.append(f['field'])

elif f['field'] != 'None':
self.ordinal_fields.append(f['field'])

self.field = list(self.fields.values())[0]

if self.colormaps:
self.colormap = self.colormaps[list(self.fields.keys())[0]]

def load_datasets(self):
print('Loading Data...')
taxi_path = self.config['file']
data_path = self.config['file']
print('Loading Data from {}...'.format(data_path))

if not path.isabs(taxi_path):
if not path.isabs(data_path):
config_dir = path.split(self.config_path)[0]
taxi_path = path.join(config_dir, taxi_path)
data_path = path.join(config_dir, data_path)

if not path.exists(taxi_path):
raise IOError('Unable to find input dataset: "{}"'.format(taxi_path))
if not path.exists(data_path):
raise IOError('Unable to find input dataset: "{}"'.format(data_path))

axes_fields = []
for f in self.axes.values():
axes_fields += [f[1], f[2]]

load_fields = list(self.fields.values()) + axes_fields
self.df = pd.read_csv(taxi_path, usecols=load_fields)
load_fields = [f for f in self.fields.values() if f is not None] + axes_fields

if data_path.endswith(".csv") :
self.df = pd.read_csv(data_path, usecols=load_fields)

# parse categorical fields
for f in self.categorical_fields:
self.df[f] = self.df[f].astype('category')

elif data_path.endswith(".castra"):
import dask.dataframe as dd
self.df = dd.from_castra(data_path)
else:
raise IOError("Unknown data file type; .csv and .castra currently supported")


class AppView(object):

def __init__(self, app_model):
Expand All @@ -180,7 +260,7 @@ def create_layout(self):
self.fig = Figure(tools='wheel_zoom,pan', x_range=self.x_range,
y_range=self.y_range)
self.fig.plot_height = 560
self.fig.plot_width = 800
self.fig.plot_width = 810
self.fig.axis.visible = False

# add tiled basemap
Expand All @@ -193,28 +273,59 @@ def create_layout(self):
extra_url_vars=self.model.shader_url_vars)
self.image_renderer = DynamicImageRenderer(image_source=self.image_source)
self.fig.renderers.append(self.image_renderer)

# add label layer
self.label_source = WMTSTileSource(url=self.model.labels_url)
self.label_renderer = TileRenderer(tile_source=self.label_source)
self.fig.renderers.append(self.label_renderer)

self.invisible_square = Square(x='x',
y='y',
fill_color=None,
line_color=None,
size=self.model.hover_size)

self.visible_square = Square(x='x',
y='y',
fill_color='#79DCDE',
fill_alpha=.5,
line_color='#79DCDE',
line_alpha=1,
size=self.model.hover_size)

cr = self.fig.add_glyph(self.model.hover_source,
self.invisible_square,
selection_glyph=self.visible_square,
nonselection_glyph=self.invisible_square)

# Add a hover tool, that selects the circle
code = "source.set('selected', cb_data['index']);"
callback = CustomJS(args={'source': self.model.hover_source}, code=code)
self.hover_tool = HoverTool(tooltips=[(self.model.fields.keys()[0], "@value")], callback=callback, renderers=[cr], mode='mouse')
self.fig.add_tools(self.hover_tool)

# add ui components
controls = []
axes_select = Select.create(name='Axes',
options=self.model.axes)
axes_select.on_change('value', self.on_axes_change)
controls.append(axes_select)

field_select = Select.create(name='Field', options=self.model.fields)
field_select.on_change('value', self.on_field_change)
self.field_select = Select.create(name='Field', options=self.model.fields)
self.field_select.on_change('value', self.on_field_change)
controls.append(self.field_select)

aggregate_select = Select.create(name='Aggregate',
self.aggregate_select = Select.create(name='Aggregate',
options=self.model.aggregate_functions)
aggregate_select.on_change('value', self.on_aggregate_change)
self.aggregate_select.on_change('value', self.on_aggregate_change)
controls.append(self.aggregate_select)

transfer_select = Select.create(name='Transfer Function',
options=self.model.transfer_functions)
transfer_select.on_change('value', self.on_transfer_function_change)
controls.append(transfer_select)

# add map components
basemap_select = Select.create(name='Basemap', value='Toner',
options=self.model.basemaps)
basemap_select.on_change('value', self.on_basemap_change)
Expand All @@ -227,13 +338,14 @@ def create_layout(self):
end=100, step=1)
basemap_opacity_slider.on_change('value', self.on_basemap_opacity_slider_change)

hover_size_slider = Slider(title="Hover Size (px)", value=8, start=4,
end=30, step=1)
hover_size_slider.on_change('value', self.on_hover_size_change)
controls.append(hover_size_slider)

show_labels_chk = CheckboxGroup(labels=["Show Labels"], active=[0])
show_labels_chk.on_click(self.on_labels_change)


controls = [axes_select, field_select, aggregate_select,
transfer_select]

map_controls = [basemap_select, basemap_opacity_slider,
image_opacity_slider, show_labels_chk]

Expand All @@ -249,12 +361,28 @@ def update_image(self):

def on_field_change(self, attr, old, new):
self.model.field = self.model.fields[new]
self.hover_tool.tooltips = [(new, '@value')]
self.update_image()

if not self.model.field:
self.aggregate_select.options = [dict(name="No Aggregates Available", value="")]

elif self.model.field in self.model.categorical_fields:
self.aggregate_select.options = [dict(name="Categorical", value="count_cat")]
else:
opts = [dict(name=k, value=k) for k in self.model.aggregate_functions.keys()]
self.aggregate_select.options = opts

def on_basemap_change(self, attr, old, new):
self.model.basemap = self.model.basemaps[new]
self.tile_renderer.tile_source = WMTSTileSource(url=self.model.basemap)

def on_hover_size_change(self, attr, old, new):
self.model.hover_size = int(new)
self.invisible_square.size = int(new)
self.visible_square.size = int(new)
self.update_image()

def on_axes_change(self, attr, old, new):
self.model.active_axes = self.model.axes[new]
self.update_image()
Expand All @@ -278,7 +406,7 @@ def on_labels_change(self, new):

if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--config', help='yaml config file (e.g. nyc_taxi.yml)', required=True)
parser.add_argument('-c', '--config', help='yaml config file (e.g. nyc_taxi.yml)', required=True)
args = vars(parser.parse_args())

APP_PORT = 5000
Expand Down
3 changes: 3 additions & 0 deletions examples/dashboard/nyc_taxi.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ axes:
yaxis: dropoff_y

summary_fields:
- name: Trip Count
field: None

- name: Passenger Count
field: passenger_count

Expand Down