Documentation for non-Ipython usage too sparse

[uahic]

Hello,

I would like to visualize a fixed set of heatmaps (envolving heatmap over time) and this works with the matplotlib backend but it takes an incredible time to write this as static HTML file (like 10 minutes (!!!) for just a few Images when choosing holomaps='gif' and fig='svg'). The produced output html file has 24MB which is large for a HTML but I've got 8GB RAM so ...

Then I've tried using the DynamicMap but I have no clue how to actually visualize this when not starting from withing an ipython notebook (I am not a big fan of them anyways). Is that even possible? Bokeh as backend would also be fine but - again - the time to assemble a static HTML including less than 100 pictures of default size is in the range of >10 minutes. What does take soo much time?

I like the idea of holoviews very much and so far it looks very clean but I highly dislike the documentation which doesnt allow to catch ideas very quickly, instead you have to read through all of the pages to get something done.

Thank you very much

[philippjfr]

Thanks for the feedback @uahic. I do agree with you on the documentation and we have plans to add a bunch of quickstart guides which will provide more concise introductions for specific topics rather than having these huge monolithic tutorials.

In terms of the speed of exporting it's not quite clear to me what's going on without seeing an example. I've exported large amounts of images, gifs and videos in the past without issue. If you could supply us with a small self-contained example, which highlights the issue that would be very helpful.

[jbednar]

Times in the range of 10 minutes are very unusual; I typically export even quite large HTML pages (more than 24MB) in under a minute. Definitely something to get to the bottom of!

Non-Jupyter usage is definitely something we need to focus on improving. Up until now, HoloViews has had financial support for specific components and features, but it has never had support for general tasks like documentation and integration. Luckily, if all goes well with some current negotiations, we will soon be able to have someone working full-time to help these big-picture issues come together more smoothly.

[uahic]

Thanks for the very fast response! Ill provide you the full script below which runs pretty fast except for renderer.save(...)

def import_renderer(backend, fig='svg', holomap='gif'):
    if backend == 'matplotlib':
        from matplotlib import pyplot
        # pyplot.switch_backend('agg')
        import holoviews.plotting.mpl
        renderer = Store.renderers['matplotlib'].instance(
            fig=fig, mode='mpld3', holomap=holomap)
    elif backend == 'bokeh':
        import holoviews.plotting.bokeh
        renderer = Store.renderers['bokeh'].instance()
    else:
        raise Exception('Backend not fount')

    return renderer


def read_voltage_data(file, n, m):
    names = ['id', 'time', 'V_m']
    with open(file, 'r') as f:
        # data = pd.read_csv(file, engine='python', sep=r"\s*",
        #                    parse_dates=['time'], date_parser=date_parser, names=names)
        data = pd.read_csv(file, engine='python', sep=r"\s*", names=names)
    data.sort_values('id')
    assert len(data) > 0, 'Multimeter record is empty'
    return data

def make_heatmap_images(data, n, m):
    grouped_data = data.groupby('time', as_index=False)
    # for time, dataframe in grouped_data:
    #     print dataframe.values[:, 2].ravel().reshape((n,m))
    # raise
    image_series = [((time), hv.Image(
        dataframe.values[:, 2].ravel().reshape((n, m)),
        group='heatmap',
        bounds=(0, 0, n, m),
        xdensity=1,
        ydensity=1)) for i, (time, dataframe) in enumerate(grouped_data)]
    return image_series

def show_heatmap(file,
                 n,
                 m,
                 fig='svg',
                 holomap='gif',
                 backend='matplotlib',
                 **renderer_args):
    renderer = import_renderer(backend=backend, fig=fig, holomap=holomap)
    voltage_data = read_voltage_data(file, n, m)
    heatmap_images = make_heatmap_images(voltage_data, n, m)
    # dynamic_callback, slider_values = make_dynamic_heatmap_images(voltage_data, n, m)
    # kdims = [hv.Dimension('time', values=slider_values)]

    RasterPlot = renderer.plotting_class(hv.Image)
    RasterPlot.colorbar = True
    RasterPlot.set_param(**renderer_args)
    obs_hmap = hv.HoloMap(heatmap_images)

    renderer.save(obs_hmap, 'heatmap')

[jbednar]

Thanks! Do you have sample data files available, so that we could run this?

It looks like you are using the mpld3 mode, which could explain things, as I generally avoid that due to it no longer being supported.

[uahic]

I tried it before without mpld3 but it was basically the same performance, hmh...

data.zip

try this data set with n = 10 , m = 10

[jbednar]

The following version of the code runs fine:

import holoviews as hv
from holoviews import Store
import pandas as pd

def import_renderer(backend, fig='svg', holomap='gif'):
    if backend == 'matplotlib':
        from matplotlib import pyplot
        import holoviews.plotting.mpl
        renderer = Store.renderers['matplotlib'].instance(
            fig=fig, holomap=holomap)
    elif backend == 'bokeh':
        import holoviews.plotting.bokeh
        renderer = Store.renderers['bokeh'].instance()
    else:
        raise Exception('Backend not fount')

    return renderer


def read_voltage_data(file, n, m):
    names = ['id', 'time', 'V_m']
    with open(file, 'r') as f:
        data = pd.read_csv(file, engine='python', sep=r"\s*", names=names)
    data.sort_values('id')
    assert len(data) > 0, 'Multimeter record is empty'
    return data

def make_heatmap_images(data, n, m):
    grouped_data = data.groupby('time', as_index=False)
    image_series = [((time), hv.Image(
        dataframe.values[:, 2].ravel().reshape((n, m)),
        group='heatmap',
        bounds=(0, 0, n, m),
        xdensity=1,
        ydensity=1)) for i, (time, dataframe) in enumerate(grouped_data)]
    return image_series

def show_heatmap(file,
                 n,
                 m,
                 fig='svg',
                 holomap='gif',
                 backend='matplotlib',
                 **renderer_args):
    renderer = import_renderer(backend=backend, fig=fig, holomap=holomap)
    voltage_data = read_voltage_data(file, n, m)
    heatmap_images = make_heatmap_images(voltage_data, n, m)

    RasterPlot = renderer.plotting_class(hv.Image)
    RasterPlot.colorbar = True
    RasterPlot.set_param(**renderer_args)
    obs_hmap = hv.HoloMap(heatmap_images)

    renderer.save(obs_hmap, 'heatmap')


if __name__ == "__main__":
        show_heatmap("voltages-117-0.copy.dat",10,10)

This takes 7 minutes and produces a 64MB HTML file:

$ time python data.py
405.217u 20.155s 7:06.11 99.8%	0+0k 8+21io 38pf+0w
$ ls -l heatmap.html 
-rw-rw-r--  1 jbednar  staff  68219895 Feb 15 07:39 heatmap.html

7 minutes is a long time, but there appear to be nearly 2000 images being rendered here, which works out to 0.2 seconds per frame. Could be faster, but isn't actually all that bad.

[uahic]

Great thanks!

Maybe I have to go with some kind of dynamic rendering solution then since the 2000 images correspond to just a few milliseconds of simulation time.

[jbednar]

Right -- definitely seems like a good case for using a dynamic solution. @philippjfr or @jlstevens may have suggestions for how to work well with a simulator, as they originally developed HoloViews for that purpose.

[uahic]

I've got a working solution with pure bokeh (basically just streaming numpy arrays to an image which is triggered by a CustomJS callback) but I really would like to use HoloView's nice features like overlays, coupling of plots and all that stuff. Could somebody provide a simple/basic example how to use DynamicMaps with a normal python interpreter (or IPython, but not inside a notebook cell). Thank you very much!

[philippjfr]

@uahic I've revised your example a little bit so it now also supports rendering widgets to an html file. Rendering a gif at 5 fps take 3.5 minutes for me and comes out at 20 MB, rendering a static html file with a bokeh plot + widgets takes 1.5 minutes, which is pretty good. Note that I've found a bug when exporting a static widgets which I'll fix in a PR shortly.

import holoviews as hv
from holoviews import Store
import pandas as pd

def import_renderer(backend, fps=5):
    if backend == 'matplotlib':
        import holoviews.plotting.mpl
    elif backend == 'bokeh':
        import holoviews.plotting.bokeh
    else:
        raise Exception('Backend not found')
    renderer = Store.renderers[backend].instance(fps=fps)
    return renderer

def read_voltage_data(file, n, m):
    names = ['id', 'time', 'V_m']
    with open(file, 'r') as f:
        data = pd.read_csv(file, engine='python', sep=r"\s*", names=names)
    data.sort_values('id')
    assert len(data) > 0, 'Multimeter record is empty'
    return data

def make_heatmap_images(data, n, m):
    grouped_data = data.groupby('time', as_index=False)
    image_series = [(time, hv.Raster(
        dataframe.values[:, 2].ravel().reshape((n, m))))
        for i, (time, dataframe) in enumerate(grouped_data)]
    return image_series

def show_heatmap(file,
                 n,
                 m,
                 backend='matplotlib',
                 widgets=False,
                 fps=5):
    renderer = import_renderer(backend=backend, fps=fps)
    voltage_data = read_voltage_data(file, n, m)
    heatmap_images = make_heatmap_images(voltage_data, n, m)
    obs_hmap = hv.HoloMap(heatmap_images, kdims=['Time'],
                          group='heatmap')

    if backend == 'bokeh' or widgets:
        with open('heatmap.html', 'w') as f:
            f.write(renderer.static_html(obs_hmap))
    elif backend == 'matplotlib':
        renderer.save(obs_hmap, 'heatmap', fmt='gif')

options = hv.Store.options(backend='matplotlib')
options.Raster = hv.Options('plot', colorbar=True)

options = hv.Store.options(backend='bokeh')
options.Raster = hv.Options('plot', colorbar=True, toolbar='above', width=425, height=400)

%timeit -r 1 -n 1 show_heatmap("./notebooks/voltages-117-0.copy.dat",10,10, backend='bokeh')

That said I think what you're looking for will be using bokeh server together with holoviews. Putting together a time scrubber like this will be really straightforward and you'll be able to start it simply with:

bokeh serve script.py

We've got a PR where that's already working along with a few examples here: #959

Edit: Here's what the static HTML output of the script above produces (once I've fixed a bug): http://philippjfr.com/ipynbs/heatmap.html

[jlstevens]

We should make some of this material into a user guide. Tentatively assigning to 1.8.1.