Memory leak with bokeh serve and datashader

[jgkatz]

I believe I am encountering a memory leak when using datashader in combination with a 'bokeh serve' style application.

Here is a simplified case that will reproduce the issue:

import datashader as ds
import numpy as np
import holoviews as hv
from itertools import cycle
from holoviews.operation.datashader import datashade
from bokeh.plotting import curdoc
import logging

log = logging.getLogger(__name__)

def setup_doc():
    log.info('Loading module.')
    cat = cycle(range(22))

    # Generate a bunch of data
    module_data = [
        {
            cat.next(): hv.Curve((np.linspace(0, 5000, 500000), np.random.normal(np.ones(500000))))
            for i in range(514)
        }
    ]

    # Use datashader and create a layout
    hv_layout = []
    for item in module_data:
        hv_layout.append(
            datashade(hv.NdOverlay(item, kdims='k'), aggregator=ds.count_cat('k'))
            .opts(plot=dict(width=800))
        )
    final_layout = hv.Layout(hv_layout).cols(1)
    log.info('Done loading module.')

    # Create plot
    plot = hv.renderer('bokeh').instance(mode='server').get_plot(final_layout)
    doc = curdoc()
    doc.add_root(plot.state)

setup_doc()

Add this to a folder test and run:

mprof run bokeh serve test

then try opening up a tab or two. Close the tabs and wait a minute or so for the sessions to be destroyed.

mprof plot

Afterward it appears that memory is not reclaimed.

Though it is hard to do with non-datashaded lines if you run the same experiment (with fewer datapoints) and a normal overlay, it appears the memory is reclaimed. I tried to dig into the persistent objects in the on_session_destroyed hook, and it seems at least the datashader object is not garbage collected when the session is destroyed. Not sure what else might be hanging around.

Am I missing something in the implementation?

[philippjfr]

Apologies I never responded here. I haven't investigated but I have a few thoughts on what might be hanging around, stream callbacks being the most likely candidate. I think this is a priority to fix before 1.10.0.

[jgkatz]

Great to hear. Looking forward to a potential fix.

[popher]

Just following up on this, I am having the same problems.
I'm using bokeh server to serve an app which uses HoloViews, Pandas, Bokeh and Datashader. It loads a ~100MB file (x,y data) to plot. However, the memory cost (~100MB per session) does not appear to be reclaimed after session end and very quickly the server runs out of memory.

I made a mre which has the same bug (but smaller footprint as I generate random data rather than read in such a large amount of data).

import numpy as np
import pandas as pd

# Generate some data
mz = np.linspace(100,1000,int(1e6))
I = np.random.randint(0,int(1e6),int(1e6))
df = pd.DataFrame(columns=['mz','I'])
df['mz'] = mz
df['I'] = I
df.head()

# import plotting libraries
import matplotlib as mpl
mpl.use('Agg') # otherwise pyplot throws a 'no qt backend' error
import matplotlib.pyplot as plt

import holoviews as hv
import hvplot.pandas
import datashader as ds
from holoviews.operation.datashader import datashade, dynspread

# specify parameters
hv.extension('bokeh', 'matplotlib', width="100")
dynspread.max_px=1
dynspread.threshold=0.25

# generate holoviews plots
msfull = df.hvplot(x='mz',y='I')
msshow = dynspread(datashade(msfull,cmap=['blue']).opts(plot=dict(width=1024, height=800)))

# add to bokeh document for serving
doc = hv.renderer('bokeh').server_doc(msshow)
doc.title = 'HoloViews Bokeh App'

Served with bokeh serve --show myapp.py
Same issue running on Windows 10 or Ubuntu 18.04 LTS, Python 3.6.6 or Python 3.7. All other packages required are up-to-date.

[philippjfr]

Thank you, I recently merged a change into bokeh that will let us clean up after a session is killed more easily. I'll use your example to check whether that is enough to address these issues.

[philippjfr]

Thank you both for your examples, after a marathon debugging session I've finally addressed the issue.

[afzalmushtaque]

I am facing the same memory leak problem with panel serve using holoviews with datashading. Should I open a new issue for it?

[ceball]

Sure, but please include detailed information about your environment (versions of packages, how installed, platform, etc) and how to reproduce (e.g. one of the scripts above).

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.