distributed.utils - ERROR - 'distributed.scheduler.TaskGroup' object has no attribute '_states'

[n8henrie]

What happened: During a computation (in either JupyterLab or from standard python in a script), if I open http://127.0.0.1:8787/status I get 500: Internal Server Error in the browser and (immediately) a long traceback including the below from python / jupyter (truncated). NB: doesn't happen if I don't go to that URL.

distributed.utils - ERROR - 'distributed.scheduler.TaskGroup' object has no attribute '_states'
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/distributed/utils.py", line 659, in log_errors
    yield
  File "/usr/lib/python3.9/site-packages/distributed/dashboard/components/scheduler.py", line 1727, in update
    active_states = tp.active_states
  File "distributed/scheduler.py", line 638, in distributed.scheduler.TaskPrefix.active_states.__get__
AttributeError: 'distributed.scheduler.TaskGroup' object has no attribute '_states'
distributed.utils - ERROR - 'distributed.scheduler.TaskGroup' object has no attribute '_states'
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/distributed/utils.py", line 659, in log_errors
    yield
  File "/usr/lib/python3.9/site-packages/distributed/dashboard/components/scheduler.py", line 2103, in status_doc
    task_progress.update()
  File "/home/n8henrie/ml/.venv/lib/python3.9/site-packages/bokeh/core/property/validation.py", line 93, in func
    return input_function(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/distributed/dashboard/components/scheduler.py", line 1727, in update
    active_states = tp.active_states
  File "distributed/scheduler.py", line 638, in distributed.scheduler.TaskPrefix.active_states.__get__
AttributeError: 'distributed.scheduler.TaskGroup' object has no attribute '_states'
tornado.application - ERROR - Uncaught exception GET /status (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:8787', method='GET', uri='/status', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tornado/web.py", line 1704, in _execute
    result = await result
  File "/home/n8henrie/ml/.venv/lib/python3.9/site-packages/bokeh/server/views/doc_handler.py", line 52, in get
    session = await self.get_session()
  File "/home/n8henrie/ml/.venv/lib/python3.9/site-packages/bokeh/server/views/session_handler.py", line 120, in get_session
    session = await self.application_context.create_session_if_needed(session_id, self.request, token)
  File "/home/n8henrie/ml/.venv/lib/python3.9/site-packages/bokeh/server/contexts.py", line 218, in create_session_if_needed
    self._application.initialize_document(doc)
  File "/home/n8henrie/ml/.venv/lib/python3.9/site-packages/bokeh/application/application.py", line 171, in initialize_document
    h.modify_document(doc)
  File "/home/n8henrie/ml/.venv/lib/python3.9/site-packages/bokeh/application/handlers/function.py", line 132, in modify_document
    self._func(doc)
  File "/usr/lib/python3.9/site-packages/distributed/dashboard/components/scheduler.py", line 2103, in status_doc
    task_progress.update()
  File "/home/n8henrie/ml/.venv/lib/python3.9/site-packages/bokeh/core/property/validation.py", line 93, in func
    return input_function(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/distributed/dashboard/components/scheduler.py", line 1727, in update
    active_states = tp.active_states
  File "distributed/scheduler.py", line 638, in distributed.scheduler.TaskPrefix.active_states.__get__
AttributeError: 'distributed.scheduler.TaskGroup' object has no attribute '_states'
^C^CTraceback (most recent call last):
  File "/home/n8henrie/ml/testdask.py", line 14, in <module>
Process Dask Worker process (from Nanny):
Process Dask Worker process (from Nanny):
Process Dask Worker process (from Nanny):
Process Dask Worker process (from Nanny):
Process Dask Worker process (from Nanny):
Process Dask Worker process (from Nanny):

What you expected to happen: See the dashboard

Minimal Complete Verifiable Example:

import time

import dask.bag as db
from dask.distributed import Client


def foo(n):
    time.sleep(0.1)
    return n


if __name__ == "__main__":
    with Client():
        df = db.from_sequence(range(100_000)).map(foo).to_dataframe().compute()

    print(df)

While this is running, go to the status dashboard.

Environment:

Arch linux, using systemwide dask install in a venv with --system-site-packages. Firewall disabled.

Arch, 5.10.16-arch1-1
Python 3.9.1
$ uname -r
5.10.16-arch1-1
$ python3 --version
Python 3.9.1
$ pip freeze | grep -i -e dask -e distributed
dask==2020.12.0
distributed==2020.12.0
$ pacman -Q | grep -i -e dask -e distributed
python-dask 2020.12.0-1
python-distributed 2020.12.0-2

[jrbourbeau]

Thanks for raising an issue @n8henrie. I tried your example (thanks for providing a nice, minimal example) with the latest dask and distributed releases (2021.2.0) but wasn't able to reproduce the 500: Internal Server Error in the Dashboard. Could you try with the the latest release? Also, what version of bokeh are you using?

cc @jakirkham as distributed.utils - ERROR - 'distributed.scheduler.TaskGroup' object has no attribute '_states' may be related to the recent cython refactoring

[jakirkham]

This is probably something that got fixed in the last few releases. In particular I believe Ben ran into this previously and we fixed it with PR ( #4408 )

[n8henrie]

Thanks for the super quick responses.

Solution: pip install --upgrade --upgrade-strategy=eager dask[complete]

$ pip freeze | grep -i -e dask -e distributed
dask==2021.2.0
distributed==2021.2.0

The dashboard is now working great both on the MCVE and on my actual code. Many thanks.

[jrbourbeau]

Great, glad to hear it. Thanks for following up @n8henrie

[jakirkham]

I think the interesting thing here is that Arch Linux seems to be building python-distributed (their name for this package) with Cython