Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[QST] Verify processing across cluster #624

Closed
aclifton314 opened this issue May 25, 2021 · 8 comments
Closed

[QST] Verify processing across cluster #624

aclifton314 opened this issue May 25, 2021 · 8 comments
Labels
inactive-30d inactive-90d

Comments

@aclifton314
Copy link

If I create a LocalCudaCluster and Client:

def setup_dask():
    cmd = "hostname --all-ip-addresses"
    process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
    output, error = process.communicate()
    IPADDR = str(output.decode()).split()[0]

    cluster = LocalCUDACluster(ip=IPADDR)
    client = Client(cluster)
    return cluster, client

and run the following:

gpu_cluster, dask_client = setup_dask()
data_path = '/path/to/data.csv'
data_dask_df = dask_cudf.read_csv(data_path, encoding='ISO-8859-1')

data_series = data_dask_df['lemmatized']
tmp_series = data_series.map_partitions(process_text, meta=data_series._meta).persist()

is there anything in LocalCudaCluster or Client that could be printed out to verify that the process_text method is being ran across the cluster?
@pentschev
Copy link
Member

If you want a programmatic way to do that, you can probably check whether you have keys in each Dask worker's data object:

In [1]: from dask_cuda import LocalCUDACluster

In [2]: from distributed import Client

In [3]: import dask_cudf

In [4]: cluster = LocalCUDACluster()

In [5]: client = Client(cluster)

In [6]: a = dask_cudf.read_csv("file.csv")

In [7]: client.run(lambda dask_worker: list(dask_worker.data.keys()))
Out[7]:
{'tcp://127.0.0.1:34767': [],
 'tcp://127.0.0.1:34881': [],
 'tcp://127.0.0.1:35237': [],
 'tcp://127.0.0.1:35289': [],
 'tcp://127.0.0.1:35507': [],
 'tcp://127.0.0.1:35597': [],
 'tcp://127.0.0.1:37289': [],
 'tcp://127.0.0.1:41885': []}

In [8]: a = a.persist()

In [9]: client.run(lambda dask_worker: list(dask_worker.data.keys()))
Out[9]:
{'tcp://127.0.0.1:34767': ["('read-csv-2cf9f603f6b16ad778eb7adf04c1fa24', 0)"],
 'tcp://127.0.0.1:34881': ["('read-csv-2cf9f603f6b16ad778eb7adf04c1fa24', 4)"],
 'tcp://127.0.0.1:35237': ["('read-csv-2cf9f603f6b16ad778eb7adf04c1fa24', 1)"],
 'tcp://127.0.0.1:35289': ["('read-csv-2cf9f603f6b16ad778eb7adf04c1fa24', 6)"],
 'tcp://127.0.0.1:35507': ["('read-csv-2cf9f603f6b16ad778eb7adf04c1fa24', 2)"],
 'tcp://127.0.0.1:35597': ["('read-csv-2cf9f603f6b16ad778eb7adf04c1fa24', 7)"],
 'tcp://127.0.0.1:37289': ["('read-csv-2cf9f603f6b16ad778eb7adf04c1fa24', 3)"],
 'tcp://127.0.0.1:41885': ["('read-csv-2cf9f603f6b16ad778eb7adf04c1fa24', 5)"]}

Otherwise, looking at Dask Distributed's dashboard will provide real-time visual information of the compute happening under-the-hood.

@jacobtomlinson
Copy link
Member

The Client object also has who_has and has_what methods which do something similar.

You may also be interested in dask/distributed#4853 where I've tried to add nice HTML reprs for use in Jupyter.

@aclifton314
Copy link
Author

@jacobtomlinson and @pentschev thank you very much for your response. I tried the following:

def process_text(text_series, process_method: str, batch_size=256):
    print(dask_client.who_has())
    print('-'*75)
    print(dask_client.has_what())
    if 'no_punct_lower' in process_method:
        docs = nlp.pipe(text_series.to_pandas(), batch_size=batch_size, disable=['parser', 'ner'])
        new_strs_list = []
        for doc in docs:
            proc_list = []
            for token in doc:
                if token.pos_ != 'PUNCT' and token.pos_ != 'SPACE':
                    proc_list.append(token.text.lower())
            new_strs_list.append(' '.join(proc_list))
        return cudf.Series(new_strs_list)
    else:
        return text_series

def setup_dask():
    cmd = "hostname --all-ip-addresses"
    process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
    output, error = process.communicate()
    IPADDR = str(output.decode()).split()[0]

    cluster = LocalCUDACluster(ip=IPADDR)
    client = Client(cluster)
    return cluster, client

print('setting up gpu_cluster and dask_client...')
gpu_cluster, dask_client = setup_dask()

print('reading data...')
data_path = '/path/to/data.csv'
data_dask_df = dask_cudf.read_csv(data_path, encoding='ISO-8859-1')

run_type = 'lemma_no_punct_lower'
data_series = data_dask_df['lemmatized']

tmp_series = data_series.map_partitions(process_text, run_type, meta=data_series._meta).persist()

but get the following error:

loading spacy model...
setting up gpu_cluster and dask_client...
reading data...
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/core.py", line 108, in loads
    frames[0], object_hook=_decode_default, use_list=False, **msgpack_opts
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/core.py", line 100, in _decode_default
    sub_header, sub_frames, deserializers=deserializers
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 455, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 389, in deserialize
    return loads(header, frames)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 169, in serialization_error_loads
    raise TypeError(msg)
TypeError: Could not serialize object of type tuple.
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 56, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _asyncio.Task objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 312, in serialize
    header, frames = dumps(x, context=context) if wants_context else dumps(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 55, in pickle_dumps
    protocol=context.get("pickle-protocol", None) if context else None,
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 60, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _asyncio.Task objects

distributed.core - ERROR - Could not serialize object of type tuple.
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 56, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _asyncio.Task objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 312, in serialize
    header, frames = dumps(x, context=context) if wants_context else dumps(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 55, in pickle_dumps
    protocol=context.get("pickle-protocol", None) if context else None,
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 60, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _asyncio.Task objects
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/core.py", line 555, in handle_stream
    msgs = await comm.read()
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/comm/tcp.py", line 222, in read
    allow_offload=self.allow_offload,
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/comm/utils.py", line 79, in from_frames
    res = _from_frames()
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/comm/utils.py", line 63, in _from_frames
    frames, deserialize=deserialize, deserializers=deserializers
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/core.py", line 108, in loads
    frames[0], object_hook=_decode_default, use_list=False, **msgpack_opts
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/core.py", line 100, in _decode_default
    sub_header, sub_frames, deserializers=deserializers
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 455, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 389, in deserialize
    return loads(header, frames)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 169, in serialization_error_loads
    raise TypeError(msg)
TypeError: Could not serialize object of type tuple.
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 56, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _asyncio.Task objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 312, in serialize
    header, frames = dumps(x, context=context) if wants_context else dumps(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 55, in pickle_dumps
    protocol=context.get("pickle-protocol", None) if context else None,
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 60, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _asyncio.Task objects

distributed.worker - ERROR - Could not serialize object of type tuple.
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 56, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _asyncio.Task objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 312, in serialize
    header, frames = dumps(x, context=context) if wants_context else dumps(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 55, in pickle_dumps
    protocol=context.get("pickle-protocol", None) if context else None,
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 60, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _asyncio.Task objects
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/worker.py", line 997, in handle_scheduler
    comm, every_cycle=[self.ensure_communicating, self.ensure_computing]
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/core.py", line 555, in handle_stream
    msgs = await comm.read()
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/comm/tcp.py", line 222, in read
    allow_offload=self.allow_offload,
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/comm/utils.py", line 79, in from_frames
    res = _from_frames()
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/comm/utils.py", line 63, in _from_frames
    frames, deserialize=deserialize, deserializers=deserializers
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/core.py", line 108, in loads
    frames[0], object_hook=_decode_default, use_list=False, **msgpack_opts
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/core.py", line 100, in _decode_default
    sub_header, sub_frames, deserializers=deserializers
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 455, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 389, in deserialize
    return loads(header, frames)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 169, in serialization_error_loads
    raise TypeError(msg)
TypeError: Could not serialize object of type tuple.
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 56, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _asyncio.Task objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 312, in serialize
    header, frames = dumps(x, context=context) if wants_context else dumps(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 55, in pickle_dumps
    protocol=context.get("pickle-protocol", None) if context else None,
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 60, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _asyncio.Task objects

tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x2aac510f5310>>, <Task finished coro=<Worker.handle_scheduler() done, defined at /home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/worker.py:994> exception=TypeError('Could not serialize object of type tuple.\nTraceback (most recent call last):\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 56, in dumps\n    result = cloudpickle.dumps(x, **dump_kwargs)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps\n    cp.dump(obj)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump\n    return Pickler.dump(self, obj)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump\n    self.save(obj)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple\n    save(element)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce\n    save(args)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple\n    save(element)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict\n    self._batch_setitems(obj.items())\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems\n    save(v)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple\n    save(element)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple\n    save(element)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list\n    self._batch_appends(obj)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends\n    save(x)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list\n    self._batch_appends(obj)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends\n    save(x)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function\n    *self._dynamic_function_reduce(obj), obj=obj\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5\n    save(state)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple\n    save(element)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict\n    self._batch_setitems(obj.items())\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems\n    save(v)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict\n    self._batch_setitems(obj.items())\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems\n    save(v)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce\n    save(state)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict\n    self._batch_setitems(obj.items())\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems\n    save(v)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list\n    self._batch_appends(obj)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends\n    save(tmp[0])\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save\n    rv = reduce(self.proto)\nTypeError: can\'t pickle _asyncio.Task objects\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 312, in serialize\n    header, frames = dumps(x, context=context) if wants_context else dumps(x)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 55, in pickle_dumps\n    protocol=context.get("pickle-protocol", None) if context else None,\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 60, in dumps\n    result = cloudpickle.dumps(x, **dump_kwargs)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps\n    cp.dump(obj)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump\n    return Pickler.dump(self, obj)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump\n    self.save(obj)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple\n    save(element)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce\n    save(args)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple\n    save(element)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict\n    self._batch_setitems(obj.items())\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems\n    save(v)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple\n    save(element)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple\n    save(element)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list\n    self._batch_appends(obj)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends\n    save(x)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list\n    self._batch_appends(obj)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends\n    save(x)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function\n    *self._dynamic_function_reduce(obj), obj=obj\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5\n    save(state)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple\n    save(element)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict\n    self._batch_setitems(obj.items())\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems\n    save(v)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict\n    self._batch_setitems(obj.items())\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems\n    save(v)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save\n    self.save_reduce(obj=obj, *rv)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce\n    save(state)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict\n    self._batch_setitems(obj.items())\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems\n    save(v)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save\n    f(self, obj) # Call unbound method with explicit self\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list\n    self._batch_appends(obj)\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends\n    save(tmp[0])\n  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save\n    rv = reduce(self.proto)\nTypeError: can\'t pickle _asyncio.Task objects\n')>)
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/worker.py", line 997, in handle_scheduler
    comm, every_cycle=[self.ensure_communicating, self.ensure_computing]
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/core.py", line 555, in handle_stream
    msgs = await comm.read()
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/comm/tcp.py", line 222, in read
    allow_offload=self.allow_offload,
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/comm/utils.py", line 79, in from_frames
    res = _from_frames()
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/comm/utils.py", line 63, in _from_frames
    frames, deserialize=deserialize, deserializers=deserializers
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/core.py", line 108, in loads
    frames[0], object_hook=_decode_default, use_list=False, **msgpack_opts
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/core.py", line 100, in _decode_default
    sub_header, sub_frames, deserializers=deserializers
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 455, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 389, in deserialize
    return loads(header, frames)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 169, in serialization_error_loads
    raise TypeError(msg)
TypeError: Could not serialize object of type tuple.
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 56, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _asyncio.Task objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 312, in serialize
    header, frames = dumps(x, context=context) if wants_context else dumps(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 55, in pickle_dumps
    protocol=context.get("pickle-protocol", None) if context else None,
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 60, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 687, in _save_reduce_pickle5
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/home/aclifton/anaconda3/envs/dask-cudf/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle _asyncio.Task objects

@pentschev
Copy link
Member

I'm not entirely sure, but I think it's related to dask/distributed#4853 (comment) , @jacobtomlinson is probably best to confirm this.

@jakirkham
Copy link
Member

We are reverting that change temporarily for the Dask release. So that should clear out (if that is the cause). Also can just checkout the previous commit of Distributed

@github-actions
Copy link

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

@github-actions
Copy link

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

@pentschev
Copy link
Member

As per #624 (comment) this should have been resolved. I'm closing but please reopen if still relevant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
inactive-30d inactive-90d
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jacobtomlinson @jakirkham @pentschev @aclifton314