resolve deprecation warning about serialize_for_cli()

[jameslamb]

In recent versions of dask and distributed, some utility functions were moved from distributed to dask.

• Deprecate utilities which have moved to dask distributed#4966
• Upstream config serialization and inheritance dask#6987

For example, if you use dask=2021.7.2, distributed=20121.7.2, and dask-cloudprovider=2021.6.0, you will see the following warning.

.../python3.8/site-packages/dask_cloudprovider/generic/vmcluster.py:14: FutureWarning: serialize_for_cli is deprecated and will be removed in a future release. Please use dask.config.serialize instead.
from distributed.utils import warn_on_duration, serialize_for_cli, cli_keywords

This can be reproduced with the following (run from anywhere except the root of this repo), for example:

pip install --upgrade \
    'dask==2021.7.2' \
    'distributed==2021.7.2' \
    'dask-cloudprovider==2021.6.0'

python -c "import dask_cloudprovider.generic.vmcluster"

This PR proposes a change to suppress that warning and to protect future releases of dask-cloudprovider from being broken when serialize_for_cli() is eventually removed from distributed.

How I tested this

I used FargateCluster with dask=2021.7.2 and distributed=2021.7.2 to test these changes. Confirmed that after this change, the following code does not raise the deprecation warning, that it creates the cluster successfully, and that a small task like creating and taking the mean of a Dask array is successfully run out on the cluster.

import os
import dask.array as da
from dask.distributed import Client
from dask_cloudprovider.aws import FargateCluster

os.environ["AWS_DEFAULT_REGION"] = "us-west-2"
n_workers = 3
cluster = FargateCluster(
    image="daskdev/dask:2021.7.2",
    worker_cpu=512,
    worker_mem=4096,
    n_workers=n_workers,
    fargate_use_private_ip=False,
    scheduler_timeout="40 minutes",
    find_address_timeout=60 * 10,
)
client = Client(cluster)
client.wait_for_workers(n_workers)

print(f"View dashboard: {cluster.dashboard_link}")

X = da.random.random((1e6, 10), chunks=(1e3, 10))
X.mean().compute()

I also ran git grep -E "distributed.*utils" to check for other things that have been moved, annd didn't see any other that needed to be addressed.

Notes for Reviewers

I'm proposing introduce this _compat layer with conditional imports so that dask-cloudprovider doesn't have to bump its version floor on distributed or dask.

Thanks for your time and consideration.

[jacobtomlinson]

Thanks for working on this @jameslamb.

We frequently bump the dependency floor here anyway as enhancements often require upstream changes. So I have no problem doing that here. I appreciate you trying to maintain backward compatibility but I don't feel that the increased indirection is worth it.

[jameslamb]

I don't feel that the increased indirection is worth it.

alright no problem. Since #6987 was merged on December 18, 2020, I guess the existing floor of dask>=2021.1.0 doesn't have to be changed anyway!

dask-cloudprovider/requirements.txt

Lines 2 to 3 in 9cc73a0

	dask>=2021.01.1
	distributed>=2021.01.1

https://pypi.org/project/dask/#history

Updated in 208b43b.

[jacobtomlinson]

Thanks @jameslamb!

[dangercrow]

What's the release timeline looking like for this PR?
I was glad to see someone had fixed this as the error was starting to bug me, too!
However, I didn't see any new version of dask-cloudprovider that included this commit :(

[jacobtomlinson]

I'll likely do a release at the start of September. For now you can install from source.