Support remote schedulers

[mrocklin]

This PR includes two major changes:

1. We now query the scheduler for its state, rather than assume that it is running locally
2. We make a bunch of things async, in order to deal with the communication inherent in 1

Things seem to work well for me locally, except that the dashboard proxy seems unhappy.

I'm testing with Dask's new SSHCluster, which is based on a new internal standard which everything should hopefully be based off of in the near future.

@ian-r-rose if you have time I could use your help with the proxy bits. My attempt at this was the last commit. I get errors like the following:

/Users/mrocklin/miniconda/envs/dev2/lib/python3.7/asyncio/events.py:88: RuntimeWarning: coroutine 'ProxyHandler.proxy' was never awaited

My guess is that Jupyter-server-proxy doesn't support async functions for ProxyHandler.proxy subclasses.

For testing I'm using the following config file, which currently depends on a few not-yet-merged PRs in dask.distributed.

labextension:
  factory:
     module: 'distributed.deploy.ssh2'
     class: 'SSHCluster'
     args: [["localhost", "localhost", "localhost", "localhost", "localhost", "localhost"]]
     kwargs:
       connect_kwargs:
         known_hosts: null
       scheduler_kwargs:
         port: 0
         dashboard_address: :0
       worker_kwargs:
         nthreads: 3
         memory_limit: 4 GiB

[mrocklin]

This may still need some things in master. I plan to release distributed next week. I'll try this again after that.

[ian-r-rose]

This looks reasonable to me. I'm not sure what the issue might be with the proxy handler, but I'd be happy to look at it once the required distributed has been released.

Perhaps this needs to be awaited now? https://github.com/dask/dask-labextension/pull/72/files#diff-9dd27938c6735fe022a08088466052c1R61

[mrocklin]

Perhaps this needs to be awaited now? https://github.com/dask/dask-labextension/pull/72/files#diff-9dd27938c6735fe022a08088466052c1R61

But the super implementation hasn't changed. I suspect that it is still synchronous.

[mrocklin]

I'm also finding another workaround to this issue, so this approach may become moot.

[mrocklin]

Potentially superceded by #73 (if we choose to adopt a convention internally to dask.distributed cluster managers.

[ian-r-rose]

Perhaps this needs to be awaited now? https://github.com/dask/dask-labextension/pull/72/files#diff-9dd27938c6735fe022a08088466052c1R61

But the super implementation hasn't changed. I suspect that it is still synchronous.

Weirdly, the super implementation is asynchronous. I'm not really sure what to make of that -- this is something that @yuvipanda knows much more about than I. I know he's out for a couple of weeks, but maybe we can ask him to weigh in when he's back.