Dask lab extension unable to launch new clusters (2019.05.19) #48
Comments
|
@scottyhq Do you have access to the server logs? Do they have any interesting information in them? I'd still be wary about tornado 6... |
|
we've pinned to tornado 5.1.1. I see the following message in the log: |
|
Hmm, I'm not sure. @mrocklin does this look familiar? I wonder if it's a regression in |
|
Actually... i'm suspicious this is something to do with the a new version of repo2docker and some mixing of environments (#47). Because if I list the packages in the 'base' environment we have the following (including tornado 6.0.2): |
|
My first guess would be JupyterLab + Tornado 6 conflicts. I don't think that dask-kubernetes has changed a ton recently, but @jhamman might know more. |
|
|
|
Fair point |
|
@ian-r-rose - given multiple conda environments on a jupyterhub, which does dask labextension use by default? |
|
Whichever one is used to launch JupyterLab, I think. |
|
@scottyhq - can you try with a LocalCluster and see if that works? That will help determine if its in KubeCluster or not. |
|
Just to clarify, launching a KubeCluster programmatically works, and I can use the 'search' glass to find it and activate all buttons. It is the 'clusters +new' part that is non-responsive. If anyone wants to enter the hub and explore further, see the hub link in the first comment. import dask
from dask_kubernetes import KubeCluster
from dask.distributed import Client
from dask.distributed import wait, progress
cluster = KubeCluster(n_workers=2)
cluster |
|
Oh, yeah, launching clusters through the lab extension requires that they
support async/await. This is the whole reason for the async development
branch in dask kubernetes.
…On Tue, May 21, 2019, 6:42 PM Scott Henderson ***@***.***> wrote:
Just to clarify, launching a KubeCluster programmatically works, and I can
use the 'search' glass to find it and activate all buttons. It is the
'clusters +new' part that is non-responsive. If anyone wants to enter the
hub and explore further, see the hub link in the first comment.
import daskfrom dask_kubernetes import KubeClusterfrom dask.distributed import Clientfrom dask.distributed import wait, progress
cluster = KubeCluster(n_workers=2)
cluster
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#48?email_source=notifications&email_token=AACKZTD2ZY5RFH73PXD76UDPWSCEZA5CNFSM4HOPJQI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV5PUXI#issuecomment-494598749>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AACKZTCJQUTRWNF33G65MXTPWSCEZANCNFSM4HOPJQIQ>
.
|
|
Has that not been merged/published yet? |
No, its still sitting in the This binder does a few things:
Full specs here: https://github.com/jhamman/pangeo_test_binder/tree/dask_dev |
|
Is there anything that should be done to move forwards with an async-aware |
|
There are a couple things to do, yes. Mostly it needs to be used and bugs need to be found and fixed. I plan to write up the state of things and a few possible plans as an issue later this week. |
Merged
|
@mrocklin I should have time to push on async KubeCluster (not this week, but probably next week). Is that something you'd like me to take on? |
|
That would be very welcome. @quasiben has some work. Also, I can also
imagine a future in which we make a Worker-like class that manages a Pod,
and then just use SpecCluster around that class. I'd be happy to have a
chat any time about options here if you're interested.
…On Tue, Jun 4, 2019 at 1:56 PM Tom Augspurger ***@***.***> wrote:
@mrocklin <https://github.com/mrocklin> I should have time to push on
async KubeCluster (not this week, but probably next week). Is that
something you'd like me to take on?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#48?email_source=notifications&email_token=AACKZTAV4PXNWBSIPL3OUBDPY3JJTA5CNFSM4HOPJQI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW53IIQ#issuecomment-498840610>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AACKZTBTX3WSF6SSCHKYOP3PY3JJTANCNFSM4HOPJQIQ>
.
|
|
If it will take a significant amount of time, we can also back out the changes that make |
|
@TomAugspurger there are two PRs for async kube if you are interested:
I would suggest looking at @mrocklin 's before mine. |
|
noting that once upcoming async changes to dask-kubernetes are merged and released we should bump to dask-labextension > 1.0 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The most recent images did away with pinning most versions:
#42
Unfortunately in running these new images, the dask labextension is no longer able to launch new KubeClusters (you can select the latest image to run on this hub: https://nasa.pangeo.io)
Seeing messages such as these:
And here is a copy of the full conda environment installed:
pinging @ian-r-rose @jhamman
The text was updated successfully, but these errors were encountered: