Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dash-jobqueue #235

Open
BoPeng opened this issue Jul 12, 2019 · 2 comments
Open

Dash-jobqueue #235

BoPeng opened this issue Jul 12, 2019 · 2 comments

Comments

@BoPeng
Copy link
Contributor

BoPeng commented Jul 12, 2019

Work with cluster using a jupyterhub system on jupyterhub. Use code in notebook to create jobs.

Our approach is different in that it tries to separate detailed configuration from the notebook.

Our task is far less scalable than dark though.

Live at scipy2019.
@BoPeng
Copy link
Contributor Author

BoPeng commented Jul 12, 2019 •
edited
Loading

Their jos are essentially interactive whereas ours are mostly batch jobs.

End everything with cluster.close() and client.close().

@BoPeng BoPeng changed the title Dash-taskque Dash-jobqueue Jul 12, 2019
@BoPeng
Copy link
Contributor Author

BoPeng commented Jul 12, 2019

I asked the questions

  1. Does dask-jobqueue scale inside jupyterhub with a pre-allocated number of nodes? No.
  2. Then what if scale up but need to wait 30 minutes for your job to start? No solution, because HPC is not designed for interactive data analysis anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@BoPeng