-
-
Notifications
You must be signed in to change notification settings - Fork 143
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Override _new_worker_name to make using Job Arrays easier #480
Conversation
That's an interesting idea indeed to make it slightly less convoluted to use job arrays with Dask-Jobqueue. Although it was too invasive at one point to make it at the SpecCluster (in Could you add some tests to make sure that the cluster name reaches the worker name? I think something close to the job array use case would be: cluster = SLURMCluster(name='test-$MY_ENV_VARIABLE', env_extra="MY_ENV_VARIABLE=my-env-variable-value", ...) You can then get the worker names [w['id'] for w in client.scheduler_info()['workers'].values()] and check that they look like "my-env-variable-value-" |
By the way about the CI failures:
|
CI:
I see this now, based on the CI, I'll have to fix:
Testing:As for the tests, okay, wasn't too sure of where/what to put. Thanks for the ideas! Just to be sure, testing code should be placed in test_jobqueue_core.py, right? I'll have to look at this over the coming weekend. |
I ended up trying to add a test, remembering the testing setup was not that trivial to get up and running (docker and docker-compose setup to mimick SLURM, SGE, etc ... clusters) and eventually pushing into your branch. Because this test needs Dask workers it needs a real cluster (i.e. you need to Let's see what the CI has to say about my last commit 🤞 |
Merging thanks a lot for this! |
Bit late, but it may be worth leaving the original "NOTE" in the docs? Unless the expectation is that folks will search prior to using job arrays. |
I think people will find it in the github issue / github discussion by googling with more context. I felt like it was a bit too much info hidden in a docstring of an obscure method ... |
If you thinks this could be helpful to other people, you might think of documenting the use of Job arrays in Slurm in the docs! |
This is based on what I've found to work on a SLURM cluster (https://openmind.mit.edu) and reported in dask/dask#7070 (comment) as an attempt to address those errors when submitting Job Arrays, using a variant of #196 (comment).