-
Notifications
You must be signed in to change notification settings - Fork 178
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for installing client sites on HPC systems #2595
Comments
Hi @dirkpetersen thanks for bringing this topic up! One workaround as you said is to run a NVFlare client process (client_train.py) directly in your SLURM cluster node. We do have several ways to achieve that in HPC cluster.
As we are seeing more interests, we might write out a whole tutorial/reference implementation on this later. |
Awesome, this is very helpful and I will try that ! |
@YuanTingHsieh, it seems that both options require that client_train.py will run on the HPC login node, I think that is a reasonable assumption at least for many life sciences HPC systems that tend to have beefy login nodes and tolerant HPC admins. There are other disciplines where login nodes are guarded more strictly and it may not be allowed to run an agent. For those it would be better to have client_train.py run on a system adjacent to HPC and then submit the job via ssh and sbatch (for example using paramiko) . Perhaps a lower priority right now as I understand that FL use cases are focusing on life sciences right now ? |
@dirkpetersen thanks for the discussion! Yes, as you said, if you have a mechanism to submit the job via ssh and sbatch from machine A to your HPC system. Note that, you can also start NVFlare client using the "start.sh" in our startup kits, as you can see we add some restarting mechanisms inside that script as well. |
Is your feature request related to a problem? Please describe.
Some organizations have all their GPUs allocated in HPC systems and find it difficult to allocate dedicated GPU servers to NVFlare. Currently the use on HPC systems is undocumented.
Describe the solution you'd like
In an ideal world, a client would be installed on a virtual machine which then submits jobs to an HPC system to prevent that the GPU is allocated for long periods of time without being used.
Describe alternatives you've considered
I currently use this workaround and describe some of the issues with running on HPC systems (Slurm in this case)
https://github.com/dirkpetersen/nvflare-cancer#install-a-client-on-hpc
The text was updated successfully, but these errors were encountered: