Deploy job and start training at different times

[alx338]

Hi,

I would like to have custom code (BYOC) in my application, but the clients need to review the custom code before execution.
If I understand correctly, using the submit_job command in the moment the job is deployed (to server and clients), it is also executed.

Is there a way to separate the deployment of a job and the actual execution ? (so that in the meantime the custom code can be checked)
Or a possibility to start training using custom code that is already on the client side?

Thanks a lot for your help!

[YuanTingHsieh]

Hi @alx338 ,

Thanks for the interest!

Currently (2.2.1) NVFlare treats the submitted jobs as "REVIEWED/APPROVED" by the clients.
So when the job is submitted, it will be treated as ready to deploy (and resource will be checked against each client site to see if this job can be run)

However, I think we do have a plan to support reviewing/approval of jobs for each participants.
@chesterxgchen can you comment on this?

A hacky way to do this check right now is that, you can have your custom executor keep checking for a "file" on client site.
(for example, "approval.txt")
If this file exist then you proceed to train, if it does not exist you keep checking for like 3 minutes before the executor exits.
This means each user can have this 3 minutes to review the custom code.
And if they decide this is OK, they can generate the "approval.txt" file. (On ubuntu you can do touch approval.txt)

[parkeraddison]

Or a possibility to start training using custom code that is already on the client side?

I think you technically can do this by adding your custom Python package to the client's PYTHONPATH or by pip-installing the local package. Basically, if a Python process running on the client can successfully import your_package.module then you can use it in the code/config.

Seems like a small hassle but if you're distributing the custom package along with when you distribute the provisioned startup bundles it doesn't seem too bad.

[chesterxgchen]

This is exactly some of our clients ddid. As they don’t like custom code to be uploaded to their sites. But they are ok to pre install the code to be executed

[alx338]

Hi @parkeraddison ,
your approach would work well in our setting. thx!