Local simulation instructions before deploying to the real world and questions thereof`

[AkshayBhardwajMN]

Hi all

We need to deploy a federated learning server (microservice) to collaborate with other institutions. However, our own compute environment is airtight (no access to the internet) due to regulatory requirements. Therefore, we need to develop a prototype solution outside of this environment, however, the prototype has to be more thorough than a simple simulation as listed on the FLARE site.

Please let me know if the following workflow is possible:

1. Go to WSL2 on a local machine
2. Create a docker container to host the NVFLARE server
3. Create either two docker containers or virtual environments (whichever works better) to simulate clients (2 external sites, 1 main site that is a participant but also the controller of this server)
4. The goal is to execute provisioning as you would in a real world setting (login, PKI etc), and to capture exactly what type of data is leaving the sites and entering the server, and vice versa e.g. contents of shareables, dxo's etc files
5. From an end user perspective (the data scientists who will be using the FLARE server from their respective computes); what will the control site need to create and share with them so they can convert their python codes to FL codes by simple availing the FLARE API (python) e.g. connect to the server using username/password/cryptographic key etc
6. How can we manage the server i.e. activity profiling, managing different projects (and making sure different projects are not able to look into another site's other projects that they're not a part of), determining model convergence, and retire/decommision client sites once they're no longer a part of an active project?
7. We're trying to decide between NFLARE & OpenFL, and would like a lean low overhead solution that meets our security architecture.

Thank you

[YuanTingHsieh]

@AkshayBhardwajMN thanks for the interest.

For your assumption/setup:
So you say "computing environment is airtight (no access to the internet)"
Do you mean you want to deploy an FL server that has no access to the internet?
There got to be some communication mechanism between your FL server and clients right?

For the workflow you suggest:
So we have FL server, FL clients and admin clients.
The FL server coordinates the training.
FL clients reside on the machines with data and do the training.
Admin clients can be used anywhere to submit_job/abort_job/check server status and check logs etc. (https://nvflare.readthedocs.io/en/stable/real_world_fl/operation.html)

You don't have to run admin client and FL client on the same machine.

For your point 5, there are multiple ways to go about it.
One way is that you just have your data scientists develop their application using Flare client API and hand it over to one Admin that has the right to submit a job to the whole FL system. This way you don't need them to have their own admin client package.
Another way is that when you provision, you can add a lot more admin package, each of them can be configure to have different roles:

  - name: admin@nvidia.com
    type: admin
    org: nvidia
    role: project_admin

you can also have

  - name: user1@hello.com
    type: admin
    org: hospital1
    role: member

Each client site can control each role can perform what action as well, please refer to https://nvflare.readthedocs.io/en/stable/user_guide/security/identity_security.html

For point 6, a simple way is to use one server for each different project.

Note that, for one project, you have already 1 server and several clients involved.
You can use job level authorization to control whether a site can run this job or not: https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/job-level-authorization

thanks

[AkshayBhardwajMN]

Do you mean you want to deploy an FL server that has no access to the internet?
Correction. We will use a proxy between our local compute environement and the flare server when in production mode.

For now, I am looking to simulate a proof of concept but by using more than just the poc option given in the tutorial.

I have performed docker build with an ngc container to create the docker container for the nvflare server. Even though I am prototyping locally; I'd like to open the ports 8002 (for clients) and 8003 (for admins)? That said, I am only interested in spinning up a total of 3 docker containers - server, fl client 1, fl client 2; and none for admin clients since its just me using client docker containers to send/receive jobs.

Does this sound feasible so far?

Secondly, I am most interested in seeing what data leaves the client site i.e. is it just the model parameters or does something else leave too, and if yes, how do I embed an HE filter inside the server container so that anytime data is coming from a client, it becomes encrypted. Do I need to use the filter API inside the python code on client side, or embed it inside the JSON file on the server?

[chesterxgchen]

1. if you just want to use docker locally, you can try this : https://nvflare.readthedocs.io/en/main/getting_started.html#containerized-deployment-with-docker
   This way use docker compose, you can modify to have several dockers there.

2. If you want to try with to simulate what the deployment in real but test out locally, you can use POC mode, where you can specify the docker image
   https://github.com/NVIDIA/NVFlare/blob/main/examples/tutorials/setup_poc.ipynb

3. once you sucessfully run the experiments, you want to go to the next step, you can checkout this example for Homomorphic encryption example
   https://github.com/NVIDIA/NVFlare/blob/main/examples/advanced/cifar10/cifar10-real-world/README.md

for many other examples you can find from our tutorial category

https://nvidia.github.io/NVFlare/catalog/?