FedHyper: A Universal and Robust Learning Rate Scheduler for Federated Learning with Hypergradient Descent

Code of ICLR 2024 paper : FedHyper: A Universal and Robust Learning Rate Scheduler for Federated Learning with Hypergradient Descent.

The code include the FedHyper-G, FedHyper-SL, FedHyper-CL, and other methods and baselines in the paper. The dataset are FMNIST and CIFAR-10 in this code.

Requirments

Install all the packages from requirments.txt

• Python3
• Pytorch
• Torchvision

Data

• Download train and test datasets manually or they will be automatically downloaded from torchvision datasets.
• Experiments are run on Fashion Mnist and Cifar.
• To use your own dataset: Move your dataset to data directory and write a wrapper on pytorch dataset class.

Running the experiments

• To run the IID FedHyper algorithms with global/local schedulers, run:

python src/baseline_main.py --function FedHyper-GM --dataset cifar --epochs 50 --iid 1
python src/baseline_main.py --function FedHyper-G --dataset cifar --epochs 50 --iid 1
python src/baseline_main.py --function FedHyper-SL --dataset cifar --epochs 50 --iid 1
python src/baseline_main.py --function FedHyper-CL --dataset cifar --epochs 50 --iid 1
python src/baseline_main.py --function FedHyper-FULL --dataset cifar --epochs 50 --iid 1

To run NON-IID settings, the iid parameter should be 0. FedHyper-FULL can run both global and local schedulers

• To run the baselines in our paper, here are some examples:

python src/baseline_main.py --function FedAdam --dataset cifar --epochs 50 --iid 1
python src/baseline_main.py --function FedAdagrad --dataset cifar --epochs 50 --iid 1
python src/baseline_main.py --function Decay-G --dataset cifar --epochs 50 --iid 1
python src/baseline_main.py --function Decay-SL --dataset cifar --epochs 50 --iid 1
python src/baseline_main.py --function FedExp --dataset cifar --epochs 50 --iid 1

---