MonaiAlgo's convert_global_weights broken for multi-gpu clients

[holgerroth]

Describe the bug
If the server initializes the model in a single-GPU/CPU environment, but clients use multi-gpu training, we get a key mismatch in this function.

2023-04-05 10:40:13,775 - WARNING - No global weights converted! Received weight dict keys are ['module.features.conv0.weight', 'module.features.norm0.weight', 'module.features.norm0.bias', 'module.features.norm0.running_mean', 'module.features.norm0.running_var',...

Ideally, we can also update the below test to fail when this warning happens.

To Reproduce
Steps to reproduce the behavior:
Run test python3 -m tests.test_fl_monai_algo_dist

Expected behavior
Should add a check similar to the one in copy_model_state() util:

        if isinstance(dst, (nn.DataParallel, nn.parallel.DistributedDataParallel)):
            dst = dst.module

Screenshots
If applicable, add screenshots to help explain your problem.

Environment

Ensuring you use the relevant python executable, please paste the output of:

================================
Printing MONAI config...
================================
MONAI version: 0.4.0+1935.g0030419f.dirty
Numpy version: 1.23.5
Pytorch version: 1.13.0+cu117
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 0030419ffa67607d1ea0279fb3c831b419ac6a27
MONAI __file__: /home/hroth/Code2/monai/nic/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.3.0
Nibabel version: 5.0.0
scikit-image version: 0.19.3
Pillow version: 9.0.1
Tensorboard version: 2.12.0
gdown version: 4.6.0
TorchVision version: 0.14.0+cu117
tqdm version: 4.64.1
lmdb version: 1.4.0
psutil version: 5.9.1
pandas version: 1.5.1
einops version: 0.6.0
transformers version: 4.21.3
mlflow version: 2.1.1
pynrrd version: 1.0.0

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 22.04.1 LTS
Platform: Linux-5.19.0-38-generic-x86_64-with-glibc2.35
Processor: x86_64
Machine: x86_64
Python version: 3.10.6
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 6
Num logical CPUs: 12
Num usable CPUs: 12
CPU usage (%): [4.9, 4.5, 3.0, 3.3, 3.0, 5.6, 12.9, 17.3, 3.4, 4.9, 3.7, 100.0]
CPU freq. (MHz): 2200
Load avg. in last 1, 5, 15 mins (%): [9.1, 7.6, 6.8]
Disk usage (%): 43.8
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 94.0
Available memory (GB): 85.1
Used memory (GB): 7.8

================================
Printing GPU config...
================================
Num GPUs: 2
Has CUDA: True
CUDA version: 11.7
cuDNN enabled: True
cuDNN version: 8500
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86']
GPU 0 Name: NVIDIA TITAN Xp COLLECTORS EDITION
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 30
GPU 0 Total memory (GB): 11.9
GPU 0 CUDA capability (maj.min): 6.1
GPU 1 Name: NVIDIA TITAN Xp COLLECTORS EDITION
GPU 1 Is integrated: False
GPU 1 Is multi GPU board: False
GPU 1 Multi processor count: 30
GPU 1 Total memory (GB): 11.9
GPU 1 CUDA capability (maj.min): 6.1

Additional context
n/a

[holgerroth]

Maybe, this condition should be raising an error rather than just a warning. Then the test would have failed.

[Nic-Ma]

Hi @holgerroth ,

I think this issue is caused by wrong test code instead of MonaiAlgo itself, I already fixed it in #6158.
convert_global_weights() accepts model weights, so we should convert the multi-gpu model then get the weights:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/networks/utils.py#L446
Could you please help take a look?

Thanks.

[Nic-Ma]

Maybe, this condition should be raising an error rather than just a warning. Then the test would have failed.

Sure, changed to RuntimeError in the latest commit of PR #6158.

Thanks.