Multi-GPU Training error

[blueskywwc]

1.Multi-GPU Training：
python -m torch.distributed.launch --master_port 42342 --nproc_per_node 2 train.py --device 0,1

When I set image-weights to true, I got the error: Tensors must be CUDA and dense

When I set image-weights to false,It's normal

2.Single-GPU Training：
python train.py --device 0

When I set image-weights to true,It's normal

Why can't image-weights be set to true during multi-gpu training? Thank you ！

[glenn-jocher]

@blueskywwc sorry to hear about your training problems! The --img-weights argument has not been tested on Multi-GPU so it's possible the two may be incompatible. I'll add a TODO here to investigate, but since this is not a common use case we may not get around to fixing this for a while unfortunately.

If you could help debug and figure out a good fix to help everyone else it would much appreciated!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[glenn-jocher]

@blueskywwc good news 😃! Your original issue may now been fixed ✅ in PR #3275. This provides improved error handling to notify the user than DDP is not compatible with the --image-weights training argument. To receive this update you can:

• git pull from within your yolov5/ directory
• git clone https://github.com/ultralytics/yolov5 again
• Force-reload PyTorch Hub: model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• View our updated notebooks:

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

[blueskywwc]

thanks，I will try and update!

[blueskywwc]

@glenn-jocher
git clone https://github.com/ultralytics/yolov5 again
python 3.8.5 torch1.7.1+cu101

1.Multi-GPU Training：
python -m torch.distributed.launch --master_port 42342 --nproc_per_node 2 train.py --device 0,1
set image-weights to true, I got the error:

File "train.py", line 529, in
assert not opt.image_weights, '--image-weights argument is not compatible with DDP training'
AssertionError: --image-weights argument is not compatible with DDP training

They are still not compatible ，thanks！

[glenn-jocher]

@blueskywwc yes everything is working as intended now!

The two arguments are not compatible, you are correct. Now the error handling is improved so that now the users can understand better the cause of the problem and avoid this pairing.

[glenn-jocher]

@blueskywwc so the natural solution is to train --image-weights on single GPU.

[blueskywwc]

thanks，I see