CUDA_ERROR_ILLEGAL_ADDRESS error when many objects in one image

[Lizw14]

Hi,

I encountered the problem of CUDA_ERROR_ILLEGAL_ADDRESS error during training when the objects are densely located, i.e. when there are many(roughly >100 or 150) ships in one 6001000 image. When I split these dense images into smaller ones(control object number < 50) and pad the splitted images into 6001000, the training goes on well with no error. BTW, the same error happens in all your repositories(R2CNN, RRPN, R-DFPN).

But I think split the images with densely located objects may damage the final performance, so I want to get this problem solved. Could you please tell me which part cause this error? Maybe GPU NMS or something else?

[powermano]

Thanks for your share. When i control the objects per image, it finally works.

[FishBigOcean]

Hi,

I encountered the problem of CUDA_ERROR_ILLEGAL_ADDRESS error during training when the objects are densely located, i.e. when there are many(roughly >100 or 150) ships in one 600_1000 image. When I split these dense images into smaller ones(control object number < 50) and pad the splitted images into 600_1000, the training goes on well with no error. BTW, the same error happens in all your repositories(R2CNN, RRPN, R-DFPN).

But I think split the images with densely located objects may damage the final performance, so I want to get this problem solved. Could you please tell me which part cause this error? Maybe GPU NMS or something else?

Thanks for sharing！
Did you finally find a solution other than cropping the picture?
look forward to your reply

[zsh1234567]

May I ask what dataset set do you use?