Large batch size and multiple GPUs

[jjcao]

When I train pix2pix model with batchSize > 1, norm = batch, and multiple GPU, the results seem wrong/strange.

When I train pix2pix model with batchSize > 1, norm = batch, and single GPU, the results are correct.

Could this be solved?

Thank you.

[junyanz]

We observe that batchSize=1 with single gpu gives us the best results so far.
According to this post, It seems that pytorch calculates mean/var statistics for each gpu.
So how many images do you have per GPU? 1 per GPU might cause some problems for batchnorm.
Have you tried instancenorm with multi-gpu setting and batchSize>1?

[jjcao]

Yes. with --norm instance, it worked.

Specify the number of images per GPU? Is there an option or is it simple for changing your code?

[junyanz]

I guess it will be batchSize/#gpus.

[jjcao]

If it is batchSize/#gpus, then norm still need to be "instance" for successful training. I have tested this.