training time and batch size

[haiderasad]

@rosinality @taesungp I am training on 8 V100s(16gB) with a custom dataset of 50k images using resolution 256 and batch size 8.
Firstly in the paper, they are using batch size 64 for 256 resolution training, while in my case batch size 16 starts giving cuda oom error.
Secondly, the training time on 8 GPUs should be minimized but in my case it's giving an ETA of more than 1000 hours for 25M iterations (which is the default in @taesungp repo https://github.com/taesungp/swapping-autoencoder-pytorch and gives good results).
I am following the @rosinalty repo for training and the dataset is in lmdb format.
UPDATE: I implemented @taesungp code and it is training at 64 batch size and comparatively faster iterations

@rosinality can you please tell why the training time is taking too long? all 8 GPUs are being utilized around 100% and why bigger batch size is not fitting in memory compared to @taesungp repo

[rosinality]

In this implementation batch size corresponds to the batch size per gpus, so if you set 64 for 8 gpus total batch size will be 64 * 8.