logging question in DDP

[ktrapeznikov]

Question regarding logging from lightning module in DDP model

For example, here is a validation step function that computes accuracy.

def validation_step(self, batch, batch_idx):
       x, y = batch
       logits = self(x)
       acc = acc_fun(logits,y)     
       self.log('val_acc', acc)

So what happens in DDP, is the logged value averaged across GPUs? At the end of every epoch?

[carmocca]

It is averaged if you set sync_dist=True (https://pytorch-lightning.readthedocs.io/en/stable/lightning_module.html#pytorch_lightning.core.lightning.LightningModule.log.params.sync_dist)

You can change the op from average (mean) to something else with sync_dist_op.

Given that you are logging inside of validation_step, it will be done at the end of the epoch by default. You can change this behaviour via on_step and on_epoch.

Please have a look at the docs page I linked. Lots of useful info there 😄

[ktrapeznikov]

Thanks. I missed the sync_dist flag.