What's the difference between on_step and on_epoch of pl_module.log

[marsggbo]

https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html?highlight=on_epoch%20#pytorch_lightning.core.lightning.LightningModule.log.params.on_epoch

I'm using horovod to train the model. I wonder if on_step and on_epoch average the metrics across all GPUs automatically. In other words, do we need to explicitly average the metrics in functions like training_epoch_end and validation_epoch_end?

[tchaton]

Dear @marsggbo,

When using self.log(..., on_step=True), this will compute the metric per step locally as synchronisation adds performance hit.
When using self.log(..., on_step=True, sync_dist=True), this will compute the metric per step across GPUS.
When using self.log(..., on_epoch=True), this will compute the metrics across GPUS and epoch batches automatically.

Best,
T.C

[marsggbo]

thanks!

[marsggbo]

I have to get the total accuracy of the distributed mode after each epoch, how can we get the value returned by self.log?

[rohitgr7]

must be present in trainer.callback_metrics.