validation_epoch_end with DDP

[adarsh-kr]

What is your question?

I am trying to implement a metric which needs access to whole data. So instead of updating the metric in *_step() methods, I am trying to collect the outputs in the *_epoch_end() methods. However, the outputs contain only the output of the partition of the data each device gets. Basically if there are n devices, then each device is getting 1/n of the total outputs.

Stackoverflow Post

What's your environment?

• OS: ubuntu
• Packaging: conda
• Version [1.0.4
• Pytorch: 1.6.0

[tchaton]

Hey @adarsh-kr,

Are you using Pytorch Lightning Metric API or implementing our own. Metric API should take care of sync across gpus.

We use this function to sync values: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/utilities/distributed.py#L121

Would you mind creating a simple script with BoringModel to reproduce your use case, so we can assist you better.
Please, use this script: https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/bug_report_model.py

Best regards,
T.C

[adarsh-kr]

Is the outputs in validation_epoch_end(outputs) expected to have outputs from all the devices when running ddp? Or for that device only?

If this property is guaranteed in validation_epoch_end then I can circumvent the problem I am facing with metric.