nan metric breaking ModelCheckpoint

[ehsanmok]

🐛 Bug

Comparing any numbers to float('nan') is False in Python so as a result if a non-loss metric score is nan initially in training, then callback cannot checkpoint any scores after.

Expected behavior

Ignore a nan metric score. This is orthogonal to when grad or weights become nan.

Environment

• PyTorch Version (e.g., 1.0): 1.5.0
• OS (e.g., Linux): Linux
• How you installed PyTorch (conda, pip, source): pip
• Build command you used (if compiling from source):
• Python version: 3.6
• CUDA/cuDNN version: 10.0
• GPU models and configuration: Tesla V100

Additional context

Previous issue wasn't addressed completely #1008

[github-actions]

Hi! thanks for your contribution!, great first issue!

[Borda]

@ehsanmok mind send a PR? 🐰

[ddrevicky]

Assign this to me please.

[ddrevicky]

So I can reproduce this issue like this.

I am not exactly clear on what the expected behavior is though. In @awaelchli 's PR for nan detection and intervention, training is stopped when loss or weights contain nan or infinite values.

What do we want to do:

1. if a metric that was passed as monitor param to ModelCheckpointgoes nan/inf:

   • raise an error (like when loss/weights do the same)
   • raise a warning but continue training and do not save any more checkpoints
   • raise a warning but continue training and do not save checkpoints when monitor is nan/inf. What if it returns to non nan/inf values (as mentioned in #1008)?

2. separate from this issue perhaps if any metric goes nan/inf (regardless of whether ModelCheckpoint) is used:

   • raise a warning / error ? (if the former than this should be addressed in a different issue perhaps)

3. when do we want to detect nan/infmetrics (whether monitor or any)?

   • ASAP which would be in the first validation step when it happens (perhaps even in Trainer.run_sanity_check)
   • on_validation_end (after all batches are processed) and model checkpoint is being saved