LightningCLI docs example for EarlyStopping missing required args in config file

[Lunamos]

📚 Documentation

Error Description

In the lightning CLI tutorial (https://lightning.ai/docs/pytorch/stable/cli/lightning_cli_advanced_3.html), misleading error exists in the tutorial on how to set callbacks in yaml files.

How to fix?

The original tutorial gives the following simple configuration file that defines two callbacks

# wrong sample configuration file in tutorial
trainer:
  callbacks:
    - class_path: lightning.pytorch.callbacks.EarlyStopping
      init_args:
        patience: 5
    - class_path: lightning.pytorch.callbacks.LearningRateMonitor
      init_args:
        ...

However, this example yaml file does not work correctly. The following yaml file can get the correct results.

# Fixed config.yaml
trainer:
  callbacks:
    - class_path: EarlyStopping
      init_args:
        patience: 5
    - class_path: LearningRateMonitor
      init_args:
        logging_interval: 'epoch'

What version are you seeing the problem on?

lightning v2.1.2 & v2.2.1

How to reproduce the bug

# main.py
from lightning.pytorch.cli import LightningCLI
from lightning.pytorch.demos.boring_classes import DemoModel, BoringDataModule

def cli_main():
    cli = LightningCLI(DemoModel, BoringDataModule)

if __name__ == "__main__":
    cli_main()

# config.yaml
trainer:
  callbacks:
    - class_path: lightning.pytorch.callbacks.EarlyStopping
      init_args:
        patience: 5
    - class_path: lightning.pytorch.callbacks.LearningRateMonitor
      init_args:
        logging_interval: 'epoch'

Error messages and logs

usage: main.py [-h] [-c CONFIG] [--print_config[=flags]] {fit,validate,test,predict} ...
error: Parser key "trainer.callbacks":
  Does not validate against any of the Union subtypes
  Subtypes: (typing.List[lightning.pytorch.callbacks.callback.Callback], <class 'lightning.pytorch.callbacks.callback.Callback'>, <class 'NoneType'>)
  Errors:
    - Problem with given class_path 'lightning.pytorch.callbacks.EarlyStopping':
        Validation failed: Key "monitor" is required but not included in config object or its value is None.
    - Not a valid subclass of Callback
      Subclass types expect one of:
      - a class path (str)
      - a dict with class_path entry
      - a dict without class_path but with init_args entry (class path given previously)
    - Expected a <class 'NoneType'>
  Given value type: <class 'list'>
  Given value: [Namespace(class_path='lightning.pytorch.callbacks.EarlyStopping', init_args=Namespace(monitor=None, min_delta=0.0, patience=5, verbose=False, mode='min', strict=True, check_finite=True, stopping_threshold=None, divergence_threshold=None, check_on_train_epoch_end=None, log_rank_zero_only=False)), Namespace(class_path='lightning.pytorch.callbacks.LearningRateMonitor', init_args=Namespace(logging_interval='epoch', log_momentum=False, log_weight_decay=False))]

More Info

I confirmed that I have installed the latest jsonargparse using pip install "jsonargparse[signatures]"

This tutorial should have existed for a long time. If this error did not exist before, I suspect that jsonargparse may have updated the method of parsing callback configuration.

cc @Borda @carmocca @mauvilsa

[awaelchli]

The error occurs because the EarlyStopping callback has a required argument 'monitor' that wasn't listed there. The code snippets there are meant to illustrate the point of configuring any callback from the config file. There is no model code so it's not really possible to set a meaningful monitor value, but if you feel it's better we can include an arbitrary monitor value like monitor: val_loss or choose a different callback for demonstration purposes.

Do you want to send a PR?

[Lunamos]

You are right, this problem occurs because the EarlyStopping callback requires a monitor argument, not an address resolution problem as I thought.

Since DemoModel lacks a real validation_step, I propose to replace EarlyStopping with another callback, such as

- class_path: lightning.pytorch.callbacks.ModelCheckpoint
  init_args:
    save_weights_only: true

This configuration works fine and has been tested by me.

I will submit a PR right away.