How to merging nested configurations with Hydra

[icedpanda]

Hello,

I am trying to use Hydra to manage complex configurations for a deep learning project. I've encountered a challenge when attempting to merge nested configurations dynamically.

Directory Structure:

config
│
├── model
│   ├── task1
│   │   └── bert.yaml
│   ├──task2
│   │   └── gpt.yaml

├──trainer
│   ├── default.yaml
│   ├── mps.yaml
│   ├── ddp.yaml
│   └── tasks
│       ├── phase1.yaml
│       ├── phase2.yaml
│       └── phase3.yaml

Motivation:

• I am using different models with specific trainer configurations based on various research papers.
• For each model, the training has multiple phases, e.g., pretraining, fine-tuning, and etc.
• The trainer settings, like epoch, are model-dependent and are defined under model.task1.trainer, model.task2.trainer, via

# config/model/task1/bert.yaml
defaults:
  - /trainer@trainer: ../trainer/tasks/phase2
  - _self_
  
  #model parameters here... 
  ###########

# model specific trainer configuration
trainer:
  max_epochs: 99

• I want to retain the ability to switch the training strategy (like ddp, mps) while keeping the base trainer settings defined in the model's YAML.

However, I can only mange it to by calling python main.py model.task1.trainer.strategy=ddp model.task1.trainer.sync_batchnorm=True. It will prompt item already exist error if I do python main.py +model.task1.trainer=trainer/ddp.

Is there a native way in Hydra to overwrite nested configurations dynamically based on command line inputs? (e.g. only overwriting new key values. In this case, strategy=ddp and sync_batchnorm=True)

[odelalleau]

I believe what you're trying to achieve would look like:

# config/model/task1/bert.yaml
defaults:
  - /trainer/tasks@trainer: phase2    # <== this is the change -- might not be required, but looks cleaner
  - _self_

  #model parameters here... 
  ###########

# model specific trainer configuration
trainer:
  max_epochs: 99

And then

python main.py +trainer@model.task1.trainer=ddp

[odelalleau]

Can you please provide a minimal example to reproduce this issue? It's a bit hard to follow (the error message mentions an issue related to model.rec.trainer but I don't see anything releated to this in the configs you showed).

You can also try to follow the advice from the error message (To override a value use 'override trainer@model.rec.trainer: ddp')

[icedpanda]

sorry, rec here is the actual task name.

Please found a minimal example here:

config/default.yaml

# @package _global_

# specify here default configuration
# order of defaults determines the order in which configs override each other
defaults:
  - _self_
  - model: transformer

config/trainer/default.yaml

# default
# https://lightning.ai/docs/pytorch/stable/common/trainer.html
_target_: lightning.Trainer
# default values for all trainer parameters
accelerator: cpu # (“cpu”, “gpu”, “tpu”, “ipu”, “hpu”, “mps”, “auto”)
devices: 1
num_nodes: 1 # Number of GPU nodes for distributed training
precision: 32 # Full precision (32), half precision (16).
fast_dev_run: False # runs 1 batch of train, test  and val to find any bugs (ie: a sort of unit test)
max_epochs: ??? # Stop training once this number of epochs is reached. Must be specified !!!
min_epochs: 1 # Force training for at least these many epochs
reload_dataloaders_every_n_epochs: 0 # Set to a non-negative integer to reload dataloaders every n epochs
sync_batchnorm: False # Synchronize batch norm layers between process groups/whole world

config/trainer/ddp.yaml

defaults:
  - default

strategy: ddp
sync_batchnorm: True

config/trainer/tasks/task1.yaml

defaults:
  - default
  - _self_

default_root_dir: ${paths.output_dir}/task1/

max_epochs: 10

accelerator: gpu
devices: auto

precision: bf16-mixed
# perform a validation loop every N training epochs
check_val_every_n_epoch: 1

config/model/transformer.yaml

defaults:
  - /model@task1: task1/bert

config/model/task1/bert.yaml

defaults:
  - /trainer@trainer: tasks/task1
  - _self_

pl:
  _target_: llmcrs.models.lightning_modules.TransformerDefaultEmbeddingLightning
  is_compile: False

# model specific trainer configuration
trainer:
  max_epochs: 10

[odelalleau]

Thanks, I tried your example but it runs with no error for me, here's the final config:

model:
  task1:
    trainer:
      _target_: lightning.Trainer
      accelerator: gpu
      devices: auto
      num_nodes: 1
      precision: bf16-mixed
      fast_dev_run: false
      max_epochs: 10
      min_epochs: 1
      reload_dataloaders_every_n_epochs: 0
      sync_batchnorm: false
      default_root_dir: ${paths.output_dir}/task1/
      check_val_every_n_epoch: 1
    pl:
      _target_: llmcrs.models.lightning_modules.TransformerDefaultEmbeddingLightning
      is_compile: false

Which version of Hydra are you using?

[icedpanda]

I am using hydra 1.3.0.

Your outputs seem to just be running default/task1@trainer, which I have no problem with. But I would like the configuration composition to follow this logic while using ddp:

• model.task1.trainer initializes from ddp
• ddp then will initializes from trainer/default.
• Then initialize task-specific trainer settings.
• Finally, apply model specific trainer settings.

I can only mange it to by calling python main.py model.task1.trainer.strategy=ddp model.task1.trainer.sync_batchnorm=True, but just wondering if there is a native way in Hydra to overwrite this like.

[odelalleau]

Ok I see, thanks. I tried some stuff that I expected would work but it didn't, I think it's worth trying to simplify this example to make it really minimal and then maybe open an issue (taking note to do it later, though that may take a while).

Potential workarounds:

1. What I find easiest, when the config isn't too dynamic, is to just hardcode the paths in the configs being merged:

config/trainer/ddp.yaml
@@ -1,3 +1,5 @@
+#@package model.task1.trainer
+
 defaults:
     - default

config/default.yaml
@@ -4,5 +4,6 @@
 # order of defaults determines the order in which configs override each other
 defaults:
   - _self_
+  - optional trainer: null
   - model: transformer

Then you can use python my_app.py trainer=ddp (though it won't work exactly as you want here, because trainer/default.yaml is reloaded from task1.yaml and overwrite some ddp settings ==> instead it could be loaded only once from the base config as the first element in the defaults list)

2. Not using the : syntax (I'm not sure if you can easily manipulate it from the command line though):

config/trainer/tasks/task1.yaml
@@ -1,5 +1,6 @@
 defaults:
   - default
+  - /trainer/ddp@_here_
   - _self_

 default_root_dir: ${paths.output_dir}/task1/