How to override variables in subdirectory from another subdirectory in the same level?

[EternalGoldenBraid]

I have the following hydra setup:

config
├── config.yaml
├── experiments
│  ├── results.yaml
├── paths
│  ├── config.yaml

config.yaml:

defaults:
- paths: config

paths.config.yaml:

dbg_path: logs/debug

Given that I run my script with: python main.py +experiments=results, how do I override paths.debug_path in the experiments.config.yaml?

Something like?:
experiments.config.yaml:

experiment_name: experiment_A
${paths.dbg_path} = ${paths.dbg_path}/${experiment_name}

In my code I'm trying to only refer to config.paths.dbg_path and I'd like it to adapt to the experiment name.

Thanks,
Nick

[odelalleau]

Generally speaking, you could use #@package _global_ in experiments/config.yaml to merge it into the root (so as to override paths.dbg_path), or have it include another config through the defaults list that would be merged into the root.

What makes things tricky in your use case is that you want to set a value that depends on the value you are overriding. Here's a potential way to achieve this (but yeah, it's not super clean):

config.yaml:

defaults:
  - paths@_orig_paths: ${paths}  # backup value that we may override
  - paths: config
  - _self_

And then experiments/config.yaml:

#@package _global_

experiment:
  experiment_name: experiment_A

paths:
  dbg_paths: ${_orig_paths.dbg_paths}/${experiment.experiment_name}

Depending on the exact constraints on which configs you can change, you may be able to make this cleaner. One annoying thing is that you can't access ${paths} in the defaults list of experiments/results.yaml due to Hydra's resolution order, which is why I had to backup the values in config.yaml instead...

[EternalGoldenBraid]

Thanks for the swift reply and sorry for the delay! I'll try to try it out later today and I'll let you know how it fits :)