folder organization and package style

[aretor]

I have the following folder tree

|- configs
|- experiment
|   |- config.yaml
|- data
    |- task1
        |- dset.yaml
        |- dset_config_group
            |- option1.yaml
            |- option2.yaml

the content of the important files is the following

config.yaml

# @package _global_
defaults:
  - override data/task1/dset_config_group@data.dset_config_group: null

dset.yaml

# @package data
defaults:
  - dset_config_group: option1

task1 is just a logical category to keep all configurations related to one section of my experimental setting in the same place (I can later create another subpkg1, representing another section). I don't want task1 to be included in the hydra config object, therefore I override the package of dset.yaml to data only.

In addition, I would like to be able to select one between the two options in the config group. I can do this from the command line.

This setting however complicates the override in config.yaml, which now requires its own package override in the defaults. Now as a CLI arg I have to specify e.g. data/task1/dset_config_group@data.dset_config_group=option1. I feel this is a bit ugly and unnecessarily verbose in most cases. I would have preferred to just type data/task1/dset_config_group=option1, or data.dset_config_group=option1.

Questions:

1. Should my folder tree reflect only the structure of the cfg? In other words, should the folders be used only as config groups and not for any other kind of categorization?
2. Should I avoid CLI argument override in such cases, and rely more on config files for each experiment?
3. What is the reason behind defining such a verbose override syntax?
4. Is there a better way (in terms of folder organization or override syntax) to achieve what I want to achieve?