Specify a configuration that should not be run with multi run

[pablo2909]

Hello everyone,

I wondered when launching a multi run if there is a way to specify some configurations that should not be run if encountered in hydra. For example assume I have two models odenet and resnet and two optimisers adam and rmsprop. To launch a multi run I would do:

python main.py -m +model=odenet,resnet +optim=adam,rmsprop

Now what if I don't want to run the combination resnet + rmsprop. Is there a way of telling that to hydra ?

Thank you for any help :)

[Jasha10]

One or more of the following techniques may suit your use-case:

• Check at runtime within your @main-decorated function:
  if (cfg.model, cfg.optim) == ("resnet", "rmsprop"): raise RuntimeError(f"Aborting job with invalid config: {cfg.model=}, {cfg.optim=}")
• Use the Callback API to inspect the config in the on_job_start callback. You can throw an error from the callback to abort the job. (Edit: it turns out that a callback-raised Exception will not cause cause the job to abort; see my comment below)
• Build a plugin that checks whether one of the prohibited combinations is used. This is a heavy-weight solution and will likely only be worth your time if you need low-level customization over the launching/sweeping logic.
• Apply the Specializing configuration pattern to load a specialized config for the resnet + rmsprop combination. E.g. you could set cfg.abort == true via the file model_optim/resnet_rmsprop.yaml. Later, you can inspect this boolean value cfg.abort using one of the other techniques in this list.
• Use advanced features of the Command-line Grammar to exclude the undesirable combination from your command-line sweep:

$ python my_app.py -m '+model_optim={model:odenet,optim:adam},{model:odenet,optim:rmsprop},{model:resnet,optim:adam}'
[2022-12-09 06:22:41,703][HYDRA] Launching 3 jobs locally
[2022-12-09 06:22:41,703][HYDRA]        #0 : +model_optim={model:odenet,optim:adam}
model_optim:
  model: odenet
  optim: adam

[2022-12-09 06:22:41,757][HYDRA]        #1 : +model_optim={model:odenet,optim:rmsprop}
model_optim:
  model: odenet
  optim: rmsprop

[2022-12-09 06:22:41,815][HYDRA]        #2 : +model_optim={model:resnet,optim:adam}
model_optim:
  model: resnet
  optim: adam

[pablo2909]

Hi @Jasha10,

Thank you for your detailed answer.

I will try one of those options. However for the first two proposed solutions, as this raises an error it will interrupt every other run scheduled right ? I had more in mind something that checks and just skip if one forbidden configuration is detected.

[Jasha10]

it will interrupt every other run scheduled right ?

The first option (using an if-statement in the @main decorated function) does not interrupt other jobs:

@hydra.main(version_base=None, config_name="config", config_path=".")
def my_app(cfg: DictConfig) -> None:
    if cfg.model == "resnet" and cfg.optim == "rmsprop":
        raise RuntimeError  # this works
    print(OmegaConf.to_yaml(cfg))

Only the job that raised the error gets canceled.

You are right about the second option (using callbacks).

• If your callback raises an Exception, hydra catches the error and then the job will still run. Hydra will issue a warning but the job will not be aborted.
• If your callback raises a BaseException, Hydra will cash and subsequent jobs will not run.

[pablo2909]

Thank you very much ! this answers my question :)