Submitit plugin won't work for 2 experiments

[Aceticia]

I'm working on a project that starts from this repo. I wrote some custom experiments that uses submitit, and it correctly submits the right jobs if I run commands with only one experiments, like python src/train.py -m experiment=rnn seed=0,1,2. However, if I run 2 or more experiments, for example python src/train.py -m experiment=rnn,temporal_attn seed=0,1,2,submitit will stop working and run the job locally. Is this likely a feature or a bug (in my config or in the plugin)?

Some clues to the problem: The experiments themselves use the # @package _global_ header. Could this be what makes it behave differently?

I currently don't have a minimal example to reproduce this, but please let me know if you need anything to solve this. Thank you.

[Jasha10]

The submitit plugin runs locally vs non-locally based the config setting for the hydra/launcher group. Specifically, it runs locally if hydra/launcher=submitit_local and on slurm if hydra/launcher=submitit_slurm.

I'd suggest printing the config from each of your experiment runs to confirm that hydra/launcher is set correctly:

python src/train.py --cfg=all seed=0 experiment=rnn
python src/train.py --cfg=all seed=0 experiment=temporal_attn

If the above commands show that hydra/launcher is set to submitit_slurm, then the next step in debugging would be to check the composed config for the multirun job. Run the command python src/train.py -m experiment=rnn,temporal_attn seed=0,1,2 (which you reported is running locally) after modifying your src/train.py file so that the full config will be printed to stdout or to a log file:

# src/train.py
...
@hydra.main(...)
def app(cfg):
    # add lines printing the config for diagnostic purposes:
    print("\n JOB CONFIG:")
    print(OmegaConf.to_yaml(cfg))  # print the job config
    from hydra.core.hydra_config import HydraConfig
    print("\n HYDRA CONFIG:")
    print(OmegaConf.to_yaml(HydraConfig.get()))  # print the Hydra config

    ...

Looking at the output, can you confirm that hydra/launcher is set properly?

[Aceticia]

Thanks for the fast reply. Testing individual experiments, I can confirm that the launchers are correctly set. Surprisingly, the launcher is also set correctly when launching multiple experiments, even if the job is actually running locally. Let me try to reproduce this on a smaller set of configs...

[Aceticia]

Here's a small set of code that reproduces the issue I have. Should we keep the discussion here or should I open an issue?

From the configs, it looks like to me the issues are more on the submitit part since the configs are correctly overridden?

[Jasha10]

I've investigated the example you uploaded.
The behavior you observed is due to a limitation of Hydra. Before I explain this limitation, let me give a few ways to work around it:

1. Explicitly specify the desired launcher at the command line:
   python app.py -m experiments=base,exp1 hydra/launcher=submitit_local
2. In the configs/top.yaml defaults list, replace - slurm: null with - slurm: gpu
3. In the configs/top.yaml defaults list, replace - experiments: null with - experiments: base
4. In the configs/top.yaml defaults list, add - override hydra/launcher: submitit_local

Now I'll explain the limitation that causes the behavior you observed. When in multirun mode, Hydra performs config composition in two phases:

• First, hydra does preliminary config composition. This preliminary config does not take into account sweep overrides specified at the command line. The preliminary config is used to e.g. decide what sweeper to use, what launcher to use, what callbacks to call.
• After deciding what sweeper and launcher to use, Hydra calls the chosen sweeper, which re-composes the config once for each job. For example, with the command-line override experiments=base,exp1, there would be two rounds of config composition in this second phase (one with experiments=base and one with experiments=exp1).

The launcher is chosen after the first phase of config composition, based on the preliminary config composed during that phase. The first round of config composition does not take into account sweep parameters. The CLI override experiments=base,exp1 is a "sweep override", while experiments=base is not a sweep override. For this reason, the CLI override experiments=base may have an effect on the chosen sweeper and launcher, while the CLI override experiments=base,exp1 will not have any effect on the chosen sweeper and launcher.

The takeaway is as follows: try to make sure your sweeper and launcher can be determined from your top-level config or from basic (non-sweep) command line overrides, as sweep overrides do not factor into Hydra's choice of sweeper/launcher.

[Aceticia]

Thank you so much for the detailed explanation and proposed solution!

[Aceticia]

@Jasha10 sorry for spamming, but do you think this should be submitted as an issue? I currently don't have any solutions and it doesn't look like a trivial problem.

[Jasha10]

Apologies for the late reply on this. I'll try to get back to you asap.

[jieru-hu]

hi @Aceticia - in your example you have hydra/launcher=submitit_local which will make the launcher launch on the local machine. could you try hydra/launcher=submitit_slurm instead?

[Aceticia]

Hi, I tested that with submitit_slurm it also doesn't work. I used submitit_local to make sure it can be reproduced by anyone when using a local machine.

[jieru-hu]

@Aceticia i cannot repro your issue with your example. I installed your example and ran it on our slurm cluster and I could see the jobs were run on the cluster, am i missing something here?

$ python app.py -m experiments=exp1,exp2 hydra/launcher=submitit_slurm
[2022-09-07 23:10:41,513][HYDRA] Submitit 'slurm' sweep output dir : multirun/2022-09-07/23-10-41
[2022-09-07 23:10:41,515][HYDRA]        #0 : experiments=exp1
[2022-09-07 23:10:41,519][HYDRA]        #1 : experiments=exp2

$ tree  -a multirun/2022-09-07/23-10-41
multirun/2022-09-07/23-10-41
├── .submitit
│   ├── 104659
│   │   └── 104659_submission.sh
│   ├── 104659_0
│   │   ├── 104659_0_0_log.err
│   │   ├── 104659_0_0_log.out
│   │   ├── 104659_0_0_result.pkl
│   │   └── 104659_0_submitted.pkl
│   └── 104659_1
│       ├── 104659_1_0_log.err
│       ├── 104659_1_0_log.out
│       ├── 104659_1_0_result.pkl
│       └── 104659_1_submitted.pkl
├── 0
│   ├── .hydra
│   │   ├── config.yaml
│   │   ├── hydra.yaml
│   │   └── overrides.yaml
│   └── app.log
├── 1
│   ├── .hydra
│   │   ├── config.yaml
│   │   ├── hydra.yaml
│   │   └── overrides.yaml
│   └── app.log
└── multirun.yaml

8 directories, 18 files

if i cat the slurm log

$ cat multirun/2022-09-07/23-10-41/.submitit/104659_1/104659_1_0_log.out
submitit INFO (2022-09-07 23:10:42,751) - Starting with JobEnvironment(job_id=104659_1, hostname=hpc-230, local_rank=0(1), node=0(1), global_rank=0(1))
submitit INFO (2022-09-07 23:10:42,751) - Loading pickle: 

[Aceticia]

If I add 'hydra/launcher=submitit_slurm' in cli commands it indeed works, but my intended use case is to encapsulate it in the configs. It's a pretty niche use, but I feel like this should be supported since this works for single experiment?

[omry]

You can't sweep on the Hydra's configuration in general. If you must do it use an external script.
@Jasha10, maybe it's possible to detect this scenario and error out. IIRC sweeping on hydra parameters directly does that (hydra.foo.bar=1,2,3)

[Jasha10]

maybe it's possible to detect this scenario and error out

Good idea, thanks!
Edit: Opened issue here: #2381