Detected 1 oom-kill event(s) in StepId=125757.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.

[QishengL]

System Info

- `Accelerate` version: 0.15.0
- Platform: Linux-5.4.199-ql-generic-12.0-18-x86_64-with-glibc2.31
- Python version: 3.10.4
- Numpy version: 1.23.1
- PyTorch version (GPU?): 1.12.1 (False)

Information

• The official example scripts
• My own modified scripts

Tasks

• One of the scripts in the examples/ folder of Accelerate or an officially supported no_trainer script in the examples folder of the transformers repo (such as run_no_trainer_glue.py)
• My own task or dataset (give details below)

Reproduction

The error: Detected 1 oom-kill event(s) in StepId=125757.batch. Some of your processes may have been killed by the cgroup out-of-memory handler.
Try to load optimizer and models but got CPU OOM with code below
optimizer.load_state_dict(torch.load("optim.pt",map_location='cpu'))
of
accelerator.load_state("my_checkpoint")

If I use the code below then everything is fine.
optimizer.load_state_dict(torch.load("optim.pt",map_location=accelerator.device))

Expected behavior

Is it possible to use accelerator.load_state("my_checkpoint") and load the state to accelerator.device directly?

[sgugger]

I think this comes from the load of the optimizer state on multiple processes at once. @muellerzr we should probably load the optimizer on the device when in multi-GPU setting with num_processes>1

[muellerzr]

Should be fixed with #1220, can you try it out via pip install git+https://github.com/huggingface/accelerate and let us know? :)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.