Docker does not recover from resource saver mode

[ramki88]

Description

After the recent update, docker does not seem to recover from resource saver mode. In the dashboard CPU and Mem shows zero and all docker commands do not respond or wake the engine. Only after a restart of the engine it starts to respond

Reproduce

docker info

Expected behavior

"docker info should provide the info about docker

docker version

Client:
 Cloud integration: v1.0.35-desktop+001
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:32:30 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.22.0 (117440)
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:38 2023
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.5
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.1
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2-desktop.1
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.6
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.20.0
    Path:     /Users/ramakrishnan3.s/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 5
 Total Memory: 7.765GiB
 Name: docker-desktop
 ID: efcb88bd-4e43-4b04-be3f-89edcd9a82d7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: true

Diagnostics ID

073CCF80-81D3-426A-A5F1-1B13A4481DE2/20230804091953

Additional Info

No response

[christophermclellan]

Thanks @ramki88 we'll look in to this asap. Fyi, you can disable Resource Saver temporarily via Settings > Features in development > Experimental features.

[ctalledo]

Hi @ramki88, sorry you hit a problem and thanks for filing the issue.

Looking at the diagnostics bundle, I see Docker Desktop hit an internal error while entering resource saver mode:

[2023-08-03T17:41:10.371280000Z][com.docker.backend.idle][I] busy -> idle
[2023-08-03T17:41:10.372035000Z][com.docker.backend.idle][I] starting timer (30s)
[2023-08-03T17:41:40.373905000Z][com.docker.backend.idle][I] timer expired (30s)
[2023-08-03T17:41:40.375186000Z][com.docker.backend.idle][I] system idle for 30.002479833s -> reducing
[2023-08-03T17:41:40.380947000Z][com.docker.backend.idle][I] populating cache
ESC[91m[2023-08-03T17:42:40.381430000Z][com.docker.backend.idle][E] failed to populate cache: calling HEAD http://localhost/_ping: calling http://localhost/_ping 500ESC[39m
[2023-08-03T17:55:40.426983000Z][com.docker.backend.idle][I] idle -> busy (was idle for 14m29.996941291s; reason: [in-flight APIs: map[/_ping:1]])

We will work internally to fix it.

Out of curiosity, how often does this reproduce for you?

Thanks!

[ramki88]

Hi @ctalledo, Thanks for you response. I have noticed these behaviour about 4 times today. Till now I have not observed any pattern. Will observe and report if I find anything.

[kangaechu]

I have same problem.
I disabled "Access experimental features", but resource saver mode is still on.

I tried Factory Reset, but it doesn't changed.

[mikelax]

I am having similar issues with the 4.22.0 version that I just upgraded to this morning. Once I noticed that the app went into Resource Saver mode, I am now unable to do anything within the app. I can't exit or force quit using the Mac menu.

I have restarted my laptop, once I then start Docker Desktop it starts back in Saver mode but I am unable to do anything. I can't run docker build, restarting and clean/purge data don't respond with anything and Quit Docker Desktop from the system menu also does nothing.

[hrabe]

Have the same issue with version 4.22.0 (117440)
After reaching the resource saver state:

• cli won't work
• restart docker won't work
• force quit the Docker Desktop application seems to close UI and removes icon but can't be started afterwards
• restart of Mac will make it work for some time (but starts in resource save mode)

Please provide a work around, a config option to deactivate this or make sure, it wakes up correctly when used.

[bconfrancisco]

Hi - I installed 4.22.0 last week and it was working on Friday 8/4. Today i have the same issues as the others.
Thank you in advance for the fix.

[ctalledo]

Hi folks, Cesar Talledo from Docker here.

Sorry to hear you are having trouble with the resource-saver feature, and thank you very much for reporting the problem.

First, quick context:

The resource saver feature is meant to significantly reduce Docker Desktop's memory and CPU usage on the host when no containers are running, to improve overall user experience. It will reduce mem utilization by ~2GBs (or more) when DD goes idle. In DD v4.21, the feature would simply pause processes in the DD Linux VM, but in v4.22 we improved it to shutdown the DD Linux VM. We did a lot of testing prior to releasing the feature and for most users it's working well, but a few are reporting problems. Our apologies, we will try to fix them ASAP.

For those of you experiencing problems:

• If possible, please upload a diagnostics bundle so we can debug it.

• To disable the feature, edit the features-overrides.json file as follows:

{
  "IdleShutdown": {
    "name": "",
    "label": "Shutdown VM when idle",
    "description": "When this is set, the system will shutdown the VM when idle",
    "enabled": false,
    "type": 2
  }
}

The file can be found here:

Mac: ~/Library/Group Containers/group.com.docker/features-overrides.json
Windows: %APPDATA%\Roaming\Docker\features-overrides.json
Linux: $HOME/.docker/desktop/settings.json

After you disable the feature, make sure to quit Docker Desktop and start it again via the Whalemenu on the taskbar (don't just simply click "restart" on the Whalemenu).

If you have any issues with this, please let us know.

Thanks again and we hope to fix all issues very soon.

[bsousaa]

We are investigating the issue internally.

To help us reproducing the issue could you upload a diagnostics bundle and share the id? @bconfrancisco @hrabe @mikelax @kangaechu . Thanks in advance!

[safayildirim]

We are investigating the issue internally.

To help us reproducing the issue could you upload a diagnostics bundle and share the id? @bconfrancisco @hrabe @mikelax @kangaechu . Thanks in advance!

I have the same problem. My diagnastic id is: 17381DA9-8BEE-4201-8FF2-0FB58278F01C/20230808113242

[mikelax]

We are investigating the issue internally.

To help us reproducing the issue could you upload a diagnostics bundle and share the id? @bconfrancisco @hrabe @mikelax @kangaechu . Thanks in advance!

@bsousaa Here is an ID: D037C0D1-560C-409B-A11C-0629819A44CE/20230808114543

One item of note, I first started Docker this morning and immediately ran the Diagnostics and uploaded. During this time Docker appeared to be in regular mode. As soon as the Diagnostics ID uploaded completed then Docker switched into "Resource saver mode".

[benjaminreji]

Diagnostic ID: 8C632EC5-F963-42AB-B88E-4342E129FF2F/20230808115322

I keep getting this error
no such service: build

Docker is in resource saver mode

[bconfrancisco]

Diagnostic ID: 8E671B87-5DFC-45A8-8460-34DBFE462181/20230808152154

MacOS Monterey: Version 12.5.1

[saillingaway]

I've also been having this issue with version 4.22.0 for the last 2 days.

I can't quit/start/restart docker via the Whalemenu on the taskbar. Quitting Docker Desktop via the app makes the app/UI disappear but the taskbar Whalemenu doesn't change and shows it's still running. I'm new to Docker though so this could be the expected behavior.

Diagnostic ID: 480CE2DB-48A1-4BB8-8B7B-78C895F45DAD/20230808163825
MacOS Ventura 13.4

@ctalledo I don't seem to have ~/Library/Group Containers/group.com.docker/features-overrides.json, but I do have a settings.json. Not sure if I should create a features-overrides.json file there and add the IdleShutdown snippet, or if I should add it to settings.json.

[ctalledo]

Hi @safayildirim, apologies you hit this issue (we are on it) and thanks for uploading the diagnostics bundle, very helpful.

Not sure if I should create a features-overrides.json file there and add the IdleShutdown snippet

Yes, that's what you should do: create a new features-overrides.json file in that same directory and then quit & start Docker Desktop.

[ctalledo]

Thank you @bconfrancisco @benjaminreji @mikelax @safayildirim @safayildirim for uploading the diagnostics bundle, this helps us root cause the problem.

[kangaechu]

Diagnostic ID: 60C25128-558D-4678-9961-8E6CA9B9DE27/20230809012117

[ctalledo]

Diagnostic ID: 60C25128-558D-4678-9961-8E6CA9B9DE27/20230809012117

Thanks @kangaechu; I looked at the diagnostic bundle but did not see any failure with the resource saver feature.

Did it fail for you somehow? Or are you just looking to disable it?

Thanks.

[kangaechu]

@ctalledo
I downgraded Docker Desktop for Mac to 4.21.1 two days ago.
Today I updated it to 4.22.0 in order to send a diagnostic bundle.
As a result, the failure log for the Resource Saver feature may not be included.

[niktolis]

Same issue for me in 4.22 on macOS Ventura 13.4.1. Containers were stopped the Mac was put on sleep. When waking up the docker is in Resource Saver mode and unresponsive. Diagnostic ID: 0408424A-469D-48B6-8B3F-B92D0868B079/20230810045901

[chucheng92]

+1 same issue for me in v4.22.0 on macOS Ventura 13.4.1.
when i disable experimental feature, it can't restore.

[ctalledo]

Hi folks, Cesar from Docker here.

Thanks for the reports on on the Docker Desktop resource saver issues, particularly in Docker Desktop 4.34.

I can confirm there's a regression in Docker Desktop 4.34 where once Docker Desktop enters Resource Saver mode, some commands (e.g., docker compose up, docker ps --filter, and a few others) won't cause Docker Desktop to wake up from resource saver mode. Apologies for this.

We have a Docker Desktop 4.34.2 patch release coming soon (as early as tomorrow hopefully) that will have a fix for this.

In the meantime, you can disable resource saver mode via the Docker Desktop settings menu, under Settings -> Resources.

Thanks and apologies again for the regression.

[xdannyrobertsx]

it would also be great to be able to restart the docker daemon from the cli since the desktop app allows for it via CMD + R

this is my current workaround for this issue since resource saving mode is great otherwise

[thaJeztah]

@xdannyrobertsx for controlling docker desktop from the commandline; you can upvote docker/roadmap#172, which may help getting it prioritized.

Also see my comment on that proposal, where I left some rough ideas of what it could look like;

• Control Docker Desktop via CLI roadmap#172 (comment)

[mattccrampton]

Confirmed that after updating Docker Desktop on Mac to version 4.34.2 (167172) released this morning that this issue is solved for me. Even if Docker Desktop is running in resource saver mode that I am able to run docker commands and Docker Desktop "wakes up" appropriately. Thanks Docker team!

[ctalledo]

Thanks @mattccrampton for the confirmation, and apologies again for the regression.

[T-101]

Looks like Docker Desktop for Mac 4.34.0 has brought this issue back as a regression bug. I need to bring Docker back from sleep by opening the Dashboard and clicking the "play" button to get my stuff back up.

[manchenkoff]

Looks like Docker Desktop for Mac 4.34.0 has brought this issue back as a regression bug. I need to bring Docker back from sleep by opening the Dashboard and clicking the "play" button to get my stuff back up.

Try upgrading to 4.34.2, for me everything was fixed properly

[maduonline]

Yes the new version solved it. Thank you! 🎉

[esetnik]

I am still getting issues after upgrading to v4.34.2. I got a hang yesterday that required me to force quit the docker process. The resource saver still appears to be the culprit.

[djs55]

@esetnik it sounds like there's a different bug lurking there. Next time it gets stuck, could you upload a set of diagnostics in that state and post the id here? I can take a look and hopefully figure it out.

[esetnik]

@esetnik it sounds like there's a different bug lurking there. Next time it gets stuck, could you upload a set of diagnostics in that state and post the id here? I can take a look and hopefully figure it out.

Unfortunately I'm not even able to generate diagnostics in this state. It just hangs completely.

[djs55]

@esetnik ouch. If diagnostics don't work could you try this:

curl --unix-socket ~/Library/Containers/com.docker.docker/Data/backend.sock http://localhost/goroutines/stacks -o stacks.txt

and attach the file? Hopefully I'll be able to spot the cause of the deadlock in the stacks.

[xdannyrobertsx]

This is fixed for me until I sleep my machine.
After waking it back up this issue presents itself again.

OS: macOS Sonoma 14.6.1 arm64
Host: MacBook Pro (16-inch, 2021)
Kernel: Darwin 23.6.0
Shell: zsh 5.9
Terminal: tmux 3.4
CPU: Apple M1 Pro (10) @ 3.23 GHz
GPU: Apple M1 Pro (16) @ 1.30 GHz [Integrated]

[akhanalcs]

@djs55 @ctalledo
The issue still persists. I have to constantly restart my computer just to restart Docker Destop because restarting Docker Desktop seems impossible as it just hangs.

Below screenshot shows it hanging after it crashed and asked to do factory restart and tried to restart itself. I'm on M1 Max MBP running MacOS Sonoma 14.6.1 and Docker v4.34.2.

---

Output of this command is attached below:

curl --unix-socket ~/Library/Containers/com.docker.docker/Data/backend.sock http://localhost/goroutines/stacks -o stacks.txt

stacks.txt

Related issue that I'm having which is making Docker Desktop unusable: #7240 (comment)

[ctalledo]

Hi @akhanalcs, sorry to hear about the problems you are having, but thanks for reporting them.

I looked at the stack trace you uploaded but could not tell the root cause of the problem from it.

Question: when are you hitting that hang on "Turning off Docker Engine"? Does it occur when the resource saver feature kicks-in (i.e., turns off the VM automatically), or when you quit Docker Desktop?

If possible, could you restart Docker Desktop and upload a diagnostics bundle? If not possible, maybe attach the Docker Desktop logs (under ~/Library/Containers/com.docker.docker/Data/log) so we can take a look.

Thanks.

[akhanalcs]

Hi @ctalledo , thank you for your response. I responded to your questions in the other thread as I think my issue resonates more with the issue in the other thread. #7240 (comment)

[NiklasBr]

This just happened on 4.34.3 right as I was working actively with containers running Docker entered resource saver mode and everything became unresponsive. :(

[ctalledo]

Hi @NiklasBr, sorry to hear. If possible, could you please upload a Docker Desktop diagnostics bundle so we can take a closer look.

Thanks!

[NiklasBr]

39263439-8ED3-425A-A6C1-A4B1DD929F05/20241028071941

[ctalledo]

Hi @NiklasBr, thanks for uploading the diagnostics bundle, appreciate the willingness to do so.

Unfortunately the bundle was taken too long after the original failure, so the logs in the bundle don't have sufficient information for me to debug the root cause. I do see the Docker Desktop starting the VM multiple times, the last time at [2024-10-25T08:03:10] UTC, and it appears to start successfully (but again the bundle does not have enough logs for me to conclusively say so).

If you hit the problem again, we would really appreciate if you could get the diagnostic bundle ASAP and upload it.

Thanks!

[ctalledo]

Hi @NiklasBr,

One thing I did notice from the Docker Desktop bundle you uploaded: there seems to be some disk corruption in the Docker Desktop VM, because I can see that whenever that VM is started (either when starting Docker Desktop or after exiting resource saver mode), it's taking up to ~15 secs to do a filesystem consistency check (fsck):

[2024-10-16T08:06:09.023621507Z][init.disk             ] initial fsck ran in 15.178512132s

Normally that check should take milliseconds on a healthy filesystem, even one with dozens of GBs of data.

I don't know what could have caused the corruption, but you can try uninstalling and re-installing Docker Desktop which will give you a fresh new VM.

[djs55]

Hi @NiklasBr ,

In case you see the problem again, could you try this developer build: https://desktop-stage.docker.com/mac/main/arm64/173899/Docker.dmg ? It has increased logging around fsck, so if you upload diagnostics we might learn some more.

Thanks!

[NiklasBr]

I did uninstall and then download Docker again, it was a bit strange that the version downloaded by default on docker.com was 4.34 which was even worse then 4.35 in terms of stability, so had to download the update manually.

How long time do I have from the moment it happens until the window of opportunity ends with regards to collecting logs?

[ctalledo]

Hi @NiklasBr,

he version downloaded by default on docker.com was 4.34

Yes there's a controlled rollout of 4.35, so sometimes the download link still points to 4.34.

How long time do I have from the moment it happens until the window of opportunity ends with regards to collecting logs?

Within a few hours it should be fine, thanks!

[dhlavaty]

Still an issue in Docker Desktop for Mac 4.37.2 (179585) ; Engine: 27.4.0 ; Compose: v2.31.0-desktop.2 😢

[ctalledo]

Hi @dhlavaty, if possible please upload a Docker Desktop diagnostics bundle so we can take a look.

In general the Docker Desktop resource saver feature is quite stable, but there are always scenarios where things don't work as expected, sometimes due to a bug in Docker Desktop but also sometimes due to a problem with the underlying host. A diagnostics bundle helps us identify and resolve the issue.

Thanks!