Latest Docker upgrade crashes addons with Failed communication to port 80

[Lefuneste83]

OS Version

Debian GNU/Linux 12 (bookworm)

System Information

Linux hassio 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux

What happened?

Following recent docker upgrade published this day, Home Assistant Supervised stack is unable to start. Most container report communication error with port 80.
The upgrade concerns the following packets.

docker-buildx-plugin/bookworm 0.12.1-1~debian.12~bookworm amd64 [pouvant être mis à jour depuis : 0.11.2-1~debian.12~bookworm]
docker-ce-cli/bookworm 5:25.0.0-1~debian.12~bookworm amd64 [pouvant être mis à jour depuis : 5:24.0.7-1~debian.12~bookworm]
docker-ce-rootless-extras/bookworm 5:25.0.0-1~debian.12~bookworm amd64 [pouvant être mis à jour depuis : 5:24.0.7-1~debian.12~bookworm]
docker-ce/bookworm 5:25.0.0-1~debian.12~bookworm amd64 [pouvant être mis à jour depuis : 5:24.0.7-1~debian.12~bookworm]
docker-compose-plugin/bookworm 2.24.1-1~debian.12~bookworm amd64 [pouvant être mis à jour depuis : 2.21.0-1~debian.12~bookworm]

After the upgrade, the stack restarts completely but most addons containers are in a restart loop with the following error message.

Failed to connect to supervisor port 80 after xxx ms: Couldn't connect to server

Also Home Assistant observer page on port 4357 is filled with error message complaining about connectivity to Supervisor.

No other process is running on port 80 as it is a dedicated HA system.

Purging all components installed in the supervised procedure, including docker and affiliates, redownloading the latest components and reinstalling them does not correct the issue.

Also tried to docker prune containers and images of some crashed addons, but issue still persists at container reinstantiation.

Note that the installation procedure runs without issues and not a single error.

I've had a look at the apparmor settings across my system but cannot find any issue.

I will update the ticket as soon as I reproduce the issue as I have had to roll back the system to a previous backup.

Machine Type

generic-x86-64

Installer output

No response

Relevant log output

No response

ADR

• I have read through the ADR and have confirmed that my system is compliant with the requirements
• I understand that if my system is found to not be compliant, my issue will be closed immediately without further investigation

Code of Conduct

• I have read and agree to the Home Assistant Code of Conduct

[Zuendapp]

Hi @Lefuneste83,

I had the same problem a few hours ago.
I downgraded to the previous docker packages which were installed before the apt-update/apt-upgrade on my pi4.

You have to show the update history, via:

less /var/log/apt/history.log

There you can see what package and what version was installed before.

Now you have to check for each package, which versions are still available in the bookworm repo, for example, with:

apt list docker-compose-plugin -a

Now you only have to install the previous version, what was installed before the update for each package, for example via:

apt install docker-compose-plugin=2.21.0-1~debian.12~bookworm

After downgrading of all packages I restarted my raspberrypi and alle worked as before!

BR :)

[Lefuneste83]

Thanks for the feedback! So we do have another regression with docker... I hate when this occurs as it will kill a perfectly running HA instance in a breeze...

[shirou93]

Same issue. I Downgraded and waiting for supervisor fix :)

[ZeBen84]

Debian GNU/Linux 12 (bookworm)

Same, i had to remove all docker packages:
sudo apt remove docker*

And re-install previous versions:
sudo apt install docker-compose-plugin=2.21.0-1~debian.12~bookworm docker-ce-cli=5:24.0.7-1~debian.12~bookworm docker-ce-rootless-extras=5:24.0.7-1~debian.12~bookworm docker-buildx-plugin=0.11.2-1~debian.12~bookworm docker-ce=5:24.0.7-1~debian.12~bookworm

I then pinned the current docker until there is a fix:
sudo echo -e "Package: docker-compose-plugin\nPin: version 2.21.0-1~debian.12~bookworm\nPin-Priority: 1000\n\nPackage: docker-ce-cli\nPin: version 5:24.0.7-1~debian.12~bookworm\nPin-Priority: 1000\n\nPackage: docker-ce-rootless-extras\nPin: version 5:24.0.7-1~debian.12~bookworm\nPin-Priority: 1000\n\nPackage: docker-buildx-plugin\nPin: version 0.11.2-1~debian.12~bookworm\nPin-Priority: 1000\n\nPackage: docker-ce\nPin: version 5:24.0.7-1~debian.12~bookworm\nPin-Priority: 1000" | sudo tee /etc/apt/preferences.d/docker Package: docker-compose-plugin

Then I rebooted my server and everything was good !

[shirou93]

Debian GNU/Linux 12 (bookworm)

Same, i had to remove all docker packages:
sudo apt remove docker*

And re-install previous versions:
sudo apt install docker-compose-plugin=2.21.0-1~debian.12~bookworm docker-ce-cli=5:24.0.7-1~debian.12~bookworm docker-ce-rootless-extras=5:24.0.7-1~debian.12~bookworm docker-buildx-plugin=0.11.2-1~debian.12~bookworm docker-ce=5:24.0.7-1~debian.12~bookworm

I then pinned the current docker until there is a fix:
sudo echo -e "Package: docker-compose-plugin\nPin: version 2.21.0-1~debian.12~bookworm\nPin-Priority: 1000\n\nPackage: docker-ce-cli\nPin: version 5:24.0.7-1~debian.12~bookworm\nPin-Priority: 1000\n\nPackage: docker-ce-rootless-extras\nPin: version 5:24.0.7-1~debian.12~bookworm\nPin-Priority: 1000\n\nPackage: docker-buildx-plugin\nPin: version 0.11.2-1~debian.12~bookworm\nPin-Priority: 1000\n\nPackage: docker-ce\nPin: version 5:24.0.7-1~debian.12~bookworm\nPin-Priority: 1000" | sudo tee /etc/apt/preferences.d/docker Package: docker-compose-plugin

Then I rebooted my server and everything was good !

No need remove packages. Enough install only specific version and debian overwrite it.

[macakzsolt]

Thanks for the feedback! That's what happened to me too. Downgraded and waiting for supervisor fix.

[agners]

This issue was caused by a bug in upstream Docker 25.0.0. 25.0.1 fixed this problem. I am closing the issue as there is nothing to be done on Supervisor or Supervisor installer side.

Thanks for reporting and help getting to the bottom of this!