Unable to update docker (with watchtower)

[stanthewizzard]

Hello

Apparently there is an issue with buildx

You should pin the buildx version to v0.9.1 to fix your images (Example: cilium/cilium#23206)

Thanks

[rdwebdesign]

We don't recommend the use of Watchtower or any other automated update:

https://github.com/pi-hole/docker-pi-hole#note-on-watchtower

[dschaper]

Apparently there is an issue with buildx

How does this apply to our build process? What is the benefit of doing it, and what is the danger of not doing it?

[stanthewizzard]

this affect the ability to update in some way pi-hole when dockered.
git pull is not affected btw

[PromoFaux]

Does this affect normal updates? (In my experience... No, as I've updated between images with no issues) or is it just watchtower updates?

If it's the latter, I don't know how worried I am about this... One should not be updating Pi-hole containers with watchtower, ever...

[rdwebdesign]

I just updated my own installation from 2023.01.3 to 2023.01.5 without issues.

This is not about a problem with Pi-hole.

[nabeelr]

I just spent all day trying to figure this out.

So it is affecting a number of people. Probably good to keep an eye on it even if there is nothing you guys can do. It helps people who are searching for an answer.

[PromoFaux]

Are you able to manually update the image?

[nabeelr]

so I used docker desktop and docker for windows to pull the image, save it, and then manually copy it to the appliance I'm running containers on. The appliance is getting 404s for pihole still. Other containers work though.

[PromoFaux]

At this stage, I don't really understand the problem. I gather that those using watchtower are unable to update via watchtower (which is something we do not recommend for this image anyway, as mentioned in the readme), but what has changed in buildx to cause this issue?

Where do the 404's come into it? If you docker pull pihole/pihole on the host you have Pi-hole image running on... what happens?

To refer back to @dschaper's question:

How does this apply to our build process? What is the benefit of doing it, and what is the danger of not doing it?

Not averse to making changes, but I need to understand exactly why changes need to be made, and what benefit it will bring. Is there an upstream issue with buildx that's being tracked? I don't want to hace to pin to a specific version of that forever...

[nabeelr]

That makes two of us... I wish I could get more substantial logs out of the appliance to be able to shed some light on the situation, but if there is a way, I don't know how. I don't even know if it's using watchtower under the hood, or just exhibiting the same symptoms.

The appliance has it's own interface for pulling down docker containers from https://registry-1.docker.io/, and when you specify a container of pihole/pihole:latest it fails to download and extract, and spits out in the logs error response getting manifests: 404

For what it's worth, this seems to have only started happening in the last 10 or so hours, and was definitely working fine this morning.

In my case, I'm running it on a MikroTik CCR2116-12G-4S+ so it's pretty locked down with no traditional shell to fall back to. (that I know of anyways)

[jsermer]

I am also having issues pulling the 2023.01.4 and 2023.01.5 tags directly via podman on an arm64 system (2023.01.3 worked fine however). NOTE: I am not using watchtower.

It's interesting that using docker (not podman) to pull all the image tags referenced above work properly from a different x86_64 system. I do suspect it has something to do with the github docker builders defaulting to buildx v0.10.0 as stated above but I have not dug into exactly why that is. It might be worth a try pinning the buildx version to v0.9.1 in this repo's github workflow yaml.

I have inspected the image and it does seem structurally different in versions > 2023.01.3 and this upstream issue also seems to related: docker/setup-buildx-action#187

[stanthewizzard]

From
docker/buildx#1513 (comment)

This is a build option so in build-push-action:

  -
    name: Build and push
    uses: docker/build-push-action@v3
    with:
      push: true
      provenance: false
      tags: user/app:latest

Or if you invoke buildx directly then docker buildx build --provenance false ....

[PromoFaux]

Yeah, came across that earlier when reading through the issues. I will give it a go when I get to a computer.

It seems a better option than just pinning to an old version

[PromoFaux]

From : containrrr/watchtower#1528 (comment)

They don't want to do anything apparently because this is not a pi-hole bug.

That's not strictly true, and I think unfairly misrepresents what I've said. I have also said I will try adding disabling provenance . I wanted to get a handle on everything before we made rash changes and pinning things without first understanding what the issue was. (Please have a little patience, we're not all in the same time zone!!)

And to quote @crazy-max:

Also OCI images are around for a few years now so I think Watchtower should support them instead.

Agree with this. But I will always always always maintain though that people should not be keeping Pi-hole (a critical part of one's network) automatically updated with Watchtower. Watchtower is fine in principle... just don't use it for Pi-hole. We can and will push breaking changes (we try not to, but sometimes it just has to happen) - imagine that bringing down your whole network because you let it just upgrade itself in the middle of the night/when you were away. To repeat a link shared earlier: https://github.com/pi-hole/docker-pi-hole#note-on-watchtower

However this issue appears at the surface level to be affecting more than just users using watchtower - as there are reports of podman and a MikroTik appliance. So lets see what happens.... if disabling provenance causes no harm, then we can do that

[PromoFaux]

please try the :dev tag

[jsermer]

I agree disabling provenance is a better alternative to pinning the actual version. I saw this referenced in a few places after I had commented here. The dev tag appears to work as well. Here's the before and after (btw, my arm64 device is a UDM-Pro running the 1.x release which the native podman version which is quite old):

Model:       UniFi Dream Machine Pro
Version:     1.12.37

[UDM] root@udmpro:~# podman version
Version:            1.6.1
RemoteAPI Version:  1
Go Version:         go1.12.10
OS/Arch:            linux/arm64

[UDM] root@udmpro:~# podman pull pihole/pihole:2023.01.5
Trying to pull docker.io/pihole/pihole:2023.01.5...
  manifest unknown: OCI index found, but accept header does not support OCI indexes
Trying to pull quay.io/pihole/pihole:2023.01.5...
  unauthorized: access to the requested resource is not authorized
Trying to pull registry.fedoraproject.org/pihole/pihole:2023.01.5...
  manifest unknown: manifest unknown
Error: error pulling image "pihole/pihole:2023.01.5": unable to pull pihole/pihole:2023.01.5: 3 errors occurred:
	* Error initializing source docker://pihole/pihole:2023.01.5: Error reading manifest 2023.01.5 in docker.io/pihole/pihole: manifest unknown: OCI index found, but accept header does not support OCI indexes
	* Error initializing source docker://quay.io/pihole/pihole:2023.01.5: Error reading manifest 2023.01.5 in quay.io/pihole/pihole: unauthorized: access to the requested resource is not authorized
	* Error initializing source docker://registry.fedoraproject.org/pihole/pihole:2023.01.5: Error reading manifest 2023.01.5 in registry.fedoraproject.org/pihole/pihole: manifest unknown: manifest unknown


[UDM] root@udmpro:~# podman pull pihole/pihole:dev
Trying to pull docker.io/pihole/pihole:dev...
Getting image source signatures
Copying blob f230e18c4cac skipped: already exists
Copying blob 934ce60d1040 skipped: already exists
Copying blob 4f4fb700ef54 skipped: already exists
Copying blob a5c0dd3e9bdb done
Copying blob b0f78f1e9e5a done
Copying blob f1d94ae4ea31 done
Copying blob d1d4d39c975e done
Copying blob 420f05f8792e done
Copying blob b21fdce46b18 done
Copying config 973403f35a done
Writing manifest to image destination
Storing signatures
973403f35a640f6f623f2247f6482f9fa6ec90e7dad2700b205992a856daedf1

[PromoFaux]

Thanks for the feedback - I will tag a new version

[stanthewizzard]

From : containrrr/watchtower#1528 (comment)

They don't want to do anything apparently because this is not a pi-hole bug.

That's not strictly true, and I think unfairly misrepresents what I've said. I have also said I will try adding disabling provenance . I wanted to get a handle on everything before we made rash changes and pinning things without first understanding what the issue was. (Please have a little patience, we're not all in the same time zone!!)

And to quote @crazy-max:

Also OCI images are around for a few years now so I think Watchtower should support them instead.

Agree with this. But I will always always always maintain though that people should not be keeping Pi-hole (a critical part of one's network) automatically updated with Watchtower. Watchtower is fine in principle... just don't use it for Pi-hole. We can and will push breaking changes (we try not to, but sometimes it just has to happen) - imagine that bringing down your whole network because you let it just upgrade itself in the middle of the night/when you were away. To repeat a link shared earlier: https://github.com/pi-hole/docker-pi-hole#note-on-watchtower

However this issue appears at the surface level to be affecting more than just users using watchtower - as there are reports of podman and a MikroTik appliance. So lets see what happens.... if disabling provenance causes no harm, then we can do that

That was my understanding. My mistake. Good to see that your take the issue. Thank you thank you :)

[stanthewizzard]

And thank you for the update in the git
It works flawlessly

[nabeelr]

So I just tried :dev and :latest and they both worked.

Not sure if that's expected or not, but I'm pretty happy about it!

If it is expected, thanks very much! If it's not, please let me know if I can help ding in deeper to figure out what's going on now.

[PromoFaux]

It's expected #1290

[nabeelr]

Amazing! Thanks team! Much appreciated!