Podman auto-update with --pull=always fails

[andrew-kennedy]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I have setup a few rootless containers using systemd with the auto update label, and they fail to update using podman auto-update with errors such as the following.

Steps to reproduce the issue:

1. Create various containers in a pod.

2. Use podman generate to create rootless systemd units to start these containers.

3. Start the containers using systemctl --user start pod-test-pod.service

4. Attempt to podman auto-update

Describe the results you received:
I received many errors like this for each container:

	* error auto-updating container "e322c98940c218419dde53384aaf0c639a430b049d2e33b59613dca9e644d7d5": image check for "90a12b5c957bab94f57aadfd9e655d414ab69443495f22ef430152181ab0aede" failed: invalid repository name (90a12b5c957bab94f57aadfd9e655d414ab69443495f22ef430152181ab0aede), cannot specify 64-byte hexadecimal strings

Describe the results you expected:

I expected the containers to auto update.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      2.0.4
API Version:  1
Go Version:   go1.14.6
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64```

**Output of `podman info --debug`:**

host:
arch: amd64
buildahVersion: 1.15.0
cgroupVersion: v2
conmon:
package: conmon-2.0.19-1.fc32.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.0.19, commit: 5dce9767526ed27f177a8fa3f281889ad509fea7'
cpus: 24
distribution:
distribution: fedora
version: "32"
eventLogger: file
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 5.7.15-200.fc32.x86_64
linkmode: dynamic
memFree: 4345331712
memTotal: 33604354048
ociRuntime:
name: crun
package: crun-0.14.1-1.fc32.x86_64
path: /usr/bin/crun
version: |-
crun version 0.14.1
commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
os: linux
remoteSocket:
exists: true
path: /run/user/1000/podman/podman.sock
rootless: true
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.1.4-1.fc32.x86_64
version: |-
slirp4netns version 1.1.4
commit: b66ffa8e262507e37fca689822d23430f3357fe8
libslirp: 4.3.1
SLIRP_CONFIG_VERSION_MAX: 2
swapFree: 8582590464
swapTotal: 8589930496
uptime: 38h 4m 54.98s (Approximately 1.58 days)
registries:
search:

• registry.fedoraproject.org
• registry.access.redhat.com
• registry.centos.org
• docker.io
  store:
  configFile: /home/andrew/.config/containers/storage.conf
  containerStore:
  number: 17
  paused: 0
  running: 17
  stopped: 0
  graphDriverName: overlay
  graphOptions:
  overlay.mount_program:
  Executable: /usr/bin/fuse-overlayfs
  Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
  Version: |-
  fusermount3 version: 3.9.1
  fuse-overlayfs: version 1.1.0
  FUSE library version 3.9.1
  using FUSE kernel interface version 7.31
  graphRoot: /home/andrew/.local/share/containers/storage
  graphStatus:
  Backing Filesystem: extfs
  Native Overlay Diff: "false"
  Supports d_type: "true"
  Using metacopy: "false"
  imageStore:
  number: 21
  runRoot: /run/user/1000/containers
  volumePath: /home/andrew/.local/share/containers/storage/volumes
  version:
  APIVersion: 1
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.14.6
  OsArch: linux/amd64
  Version: 2.0.4```

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.0.4-1.fc32.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Fedora 32 Physical server.

[andrew-kennedy]

As a note, I've been looking into more docs about auto-update, and I am guessing it's a bug because I created the systemd units using the --name flag, so they aren't based on container hashes. This seems like it should be a supported use case.

[Luap99]

How did you create the container? podman inspect --format "{{ .Config.CreateCommand }}" <container>
Did you use --new flag for podman generate systemd?

[andrew-kennedy]

/usr/bin/podman run --conmon-pidfile /run/user/1000/container-caddy.pid --cidfile /run/user/1000/container-caddy.ctr-id --cgroups=no-conmon --pod-id-file /run/user/1000/pod-web-services.pod-id --name=caddy --pull=always --label io.containers.autoupdate=image --detach --restart=on-failure:10 --memory=256m -e TZ=America/Los_Angeles -v /home/andrew/services/caddy/data:/data:Z -v /home/andrew/services/caddy/config:/config:Z -v /home/andrew/services/caddy/caddyfile:/etc/caddy:Z docker.io/caddy

I created the systemd units with --new, yeah.

[vrothberg]

Hi @andrew-kennedy, thanks for opening the issue and sorry for the long silence. I was on PTO and the issue must have slipped through the inbox of other maintainers.

The last comment seems to be truncated (see docker.io/caddy). Can you repaste the full command? It seems that auto-update believes 90a12b5c957bab94f57aadfd9e655d414ab69443495f22ef430152181ab0aede is the image name.

[andrew-kennedy]

It's not truncated actually. I created these containers from the generated systemd units and the output above is still what I get from the CreateCommand podman inspect query.

[andrew-kennedy]

I think I see the problem. Podman auto-update seems to be using the Image field instead of ImageName. For example, for caddy they have these values:

"Image": "90a12b5c957bab94f57aadfd9e655d414ab69443495f22ef430152181ab0aede",
"ImageName": "docker.io/library/caddy:latest",

and the auto-uodate error mentions that image check for image 90a12b5c957bab94f57aadfd9e655d414ab69443495f22ef430152181ab0aede failed, that it couldn't find a repository with that name. I think it should be using ImageName instead during these operations.

[vrothberg]

Thanks! That should be enough information to get it fixed 👍

[vrothberg]

I can only reproduce when the container has been created with the image ID (i.e., 90a12b5c957bab94f57aadfd9e655d414ab69443495f22ef430152181ab0aede) instead of the name (i.e. docker.io/caddy).

Could you try again from scratch and make sure that names are used instead of IDs?

[andrew-kennedy]

weirdly enough, the containers are created "automatically" via generated systemd unit files which are themselves created via podman generate systemd --new --files --name caddy, which I generate after I create containers myself by image name.

So the specific failing flow is:

podman create \
--name=caddy \
--pull=always \
--label io.containers.autoupdate=image \
--detach \
--restart=on-failure:10 \
--memory=256m \
-e TZ=America/Los_Angeles \
-v ~/services/caddy/data:/data:Z \
-v ~/services/caddy/config:/config:Z \
-v ~/services/caddy/caddyfile:/etc/caddy:Z \
docker.io/caddy

and then

cd ~/.config/systemd/user
podman generate systemd --files --new --name caddy

after which I start those unit files/containers using systemctl --user start container-caddy.service, and then finally try to auto update with podman auto-update. Everything goes swimmingly until the auto-update part, where I get the above mentioned errors.

If you need the generated systemd unit files and the commands in them, I can add those here too.

[vrothberg]

To be 100 percent sure. Did you do a systemctl --user daemon-reload before starting the service?

[andrew-kennedy]

Sorry, left that step out of my summary, but yes. Had to I believe, because otherwise systemctl thinks it can't find the unit file. Here is the generated unit file for good measure:

# container-caddy.service
# autogenerated by Podman 2.0.5
# Tue Sep  8 04:50:15 PDT 2020

[Unit]
Description=Podman container-caddy.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStartPre=/bin/rm -f %t/container-caddy.pid %t/container-caddy.ctr-id
ExecStart=/usr/bin/podman run --conmon-pidfile %t/container-caddy.pid --cidfile %t/container-caddy.ctr-id --cgroups=no-conmon --name=caddy --pull=always --label io.containers.autoupdate=image --detach --restart=on-failure:10 --memory=256m docker.io/caddy
ExecStop=/usr/bin/podman stop --ignore --cidfile %t/container-caddy.ctr-id -t 10
ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile %t/container-caddy.ctr-id
PIDFile=%t/container-caddy.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

[vrothberg]

Thanks, @andrew-kennedy! I can reproduce with the service file you provided. Will take a close look now :)

[vrothberg]

I found the issue. It's related to --pull=always changing the recorded "raw" image specified by the user.