Quadlet tries to open unexisting overlays when using UserNS=keep-id

[jgottlander]

Issue Description

I just changed my home server and copied all my .container files. Compiled the latest podman with dependencies and Imported all the volumes.
The thing is, one day my containers didn't start when I rebooted the computer. I tested podman machine reset and redid everything. I got emby, lyrion and gluetun + *arr servers up and running without any problem. But homeassistant and syncthing is getting an error when I start them with systemd. If use the same config and start them with podman run they start without problem.

journalctl says Permission denied at some overlay. But when I check, the named overlay doesn't exist in .local/share/containers/storage/overlays.
I realized that the problem is UserNS-keep-id. If I remove that the container starts as expected.

Steps to reproduce the issue

Steps to reproduce the issue

1. Create .container file for syncthing:
   [Unit] Description=File synchronization tool Wants=network-online.target After=network-online.target local-fs.target [Container] Image=syncthing/syncthing:latest UserNS=keep-id Volume=syncthing_data:/var/syncthing Volume=/mnt/externa/Data:/data:rw Network=host [Service] Restart=on-failure TimeoutStartSec=900 [Install] WantedBy=default.target
2. Reload systemd and start the container with systemctl --user start syncthing.

Describe the results you received

The container fails to start.
When I check the logs I get this:
2267 Feb 22 21:48:24 omv systemd[765]: Stopped syncthing.service - File synchronization tool. 2268 Feb 22 21:48:24 omv systemd[765]: Starting syncthing.service - File synchronization tool... 2269 Feb 22 21:48:24 omv podman[3558]: 2025-02-22 21:48:24.462066345 +0100 CET m=+0.034701321 container create a8048e947a392098caba9f6c0377c3018b69d12f8376b5b7bb70cbfb03091000 (image=docker.io/syncthing/syncthing:latest, name=systemd-syncthing, org.opencontainers.image.title=Syncthing, org.opencontainers.image.licenses=MPL-2.0, org.opencontainers.image.source=https://github.com/syncthing/syncthing, PODMAN_SYSTEMD_UNIT=syncthing.service, org.opencontainers.image.authors=The Syncthing Project, org.opencontainers.image.documentation=https://docs.syncthing.net, org.opencontainers.image.vendor=The Syncthing Project, org.opencontainers.image.version=1.29.2, org.opencontainers.image.revision=516f3e29e8cc7091ea6271715308caea0fcc0778, org.opencontainers.image.url=https://syncthing.net) 2270 Feb 22 21:48:24 omv conmon[3570]: conmon a8048e947a392098caba <nwarn>: runtime stderr: open /home/XXX/.local/share/containers/storage/overlay/3f28981bb80678745b295785e62a3808eba3af724ec1fc9695d74b583cf0b722/merged: Permission denied 2271 Feb 22 21:48:24 omv conmon[3570]: conmon a8048e947a392098caba <error>: Failed to create container: exit status 1 2272 Feb 22 21:48:24 omv podman[3558]: 2025-02-22 21:48:24.540856557 +0100 CET m=+0.113491536 container remove a8048e947a392098caba9f6c0377c3018b69d12f8376b5b7bb70cbfb03091000 (image=docker.io/syncthing/syncthin:latest, name=systemd-syncthing, org.opencontainers.image.authors=The Syncthing Project, org.opencontainers.image.url=https://syncthing.net, org.opencontainers.image.documentation=https://docs.syncthing.net, org.opencontainers.image.source=https://github.com/syncthing/syncthing, org.opencontainers.image.version=1.29.2, org.opencontainers.image.title=Syncthing, PODMAN_SYSTEMD_UNIT=syncthing.service, org.opencontainers.image.licenses=MPL-2.0, org.opencontainers.image.revision=516f3e29e8cc7091ea6271715308caea0fcc0778, org.opencontainers.image.vendor=The Syncthing Project) 2273 Feb 22 21:48:24 omv podman[3558]: 2025-02-22 21:48:24.446801007 +0100 CET m=+0.019435984 image pull 78fbac91e804182ef1b9538d1a796e2a3199b095e6ee196bdabf5e7c654ff3b1 syncthing/syncthing:latest 2274 Feb 22 21:48:24 omv syncthing[3558]: Error: crun: open /home/XXX/.local/share/containers/storage/overlay/3f28981bb80678745b295785e62a3808eba3af724ec1fc9695d74b583cf0b722/merged: Permission denied: OCI permission denied 2275 Feb 22 21:48:24 omv systemd[765]: syncthing.service: Main process exited, code=exited, status=126/n/a 2276 Feb 22 21:48:24 omv systemd[765]: syncthing.service: Killing process 3570 (conmon) with signal SIGKILL. 2277 Feb 22 21:48:24 omv systemd[765]: syncthing.service: Killing process 3574 (podman) with signal SIGKILL. 2278 Feb 22 21:48:24 omv systemd[765]: syncthing.service: Killing process 3577 (podman) with signal SIGKILL. 2279 Feb 22 21:48:24 omv systemd[765]: syncthing.service: Killing process 3579 (n/a) with signal SIGKILL. 2280 Feb 22 21:48:24 omv systemd[765]: syncthing.service: Killing process 3580 (podman) with signal SIGKILL. 2281 Feb 22 21:48:24 omv systemd[765]: syncthing.service: Failed with result 'exit-code'. 2282 Feb 22 21:48:24 omv systemd[765]: Failed to start syncthing.service - File synchronization tool. 2283 Feb 22 21:48:24 omv systemd[765]: syncthing.service: Scheduled restart job, restart counter is at 6.

Describe the results you expected

I expected it to run without problems.

podman info output

host:
  arch: amd64
  buildahVersion: 1.39.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/local/libexec/podman/conmon
    version: 'conmon version 2.1.12, commit: 41e2c0dc06248ff23f67b6b8c0c03ac34bff2ceb'
  cpuUtilization:
    idlePercent: 91.36
    systemPercent: 2.01
    userPercent: 6.62
  cpus: 4
  databaseBackend: sqlite
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: journald
  freeLocks: 2016
  hostname: omv
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.12.9+bpo-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 272687104
  memTotal: 7991504896
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
    package: netavark_1.4.0-3_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/local/bin/crun
    version: |-
      crun version 1.20
      commit: 9c9a76ac11994701dd666c4f0b869ceffb599a66
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20230309.7c7625d-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 3995070464
  swapTotal: 3995594752
  uptime: 0h 23m 9.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - docker.io
store:
  configFile: /home/josef/.config/containers/storage.conf
  containerStore:
    number: 19
    paused: 0
    running: 13
    stopped: 6
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/josef/.local/share/containers/storage
  graphRootAllocated: 117019152384
  graphRootUsed: 77525331968
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 16
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/josef/.local/share/containers/storage/volumes
version:
  APIVersion: 5.4.0
  Built: 1740257908
  BuiltTime: Sat Feb 22 21:58:28 2025
  GitCommit: f9f7d48b24b1ca4403f189caaeab1cb8ff4a9aa2
  GoVersion: go1.24.0
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Running on Lenovo ThinkCentre M900

Additional information

It only happens if I add UserNS=keep-id to .container.