fuse-overlayfs is disabled but still being run by podman

[Hi-Angel]

/kind bug

Description

As part of this issue, it turned out that disabling fuse-overlayfs, so it is no longer appears in podman info output, will not make podman stop using it.

Steps to reproduce the issue:

1. Make sure to have the following ~/.config/containers/storage.conf:

   [storage]
   
   # Default Storage Driver
   driver = "overlay"
   
   [storage.options]
   # Path to a helper program to use for mounting the file system instead of mounting it
   # directly.
   mount_program = ""
   

2. Execute a podman info | grep fuse and make sure there's no output

3. In a separate terminal execute a command that will print newly spawning processes: bpftrace -e 'tracepoint:syscalls:sys_enter_exec*{ printf("pid: %d, comm: %s, args: ", pid, comm); join(args->argv); }'

4. Execute podman run --rm ubuntu:20.04 true

5. Look at the new processes in the command running since point 3

Describe the results you received:

One of new processes is a call to /usr/bin/fuse-overlayfs

Describe the results you expected:

No fuse-overlayfs should appear in the output.

Output of podman version:

Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
Go Version:   go1.18.1
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Output of podman info:

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2:2.1.5-0ubuntu22.04+obs14.3_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: '
  cpuUtilization:
    idlePercent: 96.22
    systemPercent: 0.76
    userPercent: 3.03
  cpus: 4
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: file
  hostname: node29
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 998
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 998
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
  kernel: 5.15.0-52-generic
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1288302592
  memTotal: 67404197888
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.7-0ubuntu22.04+obs47.1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.7
      commit: 40d996ea8a827981895ce22886a9bac367f87264
      rundir: /run/user/998/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/998/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-0ubuntu22.04+obs10.15_amd64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 8575172608
  swapTotal: 8589930496
  uptime: 223h 19m 11.00s (Approximately 9.29 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/gitlab-runner/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/gitlab-runner/.local/share/containers/storage
  graphRootAllocated: 983350071296
  graphRootUsed: 645746360320
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 202
  runRoot: /tmp/podman-run-998/containers
  volumePath: /home/gitlab-runner/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.1
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Package info:

$ apt list podman
Listing... Done
podman/unknown,now 4:4.3.1-0ubuntu22.04+obs64.3 amd64 [installed]
podman/unknown 4:4.3.1-0ubuntu22.04+obs64.3 arm64
podman/unknown 4:4.3.1-0ubuntu22.04+obs64.3 armhf
podman/unknown 4:4.3.1-0ubuntu22.04+obs64.3 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

[Luap99]

Did you run podman system reset? If you change storage options it is almost always required to do that.

[Hi-Angel]

Did you run podman system reset? If you change storage options it is almost always required to do that.

No, but the problem here is that podman info claims that the mount_program is no longer used. Which is untrue. If running podman system reset is required, then podman info should not claim that change in settings succeded.

[giuseppe]

fuse-overlayfs and native overlay use a slightly different format so if you've images created for fuse-overlays they won't work with native overlay. So once fuse-overlayfs is used, it won't be turned off unless you reset the storage.

podman info reflects the current configuration, but it is not aware if the configuration is changed later on because the storage has images in a format usable by fuse-overlayfs

[Hi-Angel]

Oh, I see, thanks! I think in this case it might be useful to add to man podman-info in "DESCRIPTION" paragraph a sentence Note: some configuration changes displayed here will only affect newly created images and will not affect pre-existing ones. What do you think? I could send a PR adding it.

[rhatdan]

@giuseppe do we know at the time of running podman-info that a fuse-overlay was already run? Could we print a warning stating the the storage still requires fuse-overlay, when fuse-overlay is no longer specifed.

[giuseppe]

that is now handled internally by containers/storage. We would either need to expose some information to podman or let podman query directly the storage and check whether the file .has-mount-program is present.

Could we add a debug message perhaps in c/storage (if it is not already there)?

[rhatdan]

Should we add the warning flag there when we check, IE Check if we we have .has-mount-program, yes?
If so and not overlay+mount_program logrus.Warn

[giuseppe]

we fall back to fuse-overlayfs when a mount program is not configured and native overlay is not supported.

If we make it a warning, we risk triggering the warning every time an existing setup that is using fuse-overlayfs is moved to a newer kernel and we'll have the opposite problem.

I think a debug message, as we already have, is enough.

Please reopen if you disagree