e2e tests: split imagestore test failed when setting STORAGE_FS=overlay #19748

sstosh · 2023-08-25T02:17:41Z

Issue Description

split imagestore test failed with the error message such as below.

Error: creating container storage: creating read-write layer with ID "xxx": Stat /tmp/podman_testxxx/root/overlay/xxx/diff: no such file or directory

This error looks like a bug.
However, I have not reproduced the error manually.

Steps to reproduce the issue

Run e2e tests with STORAGE_FS=overlay.

# FOCUS="podman pull and run on split imagestore" STORAGE_FS=overlay make localintegration

Describe the results you received

Error log

  > Enter [It] podman pull and run on split imagestore - /root/podman/test/e2e/pull_test.go:65 @ 08/25/23 10:58:22.034
  Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay build --pull=never --tag splitstoretest build/basicalpine
  STEP 1/1: FROM quay.io/libpod/alpine:latest
  COMMIT splitstoretest
  --> 961769676411
  Successfully tagged localhost/splitstoretest:latest
  Successfully tagged quay.io/libpod/alpine:latest
  961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
  Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay save -o /tmp/podman_test3554367878/image.tar --format oci-archive splitstoretest
  Copying blob sha256:03901b4a2ea88eeaad62dbe59b072b28b6efa00491962b8741081c5df50c65e0
  Copying config sha256:4819930274ea2bd6ab611972f9928e07b84edcbfe1d2462fb2941957308dfae2
  Writing manifest to image destination
  Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay rmi splitstoretest
  Untagged: localhost/splitstoretest:latest
  Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay load --imagestore /tmp/podman_test3554367878/splitstore -q -i /tmp/podman_test3554367878/image.tar
  Loaded image: localhost/splitstoretest:latest
  Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay tag quay.io/libpod/busybox:latest busybox-test
  Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay images --imagestore /tmp/podman_test3554367878/splitstore
  REPOSITORY                                 TAG         IMAGE ID      CREATED        SIZE
  quay.io/libpod/registry                    2.8.2       0030ba3d620c  2 weeks ago    24.6 MB
  registry.fedoraproject.org/fedora-toolbox  36          5be0bf1ecd38  4 months ago   874 MB
  quay.io/libpod/systemd-image               20230106    4f6126057b5e  7 months ago   132 MB
  quay.io/libpod/alpine_nginx                latest      ecea49d99daa  16 months ago  23.1 MB
  registry.fedoraproject.org/fedora-minimal  34          b85bdf5fbb1c  18 months ago  119 MB
  quay.io/libpod/alpine_labels               latest      5e9e9275e4d6  18 months ago  5.87 MB
  quay.io/libpod/redis                       alpine      8835e6aeca99  2 years ago    32.3 MB
  quay.io/libpod/busybox                     latest      f0b02e9d092d  2 years ago    1.45 MB
  quay.io/libpod/busybox                     latest      f0b02e9d092d  2 years ago    1.45 MB
  localhost/busybox-test                     latest      f0b02e9d092d  2 years ago    1.45 MB
  registry.k8s.io/pause                      3.2         80d28bedfe5d  3 years ago    686 kB
  localhost/splitstoretest                   latest      4819930274ea  4 years ago    5.85 MB
  quay.io/libpod/alpine                      latest      961769676411  4 years ago    5.85 MB
  quay.io/libpod/alpine                      latest      961769676411  4 years ago    5.85 MB
  quay.io/libpod/alpine_healthcheck          latest      c1103abb47d6  4 years ago    10.4 MB
  Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay rmi --imagestore /tmp/podman_test3554367878/splitstore busybox-test
  Untagged: localhost/busybox-test:latest
  Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay images
  REPOSITORY                                 TAG         IMAGE ID      CREATED        SIZE        R/O
  quay.io/libpod/registry                    2.8.2       0030ba3d620c  2 weeks ago    24.6 MB     true
  registry.fedoraproject.org/fedora-toolbox  36          5be0bf1ecd38  4 months ago   874 MB      true
  quay.io/libpod/systemd-image               20230106    4f6126057b5e  7 months ago   132 MB      true
  quay.io/libpod/alpine_nginx                latest      ecea49d99daa  16 months ago  23.1 MB     true
  registry.fedoraproject.org/fedora-minimal  34          b85bdf5fbb1c  18 months ago  119 MB      true
  quay.io/libpod/alpine_labels               latest      5e9e9275e4d6  18 months ago  5.87 MB     true
  quay.io/libpod/redis                       alpine      8835e6aeca99  2 years ago    32.3 MB     true
  quay.io/libpod/busybox                     latest      f0b02e9d092d  2 years ago    1.45 MB     false
  localhost/busybox-test                     latest      f0b02e9d092d  2 years ago    1.45 MB     false
  quay.io/libpod/busybox                     latest      f0b02e9d092d  2 years ago    1.45 MB     true
  registry.k8s.io/pause                      3.2         80d28bedfe5d  3 years ago    686 kB      true
  quay.io/libpod/alpine                      latest      961769676411  4 years ago    5.85 MB     false
  quay.io/libpod/alpine                      latest      961769676411  4 years ago    5.85 MB     true
  quay.io/libpod/alpine_healthcheck          latest      c1103abb47d6  4 years ago    10.4 MB     true
  Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay run --name test --rm splitstoretest echo helloworld
  time="2023-08-25T10:58:22+09:00" level=warning msg="The storage 'driver' option should be set in /tmp/podman_test3554367878/.config/containers/storage.conf. A driver was picked automatically."
  Error: creating container storage: creating read-write layer with ID "49199fe425879e87febccf0dcf1c03bbcd28e8aae4b367c9220966ac03b4a966": Stat /tmp/podman_test3554367878/root/overlay/03901b4a2ea88eeaad62dbe59b072b28b6efa00491962b8741081c5df50c65e0/diff: no such file or directory
  [FAILED] Expected
      <int>: 125
  to match exit code:
      <int>: 0
  In [It] at: /root/podman/test/e2e/pull_test.go:128 @ 08/25/23 10:58:22.607

  Full Stack Trace
    github.com/containers/podman/v4/test/e2e.glob..func72.5()
        /root/podman/test/e2e/pull_test.go:128 +0x102a
  < Exit [It] podman pull and run on split imagestore - /root/podman/test/e2e/pull_test.go:65 @ 08/25/23 10:58:22.607 (573ms)
  > Enter [AfterEach] TOP-LEVEL - /root/podman/test/e2e/common_test.go:119 @ 08/25/23 10:58:22.607
  Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay stop --all -t 0
  Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay pod rm -fa -t 0
  Running: /root/podman/bin/podman --storage-opt overlay.imagestore=/tmp/ginkgo2529636269/imagecachedir --root /tmp/podman_test3554367878/root --runroot /tmp/podman_test3554367878/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test3554367878 --events-backend file --db-backend boltdb --storage-driver overlay rm -fa -t 0
  < Exit [AfterEach] TOP-LEVEL - /root/podman/test/e2e/common_test.go:119 @ 08/25/23 10:58:22.702 (95ms)

Describe the results you expected

This test should succeed.

podman info output

host:
  arch: amd64
  buildahVersion: 1.32.0-dev
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 99.98
    systemPercent: 0.01
    userPercent: 0.01
  cpus: 36
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: server
    version: "38"
  eventLogger: journald
  freeLocks: 2047
  hostname: fedora38
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.2.9-300.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 4698464256
  memTotal: 8301707264
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.7.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.7.0
    package: netavark-1.7.0-1.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: Unknown
    path: /usr/local/bin/crun
    version: |-
      crun version 1.8.1.0.0.0.16-7be9e-dirty
      commit: d7ee5496991c6ad40be6dd94796643cb0def9e3c
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230625.g32660ce-1.fc38.x86_64
    version: |
      pasta 0^20230625.g32660ce-1.fc38.x86_64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8257269760
  swapTotal: 8301572096
  uptime: 19h 33m 7.00s (Approximately 0.79 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 106229137408
  graphRootUsed: 17473433600
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.7.0-dev
  Built: 1692837461
  BuiltTime: Thu Aug 24 09:37:41 2023
  GitCommit: 7c0e926820881583b181985b31f7485e1a32d24c
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.0-dev

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional information

When setting STORAGE_FS=vfs, this test successed.

# FOCUS="podman pull and run on split imagestore" STORAGE_FS=vfs make localintegration

e2e test log

  > Enter [It] podman pull and run on split imagestore - /root/podman/test/e2e/pull_test.go:65 @ 08/25/23 10:56:13.419
  Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs build --pull=never --tag splitstoretest build/basicalpine
  STEP 1/1: FROM quay.io/libpod/alpine:latest
  COMMIT splitstoretest
  --> 961769676411
  Successfully tagged localhost/splitstoretest:latest
  Successfully tagged quay.io/libpod/alpine:latest
  961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4
  Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs save -o /tmp/podman_test735789469/image.tar --format oci-archive splitstoretest
  Copying blob sha256:03901b4a2ea88eeaad62dbe59b072b28b6efa00491962b8741081c5df50c65e0
  Copying config sha256:4819930274ea2bd6ab611972f9928e07b84edcbfe1d2462fb2941957308dfae2
  Writing manifest to image destination
  Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs rmi splitstoretest
  Untagged: localhost/splitstoretest:latest
  Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs load --imagestore /tmp/podman_test735789469/splitstore -q -i /tmp/podman_test735789469/image.tar
  Loaded image: localhost/splitstoretest:latest
  Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs tag quay.io/libpod/busybox:latest busybox-test
  Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs images --imagestore /tmp/podman_test735789469/splitstore
  REPOSITORY                                 TAG         IMAGE ID      CREATED        SIZE
  quay.io/libpod/registry                    2.8.2       0030ba3d620c  2 weeks ago    24.6 MB
  registry.fedoraproject.org/fedora-toolbox  36          5be0bf1ecd38  4 months ago   874 MB
  quay.io/libpod/systemd-image               20230106    4f6126057b5e  7 months ago   132 MB
  quay.io/libpod/alpine_nginx                latest      ecea49d99daa  16 months ago  23.1 MB
  registry.fedoraproject.org/fedora-minimal  34          b85bdf5fbb1c  18 months ago  119 MB
  quay.io/libpod/alpine_labels               latest      5e9e9275e4d6  18 months ago  5.87 MB
  quay.io/libpod/redis                       alpine      8835e6aeca99  2 years ago    32.3 MB
  quay.io/libpod/busybox                     latest      f0b02e9d092d  2 years ago    1.45 MB
  quay.io/libpod/busybox                     latest      f0b02e9d092d  2 years ago    1.45 MB
  localhost/busybox-test                     latest      f0b02e9d092d  2 years ago    1.45 MB
  registry.k8s.io/pause                      3.2         80d28bedfe5d  3 years ago    686 kB
  localhost/splitstoretest                   latest      4819930274ea  4 years ago    5.85 MB
  quay.io/libpod/alpine                      latest      961769676411  4 years ago    5.85 MB
  quay.io/libpod/alpine                      latest      961769676411  4 years ago    5.85 MB
  quay.io/libpod/alpine_healthcheck          latest      c1103abb47d6  4 years ago    10.4 MB
  Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs rmi --imagestore /tmp/podman_test735789469/splitstore busybox-test
  Untagged: localhost/busybox-test:latest
  Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs images
  REPOSITORY                                 TAG         IMAGE ID      CREATED        SIZE        R/O
  quay.io/libpod/registry                    2.8.2       0030ba3d620c  2 weeks ago    24.6 MB     true
  registry.fedoraproject.org/fedora-toolbox  36          5be0bf1ecd38  4 months ago   874 MB      true
  quay.io/libpod/systemd-image               20230106    4f6126057b5e  7 months ago   132 MB      true
  quay.io/libpod/alpine_nginx                latest      ecea49d99daa  16 months ago  23.1 MB     true
  registry.fedoraproject.org/fedora-minimal  34          b85bdf5fbb1c  18 months ago  119 MB      true
  quay.io/libpod/alpine_labels               latest      5e9e9275e4d6  18 months ago  5.87 MB     true
  quay.io/libpod/redis                       alpine      8835e6aeca99  2 years ago    32.3 MB     true
  quay.io/libpod/busybox                     latest      f0b02e9d092d  2 years ago    1.45 MB     false
  localhost/busybox-test                     latest      f0b02e9d092d  2 years ago    1.45 MB     false
  quay.io/libpod/busybox                     latest      f0b02e9d092d  2 years ago    1.45 MB     true
  registry.k8s.io/pause                      3.2         80d28bedfe5d  3 years ago    686 kB      true
  quay.io/libpod/alpine                      latest      961769676411  4 years ago    5.85 MB     false
  quay.io/libpod/alpine                      latest      961769676411  4 years ago    5.85 MB     true
  quay.io/libpod/alpine_healthcheck          latest      c1103abb47d6  4 years ago    10.4 MB     true
  Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs run --name test --rm splitstoretest echo helloworld
  time="2023-08-25T10:56:14+09:00" level=warning msg="The storage 'driver' option should be set in /tmp/podman_test735789469/.config/containers/storage.conf. A driver was picked automatically."
  helloworld
  < Exit [It] podman pull and run on split imagestore - /root/podman/test/e2e/pull_test.go:65 @ 08/25/23 10:56:14.326 (907ms)
  > Enter [AfterEach] TOP-LEVEL - /root/podman/test/e2e/common_test.go:119 @ 08/25/23 10:56:14.326
  Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs stop --all -t 0
  Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs pod rm -fa -t 0
  Running: /root/podman/bin/podman --storage-opt vfs.imagestore=/tmp/ginkgo693882096/imagecachedir --root /tmp/podman_test735789469/root --runroot /tmp/podman_test735789469/runroot --runtime crun --conmon /usr/bin/conmon --network-config-dir /etc/cni/net.d --network-backend cni --cgroup-manager systemd --tmpdir /tmp/podman_test735789469 --events-backend file --db-backend boltdb --storage-driver vfs rm -fa -t 0
  < Exit [AfterEach] TOP-LEVEL - /root/podman/test/e2e/common_test.go:119 @ 08/25/23 10:56:14.442 (116ms)

The text was updated successfully, but these errors were encountered:

flouthoc · 2023-08-25T07:32:24Z

Thanks for reporting, I am able to reproduce this. One strange thing is that I cant see this issue when I use this feature manually and also in Podman CI. But when I run this specific test with overlay on my local it fails.

I'll check this.

flouthoc · 2023-08-25T11:46:41Z

Found the issue, it happens when additional image store is used along with a seperate graphroot and split store using imagestore, I'm just wondering why don't we see this failure in podman CI is it that podman CI never uses overlay ?

Anyways I'll open a PR for this.

edsantiago · 2023-08-25T12:07:30Z

e2e tests all hardcode vfs (among many, many other hardcoded settings) . It may be time to reevaluate some of those hardcodings.

mheon · 2023-08-25T18:34:57Z

If nothing else, overlay would probably make the tests run quite a bit faster

rhatdan · 2023-08-26T10:40:15Z

If the tests are running on VFS on top of xfs, then they should perform fairly well. But since almost everyone uses overlay, testing with overlay by default makes sense.

github-actions · 2023-09-26T00:06:30Z

A friendly reminder that this issue had no activity for 30 days.

We're only testing vfs in CI. That's bad. containers#18822 tried to remedy that but I'm pretty sure it's a NOP. Here we introduce CI_DESIRED_STORAGE, to be set in .cirrus.yml in the same vein as all the other CI_DESIRED_X. Since it's 2023 we default to overlay, testing vfs only in priorfedora. Skip the "split imagestore" test under overlay (containers#19748) Signed-off-by: Ed Santiago <santiago@redhat.com>

We're only testing vfs in CI. That's bad. containers#18822 tried to remedy that but that only worked on system tests, not e2e. Here we introduce CI_DESIRED_STORAGE, to be set in .cirrus.yml in the same vein as all the other CI_DESIRED_X. Since it's 2023 we default to overlay, testing vfs only in priorfedora. Skip the "split imagestore" test under overlay (containers#19748) Signed-off-by: Ed Santiago <santiago@redhat.com>

We're only testing vfs in CI. That's bad. containers#18822 tried to remedy that but that only worked on system tests, not e2e. Here we introduce CI_DESIRED_STORAGE, to be set in .cirrus.yml in the same vein as all the other CI_DESIRED_X. Since it's 2023 we default to overlay, testing vfs only in priorfedora. Fixes required: - in cleanup, umount ROOT/overlay to avoid leaking mounts - e2e: - Skip the "split imagestore" test: it doesn't work, and nobody knows why (containers#19748) - system tests: - fix a few badly-written tests that assumed/hardcoded overlay - buildx test: add weird exception to device-number test - mount tests: add special case code for vfs - unprivileged test: weird failure. Add debugging Signed-off-by: Ed Santiago <santiago@redhat.com>

We're only testing vfs in CI. That's bad. containers#18822 tried to remedy that but that only worked on system tests, not e2e. Here we introduce CI_DESIRED_STORAGE, to be set in .cirrus.yml in the same vein as all the other CI_DESIRED_X. Since it's 2023 we default to overlay, testing vfs only in priorfedora. Fixes required: - in cleanup, umount ROOT/overlay to avoid leaking mounts - e2e: - Skip the "split imagestore" test: it doesn't work, and nobody knows why (containers#19748) - system tests: - fix a few badly-written tests that assumed/hardcoded overlay - buildx test: add weird exception to device-number test - mount tests: add special case code for vfs - unprivileged test: disable one section that is N/A on vfs Signed-off-by: Ed Santiago <santiago@redhat.com>

sstosh added the kind/bug Categorizes issue or PR as related to a bug. label Aug 25, 2023

flouthoc self-assigned this Aug 25, 2023

github-actions bot added the stale-issue label Sep 26, 2023

edsantiago mentioned this issue Sep 26, 2023

CI: test overlay and vfs #20161

Merged

edsantiago removed the stale-issue label Sep 27, 2023

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Jan 17, 2024

github-actions bot locked as resolved and limited conversation to collaborators Jan 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

e2e tests: split imagestore test failed when setting STORAGE_FS=overlay #19748

e2e tests: split imagestore test failed when setting STORAGE_FS=overlay #19748

sstosh commented Aug 25, 2023 •

edited

Loading

flouthoc commented Aug 25, 2023

flouthoc commented Aug 25, 2023

edsantiago commented Aug 25, 2023

mheon commented Aug 25, 2023

rhatdan commented Aug 26, 2023

github-actions bot commented Sep 26, 2023

e2e tests: split imagestore test failed when setting STORAGE_FS=overlay #19748

e2e tests: split imagestore test failed when setting STORAGE_FS=overlay #19748

Comments

sstosh commented Aug 25, 2023 • edited Loading

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

flouthoc commented Aug 25, 2023

flouthoc commented Aug 25, 2023

edsantiago commented Aug 25, 2023

mheon commented Aug 25, 2023

rhatdan commented Aug 26, 2023

github-actions bot commented Sep 26, 2023

sstosh commented Aug 25, 2023 •

edited

Loading