[DO NOT MERGE][CI ONLY] EBUSY flake investigation

[giuseppe]

No description provided.

[openshift-ci]

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: giuseppe

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [giuseppe]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[giuseppe]

interestingly there is no mountpoint in the current mount namespace that could block the removal:

[+1136s] # Error: removing storage for container "fefaeb3391e2": removing mount point "/home/some20270dude/.local/share/containers/storage/overlay/6393565c8cfc88fea81b01f3f0d789a1ab0a03337e294b313f4de7d1e89f79c9/merged": device or resource busy
[+1136s] # time="2023-10-03T07:37:13-05:00" level=debug msg="Shutting down engines"
[+1136s] # MOUNTINFO
[+1136s] # 61 1 0:30 /root / rw,relatime shared:1 - btrfs /dev/sda5 rw,seclabel,compress=zstd:1,discard=async,space_cache=v2,subvolid=256,subvol=/root
[+1136s] # 32 61 0:5 / /dev rw,nosuid shared:2 - devtmpfs devtmpfs rw,seclabel,size=4096k,nr_inodes=494223,mode=755,inode64
[+1136s] # 33 32 0:23 / /dev/shm rw,nosuid,nodev shared:3 - tmpfs tmpfs rw,seclabel,inode64
[+1136s] # 34 32 0:24 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=000
[+1136s] # 35 61 0:22 / /sys rw,nosuid,nodev,noexec,relatime shared:5 - sysfs sysfs rw,seclabel
[+1136s] # 36 35 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:6 - securityfs securityfs rw
[+1136s] # 37 35 0:26 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:7 - cgroup2 cgroup2 rw,seclabel,nsdelegate,memory_recursiveprot
[+1136s] # 38 35 0:27 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:8 - pstore pstore rw,seclabel
[+1136s] # 39 35 0:28 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:9 - bpf bpf rw,mode=700
[+1136s] # 40 35 0:29 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:10 - configfs configfs rw
[+1136s] # 41 61 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:12 - proc proc rw
[+1136s] # 42 61 0:25 / /run rw,nosuid,nodev shared:13 - tmpfs tmpfs rw,seclabel,size=799080k,nr_inodes=819200,mode=755,inode64
[+1136s] # 23 35 0:20 / /sys/fs/selinux rw,nosuid,noexec,relatime shared:11 - selinuxfs selinuxfs rw
[+1136s] # 24 41 0:33 / /proc/sys/fs/binfmt_misc rw,relatime shared:14 - autofs systemd-1 rw,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16780
[+1136s] # 25 32 0:34 / /dev/hugepages rw,nosuid,nodev,relatime shared:15 - hugetlbfs hugetlbfs rw,seclabel,pagesize=2M
[+1136s] # 26 32 0:19 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:16 - mqueue mqueue rw,seclabel
[+1136s] # 27 35 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:17 - debugfs debugfs rw,seclabel
[+1136s] # 29 35 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:18 - tracefs tracefs rw,seclabel
[+1136s] # 30 35 0:35 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:19 - fusectl fusectl rw
[+1136s] # 44 61 0:30 /home /home rw,relatime shared:71 - btrfs /dev/sda5 rw,seclabel,compress=zstd:1,discard=async,space_cache=v2,subvolid=257,subvol=/home
[+1136s] # 47 61 8:2 / /boot rw,relatime shared:75 - ext4 /dev/sda2 rw,seclabel
[+1136s] # 50 47 8:3 / /boot/efi rw,relatime shared:78 - vfat /dev/sda3 rw,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro
[+1136s] # 53 24 0:39 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:81 - binfmt_misc binfmt_misc rw
[+1136s] # 236 61 0:44 / /var/lib/nfs/rpc_pipefs rw,relatime shared:123 - rpc_pipefs sunrpc rw
[+1136s] # 62 42 0:48 / /run/user/5063 rw,nosuid,nodev,relatime shared:249 - tmpfs tmpfs rw,seclabel,size=399536k,nr_inodes=99884,mode=700,uid=5063,gid=3660,inode64
[+1136s] # setup(): removing stray external container 6bf290250e71 (test_pod-test)

[giuseppe]

this is interesting:

[+1333s] # 369 61 0:49 / /var/lib/containers/storage/overlay/acf095f45379b10140e702f8182aeb18aee8d2fa38ef2c0b94f12e2cb00e1799/merged rw,nodev,relatime shared:349 - overlay overlay rw,context="system_u:object_r:container_file_t:s0:c12,c800",lowerdir=/var/lib/containers/storage/overlay/l/CTTRJU3QLQ7N53FFC4WOXZPZ45:/var/lib/containers/storage/overlay/l/7LHTJWBOGXR7EJER2OE7ZGWGCM,upperdir=/var/lib/containers/storage/overlay/acf095f45379b10140e702f8182aeb18aee8d2fa38ef2c0b94f12e2cb00e1799/diff,workdir=/var/lib/containers/storage/overlay/acf095f45379b10140e702f8182aeb18aee8d2fa38ef2c0b94f12e2cb00e1799/work,redirect_dir=on,uuid=on,metacopy=on
[+1333s] # 415 61 0:62 / /var/lib/containers/storage/overlay/ee85763940b49e2a10480ead6cd98765add15d67c1862a41a22a04f81b7142bd/merged rw,nodev,relatime shared:367 - overlay overlay rw,context="system_u:object_r:container_file_t:s0:c152,c444",lowerdir=/var/lib/containers/storage/overlay/l/ETMWARDSFGAHWJEZL6LRHJ6SXI,upperdir=/var/lib/containers/storage/overlay/ee85763940b49e2a10480ead6cd98765add15d67c1862a41a22a04f81b7142bd/diff,workdir=/var/lib/containers/storage/overlay/ee85763940b49e2a10480ead6cd98765add15d67c1862a41a22a04f81b7142bd/work,redirect_dir=on,uuid=on,metacopy=on
[+1333s] # 475 61 0:74 / /var/lib/containers/storage/overlay/40257c3f4a420ecc6ceb76f29cf9f8b45dc0ffcc6b6cafc316108a93635baa38/merged rw,nodev,relatime shared:393 - overlay overlay rw,context="system_u:object_r:container_file_t:s0:c1,c2",lowerdir=/var/lib/containers/storage/overlay/l/CTTRJU3QLQ7N53FFC4WOXZPZ45:/var/lib/containers/storage/overlay/l/7LHTJWBOGXR7EJER2OE7ZGWGCM,upperdir=/var/lib/containers/storage/overlay/40257c3f4a420ecc6ceb76f29cf9f8b45dc0ffcc6b6cafc316108a93635baa38/diff,workdir=/var/lib/containers/storage/overlay/40257c3f4a420ecc6ceb76f29cf9f8b45dc0ffcc6b6cafc316108a93635baa38/work,redirect_dir=on,uuid=on,metacopy=on
[+1333s] # 472 61 0:30 /root/var/lib/containers/storage/overlay /var/lib/containers/storage/overlay rw,relatime - btrfs /dev/sda5 rw,seclabel,compress=zstd:1,discard=async,space_cache=v2,subvolid=256,subvol=/root

we are mounting the home mount on top of existing mounts (which are not on a home mount). This is likely the reason for the flake, but not sure how we can end up this way

[giuseppe]

fix in containers/storage#1724