"Operation not permitted" in Singularity containers #232

bedroge · 2020-08-31T13:50:40Z

I'm trying to make use of fuse-overlayfs inside a Singularity container (based on CentOS 7) by using its new --fusemount option. Ideally I want to use this to make a writable overlay on top of a read-only CVMFS mount, but basically I'm just trying to do something like this:

singularity shell -B /overlay --fusemount "container:fuse-overlayfs -o lowerdir=/overlay/lower -o upperdir=/overlay/upper -o workdir=/overlay/work /merged" fuse-overlay.sif

With recent version of fuse-overlayfs this seems to work partly: I can do some operations, like making new files, but some stuff does not work. For instance, making directories with mkdir -p (and sometimes even just cd'ing to the merged mountpoint) usually results in an Operation not permitted, e.g.:

Singularity> mkdir -p /merged/my/new/dir
mkdir: cannot create directory '/merged/my/new': Operation not permitted

Today I more or less accidentally found out that with a Debian 10 container everything does work fine. The default version in the Debian repo is quite old though, it installs fuse-overlayfs 0.3. Testing a bit more with various manually compiled versions, I found out that for both CentOS and Debian containers everything seems to work fine with versions up to and including 0.4.1, but with version 0.5 and newer I consistently get these errors.

Do you have any idea what could cause these Operation not permitted errors?

The text was updated successfully, but these errors were encountered:

dtrudg · 2020-08-31T14:18:46Z

edit - sorry, I missed you used manually compiled versions also.

@bedroge - Fedora has the latest 1.1.2 available, and this seems to work for me with Singularity, so whatever is causing your issue may already have been addressed / changed if you didn't get that far up?

Also, what is the host you are using? I wonder if that is having an impact here. I'm on Fedora 32 on the host too.

Bootstrap: docker
From: fedora:32

%post
    dnf install -y fuse-overlayfs

dave@dev-fedora:~
09:15 AM $ singularity shell -B /overlay --fusemount "container:fuse-overlayfs -o lowerdir=/overlay/lower -o upperdir=/overlay/upper -o workdir=/overlay/work /merged" fuse-fedora.sif 
Singularity> cd /merged/
Singularity> ls
Singularity> mkdir bob
Singularity> cd bob
Singularity> mkdir -p /merged/my/new
Singularity> touch /merged/my/new/file
Singularity> ls /merged/my/new
file
Singularity> mount | grep merged
fuse on /merged type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

bedroge · 2020-08-31T14:57:09Z

This is on an Ubuntu 20.04.1 (edit: kernel version 5.4.0-42-generic) host with Singularity 3.6.1, but I think I also tried this in a CentOS VM a few days ago, and had the same issues. I just tried it with a Fedora container, but unfortunately that does not work either:

bob@bob-XPS-13-9360 ~/a/s/fedora> singularity shell -B /overlay --fusemount "container:fuse-overlayfs -o lowerdir=/overlay/lower -o upperdir=/overlay/upper -o workdir=/overlay/work /merged" fuse-fedora.sif
Singularity> mkdir -p /merged/my/new/dir
mkdir: cannot create directory '/merged/my/new': Operation not permitted
Singularity> mkdir -p /merged/my/new/dir
mkdir: cannot create directory '/merged/my/new': Operation not permitted

The upper dir does contain my/new afterwards, but not dir... When I then relaunch the container, the command does work:

bob@bob-XPS-13-9360 ~/a/s/fedora [1]> singularity shell -B /overlay --fusemount "container:fuse-overlayfs -o lowerdir=/overlay/lower -o upperdir=/overlay/upper -o workdir=/overlay/work /merged" fuse-fedora.sif
Singularity> mkdir -p /merged/my/new/dir
Singularity>

Really weird...

bedroge · 2020-08-31T16:26:07Z

I just tried it on one of our cluster nodes, which has CentOS 7.8.2003 (kernel 3.10.0-1127.13.1.el7.x86_64) and Singularity 3.5.3-1.1.el7, but then I get even weirder behavior:

$ singularity shell -B /local/$USER --fusemount "container:fuse-overlayfs -o lowerdir=/local/$USER/overlay/lower -o upperdir=/local/$USER/overlay/upper -o workdir=/local/$USER/overlay/work /merged" fuse-fedora.sif 
Singularity> cd /merged 
bash: cd: /merged: Operation not permitted
Singularity> ls /merged
ls: cannot access '/merged': Invalid argument
Singularity> ls -l / | grep merged
ls: cannot access '/merged': Invalid argument
d??????????   ? ?       ?            ?            ? merged

boegel · 2020-09-21T08:34:33Z

I'm seeing the same issues like @bedroge is seeing, with:

Singularity 3.6
Singularity container with CentOS 7.8.2003
latest fuse-overlayfs (downloaded the binary available at https://github.com/containers/fuse-overlayfs/releases/tag/v1.1.2 and copied it to /usr/bin/fuse-overlayfs)
also having these packages installed: cvmfs cvmfs-config-default cvmfs-fuse3

I can do the same with the minimal Fedora container that @dctrud was using:

$ mkdir -p /tmp/$USER/fuse-test/{lower,upper,work}

$ singularity shell -B /tmp/$USER/fuse-test --fusemount "container:fuse-overlayfs -o lowerdir=/tmp/$USER/fuse-test/lower -o upperdir=/tmp/$USER/fuse-test/upper -o workdir=/tmp/$USER/fuse-test/work /tmp/$USER/fuse-test/merged" fedora.sif
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Singularity>
Singularity> mkdir /tmp/$USER/fuse-test/merged/foo
mkdir: cannot create directory '/tmp/kehoste/fuse-test/merged/foo': Invalid argument
/tmp/$USER/fuse-test/merged

Maybe it's related to using /tmp for the overlay directories?

bedroge · 2020-11-24T14:51:43Z

Since we still don't have a proper solution for this, besides using the old version 0.4.1, I was looking into this issue again today.

I'm using a basic CentOS 7 image (in the following examples as sandbox, but the issue also occurs with regular sif images), and compiled fuse-overlayfs manually. The following shows what happens when I run a failing mkdir -p command with strace:

$ singularity shell -B /data --fusemount "container:/data/fuse-overlayfs-1.2.0/fuse-overlayfs -o lowerdir=/data/overlay/lower -o upperdir=/data/overlay/upper -o workdir=/data/overlay/work /overlay" ./centos7

Singularity> strace mkdir -p /overlay/newdir1/newdir2/newdir3
execve("/usr/bin/mkdir", ["mkdir", "-p", "/overlay/newdir1/newdir2/newdir3"], 0x7ffc825c5700 /* 66 vars */) = 0
brk(NULL)                               = 0x2612000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f453a22f000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/.singularity.d/libs/tls/x86_64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/.singularity.d/libs/tls/x86_64", 0x7ffd9f88a940) = -1 ENOENT (No such file or directory)
open("/.singularity.d/libs/tls/libselinux.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/.singularity.d/libs/tls", 0x7ffd9f88a940) = -1 ENOENT (No such file or directory)
open("/.singularity.d/libs/x86_64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/.singularity.d/libs/x86_64", 0x7ffd9f88a940) = -1 ENOENT (No such file or directory)
open("/.singularity.d/libs/libselinux.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/.singularity.d/libs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=13316, ...}) = 0
mmap(NULL, 13316, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f453a22b000
close(3)                                = 0
open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220j\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=155744, ...}) = 0
mmap(NULL, 2255216, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4539de8000
mprotect(0x7f4539e0c000, 2093056, PROT_NONE) = 0
mmap(0x7f453a00b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f453a00b000
mmap(0x7f453a00d000, 6512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f453a00d000
close(3)                                = 0
open("/.singularity.d/libs/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`&\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2156272, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f453a22a000
mmap(NULL, 3985920, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4539a1a000
mprotect(0x7f4539bde000, 2093056, PROT_NONE) = 0
mmap(0x7f4539ddd000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c3000) = 0x7f4539ddd000
mmap(0x7f4539de3000, 16896, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f4539de3000
close(3)                                = 0
open("/.singularity.d/libs/libpcre.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\25\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=402384, ...}) = 0
mmap(NULL, 2494984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f45397b8000
mprotect(0x7f4539818000, 2097152, PROT_NONE) = 0
mmap(0x7f4539a18000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x60000) = 0x7f4539a18000
close(3)                                = 0
open("/.singularity.d/libs/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19248, ...}) = 0
mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f45395b4000
mprotect(0x7f45395b6000, 2097152, PROT_NONE) = 0
mmap(0x7f45397b6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f45397b6000
close(3)                                = 0
open("/.singularity.d/libs/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200m\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=142144, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f453a229000
mmap(NULL, 2208904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4539398000
mprotect(0x7f45393af000, 2093056, PROT_NONE) = 0
mmap(0x7f45395ae000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f45395ae000
mmap(0x7f45395b0000, 13448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f45395b0000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f453a228000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f453a226000
arch_prctl(ARCH_SET_FS, 0x7f453a226840) = 0
mprotect(0x7f4539ddd000, 16384, PROT_READ) = 0
mprotect(0x7f45395ae000, 4096, PROT_READ) = 0
mprotect(0x7f45397b6000, 4096, PROT_READ) = 0
mprotect(0x7f4539a18000, 4096, PROT_READ) = 0
mprotect(0x7f453a00b000, 4096, PROT_READ) = 0
mprotect(0x611000, 4096, PROT_READ)     = 0
mprotect(0x7f453a230000, 4096, PROT_READ) = 0
munmap(0x7f453a22b000, 13316)           = 0
set_tid_address(0x7f453a226b10)         = 19
set_robust_list(0x7f453a226b20, 24)     = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x7f453939e860, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f45393a7630}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x7f453939e8f0, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f45393a7630}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
statfs("/sys/fs/selinux", 0x7ffd9f88c130) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7ffd9f88c130)      = -1 ENOENT (No such file or directory)
brk(NULL)                               = 0x2612000
brk(0x2633000)                          = 0x2633000
open("/proc/filesystems", O_RDONLY)     = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f453a22e000
read(3, "nodev\tsysfs\nnodev\ttmpfs\nnodev\tbd"..., 1024) = 417
stat("/etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned", 0x7ffd9f88bcb0) = -1 ENOENT (No such file or directory)
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7f453a22e000, 4096)            = 0
access("/etc/selinux/config", F_OK)     = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3489392, ...}) = 0
mmap(NULL, 3489392, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f4539044000
close(3)                                = 0
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2502, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f453a22e000
read(3, "# Locale name alias data base.\n#"..., 4096) = 2502
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f453a22e000, 4096)            = 0
open("/usr/lib/locale/nl_NL.UTF-8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl_NL.utf8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl_NL/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl.UTF-8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl.utf8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
umask(000)                              = 022
umask(022)                              = 000
mkdir("/overlay", 0777)                 = -1 EEXIST (File exists)
chdir("/overlay")                       = 0
mkdir("newdir1", 0777)                  = 0
open("newdir1", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_DIRECTORY) = 3
fchdir(3)                               = 0
close(3)                                = 0
mkdir("newdir2", 0777)                  = 0
open("newdir2", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_DIRECTORY) = 3
fchdir(3)                               = -1 EPERM (Operation not permitted)	
close(3)                                = 0
open("/usr/lib64/charset.alias", O_RDONLY|O_NOFOLLOW) = -1 ENOENT (No such file or directory)
write(2, "mkdir: ", 7mkdir: )                  = 7
write(2, "cannot create directory '/overla"..., 50cannot create directory '/overlay/newdir1/newdir2') = 50
write(2, ": Operation not permitted", 25: Operation not permitted) = 25
write(2, "\n", 1
)                       = 1
close(1)                                = 0
close(2)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++

Just for completeness, this is with the old version that does work:

$ singularity shell -B /data --fusemount "container:/data/fuse-overlayfs-0.4.1/fuse-overlayfs -o lowerdir=/data/overlay/lower -o upperdir=/data/overlay/upper -o workdir=/data/overlay/work /overlay" ./centos7/

Singularity> strace mkdir -p /overlay/newdir1/newdir2/newdir3
execve("/usr/bin/mkdir", ["mkdir", "-p", "/overlay/newdir1/newdir2/newdir3"], 0x7ffc01312380 /* 66 vars */) = 0
brk(NULL)                               = 0x1832000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14dc4d3000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/.singularity.d/libs/tls/x86_64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/.singularity.d/libs/tls/x86_64", 0x7ffc26bd5310) = -1 ENOENT (No such file or directory)
open("/.singularity.d/libs/tls/libselinux.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/.singularity.d/libs/tls", 0x7ffc26bd5310) = -1 ENOENT (No such file or directory)
open("/.singularity.d/libs/x86_64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/.singularity.d/libs/x86_64", 0x7ffc26bd5310) = -1 ENOENT (No such file or directory)
open("/.singularity.d/libs/libselinux.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/.singularity.d/libs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=13316, ...}) = 0
mmap(NULL, 13316, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f14dc4cf000
close(3)                                = 0
open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220j\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=155744, ...}) = 0
mmap(NULL, 2255216, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f14dc08c000
mprotect(0x7f14dc0b0000, 2093056, PROT_NONE) = 0
mmap(0x7f14dc2af000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f14dc2af000
mmap(0x7f14dc2b1000, 6512, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f14dc2b1000
close(3)                                = 0
open("/.singularity.d/libs/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`&\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2156272, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14dc4ce000
mmap(NULL, 3985920, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f14dbcbe000
mprotect(0x7f14dbe82000, 2093056, PROT_NONE) = 0
mmap(0x7f14dc081000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c3000) = 0x7f14dc081000
mmap(0x7f14dc087000, 16896, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f14dc087000
close(3)                                = 0
open("/.singularity.d/libs/libpcre.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\25\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=402384, ...}) = 0
mmap(NULL, 2494984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f14dba5c000
mprotect(0x7f14dbabc000, 2097152, PROT_NONE) = 0
mmap(0x7f14dbcbc000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x60000) = 0x7f14dbcbc000
close(3)                                = 0
open("/.singularity.d/libs/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19248, ...}) = 0
mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f14db858000
mprotect(0x7f14db85a000, 2097152, PROT_NONE) = 0
mmap(0x7f14dba5a000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f14dba5a000
close(3)                                = 0
open("/.singularity.d/libs/libpthread.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200m\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=142144, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14dc4cd000
mmap(NULL, 2208904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f14db63c000
mprotect(0x7f14db653000, 2093056, PROT_NONE) = 0
mmap(0x7f14db852000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f14db852000
mmap(0x7f14db854000, 13448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f14db854000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14dc4cc000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14dc4ca000
arch_prctl(ARCH_SET_FS, 0x7f14dc4ca840) = 0
mprotect(0x7f14dc081000, 16384, PROT_READ) = 0
mprotect(0x7f14db852000, 4096, PROT_READ) = 0
mprotect(0x7f14dba5a000, 4096, PROT_READ) = 0
mprotect(0x7f14dbcbc000, 4096, PROT_READ) = 0
mprotect(0x7f14dc2af000, 4096, PROT_READ) = 0
mprotect(0x611000, 4096, PROT_READ)     = 0
mprotect(0x7f14dc4d4000, 4096, PROT_READ) = 0
munmap(0x7f14dc4cf000, 13316)           = 0
set_tid_address(0x7f14dc4cab10)         = 19
set_robust_list(0x7f14dc4cab20, 24)     = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x7f14db642860, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f14db64b630}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x7f14db6428f0, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f14db64b630}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
statfs("/sys/fs/selinux", 0x7ffc26bd6b00) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7ffc26bd6b00)      = -1 ENOENT (No such file or directory)
brk(NULL)                               = 0x1832000
brk(0x1853000)                          = 0x1853000
open("/proc/filesystems", O_RDONLY)     = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14dc4d2000
read(3, "nodev\tsysfs\nnodev\ttmpfs\nnodev\tbd"..., 1024) = 417
stat("/etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned", 0x7ffc26bd6670) = -1 ENOENT (No such file or directory)
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7f14dc4d2000, 4096)            = 0
access("/etc/selinux/config", F_OK)     = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3489392, ...}) = 0
mmap(NULL, 3489392, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f14db2e8000
close(3)                                = 0
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2502, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f14dc4d2000
read(3, "# Locale name alias data base.\n#"..., 4096) = 2502
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f14dc4d2000, 4096)            = 0
open("/usr/lib/locale/nl_NL.UTF-8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl_NL.utf8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl_NL/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl.UTF-8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl.utf8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
umask(000)                              = 022
umask(022)                              = 000
mkdir("/overlay", 0777)                 = -1 EEXIST (File exists)
chdir("/overlay")                       = 0
mkdir("newdir1", 0777)                  = 0
open("newdir1", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_DIRECTORY) = 3
fchdir(3)                               = 0
close(3)                                = 0
mkdir("newdir2", 0777)                  = 0
open("newdir2", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_NOFOLLOW|O_DIRECTORY) = 3
fchdir(3)                               = 0
close(3)                                = 0
mkdir("newdir3", 0777)                  = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

So, for some reason, I can create a directory newdir1 in the existing /overlay, and can chdir into it, also the second directory newdir2 can be created, but for some reason it cannot change into that directory.

This can also be reproduced by doing the operations:

Singularity> cd /overlay/
Singularity> mkdir newdir1
Singularity> cd newdir1/
Singularity> mkdir newdir2
Singularity> cd newdir2
bash: cd: newdir2: Operation not permitted
Singularity> strace cd newdir2
execve("/usr/bin/cd", ["cd", "newdir2"], 0x7fff293a1478 /* 67 vars */) = 0
brk(NULL)                               = 0x1bb9000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5df41a4000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/.singularity.d/libs/tls/x86_64/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/.singularity.d/libs/tls/x86_64", 0x7ffdc2e401c0) = -1 ENOENT (No such file or directory)
open("/.singularity.d/libs/tls/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/.singularity.d/libs/tls", 0x7ffdc2e401c0) = -1 ENOENT (No such file or directory)
open("/.singularity.d/libs/x86_64/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/.singularity.d/libs/x86_64", 0x7ffdc2e401c0) = -1 ENOENT (No such file or directory)
open("/.singularity.d/libs/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/.singularity.d/libs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=13316, ...}) = 0
mmap(NULL, 13316, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5df41a0000
close(3)                                = 0
open("/lib64/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\316\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=174576, ...}) = 0
mmap(NULL, 2268928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f5df3d5a000
mprotect(0x7f5df3d7f000, 2097152, PROT_NONE) = 0
mmap(0x7f5df3f7f000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x25000) = 0x7f5df3f7f000
close(3)                                = 0
open("/.singularity.d/libs/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19248, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5df419f000
mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f5df3b56000
mprotect(0x7f5df3b58000, 2097152, PROT_NONE) = 0
mmap(0x7f5df3d58000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f5df3d58000
close(3)                                = 0
open("/.singularity.d/libs/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`&\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2156272, ...}) = 0
mmap(NULL, 3985920, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f5df3788000
mprotect(0x7f5df394c000, 2093056, PROT_NONE) = 0
mmap(0x7f5df3b4b000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c3000) = 0x7f5df3b4b000
mmap(0x7f5df3b51000, 16896, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5df3b51000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5df419e000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5df419c000
arch_prctl(ARCH_SET_FS, 0x7f5df419c740) = 0
mprotect(0x7f5df3b4b000, 16384, PROT_READ) = 0
mprotect(0x7f5df3d58000, 4096, PROT_READ) = 0
mprotect(0x7f5df3f7f000, 16384, PROT_READ) = 0
mprotect(0x6dd000, 4096, PROT_READ)     = 0
mprotect(0x7f5df41a5000, 4096, PROT_READ) = 0
munmap(0x7f5df41a0000, 13316)           = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
open("/dev/tty", O_RDWR|O_NONBLOCK)     = 3
close(3)                                = 0
brk(NULL)                               = 0x1bb9000
brk(0x1bda000)                          = 0x1bda000
brk(NULL)                               = 0x1bda000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3489392, ...}) = 0
mmap(NULL, 3489392, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5df3434000
close(3)                                = 0
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2502, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5df41a3000
read(3, "# Locale name alias data base.\n#"..., 4096) = 2502
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f5df41a3000, 4096)            = 0
open("/usr/lib/locale/nl_NL.UTF-8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl_NL.utf8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl_NL/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl.UTF-8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl.utf8/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl/LC_IDENTIFICATION", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
getuid()                                = 1000
getgid()                                = 1000
geteuid()                               = 1000
getegid()                               = 1000
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
brk(NULL)                               = 0x1bda000
open("/proc/meminfo", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5df41a3000
read(3, "MemTotal:       16166560 kB\nMemF"..., 1024) = 1024
close(3)                                = 0
munmap(0x7f5df41a3000, 4096)            = 0
rt_sigaction(SIGCHLD, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f5df37be400}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGCHLD, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f5df37be400}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f5df37be400}, 8) = 0
rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f5df37be400}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f5df37be400}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f5df37be400}, 8) = 0
rt_sigaction(SIGQUIT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f5df37be400}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGQUIT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f5df37be400}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f5df37be400}, 8) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigaction(SIGQUIT, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f5df37be400}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f5df37be400}, 8) = 0
uname({sysname="Linux", nodename="bob-Latitude-5300", ...}) = 0
stat("/overlay/newdir1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat(".", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getpid()                                = 21
getppid()                               = 19
getpgrp()                               = 19
rt_sigaction(SIGCHLD, {sa_handler=0x4414a0, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f5df37be400}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f5df37be400}, 8) = 0
getrlimit(RLIMIT_NPROC, {rlim_cur=62810, rlim_max=62810}) = 0
open("/usr/lib/locale/nl_NL.UTF-8/LC_NUMERIC", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl_NL.utf8/LC_NUMERIC", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl_NL/LC_NUMERIC", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl.UTF-8/LC_NUMERIC", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl.utf8/LC_NUMERIC", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl/LC_NUMERIC", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl_NL.UTF-8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl_NL.utf8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl_NL/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl.UTF-8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl.utf8/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/nl/LC_TIME", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=26970, ...}) = 0
mmap(NULL, 26970, PROT_READ, MAP_SHARED, 3, 0) = 0x7f5df4195000
close(3)                                = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
open("/usr/bin/cd", O_RDONLY)           = 3
ioctl(3, TCGETS, 0x7ffdc2e41850)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3, 0, SEEK_CUR)                   = 0
read(3, "#!/bin/sh\nbuiltin cd \"$@\"\n", 80) = 26
lseek(3, 0, SEEK_SET)                   = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=1024*1024}) = 0
fcntl(255, F_GETFD)                     = -1 EBADF (Bad file descriptor)
dup2(3, 255)                            = 255
close(3)                                = 0
fcntl(255, F_SETFD, FD_CLOEXEC)         = 0
fcntl(255, F_GETFL)                     = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(255, {st_mode=S_IFREG|0755, st_size=26, ...}) = 0
lseek(255, 0, SEEK_CUR)                 = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
read(255, "#!/bin/sh\nbuiltin cd \"$@\"\n", 26) = 26
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
stat("/overlay", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
stat("/overlay/newdir1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/overlay/newdir1/newdir2", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
chdir("/overlay/newdir1/newdir2")       = -1 EPERM (Operation not permitted)
chdir("newdir2")                        = -1 EPERM (Operation not permitted)
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
fstat(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 6), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5df41a3000
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "/usr/bin/cd: line 2: cd: newdir2"..., 58/usr/bin/cd: line 2: cd: newdir2: Operation not permitted
) = 58
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
read(255, "", 26)                       = 0
exit_group(1)                           = ?
+++ exited with 1 +++

And, to make it even weirder: when I restart the container, leaving the upper and workdir as they are, I can suddenly enter that directory. But when I then try to make another subdirectory, that one will give issues:

Singularity> exit
$ singularity shell -B /data --fusemount "container:/data/fuse-overlayfs-1.2.0/fuse-overlayfs -o lowerdir=/data/overlay/lower -o upperdir=/data/overlay/upper -o workdir=/data/overlay/work /overlay" ./centos7
Singularity> cd /overlay/
Singularity> cd newdir1/
Singularity> cd newdir2/
Singularity> mkdir newdir3
Singularity> cd newdir3/
bash: cd: newdir3/: Operation not permitted

Don't know if this information helps in any way, but I really hope someone can find out what's causing this issue...

bedroge · 2020-11-24T15:43:27Z

The issue can be reproduced in the following way:

export MYTMPDIR=$(mktemp -d)
export OVERLAY_BASEDIR=${MYTMPDIR}/overlay
export OVERLAY_UPPERDIR=${OVERLAY_BASEDIR}/upper
export OVERLAY_WORKDIR=${OVERLAY_BASEDIR}/work
export OVERLAY_LOWERDIR=${OVERLAY_BASEDIR}/lower
export OVERLAY_MOUNTPOINT=/overlay

mkdir -p ${OVERLAY_LOWERDIR} ${OVERLAY_UPPERDIR} ${OVERLAY_WORKDIR}

wget -O ${OVERLAY_BASEDIR}/fuse-overlayfs https://github.com/containers/fuse-overlayfs/releases/download/v1.2.0/fuse-overlayfs-x86_64
chmod +x ${OVERLAY_BASEDIR}/fuse-overlayfs

singularity exec -B ${OVERLAY_BASEDIR} --fusemount "container:${OVERLAY_BASEDIR}/fuse-overlayfs -o lowerdir=${OVERLAY_LOWERDIR} -o upperdir=${OVERLAY_UPPERDIR} -o workdir=${OVERLAY_WORKDIR} ${OVERLAY_MOUNTPOINT}" docker://centos:7 mkdir -p /overlay/newdir1/newdir2/newdir3

For me this results in:

/usr/bin/mkdir: cannot create directory '/overlay/newdir1/newdir2': Operation not permitted

DrDaveD · 2020-11-24T22:34:41Z

FYI I cannot reproduce this on CentOS7 with singularity 3.6.4; instead, singularity prints a message during startup (in runFuseDrivers() according to -d) of

lstat //overlay: invalid argument

I can however reproduce it on CentOS8 with singularity 3.6.4, both with and without the --user option to run it unprivileged.

Even better, I am able to reproduce it without singularity, by using my simple program to do fuse3 premounting instead.

To compile the program on CentOS8, do the following steps:

wget -qO fuse-premount.c https://github.com/containers/fuse-overlayfs/files/5593000/fuse-premount.c.txt
cc -o fuse-premount fuse-premount.c
sudo chown root:root fuse-premount
sudo chmod 4755 fuse-premount

Then do all of Bob's setup above except running singularity, and do these commands instead:

export OVERLAY_MOUNTPOINT=/dev/shm/overlay
mkdir ${OVERLAY_MOUNTPOINT}
./fuse-premount ${OVERLAY_BASEDIR}/fuse-overlayfs -o lowerdir=${OVERLAY_LOWERDIR} -o upperdir=${OVERLAY_UPPERDIR} -o workdir=${OVERLAY_WORKDIR} ${OVERLAY_MOUNTPOINT}
mkdir -p ${OVERLAY_MOUNTPOINT}/newdir1/newdir2/newdir3

To unmount do fusermount -u ${OVERLAY_MOUNTPOINT}.

This works for me on CentOS8, but just like with singularity, if I do the fuse-premount on CentOS7 and then try to do ls ${OVERLAY_MOUNTPOINT}, I get an "Invalid argument" error.

srd424 · 2021-01-15T09:53:59Z

I'm seeing possibly related problems .. my development environment is Ubuntu 18.04 running in systemd-nspawn .. I've just upgraded the host environment to 20.04 and now I'm seeing operation not permitted errors. Stracing overlayfs:

mkdirat(5, "2", 0770)                   = 0
syscall_0x1b5(0x5, 0x7ffe70c23f50, 0x7ffe70c23ec0, 0x18, 0, 0) = -1 (errno 1)
unlinkat(5, "2", AT_REMOVEDIR)          = 0
write(2, "   unique: 14, error: -1 (Operat"..., 64) = 64

syscall 0x1b5 is openat2 which is relatively new .. chances are for me nspawn is incorrectly blocking that syscall (no idea why it worked in previous configuration.)

Might explain the failures in singularity if it similarly does syscall filtering? (I'm not familiar with it.) Not sure it explains @DrDaveD's results outside of a container manager though.

DrDaveD · 2021-01-19T22:02:05Z

singularity doesn't do syscall filtering by default. It can with --security Seccomp.

bedroge · 2022-01-12T09:53:06Z

I've tested this with some newer versions last year (don't remember which ones, but I guess 1.4.0, 1.5.0, and/or 1.6), but those didn't solve the issue. However, today I tried version 1.8, and that one does seem to solve the Operation not permitted issues for me. Also 1.7 seems to work fine. I'll do some more tests to confirm this, and then close this issue.

sassy-crick · 2022-01-27T19:15:34Z

Just to add to this confusion more: On my home machine, running Debian 10 (Buster), using Singularity version 3.8.5 or 3.8.4 and fuse-overlayfs version 1.7.1 or 1.8.1, it is all working as expected. The same set of scripts is also working on our HPC cluster running Ubuntu-20-LTR and Singularity version 2.8.4.
However, the same scripts with the same Singularity containers have problems when running on Rocky Linux release 8.5:
Within the container upgrading EasyBuild to the latest fails with the following error message:

ERROR: Build of /dev/shm/easybuild/eb-kwdk5vzy/tmp5x62k1bu/easybuilders/easybuild-easyconfigs-develop/easybuild/easyconfigs/e/EasyBuild/EasyBuild-4.5.2.eb failed (err: "build failed (first 300 chars): Failed to create directory /apps/easybuild/e/EasyBuild: [Errno 524] Unknown error 524: '/apps/easybuild/e'")

For what it is worth, on Rocky I get:

$ ./fuse-overlayfs --version
fuse-overlayfs: version 1.7.1
FUSE library version 3.4.1
using FUSE kernel interface version 7.27
$ uname -r
4.18.0-348.12.2.el8_5.x86_64

On Debian I get:

$ ./fuse-overlayfs- --version
fuse-overlayfs: version 1.7.1
FUSE library version 3.4.1
using FUSE kernel interface version 7.27
$ uname -v
#1 SMP Debian 5.10.46-4~bpo10+1 (2021-08-07)

Right now I got the feeling there is something we are missing and depending on where fuse-overlayfs is running on, it either works, or I get the error message above. It would be nice to be able to use a newer version (see comments from @boegel and @bedroge) for a number of reasons.

Happy to provide more information.

boegel · 2022-01-29T17:08:41Z

I've tested this with some newer versions last year (don't remember which ones, but I guess 1.4.0, 1.5.0, and/or 1.6), but those didn't solve the issue. However, today I tried version 1.8, and that one does seem to solve the Operation not permitted issues for me. Also 1.7 seems to work fine. I'll do some more tests to confirm this, and then close this issue.

Looks like Bob was a tad bit too optimistic here, we had to revert back to fuse-overlayfs 0.3 (see EESSI/filesystem-layer#114), since we were still having trouble with version 1.8 in the end... :(

DrDaveD · 2022-10-24T19:57:39Z

This has become a bigger problem for apptainer (the new name for singularity) now while running in non-setuid mode because it causes the --writable-tmpfs option to fail wherever the fuse-overlayfs version is 1.7 or greater (see apptainer/apptainer#796). By a binary search I narrowed it down to this commit that enables FUSE_CAP_POSIX_ACL. Fortunately, in #323 which went into version 1.8 a noacl option was added to turn that off, and the older version I tested (0.7.2-6.el7_8) quietly ignored the option, so for apptainer I plan on always adding that option. The only version that won't fix is 1.7.

boegel · 2023-06-06T09:20:34Z

More info:

We didn't observe this problem while we were using fuse-overlayfs v1.9 in a Debian 11.5 container (cfr. update build container to Debian 11.5 + CernVM-FS 2.10.0 + fuse-overlayfs 1.9 + awscli 1.27.8 EESSI/filesystem-layer#133).
Problems re-emerged with fuse-overlayfs v1.12 in a Debian 11.7 container (cfr. update build container to use Debian 11.7 + fuse-overlayfs 1.12 EESSI/filesystem-layer#152), where we observed a simply cd (though in Python) failing:

== 2023-06-06 07:22:49,779 build_log.py:171 ERROR EasyBuild crashed with an error (at easybuild/tools/build_log.py:111 in caller_info): Failed to change from /tmp/vsc40023/easybuild/build/EasyBuild/4.7.2/system-system/easybuild-easyconfigs-4.7.2 to /cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/amd/zen3/software/EasyBuild/4.7.2: [Errno 1] Operation not permitted: '/cvmfs/pilot.eessi-hpc.org/versions/2023.04/software/linux/x86_64/amd/zen3/software/EasyBuild/4.7.2' (at easybuild/tools/filetools.py:427 in change_dir)

edit:

fuse-overlayfs v1.10 works fine, v1.11 doesn't (same Operation not permitted error)

sven-hansen · 2023-06-09T10:33:10Z

We tripped over this issue as well. Our installation environment is very similar to EESSI but we use Apptainer (1.1.8) as the runtime. The commit history made me suspicious and a quick bisect confirmed that this issue was introduced with 2666df2

DrDaveD · 2023-06-09T19:43:04Z

@sven-hansen Could you please also submit an apptainer issue with a reproducer? I have fuse-overlayfs-1.11 on my el8 VM but haven't run across this, I don't think. As mentioned above, apptainer always uses the noacl option.

boegel mentioned this issue Jun 25, 2021

Fixes for fuse-overlayfs in container images EESSI/filesystem-layer#87

Merged

bedroge mentioned this issue Jan 12, 2022

Use latest version of prebuilt fuse-overlayfs binary, and bump versions of CVMFS, awscli, Debian for client and build containers EESSI/filesystem-layer#113

Merged

afbjorklund mentioned this issue Oct 17, 2022

Using --writable-tmpfs fails run with fuse-overlayfs apptainer/apptainer#796

Closed

boegel mentioned this issue Nov 12, 2022

update build container to Debian 11.5 + CernVM-FS 2.10.0 + fuse-overlayfs 1.9 + awscli 1.27.8 EESSI/filesystem-layer#133

Merged

boegel mentioned this issue Jun 6, 2023

stick to fuse-overlayfs v1.10 in build container, since v1.12 again triggers 'Operation not permitted' error for basic operations EESSI/filesystem-layer#153

Merged

stsp mentioned this issue Aug 1, 2024

"Operation not permitted" when trying to open file for write #428

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Operation not permitted" in Singularity containers #232

"Operation not permitted" in Singularity containers #232

bedroge commented Aug 31, 2020

dtrudg commented Aug 31, 2020 •

edited

Loading

bedroge commented Aug 31, 2020 •

edited

Loading

bedroge commented Aug 31, 2020

boegel commented Sep 21, 2020

bedroge commented Nov 24, 2020

bedroge commented Nov 24, 2020

DrDaveD commented Nov 24, 2020 •

edited

Loading

srd424 commented Jan 15, 2021

DrDaveD commented Jan 19, 2021

bedroge commented Jan 12, 2022

sassy-crick commented Jan 27, 2022

boegel commented Jan 29, 2022

DrDaveD commented Oct 24, 2022

boegel commented Jun 6, 2023 •

edited

Loading

sven-hansen commented Jun 9, 2023

DrDaveD commented Jun 9, 2023

"Operation not permitted" in Singularity containers #232

"Operation not permitted" in Singularity containers #232

Comments

bedroge commented Aug 31, 2020

dtrudg commented Aug 31, 2020 • edited Loading

bedroge commented Aug 31, 2020 • edited Loading

bedroge commented Aug 31, 2020

boegel commented Sep 21, 2020

bedroge commented Nov 24, 2020

bedroge commented Nov 24, 2020

DrDaveD commented Nov 24, 2020 • edited Loading

srd424 commented Jan 15, 2021

DrDaveD commented Jan 19, 2021

bedroge commented Jan 12, 2022

sassy-crick commented Jan 27, 2022

boegel commented Jan 29, 2022

DrDaveD commented Oct 24, 2022

boegel commented Jun 6, 2023 • edited Loading

sven-hansen commented Jun 9, 2023

DrDaveD commented Jun 9, 2023

dtrudg commented Aug 31, 2020 •

edited

Loading

bedroge commented Aug 31, 2020 •

edited

Loading

DrDaveD commented Nov 24, 2020 •

edited

Loading

boegel commented Jun 6, 2023 •

edited

Loading