Stacked, or bind mounts lead to kernel lockout on write files #45
Comments
|
Hello mirek967,
mirek967:
mount -t aufs
mount -t aufs -o br:/mnt/test/overlay.root=rw:/=ro none /mnt/test/root
mount -t aufs -o br:/mnt/test/overlay.var=rw:/var=ro none /mnt/test/root/var
Now when I try to write something in any file in /mnt/test/root/var
the job get hard lock, and aufs is gone, without any message.
Thanks for the report.
Now gimme more info.
(from aufs README)
----------------------------------------
- /proc/mounts (instead of the output of mount(8))
- /sys/module/aufs/*
- /sys/fs/aufs/* (if you have them)
- /debug/aufs/* (if you have them)
- linux kernel version
if your kernel is not plain, for example modified by distributor,
the url where i can download its source is necessary too.
- aufs version which was printed at loading the module or booting the
system, instead of the date you downloaded.
- configuration (define/undefine CONFIG_AUFS_xxx)
- kernel configuration or /proc/config.gz (if you have it)
- LSM (linux security module, if you are using)
- behaviour which you think to be incorrect
- actual operation, reproducible one is better
----------------------------------------
J. R. Okajima
|
|
Let me start from the kernel. |
|
mirek967:
Let me start from the kernel.
OK thanx, but do you have a plan to fulfil other infos?
Can I be waiting for them?
J. R. Okajima
|
|
Yes, of course. I think I will do it on weekend. |
|
"J. R. Okajima":
mirek967:
> Let me start from the kernel.
OK thanx, but do you have a plan to fulfil other infos?
Can I be waiting for them?
And I'd ask you to identify the systemcall which blocks.
For example,
- fix a command to "write something in any file in /mnt/test/root/var"
- run "strace -f -o /tmp/s the_command_you_chose"
- and give me "/tmp/s"
J. R. Okajima
|
|
I stored requested files. |
|
I made one more try. According to information from strace, the last command was chown. |
|
mirek967:
How did I repeat the problem:
open a text file in mc editor. Change some text in that file, and it get locked when I try to save the file.
----------------------------------------
I made one more try. According to information from strace, the last command was chown.
I tried to make chown directly on that file, with the same result. Command got locked.
Thanks.
Then let's fix that the command is chown.
I'd want you to try
- enable CONFIG_AUFS_DEBUG and recompile
- try this sequence
# echo 8 > /proc/sys/kernel/printk
# echo 1 > /sys/module/aufs/parameters/debug
# chown brabra /var/tmp/opis.txt (or whatever)
# echo 0 > /sys/module/aufs/parameters/debug
and show me the kernel log please.
J. R. Okajima
|
|
I did it. Please see attached file. |
|
mirek967:
[log.txt](https://github.com/user-attachments/files/17033236/log.txt)
I did it. Please see attached file.
Thanks, but was it the case 'chown' locked?
I see no errors, or some last part of the log were gone?
J. R. Okajima
|
|
Yes, it is locked. I still didn't reboot. |
|
mirek967:
Yes, it is locked. I still didn't reboot.
In dmesg I don't see anything more than that.
OK thanks.
It means that the easy debug print method doesn't give us good
info. Let's try another one.
- enable CONFIG_MAGIC_SYSRQ
- enable CONFIG_AUFS_DEBUG and CONFIG_AUFS_MAGIC_SYSRQ
- run "chown brabra..." and make your system locked
- press SysRq (Alt+PrintScreen) + w,
SysRq + t,
SysRq + d,
SysRq + a,
and show me the kernel log.
J. R. Okajima
|
|
I don't know if it will be helpfull. Because I don't have old kernel, I made it on newly compiled one. Right now it crashes directly when I change owner of the file. So there is crash dump.
|
|
mirek967:
I don't know if it will be helpfull. Because I don't have old kernel, I made it on newly compiled one. Right now it crashes directly when I change owner of the file. So there is crash dump.
Maybe the reason is, because I compiled in debugfs.
It is not helpful.
It's a different kernel and different symptom. Let's address the
problems one by one, step by step. Shall we focus the 6.6.51-gentoo
lockup problem? And would you try Magic SysRq which I posted previously?
J. R. Okajima
|
|
Please see attached file. |
|
mirek967:
Please see attached file.
There is SysReq W, and T.
SysReq D, and A didn't work.
[kern.log.zip](https://github.com/user-attachments/files/17340626/kern.log.zip)
Thanks for the log.
Now I can see the suspicious change is f2fs in v6.6.44.
f44a25a8bfe0 2024-08-03 f2fs: fix to truncate preallocated blocks in f2fs_file_open()
The commit adds inode_lock() in file-open, and it causes a deadlock in
aufs internal copy-up.
Here is a workaround for this problem which unlocks the inode
termporarly, while I don't think it a good solution. Anyway please test
this patch.
J. R. Okajima
diff --git a/fs/aufs/cpup.c b/fs/aufs/cpup.c
index 8fb1f49bbe8e..6aeae7cba6db 100644
--- a/fs/aufs/cpup.c
+++ b/fs/aufs/cpup.c
@@ -530,7 +530,7 @@ static int au_do_cpup_regular(struct au_cp_generic *cpg,
goto out;
}
h_src_attr->valid = 1;
- if (!au_test_nfs(h_src_inode->i_sb)) {
+ if (!au_test_fs_unlock_for_open(h_src_inode->i_sb)) {
err = au_cp_regular(cpg);
inode_unlock_shared(h_src_inode);
} else {
diff --git a/fs/aufs/fstype.h b/fs/aufs/fstype.h
index 802f368431ad..b351ff170ca6 100644
--- a/fs/aufs/fstype.h
+++ b/fs/aufs/fstype.h
@@ -234,6 +234,15 @@ static inline int au_test_hfsplus(struct super_block *sb __maybe_unused)
#endif
}
+static inline int au_test_f2fs(struct super_block *sb __maybe_unused)
+{
+#if IS_ENABLED(CONFIG_F2FS_FS)
+ return sb->s_magic == F2FS_SUPER_MAGIC;
+#else
+ return 0;
+#endif
+}
+
/* ---------------------------------------------------------------------- */
/*
* they can't be an aufs branch.
@@ -401,5 +410,14 @@ static inline int au_test_nfs_noacl(struct inode *inode)
&& !nfs_server_capable(inode, NFS_CAP_ACLS);
}
+/*
+ * filesystems which requires unlocking inode before open.
+ */
+static inline int au_test_fs_unlock_for_open(struct super_block *sb)
+{
+ return au_test_nfs(sb)
+ || au_test_f2fs(sb);
+}
+
#endif /* __KERNEL__ */
#endif /* __AUFS_FSTYPE_H__ */
|
|
With this patch the problem does not occur anymore. |
|
mirek967:
With this patch the problem does not occur anymore.
The good thing is, that the root of the problem is found.
Thank you.
Thank you very much.
I will create and release aufs6.6.44 for this patch.
Also the patch will be included into aufs6.11.
J. R. Okajima
|
|
------- Blind-Carbon-Copy
From: "J. R. Okajima" ***@***.***>
To: ***@***.***
Subject: aufs6 GIT release (v6.12-rc3)
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: ***@***.***>
Date: Mon, 21 Oct 2024 00:22:11 +0900
Message-ID: ***@***.***>
o news
- - new branch aufs6.6.44 is released to support f2fs branch, though I
don't use f2fs. the commit is backported from aufs6.11. reported by
mirek967 on github.
J. R. Okajima
- ----------------------------------------
- - aufs6-linux.git#aufs6.6..aufs6.6.4
nothing
- - aufs6-linux.git#aufs6.6.44..aufs6.6.54
aufs: linux-v6.11, f2fs inode_lock() in file-open
- - aufs6-linux.git#aufs6.7..aufs6.10.13
nothing
- - aufs6-linux.git#aufs6.11..aufs6.x-rcN
aufs: linux-v6.11, f2fs inode_lock() in file-open
- - aufs6-standalone.git
ditto
- - aufs-util.git
nothing
…------- End of Blind-Carbon-Copy
|
|
Thank you. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I had situation, where I mount aufs filesystem inside other aufs filesystem. This leads into lockout.
Lets take following situation.
mount -t aufs
mount -t aufs -o br:/mnt/test/overlay.root=rw:/=ro none /mnt/test/root
mount -t aufs -o br:/mnt/test/overlay.var=rw:/var=ro none /mnt/test/root/var
Now when I try to write something in any file in /mnt/test/root/var
the job get hard lock, and aufs is gone, without any message.
The text was updated successfully, but these errors were encountered: