XFS Quota Enforcement Issue: Only Applied at Volume Root, Not Inside _data for Second and Subsequent Podman Volumes #25368
Comments
|
It's not Still, I find it very curious that |
|
Alternatively we could just add more native support for |
It's not - I made a typo in the theory part. Will omit it. |
ak89224
changed the title
And yes I can confirm that for this syntax |
|
Your reproducer says
But you logs shows
These are two different things, the first creates a normal bind mount and tries to enable quotas, the second however mounts a new filesystem (xfs) and all the So I would say it is very much expected that the second command does not work. The first however should work per docs. |
Thanks, @Luap99 With the 1st option where it's creating a normal bind it doesn't pass the mount options forward. But with this I ran into another issue, that the volume quota option ( Steps to reproduce:
My Hypothesis:
@Luap99 , @mheon |
|
stripProjectInherit is only used for the top-level directory. The |
Ahh...I see. Yes, but still this behavior is unexpected ! Also direct write on the host confirms that at filesystem level xfs quotas are only enforced on volume name directory not on its children subdirectories ( @mheon @Luap99 -------------- Logs Below (Concerned parts are marked with **) ------------ |
|
I wanted to kindly follow up on this bug. I understand you are busy, but I'm still encountering this issue and would greatly appreciate any guidance when you have a moment. Revalidation StepsI set up a fresh environment on a different machine and followed the documented steps for XFS quota setup, ensuring project ID offsets were configured as per the Podman Quota documentation. Observations
Unexpected Behavior
Hypothesis
Would appreciate any insights or suggestions on this. Happy to provide more details or run additional tests if needed. Thanks! |
|
You will note that both include a log line about setting a quota: What do you mean by "at the volume name directory level"? The outer directory is unused aside from containing |
By volume named directory I mean the the outer directory, Agree both the volumes are being assigned the expected unique id's but for the 2nd volume if you see, And this part of code is only triggered for 2nd volume and onwards. You can see this line of debug log is missing from the 1st volume creation. |
|
That log line simply means we already have the quota ID to use cached; it doesn't indicate any difference in functionality, so long as the correct quota ID is set in |
Agree, but form the 2nd volume onwards, quota enforcement is broken and what's different is that it call the setProjectID which sets the inherit flag on the That's why I'm saying that might me an area of concern. |
|
Reproduces on 5.3.2. It's not PROJINHERIT, the flag is set properly. Current suspicion is it is due to the project quota of the base /var/lib/containers/storage/volumes directory being shared with the first volume created. |
|
So Smells like a potential kernel bug to me? The flag has been set properly, but the expected inheritance of the project ID by child directories and files is not occurring. |
Based on a quick read of the XFS project quota documentation and the way extended attributes (like the 'P' flag) work in Linux. Here's my understanding of Removing the Inherit Flag on the Parent: What happens: Removing the 'P' flag from the parent directory (e.g., Reason: Each directory (or inode) holds its own attributes at creation, and changes to a parent might not cascade to children that already have a set flag. Manually Enabling the Inherit Flag on a Child: What happens: If a child directory (like Impact on Existing Subdirectories: My assumption based on the fact that documentation does not mention any retroactive inheritence for children after creation. So, Existing subdirectories (like So, we could say that flag is being set properly but "no retroactive inheritence for existing children after creation" might also be a probable cause as well. |
|
Retroactive inheritance is likely the cause, now that I've been able to think about this for a day. We need to make @Luap99 This is the third major bug in quota support by my count. We need to get testing for this into CI. I think adding a system test with storage mounted on a loop device could work - does that sound like it would work? |
ak89224
changed the title
Yes some loop device created as root then formatted with xfs should work fine I think, that should also allow us devs to also run the test locally without requiring a special disk setup otherwise. |
We have previously relied on the PROJINHERIT flag for XFS quotas, which causes the ID of the parent directory to be recursively applied to subdirectories under the volume's parent directory. However, PROJINHERIT only applies to directories created after the project ID was first set. Pre-existing directories do not get the project ID if we only set it on the parent. This means that quota enforcement is not complete unless we recurse to subdirectories and apply the project ID to those as well. Fixes containers/podman#25368 Signed-off-by: Matt Heon <mheon@redhat.com>
|
@mheon thanks for the quick fix. |
We have previously relied on the PROJINHERIT flag for XFS quotas, which causes the ID of the parent directory to be recursively applied to subdirectories under the volume's parent directory. However, PROJINHERIT only applies to directories created after the project ID was first set. Pre-existing directories do not get the project ID if we only set it on the parent. This means that quota enforcement is not complete unless we recurse to subdirectories and apply the project ID to those as well. Fixes containers/podman#25368 Signed-off-by: Matt Heon <mheon@redhat.com>
We have previously relied on the PROJINHERIT flag for XFS quotas, which causes the ID of the parent directory to be recursively applied to subdirectories under the volume's parent directory. However, PROJINHERIT only applies to directories created after the project ID was first set. Pre-existing directories do not get the project ID if we only set it on the parent. This means that quota enforcement is not complete unless we recurse to subdirectories and apply the project ID to those as well. Fixes containers/podman#25368 Signed-off-by: Matt Heon <mheon@redhat.com>
We have previously relied on the PROJINHERIT flag for XFS quotas, which causes the ID of the parent directory to be recursively applied to subdirectories under the volume's parent directory. However, PROJINHERIT only applies to directories created after the project ID was first set. Pre-existing directories do not get the project ID if we only set it on the parent. This means that quota enforcement is not complete if we allow quotas to be set on directories that are not empty. We could set recursively but that comes with its own problems; quotas on directories that contain pre-existing files behave strangely. Relevant to containers/podman#25368 but is not a fix for that PR, more of a cleanup to make sure we don't make the same mistake elsewhere. Signed-off-by: Matt Heon <mheon@redhat.com>
Issue Description
When using the local volume driver with the
--opt o=size=10Moption on an XFS-backed storage (with project quotas enabled), Podman correctly creates the volume and assigns an XFS project quota. However, during container startup, Podman erroneously passes the “size” option as a mount parameter to the XFS mount command. Since XFS does not recognize a “size” mount option, the container fails to start with an error:mount: /mnt/data/containers/storage/volumes/testVolume/_data: fsconfig system call failed: xfs: Unknown parameter 'size'Steps to reproduce the issue
/dev/sdb1) as XFS and mount it at/mnt/datawithprjquotaenabled./etc/fstabentry:/etc/containers/storage.confto set graphroot:podman volume create --driver local --opt o=size=10M testVolumexfs_quota -x -c "report -p" /mnt/datapodman run --rm -v testVolume:/data busybox sh -c "dd if=/dev/zero of=/data/testfile bs=1M count=15"Describe the results you received
Actual Behavior:
--opt o=size=10M, Podman sets the project quota as expected.-o size=10M, which is rejected by the XFS mount system call, causing the container to fail to start.Debug Logs and Analysis:
xfs_quota), it later passes the “size=10M” mount option when mounting the volume inside the container.--opt o=size=10Moption, Podman mounts the volume successfully, and quotas can be manually enforced using XFS tools.Describe the results you expected
Expected Behavior:
--opt o=size=10Mparameter to set the XFS project quota on the volume (which it does) but should not pass a “size=10M” option to the mount system call when starting a container.podman info output
Podman in a container
No
Privileged Or Rootless
Privileged
Upstream Latest Release
Yes
Additional environment details
Environment:
prjquota)graphroot = "/mnt/data/containers/storage"Additional information
Impact:
Additional Information:
--opt o=size=10Moption into mount options for XFS.The text was updated successfully, but these errors were encountered: