btrfs subvolume handling inconsistency

[davide125]

Problem description

In the Fedora Asahi Remix we use

<systemdisk name="fedora">
  <volume name="home"/>
  <volume name="root"/>
</systemdisk>

expecting to get two top-level subvolumes, root and home. Instead, kiwi creates a @ root subvolume and nests root and home under it. Compare the layout we get:

ID 256 gen 494 top level 5 path @
ID 257 gen 492 top level 256 path home
ID 258 gen 487 top level 256 path root
ID 259 gen 487 top level 256 path var/lib/machines

with the one on a stock Anaconda-installed Fedora system

ID 256 gen 519138 top level 5 path home
ID 257 gen 519129 top level 5 path root
ID 262 gen 519071 top level 257 path var/lib/machines

Expected behaviour

home and root are created as top-level subvolumes

Steps to reproduce the behaviour

https://pagure.io/fedora-asahi/kiwi-descriptions/blob/rawhide/f/platforms/workstation.xml is the full config snippet we're using

OS and Software information

• KIWI version: 9.24.59
• Operating system host version: Fedora Linux 36
• Operating system target version: Fedora Linux 38 and Rawhide
• Open Build Service version (N/A if not using OBS): N/A
• Koji version (N/A if not using Koji): N/A

[Conan-Kudo]

We should not assume SUSE-style btrfs hierarchies. Does adding mountpoint="/" (for root) and mountpoint="/home" (for home) fix this?

[schaefi]

We should not assume SUSE-style btrfs hierarchies. Does adding mountpoint="/" (for root) and mountpoint="/home" (for home) fix this?

Hmm, I think you are right. Looking at kiwi/volume_manager/btrfs.py you can see

def setup(self, name=None):
    """
    Setup btrfs volume management

    In case of btrfs a toplevel(@) subvolume is created and marked
    as default volume. ...
    """
    ...

    root_volume = self.mountpoint + '/@'        

So far there is no config option to influence this.

I think I remember at the time of the implementation for the volume layout that this was a recommendation from the btrfs people and not so much a SUSE style hierarchy. In any case to fix this without regressions we would need a new attribute

[davide125]

Does adding mountpoint="/" (for root) and mountpoint="/home" (for home) fix this?

It doesn't:

[ INFO    ]: 20:31:16 | Reading runtime config file: '/etc/kiwi.yml'
[ DEBUG   ]: 20:31:16 | EXEC: [mkdir -p /home/fedora/kiwi-descriptions/outdir/build]
[ INFO    ]: 20:31:16 | Loading XML description
[ INFO    ]: 20:31:16 | Support for multiple markup descriptions available
[ INFO    ]: 20:31:16 | --> loaded .//config.xml
[ INFO    ]: 20:31:16 | --> Selected build type: oem
[ INFO    ]: 20:31:16 | --> Selected profiles: Workstation-GNOME,WorkstationCommon,BootCore,DesktopCommon,Common,GNOME-Desktop
[ ERROR   ]: 20:31:16 | KiwiRuntimeError: 

Volume setup for "/" found. The size of the root volume
must be specified via the @root volume name like the
following example shows:

<volume name="@root" size="42G"/>

A custom name or mountpoint for the root volume is not
allowed.

[schaefi]

per conversation on matrix let's add a new attribute saying

<type ... btrfs_create_toplevel_subvolume="true|false" .../>

with the default set to true to avoid regressions

[schaefi]

I'm going to work on this once I find the time.

I was thinking about the suggestion from @iammattcoleman to define the toplevel volume explicitly. As we are using <volume> definitions for both LVM and filesystems with volume support (currently only btrfs), there is no concept for nested volumes (sub-volumes) in all of the supported volume managers. Thus a generic schema as suggested won't be possible. However we have a name tag in the <systemdisk> element which defines the name of the volume group in case of LVM and I think we could use this name to define the toplevel volume in case of btrfs. So it would look like this

<systemdisk name="@">
    <volume name="home"/>
</systemdisk>

leading to @/home

or

<systemdisk name="/">
    <volume name="home"/>
</systemdisk>

leading to /home

if not specified we have to keep the default to avoid regressions though

@Conan-Kudo @iammattcoleman Thoughts ?

[Conan-Kudo]

Isn't name= used for the label of the volume? I'm already using that to be set to fedora. What about adding btrfs_toplevel_mountpoint= to the top level? If it's btrfs_toplevel_mountpoint="", then there's no toplevel volume. If there's anything else, then it creates one and everything is nested underneath. Not setting it can default to @ for v9, and we can drop it for v10.

[iammattcoleman]

I'm not a btrfs user, actually, so I'm approaching this strictly from the perspective of wanting approachable and versatile XML and being a little concerned about having the @ volume name hardcoded into Kiwi.

What about adding an optional parent attribute to <volume>?

Then, @davide125's example in this issue's problem description could remain as:

<systemdisk name="fedora">
  <volume name="home"/>
  <volume name="root"/>
</systemdisk>

...and produce the two top-level volumes that they want.

Adding a top-level volume named @ to that same hierarchy could be accomplished by adding an explicit <volume> for @ and parent attributes to the other ones:

<systemdisk name="fedora">
  <volume name="@"/>
  <volume name="home" parent="@"/>
  <volume name="root" parent="@"/>
</systemdisk>

This would allow us to represent any nested hierarchy without altering the schema to allow the <volume> elements to nest. It would also allow the name attribute to continue being used for the label.

[Conan-Kudo]

That would work, and would allow representing the infinite hierarchy potential in Btrfs properly.

[schaefi]

Isn't name= used for the label of the volume?

no, the label of the volume is set by

<type ... rootfs_label="value"/>

The name attribute in the context of <systemdisk> is only used for the LVM volume manager to name the volume group. Also see:

• https://github.com/OSInside/kiwi/blob/master/kiwi/volume_manager/btrfs.py#L88

[schaefi]

However, I like @iammattcoleman suggestion with the new parent attribute. This attribute would have no meaning for LVM though

[schaefi]

In addition for naming the root volume we have added support for that to LVM in a124d82 The information used here is only used in LVM. I suggest in a first simple change we get rid of the fixed @ toplevel volume and evaluate the existing attribute for btrfs. This means

<volume name="@root=@"/>

[schaefi]

As specifying a toplevel volume usually means the volumes are below that volume I suggest to reverse the parent logic and allow to reparent, otherwise you have to add a parent to all volumes. So it concludes to

<systemdisk>
  <volume name="@root=@"/>
  <volume name="home" reparent="/"/>
  <volume name="root"/>
  <volume name="usr/local"/>
</systemdisk>

resulting in:

@/root
@/usr/local
/home

How about that ?

[Conan-Kudo]

When bcachefs gets merged, then the design @iammattcoleman proposed will work for that too. So it'll be generic to all but LVM, really.

[Conan-Kudo]

The problem with reversing the logic is that it doesn't logically make sense when you have no singular top-level subvolume. Basically the design that we want in Fedora is that we have two separate hierarchies: one for system data (root) and one for user data (home).

The resulting fstab in Fedora looks something like this:

/dev/nvme0n1p4          /                       btrfs   subvol=root,compress=zstd:1     0 0
/dev/nvme0n1p4          /home                   btrfs   subvol=home,compress=zstd:1     0 0

[schaefi]

When bcachefs gets merged, then the design @iammattcoleman proposed will work for that too. So it'll be generic to all but LVM, really.

I'm fine with the design from @iammattcoleman I just think you have to repeat yourself a hundred times with parent=@ on each volume definition, so why not revert the logic ? Would that prevent describing any nested structure ?

[schaefi]

The problem with reversing the logic is that it doesn't logically make sense when you have no singular top-level
subvolume.

ok got it

[Conan-Kudo]

We could just as easily also include a flag to create a top level, so that parent= isn't needed. But I think we'd want to support that too for being able to build hierarchies. And I don't think repeating ourselves is all that bad in this case.

[schaefi]

The resulting fstab in Fedora looks something like this:

/dev/nvme0n1p4          /                       btrfs   subvol=root,compress=zstd:1     0 0
/dev/nvme0n1p4          /home                   btrfs   subvol=home,compress=zstd:1     0 0

@Conan-Kudo Can you give me the btrfs commands that produced you this ?

Thanks

[Conan-Kudo]

Here's an example implementation: https://pagure.io/appliance-tools/blob/main/f/appcreate/partitionedfs.py#_316-324

[Conan-Kudo]

Basically, it's:

1. mkfs.btrfs
2. mount
3. btrfs subvolume create

[iammattcoleman]

I just had a chance to really take a look at #2324. Since I'm not a btrfs user and haven't yet used LVM with Kiwi, I had to refamiliarize myself with that part of the image description syntax.

My main objection is that the syntax is confusing and involves magic values (and a packed variant involving the magic value and some extra metadata) that alter Kiwi's behavior in ways that aren't made clear in the XML description. However, I hadn't remembered that the name attribute already has special handling for the value @root until I reread the docs.

I'd prefer to see that special handling of @root entirely removed, because the XML schema already allows for everything that's doing. What could be currently written as...

<volume name="@root=foo"/>

...could be slightly more verbosely written without a packed, magic @root=foo value...

<volume name="foo" mountpoint="/"/>

Taking this approach would result in XML that clearly describes the end result and doesn't require any special understanding of Kiwi's internals.

@davide125's example in this issue's problem description would become...

<systemdisk rootfs_label="fedora">
  <volume name="home" mountpoint="/home"/>
  <volume name="root" mountpoint="/"/>
</systemdisk>

[Conan-Kudo]

The current problem is maintaining backwards compatibility with existing descriptions.

We should be able to handle this by doing the following:

• Add btrfs_create_toplevel_subvolume="true|false" attribute to systemdisk, default to true for now
• Add optional parent= attribute to volume so that nested subvolumes can be represented. This would reference the parent volume by its name attribute
• Mark the current behavior as deprecated, and warn that btrfs_create_toplevel_subvolume will flip to false in the future, and the flag will eventually go away from descriptions, requiring the usage of the parent= attribute to set up subvolume hierarchies

We do not need any special magic names to process, and we don't need to make assumptions about how volume structures will work. This will also allow us to be prepared for bcachefs once that lands, too.

[schaefi]

The current problem is maintaining backwards compatibility with existing descriptions.

yes

We should be able to handle this by doing the following:
Add btrfs_create_toplevel_subvolume="true|false" attribute

done as commit to the open PR, now

Add optional parent= attribute

we will do this in a new PR

Mark the current behavior as deprecated, and warn

done, I hope the text makes sense. I don't think that we will be removing the new attribute in the future as it can stay without conflicts to the parent setup

[schaefi]

I'd prefer to see that special handling of @root entirely removed, because the XML schema already allows for everything that's doing. What could be currently written as... <volume name="@root=foo"/> ...could be slightly more verbosely written without a packed, magic @root=foo value... <volume name="foo" mountpoint="/"/>

It's true it could be expressed better that way but it would again be a backward compatibility issue. The special @root volume name to specify properties of what is considered the root volume exists since commit a124d82 In the LVM case we use it to specify e.g the size of root volume or the name of the root volume or whatever comes next. I wanted to keep this consistent and in a toplevel based btrfs setup that root volume is somewhat similar to the root volume in a LVM setup. So the overall idea with the @root volume name in kiwi is to have an identifier for a_root_volume and assign behavior to it. When I added that I was under the impression that any root filesystem has one node where it starts (its root) and that is the case for any filesystem type.
As the changes here are already intrusive enough I'd like to keep the @root volume name even though I understand it might be received cumbersome in connection with btrfs. Hope that makes sense ?