ZFS snapshot dir not accessible in non-root neededForBoot mounts

[amarshall]

Describe the bug

ZFS provides a <mountpoint>/.zfs/snapshot directory that automounts snapshots for easy access. This does not work for non-root mounts that are neededForBoot = true.

Steps to reproduce

{
  fileSystems = {
    "/" = { type = "zfs"; device = "rootpool/root"; };
    # Should `zfs set mountpoint=legacy rootpool/foo`
    "/foo" = { type = "zfs"; device = "rootpool/foo"; neededForBoot = true; };
  };
}

touch /foo/bar
zfs snapshot create rootpool/foo@test
ls /foo/.zfs/snapshot/test

Expected behavior

Expected to find the snapshot files (e.g. bar) in /foo/.zfs/snapshot/test, but is empty.

Additional context

• /proc/spl/kstat/zfs/dbgmsg contains lines like

  zfs_ctldir.c:1109:zfsctl_snapshot_mount(): Unable to automount /.zfs/snapshot/test error=512

• This appears to be related to Unable to access snapshot files using /.zfs/snapshot/testsnapshot/ openzfs/zfs#9461, however the dataset mounted at / does not exhibit this problem, only nested mounts.

• Removing neededForBoot = true resolves

• mount -t zfs rootpool/foo -o remount resolves.

Notify maintainers

Unsure who…

Metadata

Using recent nixos-unstable (e35dcc0).

 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.54, NixOS, 23.11 (Tapir), 23.11.20230922.e35dcc0`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.0`

[matthiasdotsh]

Same here.

However, I have to mount the snapshots slightly differently.

The following works for me:

mkdir -p /tmp/zfsmount
sudo mount -t zfs rpool/encrypted/safe@zfs-auto-snap_hourly-2024-03-26-06h00 /tmp/zfsmount

Your suggestion

sudo mount -t zfs rpool/encrypted/safe@zfs-auto-snap_hourly-2024-03-26-06h00 -o remount

Gives me:

mount: rpool/encrypted/safe@zfs-auto-snap_hourly-2024-03-26-06h00: mount point does not exist.

[Shawn8901]

actually tried to double check on my system, that is using zfsutil for mounting and it seems that i can not directly reproduce with that, so either something else has been fixed or using zfsutil (so non legacy mountpoints) does not trigger the issue

   "/persist" = {
      device = "rpool/safe/persist";
      fsType = "zfs";
      options = [
        "zfsutil"
        "X-mount.mkdir"
      ];
      neededForBoot = true;
    };

─ ls /persist/.zfs/snapshot
zrepl_20240409_202739_000  zrepl_20240410_201732_000  zrepl_20240411_195737_000

[Enzime]

@matthiasdotsh Try with the dataset instead of the snapshot

sudo mount -t zfs rpool/encrypted/safe -o remount

I've managed to reproduce this issue with both the legacy stage1 as well as the systemd stage1

The easiest workaround for this issue is to run mount -a -t zfs -o remount

In my personal setup I just add it as an extra command to the zfs-mount.service:

systemd.services.zfs-mount = {
  serviceConfig = {
    ExecStart = [ "${lib.getExe' pkgs.util-linux "mount"} -a -t zfs -o remount" ];
  };
};

This works with both legacy (mountpoint=legacy) and native (mountpoint=/...) ZFS datasets that have neededForBoot = true;

cc @ElvishJerricco

[Enzime]

I did some further testing and my previous setting led to my system being broken in some weird ways, the easiest workaround I have found is to set the mountpoint on all my datasets:

$ zfs set -u mountpoint=/ rpool/root

The -u flag means it'll update the mountpoint without unmounting/mounting anything which allows you to do it on an online system

Then instead I use this snippet:

systemd.services.zfs-mount = {
  serviceConfig = {
    ExecStart = [ "${config.boot.zfs.package}/sbin/zfs mount -a -o remount" ];
  };
};

All my ZFS datasets are marked as neededForBoot, so I'm not sure if this will cause issues if you also have non-legacy ZFS datasets that aren't needed for boot

[amarshall]

For what it’s worth, this seems to not be a problem with boot.initrd.systemd.enable = true (I haven’t A/B tested it yet, but I recently changed and no longer have that problem—or a lot of other issues tbh.)

[Shawn8901]

For what it’s worth, this seems to not be a problem with boot.initrd.systemd.enable = true (I haven’t A/B tested it yet, but I recently changed and no longer have that problem—or a lot of other issues tbh.)

I am also using systems on all of my hosts, so that could be the reason that I wasn't able to reproduce it.

[Enzime]

When I tested it I believe I was able to reproduce it with systemd stage1 and with the legacy stage1.

@Shawn8901 Did you check if the contents of the /persist/.zfs/snapshot/zrepl_20240409_202739_000 is correct?

[Shawn8901]

When I tested it I believe I was able to reproduce it with systemd stage1 and with the legacy stage1.

@Shawn8901 Did you check if the contents of the /persist/.zfs/snapshot/zrepl_20240409_202739_000 is correct?

Okay now I got it. Can repro. Hadn't read the inital content careful enough.

[amarshall]

Scratch that, it is indeed not working with systemd initrd either, I made a mistake when spot checking.

[iynaix]

The remount workaround used to work for me on zfs 2.2, but sadly it no longer does on zfs 2.3rc2.

Upon bisecting, the commit that introduced this regression is: openzfs/zfs@34118eac06fba83

[n-elderbroom]

This seems to be fixed for me in 2.3.0-rc3. The snapshot dirs all appeared empty for me previously, but work after updating. I didn't implement any of the workarounds for this bug, and made no config changes other than the update.

[Enzime]

Looks like this is the PR that should fix it in 2.3.0-rc3 openzfs/zfs#16646

[Hyffer]

Same here. now upgrade to zfs 2.3.0-rc3 and it works like a charm

[ElvishJerricco]

I'll close this then. Feel free to re-open if ZFS 2.3 did not resolve the issue.