NerdCTL Fails to Find Bind Mount Path in /private/tmp on macOS Catalina

[Alexhuszagh]

Description

Version Info

The host OS is macOS, version 10.15.7, and the Lima version is 0.21.

Bug Description

On macOS, /tmp is a link to /private/tmp, and code that canonicalizes paths will use /private/tmp rather than /tmp for the host directory. A quick step to reproduce this is:

# to install grealpath
$ brew install coreutils
$ brew install lima
$ cd /tmp/lima
$ pwd
/tmp/lima
$ grealpath .
/private/tmp/lima
$ limactl start
$ cwd=$(grealpath .)
$ nerdctl.lima run --rm -it -v "${cwd}":/"${cwd}" ubuntu:20.04 bash
FATA[0000] no such file or directory
$ cwd=$(pwd)
$ nerdctl.lima run --rm -it -v "${cwd}":/"${cwd}" ubuntu:20.04 bash
# this works

My Lima info is the following:

$ nerdctl.lima info
Client:
 Namespace:	default
 Debug Mode:	false

Server:
 Server Version: v1.6.6
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Log: fluentd journald json-file
  Storage: native overlayfs stargz fuse-overlayfs
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
  rootless
 Kernel Version: 5.15.0-39-generic
 Operating System: Ubuntu 22.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.827GiB
 Name: lima-default
 ID: 73a6444c-8e28-4908-a1fa-0b1b74a3f7f2

WARNING: AppArmor profile "nerdctl-default" is not loaded.
         Use 'sudo nerdctl apparmor load' if you prefer to use AppArmor with rootless mode.
         This warning is negligible if you do not intend to use AppArmor.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

I am using all the default configurations.

Expected Behavior

Lima should properly find and then mount the /private/tmp directory, similar to Docker. This is also an issue in podman. Since it's a sticky directory, and therefore should always be present and a link to the same path, it may be possible to just remove the /private prefix if the canonicalized path starts with /private/tmp.

[jandubois]

This is working as expected; the default config maps /tmp and not /private/tmp into the VM:

lima/examples/default.yaml

Lines 85 to 88 in de9f354

	- location: "/tmp/lima"
	# 🟢 Builtin default: false
	# 🔵 This file: true (only for "/tmp/lima")
	writable: true

If you want to references files via /private/tmp/lima then you have to update the mount definition in lima.yaml.

[Alexhuszagh]

I guess then I've got 2 follow-up questions:

1. Why aren't symlinks followed if they resolved to the mounted directory, especially for cases where they're not meant to be transparent to the user (such as /private/tmp, which is meant to be compatible with UNIX-like systems)?
2. Can there be an option to do so? IE, canonicalize the mount directories first? If it's opt-in, then it would be quite nice. Allowing users to follow symlinks (opt-in though) in mounted directories outside feels like (IMO) quite a larger surface area than this.

Something like:

 - location: "/tmp/lima" 
   # 🟢 Builtin default: false 
   # 🔵 This file: true (only for "/tmp/lima") 
   writable: true 
   # Resolve any symlinks prior to mounting.
   # That is, if /tmp/a goes to /tmp/b, allow
   # either /tmp/a or /tmp/b to be mounted.
   canonicalize: true 

[jandubois]

1. Why aren't symlinks followed if they resolved to the mounted directory, especially for cases where they're not meant to be transparent to the user (such as /private/tmp, which is meant to be compatible with UNIX-like systems)?

I don't follow, the whole point of having a symlink is to be "transparent" to the user, so you can use /tmp and not worry about where the data is actually stored.

2. Can there be an option to do so? IE, canonicalize the mount directories first? If it's opt-in, then it would be quite nice. Allowing users to follow symlinks (opt-in though) in mounted directories outside feels like (IMO) quite a larger surface area than this.

Theoretically yes, but I don't see the point; why not put the target of the symlink into the lima.yaml file, if that's what you want:

mounts:
- location: /private/tmp/lima
  writable: true

This is a lot more explicit about what you want to happen.

[Alexhuszagh]

So I think we have different conceptions of what "transparent" means: if there's a symlink between /tmp and /private/tmp: you can (and should) use /tmp but the data is actually stored at /private/tmp, which is not immediately obvious until you canonicalize the path. The use of /tmp works in almost all cases, and most software never needs to consider the difference.

Users rarely should have to use canonical paths, so this indirection is great in most cases. But software sometimes does need canonical paths: it's useful both from a security and uniqueness perspective: if you need a unique path to use a lookup key in a hashtable, if you need to ensure something is not run from a certain directory that could be a user error, or if they should not have access to a resource.

The security side is almost entirely solved by Lima, which is great (thanks), but canonical paths still exist for a reason and software invoking Lima may canonicalize paths prior to passing them to Lima. The other solution, canonicalizing the paths for validation/uniqueness but then storing them as merely absolute or relative paths to interface with Lima, feels somewhat awkward.

[Alexhuszagh]

On another thought, it might be a bad idea if software mounted the canonical path into the container, since any data it tries to access using a path that isn't relative to the directory will probably break.