podman run --rootfs link/to//read/only/dir does not work

[siscia]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

It seems not possible to run a container using the --rootfs option when the exploded filesystem is in a read-only directory.

I found a couple of issues trying to run a container directly from an exploded (unpacked) filesystem directory.

I believe it is interesting for us to have them sorted, but I would like your feedback first.

The first issues is:

$ ll /cvmfs/unpacked.cern.ch/registry.hub.docker.com/library/centos\:centos7
lrwxrwxrwx 1 cvmfs cvmfs 79 Feb 17 11:02 /cvmfs/unpacked.cern.ch/registry.hub.docker.com/library/centos:centos7 -> ../../.flat/5e/5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd/
$ podman run --rootfs /cvmfs/unpacked.cern.ch/registry.hub.docker.com/library/centos\:centos7 /bin/bash
Error: /cvmfs/unpacked.cern.ch/registry.hub.docker.com/library/centos:centos7 is not an absolute path or is a symlink: OCI runtime error

Try to run an unpacked container from a symlink generate an error. The check is implemented in runc I am not sure the reasoning why this check was implemented. But it does not seems something too difficult to go around.

The error is generate here: https://github.com/opencontainers/runc/blob/4769cdf607b0d1cbd73e5407d9655da7511fe193/libcontainer/configs/validate/validator.go#L75

It would be interesting to understand why this happens and why the check was put in place, however it seems something that we can get around it in podman but passing the real path to runc instead of what we read from the input.

The second error seems more complex:

$ podman run --rootfs /cvmfs/unpacked.cern.ch/.flat/5e/5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd /bin/bash
Error: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/run/user/1001/containers/vfs-containers/a6e0915f16bcaa3d674e0897e12382f3ee0cf274afbd7afebaa77db2eb7cc046/userdata/hostname\\\" to rootfs \\\"/cvmfs/unpacked.cern.ch/.flat/5e/5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd\\\" at \\\"/cvmfs/unpacked.cern.ch/.flat/5e/5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd/etc/hostname\\\" caused \\\"open /cvmfs/unpacked.cern.ch/.flat/5e/5e35e350aded98340bc8fcb0ba392d809c807bc3eb5c618d4a0674d98d88bccd/etc/hostname: read-only file system\\\"\"": OCI runtime error

This issue seems a little more complex and I didn't manage yet to figure out the complete backtrace, but it seems that comes from runc as well. It seems like that the container cannot run if the roofs is in a read-only filesystem.

Is there a reason for this? Can we detect if we are try to run from a read-only filesystem and if we are, mount the filesystem like overlay so to make it writable?

Maybe this is an issue to open agains runc and not podman but after all I am interested that this use case works in podman.

Output of podman version:

Version:            1.9.0
RemoteAPI Version:  1
Go Version:         go1.10.1
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  gitCommit: ""
  goVersion: go1.10.1
  podmanVersion: 1.9.0
host:
  arch: amd64
  buildahVersion: 1.14.8
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.15, commit: '
  cpus: 4
  distribution:
    distribution: ubuntu
    version: "18.04"
  eventLogger: journald
  hostname: student-ThinkPad
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.3.0-46-generic
  memFree: 525975552
  memTotal: 8084377600
  ociRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 0.4.3
      commit: unknown
  swapFree: 1750331392
  swapTotal: 2147479552
  uptime: 7h 50m 32.14s (Approximately 0.29 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/smosciat/.config/containers/storage.conf
  containerStore:
    number: 42
    paused: 0
    running: 0
    stopped: 42
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/smosciat/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 1
  runRoot: /run/user/1001/containers
  volumePath: /home/smosciat/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 1.9.0~2 amd64 [installed]

[siscia]

@giuseppe , very sorry for bothering you on this one.

Do you know with who is appropriate to discuss this use case?

Should we just try to submit a PR?

[giuseppe]

hi @siscia. I have already reported the issue for runc some time ago: opencontainers/runc#586

Could you workaround it with a bind mount as suggested in the issue I've linked above?

[siscia]

Hi @giuseppe :)

yes, for my specific use case I can do it. However it would be quite hard to drive adoption if people are forced to create mount points.

For us, the problem lies mostly on the read-only limitation, than on the symlink one.

I don't know what is the strategy of podman here, but if podman could detect that the filesystem we are trying to use is read-only, it could create an overlay filesystem, and in that one execute runc. This would solve both problems.

Do you think that a PR that follow these lines would get accepted?

[giuseppe]

that is a good feature for Podman, I agree. I am just not sure about the CLI for exposing it.

We could either add a new flag --rootfs-opt or extend the rootfs argument like:

podman run ... --rootfs ... /path/to/the/rootfs:list,of,arguments

And one of those arguments could be overlay=path/to/upperdir. Have it automatically handled by the storage library is more difficult, as at the moment --rootfs just skips any access to container/storage; so we would need to move the understanding of --rootfs to containers/storage first.

[rhatdan]

Does this work with crun?

[siscia]

I see how a smart handle of the --rootfs would be complex.
Even thought it would be a quite nice interface!

Also, the use of a different runtime, might be simpler to sell than teach the correct use of the --rootfs arguments...

Thanks for the feedback!

[giuseppe]

Does this work with crun?

yes, a symlink in the rootfs path is accepted by crun

[rhatdan]

Then I would prefer to switch the OCI Runtime, or get the fix into runc. Don't really want to make podman handle this.

[giuseppe]

Then I would prefer to switch the OCI Runtime, or get the fix into runc. Don't really want to make podman handle this.

that is only one part of the issue. The part to handle in podman would be a read only rootfs where we create the overlay on top of it. That is also very useful for using exploded images that can come from any source (really interesting with network file systems) and that we can still use as a base for multiple containers.

[rhatdan]

Need to be careful there, since some people might do a
podman mount

And then attempt to run containers on the overlay mounted rootfs.

[siscia]

Hi @rhatdan

I am not sure I understand the concern here. There would be security implication?

I mean, I can always figure out where the container filesystem is with the right permission.

Are you afraid that people will change the underline filesystem while a container is running?

Cheers,
Simone

[rhatdan]

No security issue, this is more about breakage. Since overlay on Overlay is not supported, Someone doing a podman mount as a readonly mount and then doing --rootfs on it, will fail.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[hackerde]

Is there any update on this?

[giuseppe]

Is there any update on this?

what issue are you having with rootfs? The same issue as in the original post?

[hackerde]

Yes. I have flat singularity images stored on a cvmfs repo and it cannot start the container with that directory as --rootfs since it is a read-only filesystem. Copying the entire image over to a non-read-only place starts the container properly.

[rhatdan]

Do you know what inodes is it complaining about. You could add an overlay mount on the rootfs to allow podman to write to it.
Podman will attempt to modify an image if it does not have certain inodes like /etc/resolv.conf and /etc/hosts.

[giuseppe]

yes the way to work on a read-only file system is through an overlay mount.

We are currently working on a better integration between cvmfs and podman so that you won't need to set it up manually (cvmfs/cvmfs#2582).

[rhatdan]

@giuseppe We really need to document what inodes need to be present to run a podman on a read/only rootfs. I believe we should be able to work, as long as the necessary inodes are present, even without an overlay.

[giuseppe]

the easiest way to make sure all the needed directories/files are present is to run once a container in writeable mode:

$ podman run --rm --rootfs /path/to/rootfs true

The command above will create all the missing directories needed to run the container.

After that, it can be used in read only mode, by multiple containers at the same time:

$ podman run --read-only --rootfs /path/to/rootfs ....

[rhatdan]

Should we document this in the man page on --rootfs? Or elsewhere?

[giuseppe]

I am not sure, it looks like a hack to me. --rootfs is not really meant to be used in read-only mode.

[rhatdan]

I don't see it one way or another, I just think we document it and allow people to use --rootfs the way they want.

[rhatdan]

Opened docs PR to be able to close this issue.

[siscia]

Many many thanks!