Permission issue running with podman

[crcox]

Right off that bat, I realize that this is meant to work with docker and singularity. However, I have read that podman is supposed to be a drop-in replacement for docker. I have a server running RHEL 8 and docker is not supported, but podman is. If this issue cannot be replicated with docker, it's probably not an issue with fmri prep.

That said: when I try to run the container, I am immediately getting a permission error:

podman run --rm -it \
    -v /usr/local/freesurfer/license.txt:/opt/freesurfer/license.txt:ro \
    -v /data/chriscox/MRI/ds000157-download:/data:ro \
    -v /data/chriscox/MRI/ds000157-prep:/out poldracklab/fmriprep:1.4.0 \
    /data /out participant

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 10, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 303, in main
    sentry_setup(opts, exec_env)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/utils/sentry.py", line 77, in sentry_setup
    if dset_desc_path.exists():
  File "/usr/local/miniconda/lib/python3.7/pathlib.py", line 1329, in exists
    self.stat()
  File "/usr/local/miniconda/lib/python3.7/pathlib.py", line 1151, in stat
    return self._accessor.stat(self)
PermissionError: [Errno 13] Permission denied: '/data/dataset_description.json'
Sentry is attempting to send 0 pending error messages
Waiting up to 2.0 seconds
Press Ctrl-C to quit

This is my first time working with a container, so I am not sure how to debug/get any more useful information about what is going on while things are running or what the environment within the container looks like.

[effigies]

What are the permissions of /data/chriscox/MRI/ds000157-download and /data/chriscox/MRI/ds000157-download/dataset_description.json? I don't know how podman handles permissions, but the Docker container runs fMRIPrep as a non-root user. You may be able to get it to use your user with -u $UID, or perhaps by giving more liberal permissions to your data directories.

[crcox]

My permissions are set so that I own both as a non-root user:

[chriscox@relearn0 MRI]$ ls -ld ds000157-download/
drwxrwxr-x. 34 chriscox chriscox 4096 May 28 13:06 ds000157-download/
[chriscox@relearn0 MRI]$ ls -l ds000157-download/dataset_description.json
-rw-rw-r--. 1 chriscox chriscox 423 Jul 16  2018 ds000157-download/dataset_descr                                                               iption.json

When I run with -u chriscox I get:

podman run --rm -it -u chriscox -v /usr/local/freesurfer/license.txt:/opt/freesurfer/license.txt:ro -v /data/chriscox/MRI/ds000157-download:/data:ro -v /data/chriscox/MRI/ds000157-prep:/out poldracklab/fmriprep:1.4.0 /data /out participant
unable to find user chriscox: no matching entries in passwd file

[effigies]

I would try it with your actual $UID, not $USER, but that might still fail some kind of /etc/passwd check.

But anyway, if it works like Docker, you should be able to enter the container and poke around:

host$ podman run --rm -it \
    -v /data:/data:ro --entrypoint=bash \
    poldracklab/fmriprep:1.4.0

container$ ls -l /data{,/chriscox{,/MRI{/ds000157-download}}}

[crcox]

Thanks for iterating on this with me!

If I use $UID, I don't get stopped by a /etc/passwd check, but run into the same permission error.

[chriscox@relearn0 MRI]$ podman run --rm -it -u $UID \
    -v /usr/local/freesurfer/license.txt:/opt/freesurfer/license.txt:ro \
    -v /data/chriscox/MRI/ds000157-download:/data:ro \
    -v /data/chriscox/MRI/ds000157-prep:/out \
    poldracklab/fmriprep:1.4.0 /data /out participant

Traceback (most recent call last):I/ds000157-prep:/out poldracklab/fmriprep:1.4.                                                         File "/usr/local/miniconda/bin/fmriprep", line 10, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 303, in main
    sentry_setup(opts, exec_env)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/utils/sentry.py", line 77, in sentry_setup
    if dset_desc_path.exists():
  File "/usr/local/miniconda/lib/python3.7/pathlib.py", line 1329, in exists
    self.stat()
  File "/usr/local/miniconda/lib/python3.7/pathlib.py", line 1151, in stat
    return self._accessor.stat(self)
PermissionError: [Errno 13] Permission denied: '/data/dataset_description.json'
Sentry is attempting to send 0 pending error messages
Waiting up to 2.0 seconds
Press Ctrl-C to quit

If I use the command you provided above, I am able to enter the container. It looks like the permissions on the /data directory are being set to root, and the group is nogroup.

[chriscox@relearn0 MRI]$ podman run --rm -it \
>     -v /data:/data:ro --entrypoint=bash \
>     poldracklab/fmriprep:1.4.0
root@7da8257a25be:/tmp# ls
root@7da8257a25be:/tmp# cd ../data/
root@7da8257a25be:/data# ls
ls: cannot open directory '.': Permission denied
root@7da8257a25be:/data# ls -ld .
drwxrwx---. 7 root nogroup 4096 May 24 17:50 .

EDIT: Realized that the command above wasn't actually mounting the right data directory. If I point to -v /data/chriscox/MRI/ds000157-download like I should have, the out come is essentially the same. The permissions are somewhat different:

[chriscox@relearn0 MRI]$ podman run --rm -it -v /data/chriscox/MRI/ds000157-download/:/data:ro --entrypoint=bash poldracklab/fmriprep:1.4.0
root@835a016551de:/tmp# ls
root@835a016551de:/tmp# cd ../data/
root@835a016551de:/data# ls
ls: cannot open directory '.': Permission denied
root@835a016551de:/data# ls -ld .
drwxrwxr-x. 34 root root 4096 May 28 18:06 .

[effigies]

I suggested mounting /data so that you could see at what point the permissions stop you from accessing contents. It looks like the problem is that /data doesn't have the o+x permission. My suspicion is that if you had that for all directories above yours, it would work.

I would suggest debugging further with your admin. I don't think there's anything we can do in fMRIPrep to work around these limitations.

[crcox]

Thank you so much. It turns out, podman does not behave exactly like docker. I came upon the solution while reading the following issue thread for podman. In short, if I add the --privileged flag, it works as expected.

[chriscox@relearn0 MRI]$ podman run --rm -it --privileged -v /data/chriscox/MRI/ds000157-download/:/data:ro --entrypoint=bash poldracklab/fmriprep:1.4.0
root@500e48676e27:/tmp# ls /data
CHANGES                   participants.json  sub-04  sub-09  sub-14  sub-19  sub-24  sub-29
README                    participants.tsv   sub-05  sub-10  sub-15  sub-20  sub-25  sub-30
annex-uuid                sub-01             sub-06  sub-11  sub-16  sub-21  sub-26  task-passiveimageviewing_bold.json
dataset_description.json  sub-02             sub-07  sub-12  sub-17  sub-22  sub-27
derivatives               sub-03             sub-08  sub-13  sub-18  sub-23  sub-28

Thank you for all your help with this!

[jirkadanek]

Might've been a SELinux issue. What worked for me in similar situation was -v /data/chriscox/MRI:/data:ro,Z. The Z tells podman to relabel.

[Wetiqe]

Might've been a SELinux issue. What worked for me in similar situation was -v /data/chriscox/MRI:/data:ro,Z. The Z tells podman to relabel.

Hello from GPT4 era, even new Bing can't give this solution. This works well for using docker on Fedora. Many thanks to you.