pull: error setting new rlimits: operation not permitted

[EmilienM]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Podman fails to pull an image from a local docker-distribution registry (in non-rootless environment).

Steps to reproduce the issue:

1. Deploy a docker-distribution registry and pull docker.io/tripleomaster/centos-binary-haproxy:current-tripleo.

2. Now try to pull the container from the registry to local with podman: podman pull --tls-verify=false 192.168.24.1:8787/tripleomaster/centos-binary-haproxy:current-tripleo

Describe the results you received:

Pull fails with error setting new rlimits: operation not permitted error.

Logs:
http://logs.openstack.org/19/616019/23/check/tripleo-ci-centos-7-containers-multinode/5ad3bd8/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-01-08_21_46_18

Describe the results you expected:

Pull should work, as it does fine with docker pull.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:       0.12.1.2
Go Version:    go1.10.2
OS/Arch:       linux/amd64

Output of podman info:

host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-0.12.1.2-2.git9551f6b.el7.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: 97d5f1232f62307696c8b4b2d430e8b771a45873-dirty'
  Distribution:
    distribution: '"centos"'
    version: "7"
  MemFree: 857681920
  MemTotal: 8364449792
  OCIRuntime:
    package: runc-1.0.0-57.dev.git2abd837.el7.centos.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 4616089600
  SwapTotal: 8588881920
  arch: amd64
  cpus: 8
  hostname: undercloud.localdomain
  kernel: 3.10.0-957.1.3.el7.x86_64
  os: linux
  rootless: false
  uptime: 2h 3m 2.36s (Approximately 0.08 days)
insecure registries:
  registries:
  - 192.168.24.1:8787
  - 192.168.24.3:8787
registries:
  registries:
  - registry.centos.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  ContainerStore:
    number: 87
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 74
  RunRoot: /var/run/containers/storage

Additional environment details (AWS, VirtualBox, physical, etc.):
http://logs.openstack.org/19/616019/23/check/tripleo-ci-centos-7-containers-multinode/5ad3bd8/logs/undercloud/var/log/extra/podman/podman_allinfo.log.txt.gz

[EmilienM]

It probably fails here: https://github.com/containers/libpod/blob/ef29a3071202cc0fcc451ebdcaf948969bfe6306/cmd/podman/main.go#L154-L156

[rhatdan]

Are these numbers too large? Is there a kernel field, we can check to see what the root max is?

[EmilienM]

or maybe we need CAP_SYS_RESOURCE capability to perform this task; however I'm confused because we ran the process which run podman pull with sudo.

[rhatdan]

Does the number need to be less then pids_max.
cat /proc/sys/kernel/pid_max
32768

[EmilienM]

On fresh centos: 32768
But once podman has deployed containers, I have 1048576.

So since the value is already set, I wonder why it fails during the pull.

[rhatdan]

Weird. We could check if the limits are >= 1048576 and only set it in that case.

[rhatdan]

@giuseppe Any ideas?

[mheon]

EPERM for setrlimit is defined as:

An unprivileged process tried to raise the hard limit; the CAP_SYS_RESOURCE capability is required to do this. Or, the caller tried to increase the hard RLIMIT_NOFILE limit above the current kernel maximum (NR_OPEN). Or, the calling process did not have permission to set limits for the process specified by pid. 

The most likely one here seems to be CAP_SYS_RESOURCE, given that we are successfully setting this value at other points (so it's not above NR_OPEN).

[mheon]

Can we easily determine if Podman has CAP_SYS_RESOURCE?

[rhatdan]

@EmilienM Says he is just executing sudo podman. @EmilienM Is this repeatable? Any chance you attempted this while in a container?

[giuseppe]

should we just ignore the error? In most cases we don't need so many open files

[giuseppe]

I'd do something like:

diff --git a/cmd/podman/main.go b/cmd/podman/main.go
index 43804ee3..0a9e2c43 100644
--- a/cmd/podman/main.go
+++ b/cmd/podman/main.go
@@ -148,17 +148,17 @@ func main() {
                        logrus.SetLevel(level)
                }
 
-               // Only if not rootless, set rlimits for open files.
-               // We open numerous FDs for ports opened
-               if !rootless.IsRootless() {
-                       rlimits := new(syscall.Rlimit)
-                       rlimits.Cur = 1048576
-                       rlimits.Max = 1048576
+               rlimits := new(syscall.Rlimit)
+               rlimits.Cur = 1048576
+               rlimits.Max = 1048576
+               if err := syscall.Setrlimit(syscall.RLIMIT_NOFILE, rlimits); err != nil {
+                       if err := syscall.Getrlimit(syscall.RLIMIT_NOFILE, rlimits); err != nil {
+                               return errors.Wrapf(err, "error getting rlimits")
+                       }
+                       rlimits.Cur = rlimits.Max
                        if err := syscall.Setrlimit(syscall.RLIMIT_NOFILE, rlimits); err != nil {
                                return errors.Wrapf(err, "error setting new rlimits")
                        }
-               } else {
-                       logrus.Info("running as rootless")
                }

if you are fine with this approach, I can open a PR

[mheon]

I'd rather just make the Setrlimit error

[EmilienM]

if you are fine with this approach, I can open a PR

We might want to keep if !rootless.IsRootless() condition, right? Otherwise lgtm.

[giuseppe]

We might want to keep if !rootless.IsRootless() condition, right? Otherwise lgtm.

no, it will work with rootless mode as well, an unprivileged process can still bump its rlimits to max.

Opened a PR here: #2126

[rhatdan]

Im fine with this, although add some logging about what happened. At least at the Warn level.

[giuseppe]

Im fine with this, although add some logging about what happened. At least at the Warn level.

added the logging