Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

APIv2: Rootless call hangs after stopping and starting service #8762

Closed
marusak opened this issue Dec 17, 2020 · 6 comments
Closed

APIv2: Rootless call hangs after stopping and starting service #8762

marusak opened this issue Dec 17, 2020 · 6 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@marusak
Copy link
Contributor

marusak commented Dec 17, 2020

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
In version 2.2.1 doing systemctl stop podman and then trying to query anything through API keeps it hanging indefinitely.

Steps to reproduce the issue:
(all done as normal user, this does not happen with system API)

  1. systemctl --user enable --now podman.socket
  2. Doing now some call works fine:
$ curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.24/libpod/info
{"host":{"arch":"amd64","buildahVersion":"1.18.0","cgroupManager":"systemd","cgroupVersion":"v2","conmon":...
  1. systemctl --user stop podman.service podman.socket
  2. systemctl --user enable --now podman.socket
  3. curl --unix-socket /run/user/1000/podman/podman.sock http://d/v1.24/libpod/info

The last step never retrurn anything, it just hangs indefinitely. If I ^C it and try again, all works fine.
While it hangs I can query states and all seems fine:

$ systemctl --user status podman
● podman.service - Podman API Service
     Loaded: loaded (/usr/lib/systemd/user/podman.service; static; vendor preset: disabled)
     Active: active (running) since Thu 2020-12-17 08:13:43 EST; 4s ago
TriggeredBy: ● podman.socket
       Docs: man:podman-system-service(1)
   Main PID: 19477 (podman)
      Tasks: 14 (limit: 1271)
     Memory: 101.8M
        CPU: 83ms
     CGroup: /user.slice/user-1000.slice/user@1000.service/podman.service
             ├─19050 /usr/bin/podman
             ├─19477 /usr/bin/podman --log-level=info system service
             └─19483 /usr/bin/podman --log-level=info system service

Dec 17 08:13:43 m1.cockpit.lan systemd[8261]: Starting Podman API Service...
Dec 17 08:13:43 m1.cockpit.lan systemd[8261]: Started Podman API Service.
Dec 17 08:13:43 m1.cockpit.lan podman[19477]: time="2020-12-17T08:13:43-05:00" level=info msg="/usr/bin/podman filtering at log level info"
Dec 17 08:13:43 m1.cockpit.lan podman[19477]: time="2020-12-17T08:13:43-05:00" level=warning msg="Error initializing configured OCI runtime kata: no valid executable found for OCI runtime k>
Dec 17 08:13:43 m1.cockpit.lan podman[19477]: time="2020-12-17T08:13:43-05:00" level=info msg="Setting parallel job count to 4"
Dec 17 08:13:43 m1.cockpit.lan podman[19483]: time="2020-12-17T08:13:43-05:00" level=info msg="/usr/bin/podman filtering at log level info"
Dec 17 08:13:43 m1.cockpit.lan podman[19483]: time="2020-12-17T08:13:43-05:00" level=warning msg="Error initializing configured OCI runtime kata: no valid executable found for OCI runtime k>
Dec 17 08:13:43 m1.cockpit.lan podman[19483]: time="2020-12-17T08:13:43-05:00" level=info msg="Setting parallel job count to 4"
Dec 17 08:13:43 m1.cockpit.lan podman[19483]: time="2020-12-17T08:13:43-05:00" level=info msg="using API endpoint: 'unix:/run/user/1000/podman/podman.sock'"
Dec 17 08:13:43 m1.cockpit.lan podman[19483]: time="2020-12-17T08:13:43-05:00" level=info msg="API server listening on \"/run/user/1000/podman/podman.sock\""

Output of podman version:

Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.5
Built:        Tue Dec  8 09:37:50 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 1
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: journald
  hostname: m1.cockpit.lan
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.9.14-200.fc33.x86_64
  linkmode: dynamic
  memFree: 557993984
  memTotal: 1150504960
  ociRuntime:
    name: crun
    package: crun-0.16-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.16
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 1291841536
  swapTotal: 1291841536
  uptime: 10m 49.39s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/admin/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.2.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/admin/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  volumePath: /home/admin/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438270
  BuiltTime: Tue Dec  8 09:37:50 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.2.1-1.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes
@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Dec 17, 2020
@marusak
Copy link
Contributor Author

marusak commented Dec 17, 2020

damn, this is pretty much the same I reported in #6660

marusak added a commit to marusak/cockpit-podman that referenced this issue Dec 17, 2020
@marusak
test: Add hack to work around stuck service
41a81cb 
See containers/podman#8762
marusak added a commit to cockpit-project/cockpit-podman that referenced this issue Dec 17, 2020
@marusak
test: Add hack to work around stuck service
3c8a6a5 
See containers/podman#8762
@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Jan 18, 2021

@marusak Are you still seeing this when using the main branch?

@marusak marusak mentioned this issue Jan 19, 2021
Closed
@marusak
Copy link
Contributor Author

marusak commented Jan 19, 2021

Is still happening with podman 2:3.0.0-0.136.dev.gita944f90.fc34 (what is in current rawhide)

@marusak
Copy link
Contributor Author

marusak commented Jan 19, 2021

Interestingly, this is only happening on fedora and rhel, in ubuntu and debian it seems to be working just fine.

@rhatdan
Copy link
Member

rhatdan commented Feb 4, 2021

Seems to be working on

$ podman version
Version: 3.0.0-rc2
API Version: 3.0.0
Go Version: go1.15.6
Built: Fri Jan 29 18:06:12 2021
OS/Arch: linux/amd64

@rhatdan rhatdan closed this as completed Feb 4, 2021
marusak added a commit to marusak/cockpit-podman that referenced this issue Feb 6, 2021
@marusak
Timeout getInfo after 5 seconds
5ba8358 
This call should be very quick. It is the first all that cockpit-podman does.
Unfortunately often the first call gets stuck, see:
containers/podman#8762
containers/podman#9251

When the call is stuck, the whole page is stuck in `Loading...`. When we
just reject it, the `Loading...` state moves on and we are presented
with alert to 'Start' the service. Clicking the button does the trick.

It is kind of hacky, but I think it is more user friendly then just
being stuck on a page.
marusak added a commit to marusak/cockpit-podman that referenced this issue Feb 7, 2021
@marusak
test: Add hack to work around stuck services and failing first calls
7ae8676 
The first rootless call often gets stuck or fails.
In such case we have alert banner to start the service (or just empty state).
A real user would just hit the button so lets do the same as this is always getting
back to us and we waste too much time reporting to podman with mixed results.
Examples:
    containers/podman#8762
    containers/podman#9251
    containers/podman#6660
marusak added a commit to cockpit-project/cockpit-podman that referenced this issue Feb 7, 2021
@marusak
Timeout getInfo after 5 seconds
b7ac80b 
This call should be very quick. It is the first all that cockpit-podman does.
Unfortunately often the first call gets stuck, see:
containers/podman#8762
containers/podman#9251

When the call is stuck, the whole page is stuck in `Loading...`. When we
just reject it, the `Loading...` state moves on and we are presented
with alert to 'Start' the service. Clicking the button does the trick.

It is kind of hacky, but I think it is more user friendly then just
being stuck on a page.
marusak added a commit to cockpit-project/cockpit-podman that referenced this issue Feb 7, 2021
@marusak
test: Add hack to work around stuck services and failing first calls
c507b70 
The first rootless call often gets stuck or fails.
In such case we have alert banner to start the service (or just empty state).
A real user would just hit the button so lets do the same as this is always getting
back to us and we waste too much time reporting to podman with mixed results.
Examples:
    containers/podman#8762
    containers/podman#9251
    containers/podman#6660
@martinpitt martinpitt mentioned this issue Sep 14, 2021
Open
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rhatdan @marusak @openshift-ci-robot