podman-remote: there's a hang somewhere

[edsantiago]

Another semi-useless report with no reproducer nor actual details.

There's a hang somewhere in podman-remote. It's causing flakes in CI all over the place. It does not seem to be deterministic, it's happening in lots of different situations.

Examples:

• From system tests: podman-remote, image tree #7238 - looks like it's in podman wait - log
• From podman-remote send name and tag #7220 - in pause/unpause - log
• In post-merge testing - looks like wait again - log

Sorry for sparse details, I'm trying very hard to be done for the week and am failing dismally. Please add examples here as you see them, I think this is going to be an ugly one.

[edsantiago]

Another wait https://cirrus-ci.com/task/6571684087988224

[edsantiago]

This script will reproduce it, although not quickly, you have to do it in a loop and wait possibly hours:

$ cat >7241.sh <<EOF
set -e
./bin/podman-remote run -d --name foo alpine sh -c 'touch /foo;while test -e /foo; do sleep 1;done'
cid=foo
./bin/podman-remote logs $cid
./bin/podman-remote exec $cid /etc             || true
./bin/podman-remote exec $cid no-such-command  || true
./bin/podman-remote exec $cid rm /foo
timeout -v 15 ./bin/podman-remote wait $cid
./bin/podman-remote rm $cid
EOF

$ while bash -x 7241.sh;do echo;echo;done
...copious output... eventually dies
+ timeout -v 15 ./bin/podman-remote wait foo
timeout: sending signal TERM to command ‘./bin/podman-remote’

$ podman ps -a
CONTAINER ID  IMAGE                            COMMAND               CREATED         STATUS                     PORTS   NAMES
c87ff4b0836f  docker.io/library/alpine:latest  sh -c touch /foo;...  15 minutes ago  Exited (0) 10 minutes ago          foo

I find it interesting that CREATED was 15 minutes ago, but it exited 10. Implying that it took 5 minutes between the exec rm /foo and the time the container exited.

I'm heading out for the day, but am leaving the container as it is in case anyone has suggestions for me to inspect or check logs or anything.

[zhangguanzhang]

same with you in "podman-remote wait $cid", in https://storage.googleapis.com/cirrus-ci-6707778565701632-fcae48/artifacts/containers/podman/5815345191583744/html/system_test.log.html.

# # podman-remote --url unix:/tmp/podman.6CurjS exec 3b0aee1ff91b0c0440cccc3963843b935fe767c05c48cca7216eac05935f54e2 rm -f /Q5gFVgzznJnlfcj6mZzL
# # podman-remote --url unix:/tmp/podman.6CurjS wait 3b0aee1ff91b0c0440cccc3963843b935fe767c05c48cca7216eac05935f54e2
# timeout: sending signal TERM to command ‘/var/tmp/go/src/github.com/containers/podman/bin/podman-remote’
# [ rc=124 (** EXPECTED 0 **) ]

exitCode is 124, so it seem be killed by something, but I test this on my local machine, it will be ok

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[mheon]

@edsantiago Still seeing this one?

[edsantiago]

Sorry, yes. Issue still present in master @ 54a61e3. It took 35 minutes of looping on my f32 laptop, but:

...
+ timeout -v 15 ./bin/podman-remote wait foo
timeout: sending signal TERM to command ‘./bin/podman-remote’
$ ./bin/podman ps -a                                                                                                                                  -
CONTAINER ID  IMAGE                            COMMAND               CREATED        STATUS                         PORTS   NAMES
316df2501ed6  docker.io/library/alpine:latest  sh -c touch /foo;...  6 minutes ago  Exited (0) About a minute ago          foo

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago the original report said that there were hangs all over the place, but I guess this is very rare now?

[edsantiago]

I skipped the failing test, which is one reason why we're not seeing the failure in CI any more.

I tried running the 7241.sh reproducer just now, saw no failures in one hour. I will try removing the skip and we'll see what happens in CI.

[rhatdan]

I believe this is fixed now, @edsantiago reopen if I am mistaken,