APIv2: Events streaming behaves weirdly with socket activation

[marusak]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
This is collection of observations. I want to point out some finding. Also it does not work greatly with cockpit http client and I am not sure yet if this is problem on cockpit or podman side. I hope that we actually may figure out what is happening and see, what side needs fixing.

I am using current master build. Related #5158

1. Enable and start podman.socket (systemctl enable --now podman.socket)
2. Notice that podman.service is inactive (dead) (this is of course expected and correct)
3. Start listening on events. sudo curl --unix-socket /run/podman/podman.sock http://d/v1.24/events
4. Notice that podman.service is activating (start). ((1) Never running though - this is one interesting observation)
5. Wait for a bit to make sure it keeps streaming. After minute or two do some action and see it correctly logs it. (like podman run fedora and there are 6 events (from create to cleanup)) That is all correct.
6. Stop the events streaming (by ctrl+C).
7. Notice that podman.service never dies (2). It stays in activating (start) forever. (Well, at least for around 10minutes, not sure how this works, maybe there is some exponential backof?). I would guess that when no one is listening, the service should die.

Now, when I try to do the same with cockpit http client. (3)
Points 1. and 2. are the same. I create a new connection to /run/podman/podman.sock and call /events endpoint and attach streaming callback. It works for 5 seconds, and then it disconnects. It does not recognize that there is open connection. Looking into systemctl status podman I see it is in activating (start) for those 5 seconds and then goes back to inactive (dead). Streaming with Cockpit works fine when the podman system service does not die (started with -t 0).
I am not sure if this is problem in cockpit http client and in podman not recognizing connection.

If it does not ring any bells (mostly to @jwhonce), than I try to create some simple reproducer with Cockpit. Or if there is some way how I can provide some logs, please let me know.

EDIT: A few more info, how cockpit actually works:

1. It creates cockpit.channel with options: {"method":"GET","path":"/v1.12/events?","superuser":true,"payload":"http-stream2","unix":"/run/podman/podman.sock"}. http-stream2 documentation.

[marusak]

och, so after some fun debugging I found reproducer that does not need cockpit in any way.

1. systemctl enable --now podman.socket
2. make sure that podman.service is inactive
3. create executable file:

 $ cat podman_rep.sh 
sudo curl --unix-socket /run/podman/podman.sock http://d/v1.24/info
sudo curl --unix-socket /run/podman/podman.sock http://d/v1.24/libpod/images/json &
sudo curl --unix-socket /run/podman/podman.sock http://d/v1.24/events

4. /sudo ./podman_rep.sh

See that the last command won't stay around and dies in 5 seconds. If you comment out the first or the middle command, it stays around forever and when you do some podman action it will log events.
(Every time you start it make sure that podman.service is not running).

I was only able to reproduce it that one command finished, then another started and before it finishes we start to listen on events. (Thats why the middle command is run in background so I can start to listen on events before it lists all images - it is great if you have a few images locally so it is not too fast:) )

[marusak]

@jwhonce or @baude can you please try this if it reproduces for you as well or if it is a very specific timing issue that is happening on my machine. Thanks!

[marusak]

@jwhonce were you able to reproduce this or find any problem that could cause this behaviour?

[jwhonce]

@marusak I was able to reproduce the issue with your snippet. I'm debugging issue now. This is one of those infamous "That should never happen issues." The debugging output shows the server exiting with a live connection. I'll be sure and tag this issue in the PR with the fix.