cannot remove container: ... running or paused containers cannot be removed without force: container state improper

[edsantiago]

New flake seen in remote f36-aarch64 root:

# podman-remote --url unix:/tmp/podman_tmp_nLiX run -d --log-driver=k8s-file quay.io/libpod/testimage:20220615 sh -c echo test1; echo test2
SHA
# podman-remote --url unix:/tmp/podman_tmp_nLiX logs --tail 1 SHA
test2
# podman-remote --url unix:/tmp/podman_tmp_nLiX restart SHA
SHA
# podman-remote --url unix:/tmp/podman_tmp_nLiX logs --tail 1 SHA
test2
# podman-remote --url unix:/tmp/podman_tmp_nLiX rm SHA
Error: cannot remove container SHA as it is running - running or paused containers cannot be removed without force: container state improper
[ rc=2 (** EXPECTED 0 **) ]

Not sure if this is a race in the test (missing podman wait?) or in podman-remote itself. Filing for later.

My flake logs shwo three instances, all remote, all aarch64.

[sys] 79 podman logs - tail test, k8s-file

• fedora-36-aarch64 : sys remote fedora-36-aarch64 root host [remote]
  • PR fixes data race in system.Events and nil error logging #15665
    • 09-07 16:18
  • PR Fix #15243 Set AutomountServiceAccountToken to false #15607
    • 09-06 06:33
  • PR Set enableServiceLinks to false in generated yaml #15571
    • 08-31 15:28

[edsantiago]

Different test, same symptom: f36-aarch64 again but not remote:

# podman info --format {{.Host.CgroupManager}}
systemd
# podman --cgroup-manager=cgroupfs run --name myc quay.io/libpod/testimage:20220615 true
# podman container inspect --format {{.HostConfig.CgroupManager}} myc
cgroupfs
# podman start myc
myc
# podman rm myc
Error: cannot remove container SHA as it is running - running or paused containers cannot be removed without force: container state improper
[ rc=2 (** EXPECTED 0 **) ]

This one does not look like a bug in the test, it actually looks like a bug-bug. podman start container-that-only-runs-true, followed by podman rm that-container, should never fail with "container is running".

[rhatdan]

@mheon This looks like we are waiting for podman container cleanup to change the state from running to complete?

[vrothberg]

Yes, that smells like a state-transition issue.

[Luap99]

That sounds like a test problem to me. podman start will only start the container process, there is absolutely zero guarantee that it will be finished by time we call podman rm, even if it is just true. After the container process finishes conmon needs to launch the podman container cleanup process.

The timing depends on the scheduler, I think the test assumption that this should work is wrong.

If you need a guarantee you either have to use podman wait or start the process in the foreground.

[vrothberg]

If you need a guarantee you either have to use podman wait or start the process in the foreground.

Isn't the test running the container in the foreground? There's no -d.

[edsantiago]

I think the bug is missing -a in podman start. I'll open a PR to address that.

[Luap99]

# podman start myc
myc
# podman rm myc
Error: cannot remove container SHA as it is running - running or paused containers cannot be removed without force: container > state improper
[ rc=2 (** EXPECTED 0 **) ]

This one does not look like a bug in the test, it actually looks like a bug-bug. podman start container-that-only-runs-true, followed > by podman rm that-container, should never fail with "container is running".

And also in the original report you see podman restart and run -d

[vrothberg]

Aaaah, nice catch, @Luap99! I somehow thought it was a run rm sequence.