Podman 2.2.0 is broken on Ubuntu Focal #8539
Comments
openshift-ci-robot
added
the
kind/bug
|
Can you provide the output of |
|
There you go: podman-info-debug.txt |
|
Adding mine here as well |
|
@ocafebabe It seems like yours indicates Podman is working? The one from @jdockter indicates that the directory we're using for the pause process is not the actual temporary directory - Podman is configured to use |
|
@mheon well, like I said in my bug report, it works with "sudo" but not as a regular user... |
|
@ocafebabe The directories in that debug log you provided certainly look rootless? And aside from the added debug logs, it seems to have run exactly as expected. |
|
Can you try other commands with `--log-level=debug and see if they also work? |
|
I tried the same command ( I get the same error regarding the tmp directory: |
|
@ocafebabe can you try |
I just did but I get the same error... |
Thanks! Creating a symlink from /tmp/run-1000 to /tmp/podman-run-1000 fixed the problem for me, but clearly this is a kludge...is there something we need to do our side for a permanent fix or will this be fixed in the next podman update? |
|
This is definitely a bug on our end and we'll see about getting it fixed in
a 2.2.1 soon.
…On Tue, Dec 1, 2020, 19:14 Sekhar Ravinutala ***@***.***> wrote:
@ocafebabe <https://github.com/ocafebabe> It seems like yours indicates
Podman is working?
The one from @jdockter <https://github.com/jdockter> indicates that the
directory we're using for the pause process is not the actual temporary
directory - Podman is configured to use /tmp/run-1000/libpod/tmp as a
temporary files directory, but it's looking for the pause process at
/tmp/podman-run-1000/libpod/pause.pid (note the run-1000 in the first,
versus podman-run-1000 in the second). @giuseppe
<https://github.com/giuseppe> PTAL
Thanks! Creating a symlink from /tmp/run-1000 to /tmp/podman-run-1000
fixed the problem for me, but clearly this is a kludge...is there something
we need to do our side for a permanent fix or will this be fixed in the
next podman update?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#8539 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB3AOCDAN7EJA6SN6N32IXTSSWBHFANCNFSM4UJLQXLQ>
.
|
|
I am seeing the same issue but with a different file name . Is there a fix available ? |
|
Now I'm seeing this error in our builds with Ubuntu 18. It was passing just 3 hours earlier today. |
|
Seems to be some kind of residue after: 3daef2e (v2.1.0-529-g3daef2e82) --- a/pkg/util/utils_supported.go
+++ b/pkg/util/utils_supported.go
@@ -38,7 +38,7 @@ func GetRuntimeDir() (string, error) {
}
}
if runtimeDir == "" {
- tmpDir := filepath.Join(os.TempDir(), fmt.Sprintf("run-%s", uid))
+ tmpDir := filepath.Join(os.TempDir(), fmt.Sprintf("podman-run-%s", uid))
if err := os.MkdirAll(tmpDir, 0700); err != nil {
logrus.Debug(err)
}
But uses |
|
Thanks to @afbjorklund comment, I was able to found another workaround: In my case, XDG_RUNTIME_DIR isn't set by pam systemd because I'm using "sudo" and apparently this isn't supported yet. |
|
That explains why it was working on my Ubuntu 20.04 desktop (with systemd)
Seems that it sneaked back into using vfs, wonder where fuse-overlayfs went ? Shouldn't that be in the
Added #8560 for overlay, and cgroups v1 was the reason for cgroupfs (in rootless) So far it seems happy with docker runc (1.0.0-rc10) instead of crio runc (v1.0.0-rc92) |
|
So what is the best workaround at the moment? Or is 2.2.1 when we see this fixed? |
|
For the moment, symlink I think we're probably just missing an MkdirAll somewhere to ensure that directory exists on reboot. |
|
For our GitHub Action and Travis CI, the symlink approach was good enough as workaround: freeipa/freeipa-container@de17382 |
|
Alright, I think I've figured this one out, and it's unfortunately somewhat more complicated than it seems. @rhatdan altered the way we get our default runtime path for systems not using systemd as PID1 (and without |
Previously, we always computed pause path from the Rootless runtime directory. Problem: this does not match the behavior of Libpod when the directory changes. Libpod will continue to use the previous directory, cached in the database; Pause pidfiles will swap to the new path. This is problematic when the directory needs to exist to write the pidfile, and Libpod is what creates the directory. There are two potential solutions - allow the pause pidfile to move and just make the directory when we want to write it, or use the cached Libpod paths for a guaranteed location. This patch does the second, because it seems safer - we will never miss a previously-existing pidfile because the location is now consistent. Fixes containers#8539 Signed-off-by: Matthew Heon <mheon@redhat.com>
|
I made #8556 to resolve this. Would greatly appreciate testing from anyone willing to build from source. |
|
For the record, I see the issue on CI's where podman is always installed afresh, so I'm not sure about the old cached data. |
Previously, we always computed pause path from the Rootless runtime directory. Problem: this does not match the behavior of Libpod when the directory changes. Libpod will continue to use the previous directory, cached in the database; Pause pidfiles will swap to the new path. This is problematic when the directory needs to exist to write the pidfile, and Libpod is what creates the directory. There are two potential solutions - allow the pause pidfile to move and just make the directory when we want to write it, or use the cached Libpod paths for a guaranteed location. This patch does the second, because it seems safer - we will never miss a previously-existing pidfile because the location is now consistent. Fixes containers#8539 Signed-off-by: Matthew Heon <mheon@redhat.com>
|
Symlink workaround, |
Previously, we always computed pause path from the Rootless runtime directory. Problem: this does not match the behavior of Libpod when the directory changes. Libpod will continue to use the previous directory, cached in the database; Pause pidfiles will swap to the new path. This is problematic when the directory needs to exist to write the pidfile, and Libpod is what creates the directory. There are two potential solutions - allow the pause pidfile to move and just make the directory when we want to write it, or use the cached Libpod paths for a guaranteed location. This patch does the second, because it seems safer - we will never miss a previously-existing pidfile because the location is now consistent. Fixes containers#8539 Signed-off-by: Matthew Heon <mheon@redhat.com>
|
Don't know if it's related, but applying symlink workaround on a fresh install leads to another error: Manually creating this directory solves the issue. EDIT: /var/lib/aidboxdb/ is $HOME |
|
I am having the same problem. For me a temporary workaround is |
|
@mheon I've tested e8cecd7 on Alpine Linux and it works fine. I'm waiting with the upgrade until it's merged to master or 2.2.1 is out: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/15142. |
Previously, we always computed pause path from the Rootless runtime directory. Problem: this does not match the behavior of Libpod when the directory changes. Libpod will continue to use the previous directory, cached in the database; Pause pidfiles will swap to the new path. This is problematic when the directory needs to exist to write the pidfile, and Libpod is what creates the directory. There are two potential solutions - allow the pause pidfile to move and just make the directory when we want to write it, or use the cached Libpod paths for a guaranteed location. This patch does the second, because it seems safer - we will never miss a previously-existing pidfile because the location is now consistent. Fixes containers#8539 Signed-off-by: Matthew Heon <mheon@redhat.com>
Previously, we always computed pause path from the Rootless runtime directory. Problem: this does not match the behavior of Libpod when the directory changes. Libpod will continue to use the previous directory, cached in the database; Pause pidfiles will swap to the new path. This is problematic when the directory needs to exist to write the pidfile, and Libpod is what creates the directory. There are two potential solutions - allow the pause pidfile to move and just make the directory when we want to write it, or use the cached Libpod paths for a guaranteed location. This patch does the second, because it seems safer - we will never miss a previously-existing pidfile because the location is now consistent. Fixes containers#8539 Signed-off-by: Matthew Heon <mheon@redhat.com>
|
The workaround with mkdir is working in WSL2. |
|
For reference, we are planning a 2.2.1 on Monday that will include the fix - wanted a few extra days to get more fixes for 2.2 issues landed. |
|
for me it's a different directory, but similar error message: $ sudo -u otheruser podman run --rm ubuntu:focal echo hi
ERRO[0000] cannot mkdir /home/otheruser/rundir/libpod: mkdir /home/otheruser/rundir/libpod: no such file or directory |
Travis CI has changed its policy regarding builds from OSS projects, and we have consumed all the amount of minutes they have on their free tier. This commit adds a pipeline for GHA and removes the one for Travis. It is temporarily disabling the podman builds until [1] is solved and a new version is released. [1]: containers/podman#8539
|
Hey Guys, error setting up the process: open /tmp/podman-run-2000/libpod/pause.pid: no such file or directory |
|
That's the same issue. Please try the symlink workaround mentioned earlier (link |
Previously, we always computed pause path from the Rootless runtime directory. Problem: this does not match the behavior of Libpod when the directory changes. Libpod will continue to use the previous directory, cached in the database; Pause pidfiles will swap to the new path. This is problematic when the directory needs to exist to write the pidfile, and Libpod is what creates the directory. There are two potential solutions - allow the pause pidfile to move and just make the directory when we want to write it, or use the cached Libpod paths for a guaranteed location. This patch does the second, because it seems safer - we will never miss a previously-existing pidfile because the location is now consistent. Fixes containers#8539 Signed-off-by: Matthew Heon <mheon@redhat.com>
|
@mheon when the 2.2.1 release hits kubic, will we need to remove the symlink workaround? Just trying to reduce any downtime we might have in our Travis builds. |
|
It should not be necessary - we'll go back to using the old path. The symlink will no longer be used, but it existing won't harm anything. Also, it should be rolling out soon (tag is pushed, I believe OBS is building) |
This reverts commit de17382. The issue was resolved in podman 2.2.1.
github-actions
bot
added
the
locked - please file new issue/PR
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
Steps to reproduce the issue:
Install Podman 2.2.0 using documented method/repo: https://podman.io/getting-started/installation
Execute:
podman infoDescribe the results you received:
Describe the results you expected:
Info output...
Additional information you deem important (e.g. issue happens only occasionally):
It works with sudoer accounts...
Output of
podman version:Output of
podman info --debug:Package info (e.g. output of
rpm -q podmanorapt list podman):podman/unknown,now 2.2.0~1 amd64 [installed]Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?
Yes
Additional environment details (AWS, VirtualBox, physical, etc.):
physical
The text was updated successfully, but these errors were encountered: