docker: Fix userdata bind mount path inside the container

[bpradipt]

For docker provider we use systemd container which mounts /run as a tmpfs volume.
This mounting of /run happens after docker has bind mounted the userdata file inside the container and consequently the userdata file is no longer accessible.

This commit changes the path of the userdata file to not use /run for the docker provider.

[bpradipt]

I'm not sure if there is a better way to solve the issue. Open for suggestion

[mkulke]

it's probably better to change the default mount path for the ci disk rather than adding an exception for docker, no? it doesn't need to be in /run I think, it can be in /mnt or wherever

[bpradipt]

it's probably better to change the default mount path for the ci disk rather than adding an exception for docker, no? it doesn't need to be in /run I think, it can be in /mnt or wherever

Yeah, having a uniform one is better. I was not sure if there is a hard dependency on /run for any platform. Let me just use /media as the mount path for the ci disk and check through ci if anything breaks. That should help.

[bpradipt]

One challenge is how to make /media mountpath available early in the boot cycle since root (/) is RO. Only the following are available via systemd - https://freedesktop.org/wiki/Software/systemd/APIFileSystems/

[bpradipt]

One challenge is how to make /media mountpath available early in the boot cycle since root (/) is RO. Only the following are available via systemd - https://freedesktop.org/wiki/Software/systemd/APIFileSystems/

Used /var/tmp/media/cidata. /var is tmpfs mounted for mkosi and also works for docker provider

[bpradipt]

The CI failures may be fixed via #2226

[wainersm]

Hi @bpradipt !

I finally managed to test this fix. It seems to fix the original problem: /var/tmp/media/cidata/user-data is present and the process-user-data service is up and running just fine.

However, I'm still unable to get a pod running. The problem that I'm seeing is related with image pull on guest:

=== RUN   TestDockerCreateSimplePod/SimplePeerPod_test/PodVM_is_created
    assessment_runner.go:389: unexpected warning/error event(s): Error: failed to create containerd container: error unpacking image: content digest sha256:9fa9226be034e47923c0457d916aa68474cdfb23af8d4525e9baeebc4760977a: not foundError: failed to create containerd container: error unpacking image: failed to extract layer sha256:1e604deea57dbda554a168861cff1238f93b8c6c69c863c43aed37d9d99c5fed: failed to get reader from content store: content digest sha256:9fa9226be034e47923c0457d916aa68474cdfb23af8d4525e9baeebc4760977a: not found

Above is a snippet of e2e test. I got a similar error when launching another pod from a different container image manually.

[bpradipt]

Hi @bpradipt !

I finally managed to test this fix. It seems to fix the original problem: /var/tmp/media/cidata/user-data is present and the process-user-data service is up and running just fine.

However, I'm still unable to get a pod running. The problem that I'm seeing is related with image pull on guest:

=== RUN   TestDockerCreateSimplePod/SimplePeerPod_test/PodVM_is_created
    assessment_runner.go:389: unexpected warning/error event(s): Error: failed to create containerd container: error unpacking image: content digest sha256:9fa9226be034e47923c0457d916aa68474cdfb23af8d4525e9baeebc4760977a: not foundError: failed to create containerd container: error unpacking image: failed to extract layer sha256:1e604deea57dbda554a168861cff1238f93b8c6c69c863c43aed37d9d99c5fed: failed to get reader from content store: content digest sha256:9fa9226be034e47923c0457d916aa68474cdfb23af8d4525e9baeebc4760977a: not found

Above is a snippet of e2e test. I got a similar error when launching another pod from a different container image manually.

@wainersm you are hitting the known nydus-snapshotter issue. Workaround is documented here - https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/docs/troubleshooting/nydus-snapshotter.md#pod-creation-fails-with-error-unpacking-image
Alternatively you can use crio runtime which doesn't have this issue.

[wainersm]

Hi @bpradipt !
I finally managed to test this fix. It seems to fix the original problem: /var/tmp/media/cidata/user-data is present and the process-user-data service is up and running just fine.
However, I'm still unable to get a pod running. The problem that I'm seeing is related with image pull on guest:

=== RUN   TestDockerCreateSimplePod/SimplePeerPod_test/PodVM_is_created
    assessment_runner.go:389: unexpected warning/error event(s): Error: failed to create containerd container: error unpacking image: content digest sha256:9fa9226be034e47923c0457d916aa68474cdfb23af8d4525e9baeebc4760977a: not foundError: failed to create containerd container: error unpacking image: failed to extract layer sha256:1e604deea57dbda554a168861cff1238f93b8c6c69c863c43aed37d9d99c5fed: failed to get reader from content store: content digest sha256:9fa9226be034e47923c0457d916aa68474cdfb23af8d4525e9baeebc4760977a: not found

Above is a snippet of e2e test. I got a similar error when launching another pod from a different container image manually.

@wainersm you are hitting the known nydus-snapshotter issue. Workaround is documented here - https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/docs/troubleshooting/nydus-snapshotter.md#pod-creation-fails-with-error-unpacking-image Alternatively you can use crio runtime which doesn't have this issue.

Interesting that I almost never hit that problem when running in a fresh environment like the one setup by the e2e testing provisioner. Let's see how it goes on CI...

Anyway, with the workaround above I could launch a simple pod \o/

[wainersm]

One challenge is how to make /media mountpath available early in the boot cycle since root (/) is RO. Only the following are available via systemd - https://freedesktop.org/wiki/Software/systemd/APIFileSystems/

I found in https://systemd.io/TEMPORARY_DIRECTORIES/ that use of /var/tmp has some security implications that we should be aware of. I read that doc fully and concluded we should be fine, however, I'm not an expert so better a 2nd check would be nice.

[wainersm]

I tested it locally and it worked out. It seems that we safe with using /var/tmp (keep in min I'm not an expert on the topic).

[bpradipt]

Hi @bpradipt !
I finally managed to test this fix. It seems to fix the original problem: /var/tmp/media/cidata/user-data is present and the process-user-data service is up and running just fine.
However, I'm still unable to get a pod running. The problem that I'm seeing is related with image pull on guest:

=== RUN   TestDockerCreateSimplePod/SimplePeerPod_test/PodVM_is_created
    assessment_runner.go:389: unexpected warning/error event(s): Error: failed to create containerd container: error unpacking image: content digest sha256:9fa9226be034e47923c0457d916aa68474cdfb23af8d4525e9baeebc4760977a: not foundError: failed to create containerd container: error unpacking image: failed to extract layer 

Interesting that I almost never hit that problem when running in a fresh environment like the one setup by the e2e testing provisioner. Let's see how it goes on CI...

This issue is pretty consistent with docker provider. With other providers it's occasional.

Anyway, with the workaround above I could launch a simple pod \o/

[bpradipt]

One challenge is how to make /media mountpath available early in the boot cycle since root (/) is RO. Only the following are available via systemd - https://freedesktop.org/wiki/Software/systemd/APIFileSystems/

I found in https://systemd.io/TEMPORARY_DIRECTORIES/ that use of /var/tmp has some security implications that we should be aware of. I read that doc fully and concluded we should be fine, however, I'm not an expert so better a 2nd check would be nice.

Seems ok. But would be good to hear comment from @mkulke ..

[mkulke]

Seems ok. But would be good to hear comment from @mkulke ..

I mean generally the path sounds a bit odd, why var? why tmp? Can't we create a /media/cidata path statically on the rootfs at image-build time?

[bpradipt]

Seems ok. But would be good to hear comment from @mkulke ..

I mean generally the path sounds a bit odd, why var? why tmp? Can't we create a /media/cidata path statically on the rootfs at image-build time?

Yeah, that's a possibility as well. I wanted to avoid changing the rootfs partition if there was an alternative. And it turned out systemd already creates those in tmpfs. Let me know what you prefer?

[mkulke]

Seems ok. But would be good to hear comment from @mkulke ..

I mean generally the path sounds a bit odd, why var? why tmp? Can't we create a /media/cidata path statically on the rootfs at image-build time?

Yeah, that's a possibility as well. I wanted to avoid changing the rootfs partition if there was an alternative. And it turned out systemd already creates those in tmpfs. Let me know what you prefer?

personally, I'd prefer a /media/cidata mount point.

[bpradipt]

Seems ok. But would be good to hear comment from @mkulke ..

I mean generally the path sounds a bit odd, why var? why tmp? Can't we create a /media/cidata path statically on the rootfs at image-build time?

Yeah, that's a possibility as well. I wanted to avoid changing the rootfs partition if there was an alternative. And it turned out systemd already creates those in tmpfs. Let me know what you prefer?

personally, I'd prefer a /media/cidata mount point.

@mkulke I added a new test commit, can you please check and let me know if the approach looks fine?