Run docker in kata container #358
Comments
|
Thanks for the issue, @svarlamov. In this case, its s because the default kernel provided by docker doesn't have these configs enabled, AFAICT. We do provide a method for building your own and specifying a custom kernel. It seems to me that this should meet your needs. Or at least get you past this issue! We don't enable "everything" by default, as for most the current config works and added features just impact potential density for all. |
|
@egernst Thank you for the explanation -- will build my own kernel then. Thanks! |
|
I had/have the same issue. I've been able to start
(This step can probably be done from the host where the kernel was compiled, and
Then install Docker-ce, and run the daemon: Now, it starts. But I cannot run any container, it always fail with Would it be possible to have an option in |
|
@m-barthelemy Thank you for your explanation! I haven't had a chance to dig much deeper into this yet -- have you gotten the procfs and cgroups stuff sorted? |
|
Also wondering here -- it seems that we're trying to really docker in docker, as opposed to connecting to the daemon at the VM level via a shared sock/tcp. Is there any possibility of that? Does it even make sense? |
|
I really would like this function out of mainline too. One of the long outstanding issues in the community is integrating container building in a secure way. You can see all sorts of docs like: https://docs.gitlab.com/ee/ci/docker/using_docker_build.html where there are lots of different ways to deploy it, each with their major drawbacks. There just isn't a good solution currently. But some way to give a kata container to a user while giving them privilege would allow it to start fast, run fast, and get torn down quickly while still providing security via vm level isolation would be awesome. Alternately, buildah seems to be maturing rapidly and might not need as much from the runtime as docker may. So maybe at least getting that tool up to snuff in kata might work too. |
|
@egernst Since you have a bit more background on this proj, are you able to share a brief outline of what changes are likely required to make this happen? |
|
hey @svarlamov @egernst , I'd be really interested in a little more information on how to get this use-case to work! |
|
Hi @enzian - Could you paste the error you get? It seems the issue you are all seeing is that the guest environment is not providing sufficient functionality to run docker. That isn't surprising given we try to keep it as minimal as possible. But that said, we're considering ways to provide for the myriad features/services folk need. A thought has occurred to me: lxc (like docker) creates Linux containers. But lxc (unlike docker afaik) provides a really useful command ( As such, although it's not guaranteed to help your run a docker container, trying something like the following might be enlightening: ftr, I just tried this, but the command needs to be able to find the kernel config. That either means you'll have to make Related: |
|
currently this is where I am: |
|
Thanks @enzian - a few things jump out:
|
|
By the way guys, are you running docker in debug mode? I don't see any debug messages in the log snippets so I suspect you aren't. Enabling might give more clues :) |
|
ok, patched in btrfs, mounted |
|
Hi @enzian - great - definite progress there :) It might be worth keeping a gist of how you did what you did. Did you enable debug as I still don't see any |
|
sorry, for that @jodh-intel one more with debug logs: It does not leave me any smarted ... ;-) |
|
oh, and a gist of how I got it this situation :p |
|
Thanks @enzian. The error at the end is coming from boltdb: A quick test shows that That I think is: What might be worth exploring is a /cc @sboeuf, @grahamwhaley. |
|
Ah, we've seen a BoltDB issue under VM containers before - so, this may not be specific to docker-under-kata. We wanted to add it to our 'popular images test', but it looks like there was no image. I think really @mcastelino and @egernst are the folks to ask, but both are unavailable right now I think for comment. |
|
Thanks @grahamwhaley. @GabyCT - did you try https://hub.docker.com/r/rossriley/docker-bolt/? |
|
@jodh-intel no, I think that we were looking for |
|
@jodh-intel, I tried the image from |
|
interesting... clearcontainers/runtime#852 implies its a lack of feature in 9p. Is there any workarounds we could do? emptyDir maybe? Is every file system passed through 9p? |
|
buildah-bud might be a workaround here too, as it may not use boltdb. |
|
Hi @kfox1111 . All filesystem maps from the VM to the host are via 9p unless you use a graph driver on the host that supports block device mode, and then Kata can map that block mount directly into the VM. I thought we had an Issue open asking for us to document how you enable that (@amshinde - are you willing to write that up - how to set up and use block devices with kata?) Using a block backed graph driver might work @kfox1111 . You'll need I believe to use devicemapper as the graph driver on the host. |
|
@jodh-intel, @egernst @WeiZhang555 @sboeuf - I think we should add any known 9p limitations to the limitations doc - wdyt? |
|
@grahamwhaley I had added some documentation for block storage for virtcontainers library itself: |
|
@grahamwhaley I tried your approach to use
but nothing changed: |
|
I'm not set up here to try out devicemapper - @amshinde , can I ask you to have a look at this output to see if there is anything obvious? |
Nested mode is only needed if you want to get yet another level of nesting, and it is not required on the system where you have katacontainers installed (i.e., on the system where you run |
|
@enzian - please could you:
|
|
@enzian - it would also obviously be useful if you could try running on a baremetal system without hyper-v for comparative purposes ;) |
|
ok, well we found our problem. The ESX-Cluster we had did not support unrestricted guests - migrated the hosts and it works! |
|
Cheers! |
|
I think we can close this issue now, right? |
|
@enzian - nice! 😄 @svarlamov - have you managed to recreate @enzian's results? It might be useful to summarise what is required and document this somewhere (since this is a very long issue :) |
|
@jodh-intel - I will try to run a kublet in combination with a DinD container. This I will document and contribute the how-to to the kata-project. Is this something that would be welcome? |
|
@enzian looking forward to your docs |
|
Hello, and good work with all this. I'm curious to see the documentation. I have a Thank you!! |
|
@bajacondor you mean documentation on how to build your kernel? |
|
Hi @jcvenegas, I was hoping for documentation around running docker in docker or building with docker within a kata container. |
agent: add GetGuestDetails gRPC function
|
Just as an FYI here, I managed to make docker in kata work. I have written up some basic documentation on how this was done here. I also provide a containerized build for the whole thing (which I tried to make portable, even if you don't have kata installed). I would appreciate any feedback on my approach (curious to know if it works / is useful for anyone else). I will improve the quick docs and add a few other features tomorrow if I get time after work. Thanks to everyone for their work on this issue :) |
|
Wow @daniel-noland - triple nested!!! (kata-in-docker and docker-in-kata in the same stackup!). |
|
Thanks @grahamwhaley, I appreciate that. If it is useful I would be happy to do a more complete write up / walkthrough on both docker-in-kata and kata-in-docker. Not sure when I can have that done but I think that kata-in-docker in particular is a cool way to test out kata-containers without the need to mess with (and potentially break) a new user's containerization setup. I imagine that a three line "try it out risk free" could help the project grow. That said, I should likely wait for a few people to try out the current approach. It works on my machine but that doesn't mean much. Also, I am not sure how amenable the kata-containers project would be to enabling any or all of the kernel features I turned on to make docker-in-kata work. I really respect how minimal the current kata kernel build is. If any changes derived from my work were to be accepted I assume I would need to
I am happy to do any and all of that, I would just want to confirm that there is interest first 😸. I also just wrote up a quick future plans section with some details of other things I would like to try. I would be interested to know of other people's use cases for docker-in-kata or kata-in-docker. My focus is on networking and distributed systems but I am sure other people have different goals we might want to account for. Again, thanks to everyone for all their hard work on kata-containers. I have really enjoyed playing with this project 😃 |
|
I was trying to run Docker in Kata. I tried two approaches in order to prevent getting the error
With both approaches I don't get above error and the docker daemon starts successfully. However, I can't run containers. Equally with both approaches, I get the following error: Does anybody have an idea how to fix this? Additional information
|
|
After reading this, I found out that Docker in Kata works for me after executing the following commands in the container: Is there a way to start a Kata Container on Kubernetes with that mount already present? |
|
@jodh-intel Why is this closed? I think in order to get overlay filesystems working, you need to use kernel 5.6 or higher and add Furthermore, it is unclear to me why this workaround is required. And it's just a workaround: ideally the dind container should work out of the box without supplying a custom entrypoint. |
|
@haslersn, please, see http://lists.katacontainers.io/pipermail/kata-dev/2021-April/001819.html, there you'll find the explanation why the issue was closed. |
|
Description of problem
Tried using the https://hub.docker.com/r/hyperhq/docker-in-hyper/ image and instructions, however, it seems to fail due to a missing filesystem driver in the kernel (don't have aufs, overlay, or overlay2 in the container)
Expected result
I believe that a "docker-in-kata" use case should function -- ideally without having to supply the privileged flag -- similar to hyper containers. This is really critical for running docker in our CI/CD, etc.
Actual result
All of these are running with kata-runtime 1.0:
AND
I can't find the
kata-collect-data.shscript on my machine and running it directly from github source with bash fails... I installed kata via the debian package for Ubuntu 16.04The text was updated successfully, but these errors were encountered: