Docker+Wasm Integration

[mikesir87]

Tell us about your request
As a developer, I would like to have the ability to build, share, and run Wasm applications using Docker Desktop.

Which service(s) is this request for?
Docker Desktop

Additional context
This post is intended to gather feedback on the new Docker+Wasm integration. Feel free to share issues or ideas on how we can make the product better!

[diraneyya]

I probably do not understand this very well. But I will still ask.

In the browser, we use imports functions to allow the WASM binary to interact with the browser environment, for example, by passing console.log as an import, the WASM module can use that imported function to log things in the console in the browser. The concept can be further exploited to expose a full set of graphics functions to allow a WASM binary to run a legacy game (stuff I am really into). Giving the game low-level OpenGL access to an HTML canvas element.

How does this work in Docker+WASM with the WasmEdge runtime. Do I have to write a WasmEdge plugin to be able to create this link to the outside world? Is this technically a part of the so-called containerd shim provided by WasmEdge? What kind of functionality is provided by the WasmEdge runtime? I looked at the API documentation but there was nothing there that resembled, for example, networking functionality or anything of this sort.

I would like to understand this stuff more and how to make the most out of it. Thanks!

[chris-crone]

Hi @diraneyya! Glad you asked!

WasmEdge implements the WASI standard that gives you access to the outside world. There is still active work happening on this specification. You can follow Wasm spec work more broadly as it happens under the Bytecode Alliance.

@juntao and his team can give you more information about the WasmEdge runtime specifically.

[juntao]

Thanks @chris-crone and hi @diraneyya !

Yes, WASI specifies how WebAssembly Runtimes, including WasmEdge, access the operating system. That includes standard POSIX / libc APIs to access the file system / network etc.

So, you can just create a Rust program to open a file or open a socket connection or write to the console (eg print!() and dbg!() macros), compile it to wasm32/wasi target, and it will run in WasmEdge. See

https://wasmedge.org/book/en/write_wasm/rust.html

You can also do this with other languages supported in WasmEdge including JavaScript.

https://wasmedge.org/book/en/write_wasm/js/quickstart.html

[lachie83]

Hey folks - Heads up that I'm not able to get the tech preview to run on an intel Mac. I am a Docker Desktop user and the stable version works fine. Let me know if you would like any debug information to assist. I'm really excited to see this and would love to try it out

[chris-crone]

Hi @lachie83! Sorry to hear it isn't working. We'll reach out via email to get more details

[lachie83]

Thanks Chris - happy to help.

[PranavPeshwe]

@lachie83, @chris-crone, FWIW - I was able to run it on first shot.

Here are some details about my setup in case they help debug:

$ docker --version
Docker version 20.10.18, build b40c2f6
$
$ neofetch --stdout | egrep 'OS|Host|Kernel'
OS: macOS 11.6 20G165 x86_64
Host: MacBookPro16,1
Kernel: 20.6.0

[lachie83]

Looks like similar hardware and Kernal, different OS

$ neofetch --stdout | egrep 'OS|Host|Kernel'
OS: macOS 11.7.1 20G918 x86_64
Host: MacBookPro16,1
Kernel: 20.6.0
$ docker --version
Docker version 20.10.18, build b40c2f6

[TBBle]

Well done @mikesir87 and team, this is very exciting work. I'm looking forward to seeing this work as a driver for WASM through the ecosystem, e.g., resolving/landing opencontainers/image-spec#964, and seeing https://wasmedge.org/book/en/use_cases/kubernetes/docker/containerd.html filled out.

I for one welcome our new micro-microservice overlords.

I wasn't mentioned in the blog post, but where is io.containerd.wasmedge.v1? I saw a couple of older projects implementing the containerd Runtime V2 shim for other WASM runtimes, (which is what I understand was done here based on the blog post), but couldn't find this work on GitHub or the wider Internet.

So I'm assuming it's not public yet, but will be?

[tigerinus]

Can someone confirm this is the same containerd shim used under https://learn.microsoft.com/en-us/azure/aks/use-wasi-node-pools right?

Just want to make sure when developing a generic WASI app it runs in Docker WASM.

[juntao]

Hi @TBBle

Thank you for your kind words. Please do let us know if you encounter any issues. We cannot wait to see the Wasm microservices you create. :)

The containerd shim for WasmEdge is here:

https://github.com/second-state/runwasi

We are in the process of merging it upstream to DeisLabs runwasi project, and hopefully eventually to the containerd project itself.

[juntao]

It is not the same. But all standard compliant WASI app should work in both environments. If not, let us know. :)

Can someone confirm this is the same containerd shim used under https://learn.microsoft.com/en-us/azure/aks/use-wasi-node-pools right?

Just want to make sure when developing a generic WASI app it runs in Docker WASM.

[squillace]

Just chiming in quickly to confirm what @juntao says here. The wasmedge team did in fact use the core https://github.com/deislabs/runwasi shim to quickly provide wasi support in containerd with the wasmedge runtime and we're thrilled it worked perfectly for the Docker Desktop preview. As a result, @tigerinus, our path for bringing the two together and moving the shim upstream to the containerd project is pretty easy and we look forward to working on that together.

In fact, the community's collective work on https://github.com/deislabs/runwasi -- including enabling any number of open source runtimes like SecondState's wasmedge -- will continue rapidly. If you're interested, you can track that repo (which we hope to move into the containerd repo as soon as we can). Using the Docker Desktop+wasm preview with the https://github.com/second-state/runwasi will not be any problem for using future shims. It's a great way to get started here!

[nigelpoulton]

Great work everyone and very exciting!

I'd be keen to see the following from Docker Desktop going forward:

1. If we're going to package Wasm binaries inside OCI images for the foreseeable... may be a better way than layering the Wasm module on top of a scratch base image in a Dockerfile? It feels a bit hacky
2. docker buildx build to eventually create native Wasm artifacts rather than shoe-horn them into OCI images
3. Support for more Wasm runtimes and the Docker Desktop UI make it easy to pick and install them
4. Let's get this containerd shim approach into K8s

[squillace]

1. it is hacky. We'll end up using OCI Artifact graphs, which we need for signature, attestation, and sbom information anyway. But it's fine, as there are plenty of hacks that work fine in this space.
2. My guess is yes, but that's a Docker thing. LOTS of things will build to wasm, but docker buildx build is a special kind of sweetness.
3. Already in process. (Docker, SecondState, Deislabs at MS to start....)
4. Already in process. (ditto)

Anything else, @nigelpoulton ? I mean, make it at least HARD.

[squillace]

Mind you, on 1 I am of course speaking for myself here, not Docker. :-)

[jungan21]

Quick question: can we leverage HPA to autoscale WASM module like traditonal PoD?

[squillace]

you will be able to use all the normal k8s mechanisms with either version of https://github.com/deislabs/runwasi as we roll forward. At the moment, however, telemetry isn't working well -- we have to emit things to enable this. We should soon have a new version of the runwasi shim (and secondstatewill work on theirs until we coalesce) that will emit all the telemetry needed for any k8s scaling scenario.

The ONLY thing that will not yet work is mixed containers/wasm inside one pod. This precludes the usage of service messes in k8s for the time being. Nailing that scenario is in the backlog but we have no roadmap timing for that yet. Ironically, as service messes have lots of headspace we have very little evidence that lots of people use them: https://www.datadoghq.com/container-report/ says only about 10% of people use meshes..... ymmv.

[TBBle]

Although full-on service meshes might not have high penetration, I note that mixed containers/wasm is also an important use case to implement sidecars for telemetry collection, e.g. Datadog Agent, OTel Agent, Jaeger Agent. I don't know in general how the split is between (possibly auto-injected) sidecar agents and host agents (e.g., HostPort-based Daemonsets), but Datadog specifically excluded their own Agent sidecar from that report, so presumably it's widespread enough that it matters in such data. (Since they use data from their Agent for the report, presumably it'd have been present on 100% of hosts in their report if they didn't exclude it, but it would have been interesting to see the split between deployment style.)

I'll note that anecdotally the last two k8s-based applications I was involved in building both relied on auto-injected telemetry sidecars (once with Jaeger Operator, once with OTel Operator), but since I was the common factor there, that's pretty low-quality signal even for anecdotal data.

[jungan21]

Thx @squillace for your reply.

Before, we played around with deislabs/krustlet . With Krustlet, we are able to deploy WASM module into K8s cluster. But, we are NOT able to eanble HPA feature (for autoscaling).

Two questions:

1. Since both deislabs/krustlet and deislabs/runwasi are from Deislabs, what's the difference?
2. what's difference between deislabs/runwasi and secondstate's (i.e. WasmEdge Runtime-based solution)?

[squillace]

in reverse order:
2. #426 (comment) pretty much explains it. SecondState used the deislabs runwasi to implement wasmedge because it has special features that aren't available elsewhere. You can see the diff here. But the larger point is that this is merely an early set of experiments by us all in the wasm ecosystem. Both SecondState and Microsoft (Deislabs) will be working to bring multiple runtimes to the runwasi shim as we go forward. You might need to know now that there's a difference, but we assume that in the future you won't need to.

1. Krustlet was our second attempt at deislabs (after modeling them as a CRI implementation in https://github.com/deislabs/wok) to integrate wasm into Kubernetes smoothly. Krustlet DID work, as you note, but it brought with it a ton of overhead as well as making it harder to model wasm as pods easily. In the end, it is used as a rust kubelet and it implements a rust oci-distribution crate (which will be very useful moving to OCI Artifacts) as well as krator, the state engine. But neither the Fermyon maintainers, nor the Cosmonic maintainer, nor we here at Microsoft are still working on krustlet. It's just too much effort to bring it up to production quality and then you still have to treat your nodes like pets with taints and tolerations and all that messiness.

Turns out that ContainerD shims are right abstraction for that. Krustlet might still become very useful for key situations.

Are you using Krustlet successfully aside from the autoscaling?

[squillace]

@TBBle:

I note that mixed containers/wasm is also an important use case to implement sidecars for telemetry collection, e.g. Datadog Agent, OTel Agent, Jaeger Agent.

Oh yes -- because that's how they work now. With Wasm components, you'd be moving toward an in-proc OTEL component that exports such information. Sidecars are horrible unless you have only one or two -- not because the model doesn't work, but because it's just harder to use. BUT we use them!!!! For example, we've test implemented Dapr sidecars as components and WOW does that change things. It's almost as if kubernetes is good. :-P

That said....

I'll note that anecdotally the last two k8s-based applications I was involved in building both relied on auto-injected telemetry sidecars

I'm expecting that we'll make this happen. But it's not going to happen this month. First, we make these solid, enable componentized runtimes so that people can choose theirs to use, and upstream the shim work to the ContainerD project in the CNCF.

[jungan21]

in reverse order: 2. #426 (comment) pretty much explains it. SecondState used the deislabs runwasi to implement wasmedge because it has special features that aren't available elsewhere. You can see the diff here. But the larger point is that this is merely an early set of experiments by us all in the wasm ecosystem. Both SecondState and Microsoft (Deislabs) will be working to bring multiple runtimes to the runwasi shim as we go forward. You might need to know now that there's a difference, but we assume that in the future you won't need to.

1. Krustlet was our second attempt at deislabs (after modeling them as a CRI implementation in https://github.com/deislabs/wok) to integrate wasm into Kubernetes smoothly. Krustlet DID work, as you note, but it brought with it a ton of overhead as well as making it harder to model wasm as pods easily. In the end, it is used as a rust kubelet and it implements a rust oci-distribution crate (which will be very useful moving to OCI Artifacts) as well as krator, the state engine. But neither the Fermyon maintainers, nor the Cosmonic maintainer, nor we here at Microsoft are still working on krustlet. It's just too much effort to bring it up to production quality and then you still have to treat your nodes like pets with taints and tolerations and all that messiness.

Turns out that ContainerD shims are right abstraction for that. Krustlet might still become very useful for key situations.

Are you using Krustlet successfully aside from the autoscaling?

Other features works well. BTW, I recalled there is open issue mentioning this limitation too: krustlet/krustlet#470

[assambar]

Hey guys, great work!

Have one question - how do I pass arguments to the underlying runtime? In my case I want to access files on my local machine. Now, if I do this with wasmedge separately I'd just have to add a say --dir /:/ to have access to everything from WASI.

I tried mounting with -v when I run the "wasm container" but that was not enough.

E.g. using singlestore-labs' python I could do something like this

alexandrov:\>wasmedge --dir /:$(pwd) --env PYTHONPATH=/python311.zip/python3.11 --env PYTHONHOME=/python311.zip/python3.11 ./python3.11.wasm -c 'import os; print(os.listdir(""));'
['python3.11.wasm', 'python311.zip']
alexandrov:\>tree
.
├── python3.11.wasm
└── python311.zip

0 directories, 2 files

The key thing here is --dir/$(pwd) along with the PYTHONPATH and PYTHONHOME variables.

So I built myself an image that has just python3.11.wasm and python311.zip but couldn't make it work.

Even running the interpreter fails as it does not pick the env variables. Tried passing them to the container, as usual, but that did not propagate them to the wasm module.

docker run -d --runtime=io.containerd.wasmedge.v1 --platform wasi/wasm32 python-wasm -v /:$(pwd)/python-wasm -e PYTHONPATH=/python311.zip/python3.11 -e PYTHONHOME=/python311.zip/python3.11

Any help or ideas would be greatly appreciated.

[juntao]

Hi @assambar

The Wasm file is packaged in a SCRATCH OCI image. It has access to the / directory of the container. So you can mount a directory to the container and then access it from inside the Wasm app.

Let us know if it works. Thanks!

[jungan21]

you will be able to use all the normal k8s mechanisms with either version of https://github.com/deislabs/runwasi as we roll forward. At the moment, however, telemetry isn't working well -- we have to emit things to enable this. We should soon have a new version of the runwasi shim (and secondstatewill work on theirs until we coalesce) that will emit all the telemetry needed for any k8s scaling scenario.

The ONLY thing that will not yet work is mixed containers/wasm inside one pod. This precludes the usage of service messes in k8s for the time being. Nailing that scenario is in the backlog but we have no roadmap timing for that yet. Ironically, as service messes have lots of headspace we have very little evidence that lots of people use them: https://www.datadoghq.com/container-report/ says only about 10% of people use meshes..... ymmv.

Hi @squillace & Community Members,

Beside Autoscaling, are there any other known limitations? We can't wait to roll out this features to our Production env :)

Suppose we have K8s as our Application Hosting Platform, to enable user/developer to be able to deploy to WASM-based Application (i.e. WASM Module) to our K8s env, is there any special settings/configuraiotns required for my K8s cluster?

OR

all I have to do is to package WASM file/module in a docker image (i..e. SCRATCH OCI Image)?

Another question: Once the Docker image (containing WASM file) is deployed to K8s, it's running in a Container like other tradiiotnal non-wasm Docker image? from user's perspective, it's totally transparent?

Thanks in adavnce.

[TBBle]

Per https://docs.docker.com/desktop/wasm/ I believe it's now part of the main Docker Desktop distribution as a Beta feature. However, as Docker limits who has access to Beta features, it may not be accessible to you. (I recently noticed I couldn't access the Containerd Image Store feature toggle which this one depends on anymore, when I went to test a related fix in buildx).

[rumpl]

Hey all, thanks for the ping, it's an error on our side, I'm fixing it right now, will give an update quickly.

it may not be accessible to you. (I recently noticed I couldn't access the Containerd Image Store feature toggle which this one depends on anymore, when I went to test a related fix in buildx).

@TBBle the containerd image store feature is accessible to everyone, the feature is not accessible if you are using windows containers on windows, was that the case maybe?

[TBBle]

Oh, duh. Yeah, that's my mistake. Sorry for the confusion.

Edit: On reflection, I spent enough time looking at both the containerd side of the Image Store and Windows Docker/containerd integration work that this requirement should have been obvious to me, but I forgot, and I notice that the docs don't actually mention this requirement. I haven't noticed anyone else getting caught with this mistake though.

Some day we'll have a feature toggle for "WCOW Containerd integration" and I hope that will include Containerd Image Store automatically.

[suneetnangia]

When using latest (4.20.1) Docker Desktop, building image with --platform=wasi/wasm flag results in the following error, even though the image is still built:
ERROR: failed to solve: no match for platform in manifest sha256:3187cb57f434ae8ba21e9cd78b59cae54b0345bf948bb430d83a3031af82e6ff: not found

It's annoying as this fails the build script we have, but this does not happen in the first Docker Desktop version (4.15.0) where wasm support was provided.

[tianon]

I believe opencontainers/image-spec#1055 is probably relevant (wasip1/wasm for preview1, eventually wasip2/wasm for preview2, and likely only wasi/wasm after GA) 🙇

[jprendes]

@suneetnangia , thank you for the report.
This's an issue with buildkit: moby/buildkit#3891
As a workaround you can run docker build with --provenance=false or setting the env var BUILDX_NO_DEFAULT_ATTESTATIONS=1

[kenesparta]

@suneetnangia , thank you for the report. This's an issue with buildkit: moby/buildkit#3891 As a workaround you can run docker build with --provenance=false or setting the env var BUILDX_NO_DEFAULT_ATTESTATIONS=1

Thanks @jprendes It worked for me.

[auula]

Tell us about your request As a developer, I would like to have the ability to build, share, and run Wasm applications using Docker Desktop.
Which service(s) is this request for? Docker Desktop
Additional context This post is intended to gather feedback on the new Docker+Wasm integration. Feel free to share issues or ideas on how we can make the product better!

I've seen a lot of Docker + Wasm applications on the internet, but I'm sorry to say that I think the wasm runtime that Docker currently integrates with has various problems, such as the one described in this article : Running docker wasm preview fails tutorial , it's impossible to run up a full project, and it feels like this feature is simply a toy.

I would like to give feedback that you should integrate wasm runtime like wazero.io, because it is also developed in Go, while other wasm runtimes are in C or C++. This makes it necessary for developers to learn C++ or other programming languages if they want to contribute code. The integration of a runtime implemented in the same type of Go programming language as docker would help developers get started with docker more easily and do secondary development,without dependency on cgo is a good choice!

I've been doing a bit of reading lately about the built-in wasm runtime. their documentation is written very messy, and senior developers understand that the documentation is divided into two categories, the first for the use of tutorials document, the second is the API documentation. And their software promotion and traditional rogue virus software,installed in the case of the user without knowledge of your computer, without the user's permission, similar to the early PC software bundled sales, which is a disgusting behavior, bundled in the docker sales.

[imjasonh]

Are there docs covering how to run wasm docker apps without requiring Docker Desktop? All the docs at https://docs.docker.com/desktop/wasm seem to assume/require Docker Desktop specifically.

I'm working on adding support for building wasm containers using ko and I'd love to be able to test this functionality in GitHub Actions.

[jprendes]

HI @imjasonh
There's an example for enabling and using the wamtime runtime in the doc for Alternative container runtimes.
Hope that helps!

[cataggar]

It would be really useful if we could save the wasm file from the image. Currently, it possibly to save the entire OCI image. docker save -o configur.tar cataggar/configur. It would be great if we could do docker save -f confugur.wasm cataggar/configur to save the wasm file for troubleshooting.

[TBBle]

@cataggar The easiest method I can see is something like:

docker save cataggar/configur | tar x confugur.wasm

The limitation is that images are stored as tar-streams, so to do what you want, they'd basically have to implement tar x somewhere, along with a bunch of CLI options and syntax for a new command or extra overload on an existing command.

If this was from a running or stopped container rather than an image, docker cp <imagename>:confugur.wasm confugur.wasm should work, if your debugging flow supports that.

That said, this isn't Wasm integration specific, so I'd suggest opening a feature request if you want to actually pursue such a feature in Docker.

[Avey777]

I hope the Docker Engine supports wasm, not just Docker Desktop

[jprendes]

@Avey777 it does.
Please refer to the following link for the instructions to set it up: https://docs.docker.com/engine/alternative-runtimes/#wasmtime

[royaljain]

Any work/timeline on having this as part of Docker Engine and not just Docker Desktop?

I am trying to figure out how to run this on the cloud, more specifically on GCloud Run where I don't have access to docker daemon. So can't use @jprendes's suggestion.

[TBBle]

Any work/timeline on having this as part of Docker Engine and not just Docker Desktop?

I am trying to figure out how to run this on the cloud, more specifically on GCloud Run where I don't have access to docker daemon. So can't use @jprendes's suggestion.

That link is the instructions for using this with Docker Engine directly, i.e. if you aren't using Docker Desktop.

If you're using Docker Engine managed by someone else, you'll need to raise a feature request with them to offer WASM support. Right now, from their docs, Google Cloud Run only supports Linux x86_64 ABI executables.

Looking around, I'm not convinced Google Cloud Run even uses Docker Engine; from the docs, it is running either in gVisor, or as containers inside some sort of hypervisor.

But either way, this is not something Docker has any say in, you'll have to request it from Google directly. https://issuetracker.google.com/issues/271552514?pli=1 might cover it already, actually. So I'd suggest upvoting that issue, and maybe note that you expect to see something compatible with existing wasip1/wasm images and future revisions, ala opencontainers/image-spec#1055, to maximise support for externally-created images.

[jprendes]

Just to add to @TBBle 's great answer, if you are happy using Kubernetes (like GCP Kubernetes), you could try Kwasm

[royaljain]

Thanks @TBBle for the detailed answer. I've bumped the feature request on Google's issue tracker. @jprendes I'll check out Kwasm

[johnstonphilip]

I am not sure if this is the right place to mention this, but I seem to have hit a problem with the tutorial at:
https://docs.docker.com/desktop/wasm

When I run the following, I simply get Hello WasmEdge! as a response, the container shuts down, and nothing loads at http://localhost:8080

 docker run \
  --runtime=io.containerd.wasmedge.v1 \
  --platform=wasi/wasm \
  secondstate/rust-example-hello

To note:

• I am on Docker Desktop 4.24.2 (124339)
• I have enabled Use containerd for pulling and storing images
• I have enabled Enable Wasm

[TBBle]

That looks like a documentation issue, your outcome looks correct for that project's documentation.

Try secondstate/rust-example-server instead to see a server in action. Although it is slightly more featureful than this description.

Either way, your setup is working, as you were able to run the WASM-based container. This difference should not prevent you continuing to explore the Docker WASM integration beta.

[johnstonphilip]

@TBBle Thank you for the response! I attempted to run this:

docker run \
  --runtime=io.containerd.wasmedge.v1 \
  --platform=wasi/wasm \
  secondstate/rust-example-server

When I did, I got this response:
Listening on http://0.0.0.0:8080

But when I visit http://0.0.0.0:8080 or http://localhost:8080 in a browser, nothing loads. Also, in Docker Desktop, the port column for that container isn't showing anything. Apologies if I am missing something obvious!

[jprendes]

@johnstonphilip as with any image, you need to expose the port 8080, i.e.,

docker run --rm \
  --runtime=io.containerd.wasmedge.v1 \
  --platform=wasi/wasm \
  -p 8080:8080 \
  secondstate/rust-example-server

then you can do

curl http://localhost:8080

[ankianan]

I am not able to get correct docker stats for wasm container, its always near to 0%.

Also I am not able to

docker exec -it a4ef1cf740d7 top
/containerd.task.v2.Task/Exec is not supported: not found

I am trying to compare CPU usage of wasmedge-quickjs with nodejs.

Kindly suggest.

[mikkelhegn]

@ankianan - you cannot exec into a WebAssembly and run e.g. top like you're trying to in the above example. The WebAssembly is a process hosted by the runtime.

I'm not sure what types of metrics WasmEdge may serve through containerd, but you could ask that question here: https://github.com/containerd/runwasi

[TBBle]

You're trying to execute a program called top inside the container, but there's no such program inside your container. That's not surprising, as there's no OS inside your container, it's running a WASM file directly. (The error is saying you can't run extra processes inside the WASM container.)

I expect you can use top outside Docker to see the process usage of the wasmedge process, but you may need to set some options make make the process visible, as by default I expect top does not show processes inside containers.

docker stats might work, I'm not sure if it's hooked up for the WASM integration.

[savannahostrowski]

I'm going to close this issue in favour of moving the conversation around Wasm support over to https://github.com/moby/moby so we can handle incoming feedback and requests in individual issues!

If you have any issues or suggestions around support, please feel free to open an issue over there. We'd love to hear from you!