-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Windows Container Support. Fixes #1507 and #1383 #2747
Conversation
@alexec any chance someone could have a look into this or do I miss something in the PR? I think cross-plattform workflows would really add value 🙂 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for this.
You can fix the build errors by running make codegen
then make lint
.
No one in the core team runs Windows. Would you be willing to support this going forward?
Is there anyway to build the Windows image on Linux so we can publish it?
Thanks for your feedback, the build seems to be successful now. I would try to support this as far as my spare time allows. Unfortunately there is no way to build this on Linux but as mentioned we could use any CI service supporting Windows and maybe just extend your CircleCI setup with Windows orbs to build the Windows image. For the upcoming versions of the
We should also consider adding the docker build for windows as a regular CI check for this repo. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok. Thank you. Can I request you update the doc to tell people how to build the Windows executor please?
I added the documentation |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great stuff -- thank you.
Can you please run |
done :) |
SonarCloud Quality Gate failed. 0 Bugs |
🚀 |
This PR adds support for running Windows Containers in Argo workflows. Fixes #1507, fixes #1383.
What I did:
Extracted OS specific functionality into
os_specific
package underworkflow/executor/os-specific
and used*_windows.go
or*_linux.go
suffixes to enable conditional compilation. With these changesargoexec
can be compiled for Windows and Linux. Also cross-compilation works fine. As the newly introduced functions are one-liners I don't think creating unit tests for them would make sense.Extended the Makefile part for
argoexec
and wrote aDockerfile.windows
doing the whole build ofargoexec
inside of Windows containers. I tried to create the Dockerfile as similar as possible to the existing one to make maintenance more simple. Runningdocker build -t myargoexec -f .\Dockerfile.windows --build-arg=DEV=1 --target argoexec .
on a Windows Server 2019 node creates a Docker Image for Windows withargoexec.exe
like shown in this build log.Changed all examples in docs and the manifests to include the
nodeSelector
forkubernetes.io/os: linux
. This makes sure that theargo-server
and theworkflow-controller
aren't scheduled on a Windows node in a hybrid cluster.I added documentation on how to get started with Windows containers inside of workflows as well as some example manifests. I think the documentation clearly shows the advantages of such a hybrid setup. My favourite are the hybrid workflows enabling e.g. to build an application on Windows and Linux within a single Workflow!
Notes for testing:
Have a look at
docs/windows.md
. You need to set up a Kubernetes cluster, add a Windows node like described here and set up Argo.You can download the sources of the PR to the Windows Server 2019 node and build the argoexec image yourself via
docker build -t myargoexec -f .\Dockerfile.windows --build-arg=DEV=1 --target argoexec .
I also pushed a pre-built image tolippertmarkus/argoexec:v2.7.4
which you can use.For now, on your Windows node you need to re-tag the built or pulled
argoexec
image to match the one used by Argo, e.g. viadocker tag myargoexec argoproj/argoexec:v2.7.4
. Make sure to use the correct version. This ensures that Docker uses the already existingargoexec
Docker image for Windows when running the workflow instead of trying to pull the image for Linux from Docker Hub. This isn't necessary for the end user when we publish multi-platform images to Docker Hub like described below.Change your workflow controller configmap to use the
kubelet
executor and schedule some Windows or hybrid workflows with Argo CLI like described indocs/windows.md
.TODO:
As the automated Docker builds can't build Docker images for Windows, we need to use a CI service to build the image and push it to the Docker Hub. I had good experience with AppVeyor which is free for public projects. The build should create a multi-platform image so that the
argoexec
image for the correct OS will be pulled automatically when the workflow is started. I did set up an example multiarch AppVeyor build config for argoexec. You can also view an example build of the multiarch image here. This example multiarch image is pushed to my DockerHub Repo.A Windows node should be added to the environment for the e2e tests and those tests should also test the added example workflows including Windows containers. As k3s doesn't provide Windows support yet, we maybe need to introduce a separate e2e environment. I never used CircleCI, but as far as I understand it's possible to create multiple Orbs (VMs) for Linux and Windows. Nevertheless, I think it won't be possible for those VMs to communicate to each other to manually initialize a hybrid cluster. I also don't know another CI services where this would be possible, but we could create such an e2e testing environment manually within GCP, AWS or Azure or elsewhere.
This currently doesn't work with
docker
workflow executor as the hostPath volume is set to the fixedSocket
type here. For Windows the type must be set tonull
like shown in Correctly handle named pipe host mounts for Windows kubernetes/kubernetes#69484 (comment). Also to support real hybrid clusters then, we would need thedockerSockPath
in the workflow executor configmap and the just mentionedtype
to be different for Windows and Linux.