Kubernetes container

[bhack]

Support for containers running in kubernetes (pod, deployment etc..)

[Chuxel]

@bhack Where are you running Kubernetes? If it's local, you should be able to attach to it after you spin it up. F1 > Remote-Containers: Attach to Running Container...

Would that work for you?

[bhack]

Often it is remote specially when I need special (hardware/data) resources.
I need always to use kubectl cp when I edit a file in vscode.

[bhack]

Have you collaboration with https://github.com/Azure/vscode-kubernetes-tools Azure team?

[Chuxel]

@bhack Definatley something we've been thinking about which is partly why I asked about the scenario. We actually have a dev container defintion that allows you to use a Docker sandbox container with all the needed tools (kubctl, helm docker, etc) so you can containerize your build toolchain and still use Kubernetes extension and deploy to a Kubernetes cluster from there: https://github.com/Microsoft/vscode-dev-containers/tree/master/containers/kubernetes-helm

F1 > Create Container Configuration File... > Kubernetes & Helm

[bhack]

F1 > Remote-Containers: Attach to Running Container... for remote K8s could be a good initial feature here.

[bhack]

Just to clarify I meant, as first useful feature, the same use case at (but for K8s):
https://code.visualstudio.com/docs/remote/containers#_attaching-to-running-containers

While using VS Code to spin up a new container can be useful in many situations, it may not match your workflow and you may prefer to "attach" VS Code to an already running container.

Cause my K8S remote workflow "may not match" VS Code spin up.

[Chuxel]

@bhack Yep - totally makes sense. Use kubectl to get to the pod instead of the Docker CLI. K8s configs are more complicated to manage in general so starting with an attach flow makes sense.

[bhack]

I've a question about the server side/extensions. Cause the container/pod nature it is ephemeral how do you handle server side extensions over containers? Do you need to re-configure the remote container extensions every time?

[bhack]

Also I don't understand what could be the case between remote vs container for remote-container K8s case:


We need to suppose that the running code is in the container (i.e. from init container, persitent volume, remote mount, etc) but remote/local edits need to be in sync with the local/remote files.
So there in not a volume mount of a local path like in your current "local" container case.
We need to have an auto-sync over the "source code block" in the diagram.

[bhack]

About source code block sync check ernoaapa/kubectl-warp#10.
As mentioned there I don't know if a sidecar injector could be a good strategy.

[chrmarti]

Extensions running in the container need to be installed in each container. That happens automatically for the extensions listed in the devcontainer.json.

The files don't have to be local for VS Code to work on them. You could just check out the files in the container.

[bhack]

No the problem is that you work on remote files but generally you git commit changes locally. So we need a sync.

[chrmarti]

If you commit / push through the Git viewlet in VS Code, you do it on the remote side currently.

[bhack]

Generally you don't have git push credentials on remote containers.

[chrmarti]

We currently register a credentials helper for Git in the container (if git is installed) that forwards requests to the local Git to lift that limitation.

[bhack]

@chrmarti I still think that a sync is optionally required over the source code about volume block in the diagram.
A very frequent use case is that you test a change on a small scale in a local container (i.e. Docker container) and then you want to test on an active developing Pod (container in k8s).
Generally you don't commit until changes are tested locally and on the pod resources.
So if the change works locally then you want to sync "code block" on the remote existing developing pod/container. If all works also remotely probably you can commit also with your mentioned helper.
So a local to existing pod sync it is required IMHO.

[kdvolder]

@Chuxel said:

Where are you running Kubernetes? If it's local, you should be able to attach to it after you spin it up. F1 > Remote-Containers: Attach to Running Container...

Would that work for you?

If it'd allow me to somehow connect to pods inside my local k8s cluster... yes it would (at least for my current local experimenations). But I don't think that this actually works right now (is it supposed to work already?).

When I use the "Remote-Containers: Attach to Running Container" it shows a list of things that seems to correspond to the stuff you get running docker ps command locally. But the pods running in my k8s cluster (i.e. the things that you get running kubdctl get pods) is not amongst the things it lets me choose from.

So what we need really is what you suggested here:

Yep - totally makes sense. Use kubectl to get to the pod instead of the Docker CLI.

That, by the way, would then also work with any k8s cluster be it local or remote.

[bhack]

Cause this Is still not going in the montly roadmap there Is a boiloperate workaround in the meantime:
https://okteto.com/blog/vs-code-remote-development-in-kubernetes/

[mlgill]

Cause this Is still not going in the montly roadmap there Is a boiloperate workaround in the meantime:
https://okteto.com/blog/vs-code-remote-development-in-kubernetes/

To clarify, it looks like Oketo has to be installed locally and on Kubernetes and does the equivalent of running syncthing (bi-directional sync) locally and remotely as a sidecar container, correct?

Assuming this is correct, VS Code development works through file sync and it's not possible to use any of the debugging features within the remote container.

I do hope this feature gets put into the roadmap so that it will work similar to remote development over SSH.

[bhack]

@mlgill Yes but It Is better than nothing cause there Is anything planned in the next iteraton #1390

[mlgill]

I get that. Just want to be clear about the limitations.

[bhack]

@mlgill Cause It Is closed source we cannot do anything more that upvote this. So, if you can, please try to help to collect upvotes here in your GitHub network of vscode users.

[kdvolder]

I experimented a bit with using the ssh support to connect to a k8s container. Was able to get this working but the main drawback of the approach is that you have to explicitly install ssh daemon into your 'development contiainer'. It's also a fairly involved process to set it all up. Anyhow... just in case anyone is interested. I put this up here: https://github.com/kdvolder/vscode-remote-java-dev-env

[kdvolder]

@bhack said

Cause It Is closed source

Is that true? If so, I am very surprised! I assumed otherwise given that we are here, filing issues and discussing on github (which is a platform for open source software development). But perhaps it is true after all since when I go and try to find the source code, indeed I can't seem to find any.

I've filed this ticket #1402 as I think the github readme and docs for vscode-remote are really not clear on that point.

[Phantomape]

Attach Visu

Hi, I was using Attach Visual Studio Code in my daily development, it worked perfectly. However, one day I decided to use Intellij IDEA in the pod terminal(more comfortable using IDEA to develop Java program :) ), and I found that X forward is not enabled by default. I manually did a ssh -Y <user>@<host> login to the remote host and set the DISPLAY env var in the pod, and then I was able to launch Intellij. Is there a way to enable X forward and use GUI programs in the pod after I attached to the remote container? Post the same question in vscode-kubernetes-tools/vscode-kubernetes-tools#849 but it seems that I dug into the wrong place to ask.

[gaocegege]

Hi, is there any progress?

There are lots of users (internally) who want to develop deep learning models on Kubernetes with the help of GPUs. I have to help them install sshd in the docker container then run it as @bhack said. I am wondering if you have a plan to support Kubernetes in the near future, or I need to consider switching to other possible solutions like okteto.

I'd appreciate it if anyone could help me.

Thanks 🥂 🍻

[chrmarti]

@gaocegege There is an Attach Visual Studio Code action on pods in the Kubernetes Explorer, so there is no need to set up sshd. Can you give that a try?

[gaocegege]

Thanks for your reply!

It works. But I am wondering if we have the plan to support other commands like launch and so on, just as remote-container docker does.

BTW, is there any plan to open-source remote container extension?

[chrmarti]

We are tracking supporting additional commands in this issue, but there is currently no timeline.

There are no plans to change the license. (FAQ on licensing: https://code.visualstudio.com/docs/remote/faq#_license-and-privacy)

[gaocegege]

OK gotcha, thanks for your reply!

[markomitranic]

@chrmarti Thanks for the explanation, I've seen the same example somewhere in the documentation. However my context menu does not offer the option you refer to. Why would that be?

[chrmarti]

The docs show it under Pods: https://code.visualstudio.com/docs/remote/attach-container#_attach-to-a-container-in-a-kubernetes-cluster

Is the tree node you are opening the context menu on a pod?

[markomitranic]

Hey @chrmarti thanks for your swift response. The Pods subtree behaves the same as if targeting the pod from the deployments section. I get the same context menu as in the screenshot.

[chrmarti]

@markomitranic Not sure this is obvious: You need the Remote-Containers extension in addition to the Kubernetes extension. Make sure you have that installed too.

[markomitranic]

@chrmarti Haha good point, of course, it is installed. In fact, what you are looking at is not my local vscode at all, it is a remote instance SSH running on a VPS. My local computer is not used for running code or connecting to the cluster at any point.

[chrmarti]

@markomitranic Are you connected with Remote-SSH to the SSH server or is VS Code's UI running on the SSH server?

[markomitranic]

@chrmarti Connected over Remote-SSH

[chrmarti]

Filed #6054 for connecting through a SSH server.

[arashnorouzi]

Hi @chrmarti,
When I'm trying the "attach visual studio code" option, it opens a new vs code window and connects to the pod, but my source code is still open in the first window.
Is there any way to configure local source code path for this option? Maybe I'm missing a step ?
Thanks.

[chrmarti]

@arashnorouzi VS Code is not aware of any mount points inside the pod. Would you expect the existing window to be reused?

[tobru]

When using the "Attach Visual Studio Code" on a Pod it executes:

kubectl -n myns exec -it my-deployment-94bb57f74-xlkx7 -- /bin/sh -c VSCODE_REMOTE_CONTAINERS_SESSION='UUID' /bin/sh

This seems wrong as this command terminates immediately, everything after -c should be quoted. The follwowing command would be correct:

kubectl -n myns exec -it my-deployment-94bb57f74-xlkx7 -- /bin/sh -c "VSCODE_REMOTE_CONTAINERS_SESSION='UUID' /bin/sh"

Can I configure that or is this a bug?

[chrmarti]

@tobru The quoting is only needed when running the command from a shell, but we spawn the process directly, passing the parameters as an array. The part you suggest to quote is passed as a single entry in that array.

Are you seeing the command terminate immediately?

[leociociano]

I have the same problem. I followed this procedure (https://code.visualstudio.com/docs/remote/attach-container#_attach-to-a-container-in-a-kubernetes-cluster) and I get the following error. I am cluster admin.
vs-error.txt

[FractalMachinist]

I'm chasing some similar issues as @tobru mentions.

Reproduction environment:

SERVER:

• Ubuntu 22.04.1 LTS
• Running MicroK8s v1.25.3 revision 4094
  • $(microk8s kubectl version --short) => "Client Version: v1.25.3\nKustomize Version: v4.5.7\nServer Version: v1.25.3"
• MicroK8s is running with an accessible, healthy container & pod & namespace
• SERVER is the only MicroK8s node
• SERVER is on the same network as CLIENT
• MicroK8s generated kubeconfig was provided to CLIENT

CLIENT:

• Ubuntu 22.04.1 LTS
• Kubectl is installed
• $HOME/.kube/config is in place as provided by SERVER
• Kubectl version is identical to SERVER's MicroK8s.kubectl version
• Running kubectl in a bash terminal works correctly, including all commands which vscode-remote-release depends on (discussed below)
  • Specifically, kubectl interacts with SERVER correctly.
  • Side note - kubectl fails when CWD is within an NFS mount, which are on CLIENT but are not where I'm running commands. Just being thorough.
• VSCODE:
  • Version 1.73.0
  • Extensions:
    • Dev Containers v0.263.0 (Identical behavior tested in v0.262.3)
    • Docker v1.22.2
    • Kubernetes v1.3.10
    • mtsmfm.vscode-k8s-quick-attach v0.8.0 (Useful for minimum replication)
    • YAML, Prettier

I discovered these issues while debugging "Attach to Visual Studio".

Detailed Debugging of "Attach to Visual Studio"

Discovering the issue

Dev Containers provides a mixin to the Kubernetes extension, which is supposed to support "Activity Bar > Kubernetes > [CONTEXT] > Workloads > Pods > [POD NAME] > {CONTEXT MENU} > Attach to Visual Studio".

However, this command fails in on my machine. VSCode provides a popup:

Error running command remote-containers.attachToK8sContainerFromViewlet: Command failed: kubectl get pod [POD NAME] -o json --namespace [NAMESPACE]. This is likely caused by the extension that contributes remote-containers.attachToK8sContainerFromViewlet.

...and sends a similar error to the Extension Host:

[TIMESTAMP] [error] {"message":"Command failed: kubectl get pod [POD NAME] -o json --namespace qs","stdout":{"type":"Buffer","data":[]},"stderr":{"type":"Buffer","data":[]},"code":1} remote-containers.attachToK8sContainerFromViewlet {"value":"ms-vscode-remote.remote-containers","_lower":"ms-vscode-remote.remote-containers"}

Minimizing error conditions

The exit code 1 with no stderr was surprising. I pulled open $HOME/.vscode/extensions/ms-vscode-remote.remote-containers-0.263.0/dist/extension/extension.js, pushed it through Prettier, and started unminifying.

Handling remote-containers.attachToK8sContainerFromViewlet

I located the function which handles remote-containers.attachToK8sContainerFromViewlet. To locate this function, it is the only instance of the text "current-context", and was assigned the minified name V9 in v0.263.2. I renamed this function newWindowAttachedToK8sContainer.

Inside, I found a function which spawns via child_process (function originally named ze in v0.263.2, I renamed to execOnArgs), and which is used to execute kubectl to gather information about context and the selected pod. I added more calls to execOnArgs to test its behavior. My expectation for correct behavior is having similar stdout/stderr and identical exitcode vs running the equivalent command in a bash terminal, and not leading to errors while attaching to a container.

execOnArgs Tests

Format: `${cmd} ${argsArray}`
All tests use the existing env and output.

• Base Cases
  • kubectl ['config', 'current-context']
    • Ends with code 0 and no stderr/stdout
    • bash exits code 0 and returns a string naming the current context.
    • This silent failure seems related to:
      • Attach Visual Studio Code via Kubernetes fails to connect remoteContainers [resolved] #7108
      • possibly 'Attach Visual Studio Code' does nothing, throws no error, no logging #5100
      • possibly "Attach Visual Studio Code" doesn't use kubeconfig.yaml in current project #4998
  • kubectl ['get', 'pod', 'POD NAME', '-o', 'json', '--namespace', 'NAMESPACE']
    • This is the error condition I originally discovered
    • Exit code 1, no stderr/stdout
    • bash gives a JSON description of the named pod, and exit code 0.
• Test Cases
  • echo ['echoed_text']
    • Behaves correctly
  • bash ['-c', 'echo echoed_text']
    • Behaves correctly
  • bash ['-c', 'kubectl config current-context']
    • Exit code 0, no stderr/stdout
    • Bash handles this correctly (gives stdout w/ exit 0)
    • Incidentally, this works in Python on the same machine

    • python3 -c "import subprocess as sc;print(sc.run(['bash', '-c', 'kubectl config current-context'], capture_output=True).stdout)"

    • b'microk8s\n'

  • bash ['-c', 'kubectl config current-context | tee /dev/null']
    • WORKS!
    • Exit code 0, stdout indicates current context, empty stderr
    • Perfectly matches Bash behavior.
  • bash ['-c', 'kubectl get pod PODNAME -o json --namespace NAMESPACE | tee /dev/null']
    • Works again
    • Exit code 0, stdout describes pod, empty stderr
    • Perfectly matches Bash behavior.

Correcting remote-containers.attachToK8sContainerFromViewlet

I modified newWindowAttachedToK8sContainer's calls to execOnArgs to match the successful format (the last 2 tests). With these changes, the workflow described above ("Activity Bar > Kubernetes > ... > Attach to Visual Studio") reaches a different error related to kubectl exec.

Directly testing child_process.spawn

Within VSCode

Digging into traces of execOnArgs, I found a function which calls child_process.spawn. This function is named rs in v0.262.3, and is the first instance of the text "SIGKILL". Based on this, I repeated the above tests by directly constructing calls to the child_process module as exposed in the extension. To summarize, the results are identical.

VSCode in debug VS Node in gnome-terminal

I built a test module to run in a pure Node environment and in VSCode. Running node child_proc_kubectl_test.js in a bash terminal results in the expected behavior (the current context is produced). However, if you add a breakpoint to a VSCode extension and require/run the test script, it mimics the behavior when testing in VSCode directly - kubectl config current-context returns code 0 and no stdout/stderr, kubectl get pod ... returns code 1 and no stdout/stderr.

gnome-terminal VS VSCode Terminal

Using the above test module, I ran node child_proc_kubectl_test.js in an instance of gnome-terminal and in the terminal within VSCode (CTRL-`). gnome-terminal produced the expected output, and VSCode's terminal did not.

This is, to me, the strangest claim I make. I've embedded evidence.

Screencast of me running this test
terminal_comparison.webm

Test scripts
child_proc_kubectl_test.js.txt
demo_vscode_kubectl.sh.txt

Collected test output
terminal_test.md
vscode_test.md

diff between test outputs
_test.md.diff.txt

VSCode verbose output (elided). Produced by $(code --verbose . |grep -vwE 'File Watcher|pointer_event_factory' > vscode.elided.log).
vscode.elided.log

diff of environment variables between gnome_terminal and VSCode Terminal
env.diff.txt

VSCode's terminals run a bash subprocess which is provided an argument `--init-file /snap/code/112/usr/share/code/resources/app/out/vs/workbench/contrib/terminal/browser/media/shellIntegration-bash.sh', which (at first glance) seems to focus on managing history and debugging tie-ins, but I didn't go into depth.

Debugging Conclusions

Some aspect(s) of VSCode cause kubectl subprocesses to give no stderr/stdout, and some subcommands/args to fail entirely, in a variety of settings:

• While attempting to "Attach to Visual Studio" re: Kubernetes pods.
• While debugging the Dev Containers extension and attempting to spawn kubectl directly.
  • eg spawn('kubectl', ['get', 'pod', '-o', 'json'])
• While debugging the Dev Containers extension and wrapping kubectl ... in a bash subprocess.
  • eg spawn('bash', ['-c', 'kubectl get pod'])
• While spawning a kubectl ... subprocess inside a Node process inside the bash subprocess backing the VSCode terminal.
  • See above screencast in 'Detailed Debugging of "Attach to Visual Studio"'

In some other settings, kubectl and VSCode get along perfectly well:

• While called directly from the VSCode bash terminal
  • eg. CTRL-` then 'kubectl get pod -o json' works fine
• While debugging the Dev Containers extension and wrapping kubectl ... | tee /dev/null in a bash subprocess.
  • See "execOnArgs Tests" in 'Detailed Debugging of "Attach to Visual Studio"'
• While spawning a kubectl ... | tee /dev/null subprocess inside a Node process inside the bash subprocess backing the VSCode terminal.
  • See above screencast in 'Detailed Debugging of "Attach to Visual Studio"'

The success and failure characteristics are identical across all these different settings. The behavior in the minimal case screencast is identical to the behavior leading to errors while running "Attach to Visual Code".

Minimal Reproducible Error

This is not the error discussed above, but is closely related. This error arises when something in VSCode calls vscode.commands.executeCommand(..., uri, ...) where uri starts with k8s-container. When that URI is interpreted by remote-containers, kubectl exec ... is launched as a subprocess to interact with the container, and fails. This same failure arises even when the errors discussed above are resolved or sidestepped.

This is, I believe, the specific error @tobru mentioned.

1. On CLIENT, follow the 'mtsmfm.vscode-k8s-quick-attach' command palette workflow
   • Select the context, namespace, pod, container, and target root to attach to.
   • This particular implementation avoids the issues in 'Attach to Visual Studio'.
2. A new VSCode window opens
   • An error modal shows "An error occured attaching to the container".
   • Behind that modal, the 'Dev Containers' terminal output shows attempts to call kubectl exec ... on the desired pod.
   • Following these attempts, an error message "Shell server terminated (code: 0, signal: null)".

shell_server_terminated.webm

Transcript of kubectl exec & errors

After verifying the error, I added some debugging to validate @chrmarti 's explanation in response - specifically, deconstructing the command into the args which get passed to spawn.

[60 ms] Dev Containers 0.263.0 in VS Code 1.73.0 (8fa188b2b301d36553cbc9ce1b0a146ccb93351f).
[60 ms] Start: Resolving Remote
[64 ms] Start: Run: kubectl exec -it qs-pod --context microk8s --namespace qs --container server -- /bin/sh -c VSCODE_REMOTE_CONTAINERS_SESSION='546faf54-f639-4dcc-a97d-dd5caf229e281667616251430' /bin/sh (deconstructed: 'kubectl' ["exec", "-it", "qs-pod", "--context", "microk8s", "--namespace", "qs", "--container", "server", "--", "/bin/sh", "-c", "VSCODE_REMOTE_CONTAINERS_SESSION='546faf54-f639-4dcc-a97d-dd5caf229e281667616251430' /bin/sh"])
[67 ms] Start: Run in container: echo $PATH
[252 ms] Stop (188 ms): Run: kubectl exec -it qs-pod --context microk8s --namespace qs --container server -- /bin/sh -c VSCODE_REMOTE_CONTAINERS_SESSION='546faf54-f639-4dcc-a97d-dd5caf229e281667616251430' /bin/sh (deconstructed: 'kubectl' ["exec", "-it", "qs-pod", "--context", "microk8s", "--namespace", "qs", "--container", "server", "--", "/bin/sh", "-c", "VSCODE_REMOTE_CONTAINERS_SESSION='546faf54-f639-4dcc-a97d-dd5caf229e281667616251430' /bin/sh"])
[259 ms] Shell server terminated (code: 0, signal: null)

Potential fixes

1. Transition to @kubernetes/client-node instead of constructing commands as strings/subprocesses
   • Pros:
     • Likely to reduce cross-platform compatibility issues (eg WSL)
     • This approach is used in 'mtsmfm.vscode-k8s-quick-attach', which is the reason that extension isn't running into errors.
     • Potentially provide marginally better tools for typechecking during development
     • Fix my issue in particular :) and several others
   • Cons:
     • Depend on @kubernetes/client-node's featureset instead of directly relating to the host kubectl's featureset
     • Invites some likelihood of version mismatches between @kubernetes/client-node and cluster API (same risk as host kubectl, but still another update to manage)
     • Potentially significant rewrite effort
2. Determine why VSCode causes wide-reaching interference with kubectl subprocesses, and fix that directly

[xgalaxy]

On Windows 10. I've been trying this attach method from the context menu but it fails to establish a connection. I get a websocket 1006 error. I'm able to pull up the remote terminal via ssh from the kubernetes plugin however. But attach doesn't work at all.

If I switch to MacOs it works perfectly.