Debugging Docker Container -- Not properly attaching to container #3751

akumareog · 2022-12-07T15:07:08Z

Type: Bug

References issue in vscode repo here

Debugging with the launch.json and tasks.json set up as prescribed by the docs here was working up until recently. It seems it's able to spin up the container. But afterwards it's not able to attach properly and output container logs to the Debug Console, nor is it able to respect the breakpoints I insert in VS Code.

Not sure it's relevant here, but the docker image has an entrypoint.sh where depending on the "foo" environment variable will execute either node --inspect=0.0.0.0 app.js or node app.js.

My current debug setup:

launch.json

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Docker: Node Debug Service",
      "type": "docker",
      "request": "launch",
      "platform": "node",
      "preLaunchTask": "docker-run:debug service"
    }
  ]
}

tasks.json

{
  "version": "2.0.0",
  "tasks": [
    {
      "type": "docker-build",
      "label": "docker-build",
      "platform": "node",
      "dockerBuild": {
        "context": "${workspaceFolder}",
        "dockerfile": "${workspaceFolder}/Dockerfile",
        "customOptions": "--platform linux/amd64",
        "tag": "node-app",
        "pull": true
      }
    },
    {
      "type": "docker-run",
      "label": "docker-run:debug service",
      "dependsOn": ["docker-build"],
      "platform": "node",
      
      "dockerRun": {
        "remove": true,
        "env": {
          "foo": "bar"
        },
        "ports": [{ "containerPort": 8080, "hostPort": 8080 }],
        "containerName": "node-debug-service"
      },
      "node": {
        "enableDebugging": true
      }
    }
  ]
}

VS Code version: Code 1.73.1 (Universal) (6261075646f055b99068d3688932416f2346dd3b, 2022-11-09T02:08:38.961Z)
OS version: Darwin arm64 22.1.0
Modes:
Sandboxed: No

System Info

Item	Value
CPUs	Apple M1 Max (10 x 24)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	3, 3, 4
Memory (System)	64.00GB (13.35GB free)
Process Argv	--crash-reporter-id 6b794953-834c-45f8-8b9f-19b2e3c1868e
Screen Reader	no
VM	0%

Extensions (42)

Extension	Author (truncated)	Version
better-comments	aar	3.0.2
vscode-sql-formatter	adp	1.4.4
vscode-sqlite	ale	0.14.1
vscode-fish	bma	1.0.31
npm-intellisense	chr	1.4.3
vscode-markdownlint	Dav	0.48.1
es7-react-js-snippets	dsz	4.4.3
gitlens	eam	13.1.1
vscode-html-css	ecm	1.13.1
EditorConfig	Edi	0.16.4
vscode-npm-script	eg2	0.3.29
prettier-vscode	esb	9.10.3
shell-format	fox	7.2.2
go	gol	0.36.0
cloudcode	goo	1.20.5
hcl	has	0.3.1
terraform	has	2.25.1
file-downloader	min	1.0.12
mindaro	min	1.0.120221201
theme-monokai-pro-vscode	mon	1.2.0
vscode-docker	ms-	1.23.1
vscode-dotnet-runtime	ms-	1.6.0
vscode-kubernetes-tools	ms-	1.3.11
data-workspace-vscode	ms-	0.3.0
mssql	ms-	1.16.0
sql-bindings-vscode	ms-	0.3.0
sql-database-projects-vscode	ms-	0.19.0
isort	ms-	2022.8.0
python	ms-	2022.18.2
vscode-pylance	ms-	2022.11.40
remote-containers	ms-	0.262.3
indent-rainbow	ode	8.3.1
oracledevtools	Ora	21.5.0
material-icon-theme	PKi	4.22.0
ruby	reb	0.28.1
vscode-yaml	red	1.10.1
fish-vscode	sky	0.2.1
tabnine-vscode	Tab	3.6.31
ayu	tea	1.0.5
pdf	tom	1.2.0
vscode-ruby	win	0.28.0
plsql-language	xyz	1.8.2

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492cf:30256860
vslsvsres303:30308271
pythonvspyl392:30443607
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263cf:30335440
vscorecescf:30445987
pythondataviewer:30285071
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
cmake_vspar411:30581797
vsaa593:30376534
pythonvs932:30410667
cppdebug:30492333
vscaat:30438848
vsclangdf:30486550
c4g48928:30535728
dsvsc012cf:30540253
azure-dev_surveyone:30548225
pyindex848cf:30577861
nodejswelcome1cf:30587006
282f8724:30602487
gswce1:30612156
3d0df643:30613357
dbltrim-noruby:30604474
f6dab269:30613381
fim-prod:30623723

The text was updated successfully, but these errors were encountered:

bwateratmsft · 2022-12-07T15:09:28Z

Can you share the output from the tasks terminal window for the docker-build and docker-run tasks?

akumareog · 2022-12-07T15:18:32Z

Sure. I removed some sensitive information but if needed I can format and include here as well.

EDIT: Removed links to other issues....

 *  Executing task: docker-build 

> docker image build --pull --file 'repo/code/Dockerfile' --tag 'node-app' --label 'com.microsoft.created-by=visual-studio-code' --platform linux/amd64 'repo/code' <

1 [internal] load build definition from Dockerfile
1 sha256:...1 transferring dockerfile: 40B done
1 DONE 0.0s

2 [internal] load .dockerignore
2 sha256:...
2 transferring context: 34B done
2 DONE 0.0s

--- several docker build steps ---

32 exporting to image
32 sha256:...
32 exporting layers done
32 writing image sha256:... done
32 naming to docker.io/library/node-app done
32 DONE 0.0s *  Terminal will be reused by tasks, press any key to close it. 

 *  Executing task: docker-run:debug service 

> docker container run --detach --tty --rm --name 'node-debug-service' --publish '8080:8080' --publish '9229:9229' --label 'com.microsoft.created-by=visual-studio-code' --env 'foo=bar' node-debug-service <

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested ...

bwateratmsft · 2022-12-07T15:21:43Z

@philliphoff were there any issues with debugging Node on M1's, back when you were looking into it? Especially cross-platform (if that's the right term) with amd64 containers.

akumareog · 2022-12-07T16:22:41Z

Just to add: I tried debugging a similar set up I have but for a Python app and I'm seeing similar issues. Both of these debuggers worked alright within the past month even with the warning about mismatched platforms. In both cases, after the docker container run log statement the terminal just hangs with a spinning circle while the container successfully runs in the background. Details below:

launch.json

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Docker: Python App Debug",
      "type": "docker",
      "request": "launch",
      "preLaunchTask": "docker-run: debug",
      "python": {
        "pathMappings": [
          {
            "localRoot": "${workspaceFolder}/src",
            "remoteRoot": "/opt/app/src"
          }
        ],
        "projectType": "general"
      }
    }
  ]
}

tasks.json

{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "docker-build",
      "type": "docker-build",
      "dockerBuild": {
        "context": "${workspaceFolder}",
        "dockerfile": "${workspaceFolder}/Dockerfile",
        "tag": "enterprise-registry.com/python-app:latest",
        "pull": true,
        "customOptions": "--platform linux/amd64"
      }
    },
    {
      "type": "docker-run",
      "label": "docker-run: debug",
      "dependsOn": ["docker-build"],
      "dockerRun": {
        "containerName": "python-app-debug",
        "image": "enterprise-registry.com/python-app:latest",
        "env": {
          "foo": "bar"
        }
      },
      "python": {
        "file": "app.py"
      }
    }
  ]
}

Build and run terminal outputs:

 *  Executing task: docker-build 

> docker image build --pull --file 'repo-code/Dockerfile' --tag 'enterprise-registry.com/python-app:latest' --label 'com.microsoft.created-by=visual-studio-code' --platform linux/amd64 'repo/code' <

1 [internal] load build definition from Dockerfile 1 sha256:...
1 transferring dockerfile: 874B done
1 DONE 0.0s

2 [internal] load .dockerignore
2 sha256:...
2 transferring context: 2B done
2 DONE 0.0s

10 [internal] load build context
10 sha256:...
10 transferring context: 149.72kB 0.0s done
10 DONE 0.0s

17 exporting to image
17 sha256:...
17 exporting layers 17 exporting layers 0.7s done17 writing image sha256:...
17 naming to  enterprise-registry.com/python-app:latest done
17 DONE 0.7s *  Terminal will be reused by tasks, press any key to close it. 

 *  Executing task: docker-run: debug 

> docker container run --detach --tty --name 'python-app' --publish-all --mount 'type=bind,source=/repo/code/.vscode/extensions/ms-python.python-2022.18.2/pythonFiles/lib/python/debugpy,destination=/debugpy,readonly' --label 'com.microsoft.created-by=visual-studio-code' --env 'foo=bar' --entrypoint 'python3' enterprise-registry.com/python-app:latest <

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested ...

bwateratmsft · 2022-12-07T16:37:32Z

Can you try rolling back the Docker extension to 1.22.2, to see if that fixes it? That would help narrow down a potential cause. You can do so by running the command "Extensions: Install Specific Version of Extension..." in the command palette.

akumareog · 2022-12-07T16:40:38Z

The rollback worked. Getting outputs to the debug console now

bwateratmsft · 2022-12-07T16:43:38Z

(Sorta thinking out loud here) I wonder if the change to the bind mount syntax is causing this. I think that's the most substantial change to debugging--before, we used -v, now we're using --mount.

EDIT: well, Node doesn't even use mounts, so I don't think it's that...

akumareog · 2022-12-07T16:52:24Z

Could be something to do with the docker run command. The latest version of the Docker extension uses what I'm guessing is the latest syntax for spinning up the container. Here's what was executed for the Node app with v1.22.2 of the Docker extension that was successful:

 *  Executing task: docker-run:debug service 

> docker run -dt --name "node-debug-service" -e "foo=bar" --label "com.microsoft.created-by=visual-studio-code" -p "8080:8080" -p "9229:9229" --rm "node-debug-service" <

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
...
 *  Terminal will be reused by tasks, press any key to close it.

EDIT: Only real difference I see is the --tty argument

bwateratmsft · 2022-12-07T16:54:37Z

In theory that is supposed to be identical to the t in the -dt

bwateratmsft · 2022-12-07T17:29:41Z

I tried running one container each with the old and new command, and then inspecting them, but they are identical in every way (except for the ways you'd expect, like container ID and similar). I have another theory though. We've made some considerable changes to how processes are run in the background, including potentially some different handling for output to stderr. I was able to reproduce something that looked more or less the same symptoms by running a linux/arm64 container (on my Windows x64 machine).

I need to keep digging into this, but I suspect that warning is being emitted to stderr, which might be causing the extension to erroneously think the docker run command failed. From there, it seems like the task system is never realizing that the task has completed, so it spins forever.

bwateratmsft · 2022-12-07T17:30:09Z

This could be the same issue causing #3750.

bwateratmsft · 2022-12-07T18:06:17Z

I think I found the root cause underneath everything. In short, we had two bugs in 1.22.2 (and many versions prior)--but they cancelled each other out. In 1.23.0, we fixed one of those bugs, which caused the other one to show up.

Throughout older versions and 1.23.0+, we had a flag to reject if anything was emitted to stderr from docker run. This was a mistake; we should have only rejected if the process exited with a nonzero code.

vscode-docker/src/tasks/DockerRunTaskProvider.ts

Line 64 in 959987e

true, // rejectOnStderr

The way this rejection occurred was to throw on stderr here:

vscode-docker/src/tasks/DockerPseudoterminal.ts

Lines 76 to 82 in 959987e

    
           (stderr: string) => { 
        
               this.writeError(stderr); 
        
               if (rejectOnStderr) { 
        
                   throw new Error(stderr); 
        
               } 
        
           },

But that doesn't throw out of that executeCommandInTerminal() method--it's in a callback, given to spawnAsync, and throws out of the handler to that callback...which Node ignores entirely, cancelling out the erroneous rejectOnStdErr above:

vscode-docker/src/utils/spawnAsync.ts

Lines 98 to 110 in 959987e

    
           if (onStderr || stderrBuffer) { 
        
               subprocess.stderr.on('data', (chunk: Buffer) => { 
        
                   const data = chunk.toString(); 
        
                   if (onStderr) { 
        
                       onStderr(data, subprocess); 
        
                   } 
        
                   if (stderrBuffer) { 
        
                       stderrBytesWritten += stderrBuffer.write(data, stderrBytesWritten); 
        
                   } 
        
               }); 
        
           }

So the fix is two-fold.

Stop rejecting on stderr from docker run.
Don't throw in the callback, throw from outside the callback, if rejectOnStdErr is true. Or, better yet, get rid of it because it's no longer necessary for anything.

philliphoff · 2022-12-07T20:09:16Z

@philliphoff were there any issues with debugging Node on M1's, back when you were looking into it? Especially cross-platform (if that's the right term) with amd64 containers.

I know the .NET debugger doesn't like operating cross-architecture (e.g. using amd64 containers on an M1 machine), but I don't believe Node.js has that limitation.

danegsta · 2022-12-19T19:32:37Z

A fix for this has been released in 1.23.3.

bwateratmsft added the info-needed label Dec 7, 2022

bwateratmsft added investigate Node/JS Python and removed info-needed labels Dec 7, 2022

bwateratmsft added bug and removed investigate labels Dec 7, 2022

bwateratmsft self-assigned this Dec 7, 2022

bwateratmsft added this to the 1.23.3 milestone Dec 7, 2022

bwateratmsft mentioned this issue Dec 7, 2022

Don't reject on stderr for docker run #3754

Merged

bwateratmsft closed this as completed in #3754 Dec 12, 2022

bwateratmsft removed their assignment Dec 12, 2022

bwateratmsft added the fixed in next release label Dec 12, 2022

danegsta added fix released and removed fixed in next release labels Dec 19, 2022

microsoft locked and limited conversation to collaborators Jan 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Debugging Docker Container -- Not properly attaching to container #3751

Debugging Docker Container -- Not properly attaching to container #3751

akumareog commented Dec 7, 2022

bwateratmsft commented Dec 7, 2022 •

edited

Loading

akumareog commented Dec 7, 2022 •

edited

Loading

bwateratmsft commented Dec 7, 2022

akumareog commented Dec 7, 2022

bwateratmsft commented Dec 7, 2022

akumareog commented Dec 7, 2022

bwateratmsft commented Dec 7, 2022 •

edited

Loading

akumareog commented Dec 7, 2022 •

edited

Loading

bwateratmsft commented Dec 7, 2022

bwateratmsft commented Dec 7, 2022

bwateratmsft commented Dec 7, 2022

bwateratmsft commented Dec 7, 2022

philliphoff commented Dec 7, 2022

danegsta commented Dec 19, 2022

Debugging Docker Container -- Not properly attaching to container #3751

Debugging Docker Container -- Not properly attaching to container #3751

Comments

akumareog commented Dec 7, 2022

bwateratmsft commented Dec 7, 2022 • edited Loading

akumareog commented Dec 7, 2022 • edited Loading

bwateratmsft commented Dec 7, 2022

akumareog commented Dec 7, 2022

bwateratmsft commented Dec 7, 2022

akumareog commented Dec 7, 2022

bwateratmsft commented Dec 7, 2022 • edited Loading

akumareog commented Dec 7, 2022 • edited Loading

bwateratmsft commented Dec 7, 2022

bwateratmsft commented Dec 7, 2022

bwateratmsft commented Dec 7, 2022

bwateratmsft commented Dec 7, 2022

philliphoff commented Dec 7, 2022

danegsta commented Dec 19, 2022

bwateratmsft commented Dec 7, 2022 •

edited

Loading

akumareog commented Dec 7, 2022 •

edited

Loading

bwateratmsft commented Dec 7, 2022 •

edited

Loading

akumareog commented Dec 7, 2022 •

edited

Loading