Debugging inside a container: debugger does not hit breakpoints anymore

[wasphub]

Type: Bug

Behaviour

Expected vs. Actual

Context: Debugging code inside a container

Expected: breakpoint are hit when code is executed.
Actual: breakpoints are not hit, and when the debug session the Debug Console shows:

pydev debugger: unable to find translation for: "c:\dev\qxc\model-sample\qxc\app\run.py" in ["C:\dev\qxc\model-sample/qxc/", "C:\dev\qxc\model-sample/qxc"] (please revise your path mappings).

Worth mentioning: rolling back to version v2022.18.2 of the extension fixes the problem without any change in the path mappings. Those mappings were defined weeks ago, and just stopped working with the latest release.

Steps to reproduce:

1. Configure a project to run and debug inside a container
2. Set some breakpoints
3. Run the project
4. Observe the warning describe above in the Debug Console, and verify that breakpoints are not hit

What seems strange from the message above is that the path inside the container c:\dev\qxc\model-sample\qxc\app\run.py looks like a Windows path, but the container I'm running is from a Linux-based image. I'm on a Windows box with Docker Desktop on WSL2.

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.7.4
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Global
• Value of the python.languageServer setting: Default

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

User Settings

languageServer: "Pylance"

Extension version: 2022.18.2
VS Code version: Code 1.74.0 (5235c6bb189b60b01b1f49062f4ffa42384f8c91, 2022-12-05T16:38:16.075Z)
OS version: Windows_NT x64 10.0.19044
Modes:
Sandboxed: No

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz (8 x 2112)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	undefined
Memory (System)	31.86GB (11.04GB free)
Process Argv	--folder-uri file:///c%3A/dev/qxc/model-sample --crash-reporter-id d3ea3600-8e75-4ca7-a1d1-6a498426b4ab
Screen Reader	no
VM	0%

[brettcannon]

Are you using the Docker extension? If so then you may be running into microsoft/vscode-docker#3750 .

[isears]

I have encountered a similar issue, but using remote debugger features without docker. After reading this SO post I solved the issue by reverting from 2022.20.0 to 2022.18.2.

Seems like whatever broke back then broke again in the latest 2022.20.0

Note: it seems like the debugged program actually halts at the breakpoint, but it's never registered on the clientside within VScode (i.e. no debug control panel pops up as you would normally expect). Once I detach from the debugged process execution continues as normal and I can see output.

[ph03]

I can confirm all that @isears reported on my end also - reverting to 2022.18.2 solved issues with the debugger not attaching to the remote process to debug

[wasphub]

@brettcannon I do use the Docker extension, but it seems to me it's unrelated to the issue. As others posted, reverting to 2022.18.2 solves it, and it's been working fine for weeks until the upgrade to 2022.20.0. Also, the issue you mention shows side effect I do not experience, like the code not running at all, in my case the code runs normally from the container, only breakpoints are not hit.

[brettcannon]

2022.20.0 also updated debugpy, so there was a potential bug involved there. But 2022.20.1 went out yesterday with a rollback to debugpy 1.6.3, so you can unpin the Python extension and see if it works again.

I'm going to transfer this to debugpy in case they know what's going on.

[int19h]

Can you share your launch.json config for this?

[wasphub]

I confirm 2022.20.1 solves the problem, so as you say it might well be an issue with debugpy. I'll let you decide if you want to keep this open or not. Thx for your help!

[wasphub]

Can you share your launch.json config for this?

{
    "configurations": [
        {
            "name": "App Local",
            "type": "docker",
            "request": "launch",
            "preLaunchTask": "docker-run-app-local",
            "python": {
                "pathMappings": [
                    {
                        "localRoot": "${workspaceFolder}/qxc",
                        "remoteRoot": "/qxc",
                    }
                ],
                "projectType": "general"
            }
        }
    ]
}

[fabioz]

@wasphub do you think you can provide the logs when running it in the case where it's failing?
-- I've investigated a bit but I couldn't reproduce it and I'm hoping the logs could help me find what's the actual issue there.

Please see: https://github.com/microsoft/debugpy/wiki/Enable-debugger-logs for details on collecting the debugger logs.

[fabioz]

I was able to reproduce it by setting the wrong clientOS... I was reviewing the code and one thing that stood out which could affect this is that the client os is expected to be passed on the launch config -- previously this was already the case, but the default value was computed in the adapter.

i.e.: in 1.6.3, we have the following code to set the ide OS: https://github.com/microsoft/debugpy/blob/v1.6.3/src/debugpy/adapter/clients.py#L266 whereas in 1.6.4 this code is no longer in the adapter and was moved to the server (where the client is expected to set the clientOS from the arguments passed: https://github.com/microsoft/debugpy/blob/v1.6.4/src/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_json_debug_options.py#L100).

Now, in this particular case the adapter appears to be spawned in the client so, it got the client OS properly, whereas the new code will use the client OS as the same one in the server by default... note that this was probably already a latent bug we had because if the adapter was started on the server through a listen with a remote attach to the port it'd fail even in 1.6.3.

The solution I see here is that the client (in this case vscode-python) should pass the proper clientOS in the launch or attach arguments (since the client always knows where it's running, whereas the adapter may do the right thing or not depending on where it's started).

@int19h @karthiknadig do you think that's reasonable?

If that's the case this needs to be moved to vscode-python so that the clientOS is properly sent in the launch/attach.

As a note, users can already use a work-around by setting "clientOS": "Windows" in the launch configuration (or "clientOS": "Unix" if it's the other way around).

@wasphub can you install the v2022.20.0 version and check if it works if you set "clientOS": "Windows" in your launch configuration (just to make sure this is actually the issue you're seeing and whether it fixes it for you?)

[brettcannon]

/cc @paulacamargo25

[int19h]

That sounds right - the whole purpose of "clientOS" was to allow the client to set it properly in cases where it cannot be detected.

[wasphub]

@fabioz sure, happy to try that, but could you give me a json snippet illustrating where clientOS should be added in launch.json, so to minimize trial and error? I tried to google for that attribute and I could not find any good lead, it's also fine if you point me to some docs explaining how to use it. Thx!

[fabioz]

{
    "configurations": [
        {
            "name": "App Local",
            "type": "docker",
            "request": "launch",
            "preLaunchTask": "docker-run-app-local",
            "clientOS": "Windows", 
            "python": {
                "pathMappings": [
                    {
                        "localRoot": "${workspaceFolder}/qxc",
                        "remoteRoot": "/qxc",
                    }
                ],
                "projectType": "general"
            }
        }
    ]
}

[wasphub]

I confirm that adding "clientOS": "Windows" fixes the issue even with v2022.20.0. The clientOS attribute is underlined with a squiggly by VS Code, but it does work.

[fabioz]

As a note, the fact that it's underlined is Ok. The idea is that users shouldn't really need to see that, the client (in this case vscode-python) should always set clientOS and pass it under the hood when launching/attaching the debugger.

[fabioz]

@karthiknadig can you move this to vscode-python?

[karthiknadig]

@paulacamargo25 Can you verify that this is set correctly? This used to be set in the config resolver.

[neeteshbhati]

Seems to be a recurring issue with almost every new version of the extension. It was working perfectly fine last night, woke up to realize my debug points are not being hit. My current version is v2022.20.2. Rolling back to v2022.20.1 seems to resolve the issue.

It happened a week earlier as well, around VSCode 1.74.0 which force updated this extension version. Had to roll back VSCode version to roll back extension version.

[retatu]

I'm having a similar issue. I run a aws python lambda image on docker, the debug is being attached but the code doesn't stop in the breakpoints :/

[de846]

This is also an issue when using the Python debugger not in a container, but with Remote SSH. If I use a launch configuration like the following in Run and Debug using SSH to my backend, setting any breakpoint causes that run to spin indefinitely. It will run to completion if no breakpoints are set, however.

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "justMyCode": true,
        }
    ]
}

This fails consistently (100% of the time) in v2022.20.2 and the current Pre-Release version (v2023.3.10451008). Rolling back to v2022.20.1 works consistently.

I also tried setting clientOS to unix or windows, but it doesn't unbreak debugging with breakpoints.

[jonathonfletcher]

+1 for #20407 (comment)

MacOS 13.2 using "Remote - SSH" v0.96.0 (to an Ubuntu 22LTS remote) and VS Code 1.75.1. This started consistently breaking with the v2022.20.2 and also consistently breaks with v2023.2.0. This setup has previously been stable / usable for a while with previous versions.

Setting breakpoints and then attempting to debug a python file with the same launch.json as above fails on breakpoints. I get "The editor could not be opened because the file was not found" where the file / breakpoint should be displayed in the editor window.

Using a remote user 'user' with a home directory of '/home/user', a project directory of '/home/user/project' and attempting to debug 'thing.py' with 'foo/bar.py' as the file with the breakpoint, the top-left of the editor window (above the error) shows a path of 'user/project/foo/bar.py' . This does not work as an absolute path or as a path relative to the file being debugged. '/home/user/project/foo/bar.py' or 'foo/bar.py' would be "correct" in this example.

Have not yet managed to revert the python extension back to to v2022.20.1 - mostly due to trying to remove the extraneous jupyter-related extensions that get installed on the remote with v2023.2.0. A guide to reverting back correctly from a fresh install with this setup would be helpful.

[int19h]

@jonathonfletcher Does "clientOS" make any difference for you?

[jonathonfletcher]

@jonathonfletcher Does "clientOS" make any difference for you?

Hi, thanks for the suggestion. I just tried with

"clientOS": "Unix",

in the launch.json. I get a yellow squiggly underneath and the setting seems to be ignored.

Unrelated, but potentially interesting, if I add

"env": { "PYDEVD_DEBUG": "True" },

in the launch.json, I see a possible difference between the paths of files that do open vs ones that do not in the pydevd output.

I get some references to (e.g.): /home/user/project/thing.py which is the main that I am debugging. This is the only editor tab open when I start with F5 for debugging. The python package foo/bar.py appears in the pydevd output as //home/user/project/foo/bar.py - double / at the beginning. This is a valid unix path but is it possible that this is tripping up some url-opener (i.e. file:////home/user/project/foo/bar.py instead of file:///home/user/project/foo/bar.py) ?

Again, I'm just noticing this / vs // difference. May be not relevant.

Edit: The // prefix was due to a PYTHONPATH setting. Correcting it makes no difference.

When "The editor could not be opened because the file was not found" error is shown, there is a "Create File" option. Selecting this gives me a further model dialog error:

Unable to write file 'vscode-remote://ssh-remote+server/user/project/foo/bar.py' (NoPermissions (FileSystemError): Error: EACCES: permission denied, mkdir '/user')

ie the extension is not managing the paths on the remote correctly. The vscde-remote url should be vscode-remote://ssh-remote+server/home/user/project/foo/bar.py

[int19h]

Earlier, you've mentioned that the path that the editor shows is "user/project/foo/bar.py" - without the leading slash. That would seem to imply that it's getting a relative path from somewhere and failing to properly translate that to an absolute path. If you print(__file__) inside bar.py, what does that show under the debugger?

[jonathonfletcher]

Thanks for the message. The "user/project/foo/bar.py" shown in the editor (top-left) is shown with the "The editor could not be opened because the file was not found" error.

pydevd's debug shows the file as "/home/user/project/foo/bar.py" and print(__file__) will show "/home/user/project/foo/bar.py".

I am beginning to suspect that this is actually a "Remote - SSH" extension bug and not a "Python" extension bug because the "The editor could not be opened because the file was not found" error's "Create File" button has the wrong url for the file: vscode-remote://ssh-remote+server/user/project/foo/bar.py should be vscode-remote://ssh-remote+server/home/user/project/foo/bar.py

If I force a workaround on the remote host with sudo ln -s /home/user / in order to make the vscde-remote url "valid" on the remote (ie /user/project/foo/bar.py -> /home/user/project/foo/bar.py) then the editor window comes up with the correct file and the python debugger stops and renders at the breakpoint.

[tomunger]

I had the same problem with breakpoints not working and this was fixed by rolling back to v2022.20.1.

From a windows 11 system I am trying to debug python code running in a linux container. My application runs correctly in the container (except one bug that appears only under linux). Break on raised exception works and drops me into the current code. However, line breakpoints are ignored. Toggling a breakpoint shows the "not really ready" image.

At session start, the debug console gives these errors for every file with a breakpoint:

3.68s - pydev debugger: unable to find translation for: "c:\Users\tunge\Documents\intg\intg-core\modules\CE_TO_CEN.py" in ["C:\Users\tunge\Documents\intg\intg-core/", "C:\Users\tunge\Documents\intg\intg-core"] (please revise your path mappings).

0.01s - pydev debugger: unable to find translation for: "c:\Users\tunge\Documents\intg\intg-core\modules\mq_receive.py" in ["C:\Users\tunge\Documents\intg\intg-core/", "C:\Users\tunge\Documents\intg\intg-core"] (please revise your path mappings).

It looks like the list of target directories is incorrect.

pathMappings in launch.json appear to be correct and is what is shown in the documentation:

      {
            "name": "Docker: modules/mq-receive CE_TO_CEN docker",
            "type": "docker",
            "request": "launch",
            "preLaunchTask": "ce-to-cen-docker-run: debug",
            "python": {
                "pathMappings": [
                    {
                        "localRoot": "${workspaceFolder}",
                        "remoteRoot": "/app"
                    }
                ],
                "projectType": "general"
            }
        },

Environment where breakpoints don't work is:

• Windows 11 Pro (recently updated)
• VS Code 1.74.3 (because of docker extension bug #3823 in 1.75.1)
• Python extension v2022.20.2
• Docker extension v1.23.3
• Container is based on python:3.10

In python extension version v2022.20.2 I tried setting "clientOS": "Unix", but this resulted in the debugger not starting.

[int19h]

this resulted in the debugger not starting

This is concerning, since it would likely do the same once VSCode starts supplying this setting automatically. Can you please share the error message that it produces, if any?

[tomunger]

Running with extension v2022.20.1

"clientOS": "Unix",

results in the error "clientOS" must be one of "windows" or "unix". Different than before so that casts doubt on previous results.

When I change the value to "unix" the debugger runs, but does not stop at breakpoints.

Remove the setting and the breakpoint works again.

With extension version v2022.20.2

With the setting, the debugger runs but does not stop at break points.

Without the setting, the debugger runs but does not stop at break points.

(Back to v2022.20.1 without the setting and breakpoints work again. That's my cozy place for development.)

[int19h]

My apologies, I should have caught this earlier - since your VSCode is running on Windows, it should correspond to "clientOS": "windows" for proper path mapping.

[tomunger]

Ah, thanks. With the correct value, extension version v2002.20.2 will stop at breakpoints.

[int19h]

Thank you for confirming!

[paulacamargo25]

We need to set clientOS flag in all scenarios (launch and attach) in the debugOptions.

[rjra2611]

We need to set clientOS flag in all scenarios (launch and attach) in the debugOptions.

Hey, I'm running into this issue when upgrading to v2023.4.1 or the latest v2023.6.0

I'm getting the following error when trying to attach to a remote process using the following configurations

Error:
pydev debugger: unable to find translation for: "d:\vscode-init\debuger-test\main.py" in ["D:\vscode-init\debuger-test/", "D:\vscode-init\debuger-test"] (please revise your path mappings).

Launch.json

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug",
            "type": "python",
            "request": "attach",
            "clientOS": "windows",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceRoot}",
                    "remoteRoot": "/app"
                }
            ]
        }
    ]
}

Environment Information:

• Windows 10 Enterprise
• vscode-python v2023.4.1

NOTE: It works with/without using "clientOS": "windows" when using v2023.4.0

[karthiknadig]

@rjra2611 I created microsoft/debugpy#1257 on debugpy, please follow up there on this issue.