IntelliSense process crash detected

[dmitttri]

Environment

• OS and Version: Fedora 37
• VS Code Version: 1.76.2
• C/C++ Extension Version: v1.14.4
• If using SSH remote, specify OS of remote machine: No

Hi everyone,
I am facing a blocking issue with the IntelliSense. Currently woking on a proprietary code base, of decent complexity, and for many of the source files (but not all), I constantly get the message IntelliSense process crash detected, and IntelliSense features simply don't work. This is repeatble, once a source file is open, and I start editing, crashes will happen.

Downgrading VS Code or C/C++ Extension versions has no effect, same crashes happens again.

I was able to collect debug output and call traces, please find them below.

Bug Summary and Steps to Reproduce

Bug Summary: Some source files (but not all) causes IntelliSense process crash detected, repeatedly, when being open and edited.

Steps to reproduce:
In the case of the code base I am working one, it's enough to simply open one of these problematic source files, and start editing.

Unfortunatley, I can't provide any portions of the proprietary source code. I have tried to reproduce the crash with other open source code bases I have found on GitHub, but I was not able to reproduce it.

Expected behavior:
IntelliSense shall report at least sytax errors, while editing source files.

Configuration and Logs

c_cpp_properties.json

{
    "configurations": [
        {
            "name": "Linux",
            "includePath": [
                "${workspaceFolder}/engine/include"
            ],
            "defines": [],
            "compilerPath": "/usr/bin/gcc",
            "cStandard": "c17",
            "cppStandard": "gnu++17",
            "intelliSenseMode": "linux-gcc-x64"
        }
    ],
    "version": 4
}

C/C++ Debug Output (just a portion)

...
textDocument/hover: /home/dimitrije/devel-cpp/trcc/engine/include/engine/core/engine_ops.h (id: 6)
uv_read_cb failed with nread = -104
IntelliSense process crash detected.
...

Other Extensions

CMake v0.0.17
CMake Tools v1.13.45

Additional context

Attaching to cpptools-srv don't produce any exceptions and stack traces, which is strange because C/C++ Debug output reports crashes.

The only stack trace I was able to produce with cpptools call trace (on of the threads)

__cxa_throw (Unknown Source:0)
scout_parser_cpp_generated::init_declarator(a_type_specification&, a_declarator_info&, an_identifier&) (Unknown Source:0)
scout_parser_cpp_generated::statement() (Unknown Source:0)
scout_parser_cpp_generated::statement_block(bool, a_source_point&) (Unknown Source:0)
scout_parser_cpp_generated::compound_statement(bool, a_source_point&) (Unknown Source:0)
scout_parser_cpp_generated::function_body(an_identifier const&, a_source_point&) (Unknown Source:0)
scout_parser_cpp_generated::init_declarator_list_or_function_definition(a_source_point const&, a_source_point&, a_type_specification&, a_declaration_info const&) (Unknown Source:0)
scout_parser_cpp_generated::simple_declaration(a_type_specification&, antlr::TokenRefCount<antlr::Token>, a_declaration_info const&) (Unknown Source:0)
scout_parser_cpp_generated::property_or_event_or_simple_declaration(a_type_specification&, antlr::TokenRefCount<antlr::Token>, a_declaration_info const&) (Unknown Source:0)
scout_parser_cpp_generated::general_declaration(a_type_specification&, antlr::TokenRefCount<antlr::Token>, a_declaration_info const&) (Unknown Source:0)
scout_parser_cpp_generated::general_declaration_with_error_recovery_inner(a_type_specification&, antlr::TokenRefCount<antlr::Token>, a_declaration_info const&) (Unknown Source:0)
scout_parser_cpp_generated::general_declaration_with_error_recovery(a_type_specification&, antlr::TokenRefCount<antlr::Token>, a_declaration_info const&) (Unknown Source:0)
scout_parser_cpp_generated::declaration(antlr::TokenRefCount<antlr::Token>, a_declaration_info const&, bool) (Unknown Source:0)
scout_parser_cpp_generated::declarations(bool) (Unknown Source:0)
scout_parser_cpp_generated::namespace_definition_or_alias(antlr::TokenRefCount<antlr::Token>) (Unknown Source:0)
scout_parser_cpp_generated::general_declaration(a_type_specification&, antlr::TokenRefCount<antlr::Token>, a_declaration_info const&) (Unknown Source:0)
scout_parser_cpp_generated::general_declaration_with_error_recovery_inner(a_type_specification&, antlr::TokenRefCount<antlr::Token>, a_declaration_info const&) (Unknown Source:0)
scout_parser_cpp_generated::general_declaration_with_error_recovery(a_type_specification&, antlr::TokenRefCount<antlr::Token>, a_declaration_info const&) (Unknown Source:0)
scout_parser_cpp_generated::declaration(antlr::TokenRefCount<antlr::Token>, a_declaration_info const&, bool) (Unknown Source:0)
scout_parser_cpp_generated::declarations(bool) (Unknown Source:0)

[sean-mcmanus]

Are you able to reduce the crash such that it can be reproduced with a non-proprietary code sample (it's possible it may be occurring from system header code as well). From the call stack, it looks like it's failing to parse some declaration initializer.

[sean-mcmanus]

Actually, it looks like you may not have the correct call stack. Did you attach to cpptools or cpptools-srv? I think scout_parser_cpp_generated only occurs in cpptools and it throws exceptions as part of it normal flow of control. Your crash call stack does not specifically indicate an access violation or crash occurred (just a c++ exception).

[dmitttri]

Hi Sean, thanks for your valuable feedback!

Are you able to reduce the crash such that it can be reproduced with a non-proprietary code sample (it's possible it may be occurring from system header code as well). From the call stack, it looks like it's failing to parse some declaration initializer.

Unfortunately not, I have tried to reproduce the crash with couple of open source CMake/C++ code bases, with decent complexity, but IntelliSense was working fine.

[dmitttri]

Actually, it looks like you may not have the correct call stack. Did you attach to cpptools or cpptools-srv? I think scout_parser_cpp_generated only occurs in cpptools and it throws exceptions as part of it normal flow of control. Your crash call stack does not specifically indicate an access violation or crash occurred (just a c++ exception).

I actually attached both cpptools and cpptools-srv, and I was quite sure the exception hit one of the cpptools-srv thread. But let me check once again, and update the stack trace, if necessary

[dmitttri]

Actually you are right, now I can confirm that it was cpptools that caught the C++ exception (not crash), and the call trace I pasted is form this process (one of it's threads).

Now what is quite strange is that if I attach only to cpptools-srv, I won't get any crashes or stack trace, while C/C++ Debug output will report numerous IntelliSense process crash detected.

What I could observe is that cpptools-srv process has fixed amount of threads being started (74 including the main process), and by checking their number and TID's, before and after reported crashes its exactly the same. No one of them actually crashes.

[sean-mcmanus]

Do IntelliSense features function, hoving over 0 shows (int)0)?

Could it be that the logging is incorrect?

Is the "program" in launch.json set to "cpptools-srv" instead of "cpptools"?

[dmitttri]

Do IntelliSense features function, hoving over 0 shows (int)0)?

No, for the affected source files, it just shows "Loading" for couple of seconds.

Could it be that the logging is incorrect?

I don't really know, Debug output simply reports "IntelliSense process crash detected" many times, pretty much on every action I take on the source file, like hover or deliberately inducing a syntax error

Do IntelliSense features function, hoving over 0 shows (int)0)?

Could it be that the logging is incorrect?

Is the "program" in launch.json set to "cpptools-srv" instead of "cpptools"?

It's set to "cpptools-srv" executable.

[sean-mcmanus]

If you see "Loading" that would suggest the IntelliSense process is either not launching or crashing, so it's probably crashing, giving the logging. I'm guessing you may not be attached to the correct cpptools-srv process? Can you comment out all the code in the source file that is open/crash, attach, and the uncomment the code, then it should crash with the correct call stack.

[dmitttri]

Can you comment out all the code in the source file that is open/crash, attach, and the uncomment the code, then it should crash with the correct call stack.

I did it now, commented out all the code, then attached to cpptools-srv process, then uncommented code, got Debug output message that IntelliSense crashed, but as before, none of the attached cpptools-srv process/threads crashed. They just continue to work, while IntelliSense do not work.

I tried the same thing before, just with closing the problematic file, attaching, and then re-opening the file to produce crash. Result is the same, can't get cpptools-srv (or any of it's threads) to crash

[dmitttri]

Now I am thinking to setup the full cpptools debugging environment, by followinfg the provided tutorials, and try to figure out which parsing definition causes the crash. Since this happens with many of the source files, there must be something like a common definition which is used widely in the code. It may help me to setup a lot simpler code example which will cause the crash.

[dmitttri]

@sean-mcmanus I have managed to setup full debugging environment for cpptools extension. Although I am not familiar with TypeScrip and JavaScript, i took the following path:

• Search for "IntelliSense process crash detected" and concluded it refers to Extension/src/NativeStrings.ts line 1173, where I placed the breakpoint
• on breakpoint stop, I got CallStack, took a brief look at it and decided to place a new breakpoint in /home/dimitrije/devel-cpp/vscode-cpptools/Extension/node_modules/vscode-jsonrpc/lib/common/connection.js line 362
• Recorded message contents, copule of messages before the the "IntelliSense process crash detected" was received.

[dmitttri]

This is the train of recorded message contents, json format:

{
  jsonrpc: "2.0",
  method: "cpptools/publishRefactorDiagnostics",
  params: {
    uri: "file:///home/dimitrije/devel-cpp/trcc/engine/include/engine/core/engine_ops.h",
    diagnostics: [
    ],
  },
}

{
  jsonrpc: "2.0",
  method: "cpptools/logTelemetry",
  params: {
    event: "Language_Server_Stats",
    metrics: {
      "$/cancelRequest": 1,
      "cpptools/getCodeActions": 1,
      "cpptools/getDocumentSymbols": 1,
      "cpptools/getFoldingRanges": 1,
      "cpptools/getInlayHints": 1,
      "cpptools/textEditorSelectionChange": 1,
      "textDocument/didChange": 1,
      "textDocument/signatureHelp": 1,
    },
  },
}

{
  jsonrpc: "2.0",
  id: 53,
  result: {
    signatures: [
    ],
    activeSignature: 0,
    activeParameter: 0,
  },
}

{
  jsonrpc: "2.0",
  id: 54,
  result: {
    commands: [
    ],
  },
}

{
  jsonrpc: "2.0",
  id: 49,
  result: {
    code: -32802,
    message: "",
  },
}

{
  jsonrpc: "2.0",
  id: 57,
  result: {
    ranges: [
      {
        kind: 0,
        range: {
          startLine: 0,
          endLine: 108,
        },
      },
      {
        kind: 2,
        range: {
          startLine: 3,
          endLine: 9,
        },
      },
      {
        kind: 0,
        range: {
          startLine: 11,
          endLine: 107,
        },
      },
      {
        kind: 0,
        range: {
          startLine: 12,
          endLine: 106,
        },
      },
      {
        kind: 0,
        range: {
          startLine: 14,
          endLine: 19,
        },
      },
      {
        kind: 0,
        range: {
          startLine: 23,
          endLine: 80,
        },
      },
      {
        kind: 0,
        range: {
          startLine: 33,
          endLine: 36,
        },
      },
      {
        kind: 0,
        range: {
          startLine: 48,
          endLine: 64,
        },
      },
      {
        kind: 0,
        range: {
          startLine: 65,
          endLine: 66,
        },
      },
      {
        kind: 0,
        range: {
          startLine: 82,
          endLine: 104,
        },
      },
      {
        kind: 0,
        range: {
          startLine: 85,
          endLine: 102,
        },
      },
      {
        kind: 0,
        range: {
          startLine: 89,
          endLine: 101,
        },
      },
    ],
  },
}

{
  jsonrpc: "2.0",
  method: "cpptools/reportStatus",
  params: {
    status: "C_Cpp: IntelliSense",
  },
}

{
  jsonrpc: "2.0",
  method: "cpptools/publishIntelliSenseDiagnostics",
  params: {
    uri: "file:///home/dimitrije/devel-cpp/trcc/engine/include/engine/core/engine_ops.h",
    diagnostics: [
    ],
  },
}

{
  jsonrpc: "2.0",
  method: "cpptools/debugLog",
  params: {
    text: "IntelliSense process crash detected.",
    stringId: 224,
    stringArgs: [
    ],
    indentSpaces: 0,
  },
}

[dmitttri]

The basic idea was to try to find out in which part of the source code I am working on, causes the crash, like which file or definition cpptols is not able to parse.

I don't find the previous message conetnets informative, so any other tip form your side woudl be very useful.

If the parsing processing and crash originates in cpptools-srv, then I can't really contribute to finding a bug, since it is the executable I took form the installed extension, and it's a kind of black box for me, But I am not sure if I got the extension architecture, my knowlegde is very limited on this code base.

[sean-mcmanus]

Those are just messages related to communication with cpptools. The cpptools process communicates with cpptools-srv (which is crashing). If you can attach a debugger to the cpptools-srv process before it crashes (i.e. comment out the code, attach, and the uncomment it) then you could get a crash call stack: https://github.com/microsoft/vscode-cpptools/wiki/Attaching-debugger-to-cpptools-or-cpptools%E2%80%90srv . It's also possible you could narrow down the code that is triggering the crash via comment out code, i.e. if you remove a particular function and it stops crashing, then the crashing code is being triggered by some code in that function (although it may require other dependencies too).

[dmitttri]

If you can attach a debugger to the cpptools-srv process before it crashes (i.e. comment out the code, attach, and the uncomment it) then you could get a crash call stack: https://github.com/microsoft/vscode-cpptools/wiki/Attaching-debugger-to-cpptools-or-cpptools%E2%80%90srv

I tried this already, according to your advice. Attaching to cpptools-srv leads to the strange situation, which I tried to explain in one of my previous messages. As you suggested, I commented out code in the problematic source file, attach to cpptools-srv process (which has 73 threads), then open the prolematic file and start editing. I get regular "IntelliSense process crash detected", but debugger don't stop on any of the cpptools-srv threds, all of them just continue to run. This is stange and unexpected result.

It's also possible you could narrow down the code that is triggering the crash via comment out code, i.e. if you remove a particular function and it stops crashing, then the crashing code is being triggered by some code in that function (although it may require other dependencies too).

I have tried this too. The thing is, crash happens for many of source files, it's not one or just couple of them. My guess is that it has something to do with common headers, but commenitng includes did not result in IntelliSense becoming operational on a problematic file

[sean-mcmanus]

It's possible you could be hitting the IntelliSense memory limit -- the logging message about a crash may be incorrect and we could just be killing the process due to hitting the memory limit. See my comment at #10636 (comment) .

[dmitttri]

Thanks for the tip. I have tried tried "C_Cpp.intelliSenseMemoryLimit": 16384 but the situation is the same. I have also check for running the cpptools-srv, it uses approx 200MB

Also tried the suggested "C_Cpp.intelliSenseCacheSize": 0, with the same result.

[dmitttri]

In my current situation, extension receives the message "IntelliSense process crash detected" but cpptools-srv remains active (with all it's threads). It may be that I don't understand in which component of cpptools, IntelliSense functionality (Language Server) is implemented. I thought it was cpptools-srv

I will take the path of trying to narrow down the source code section(s) which contribute to this issue, and make it reproducible for you.

[sean-mcmanus]

Yeah, "IntelliSense process crash detected" is supposed to mean "cpptools-srv" is crashed. IntelliSense functionality is implemented in that process. I don't know why the message appears while the process appears to not be crashing. You could try manually killing the cpptools-srv process in a non-crashing scenario and confirm the "IntelliSense process crash detected" message appears.

[dmitttri]

You could try manually killing the cpptools-srv process in a non-crashing scenario and confirm the "IntelliSense process crash detected" message appears.

I killed cpptools-srv process manually, and then I could see in the C/C++ Output the message "IntelliSense process crash detected", and then cpptools-srv is automatically restarted

[dmitttri]

What may happen is that cpptools-srv hangs (and stop sending messages) due to some internal issue, and I get the crash report.

[github-actions]

Hey @sean-mcmanus, this issue might need further attention.

@dmitttri, you can help us out by closing this issue if the problem no longer exists, or adding more information.

[daniel-trnk]

Hi @sean-mcmanus, @dmitttri isn't working on this anymore. Apologies for going silent on you.
The issue persists: when editing a heavily templated C++ file, cpptools-srv reliably crashes.

• We are currently on ms-vscode.cpptools v1.16.3
• We are running a devcontainer session using Rocky 9 as the base image with gcc-toolset-11, but the issue is consistent across all environments we tried)

The steps are

• Open VSCode and a non-problematic file in the workspace.

  • Hovering works, pop-up shows.

• Attach debugger to cpptools and cpptools-srv (All C++ Exceptions checked)

  • And yes, the issue exists without gdb.

• Open file that triggers the problem

  • Debug output:

    
    LSP: textDocument/didOpen: file:///workspaces/app1/core/include/core/tasks/a_session_task.h
    LSP: cpptools/activeDocumentChange: file:///workspaces/app1/core/include/core/tasks/a_session_task.h
    LSP: cpptools/getCodeActions: file:///workspaces/app1/core/include/core/tasks/a_session_task.h (id: 25)
    LSP: cpptools/getDocumentSymbols: file:///workspaces/app1/core/include/core/tasks/a_session_task.h (id: 26)
    LSP: cpptools/getSemanticTokens: file:///workspaces/app1/core/include/core/tasks/a_session_task.h (id: 27)
    LSP: cpptools/getFoldingRanges: file:///workspaces/app1/core/include/core/tasks/a_session_task.h (id: 28)
    LSP: $/cancelRequest (cpptools/getCodeActions, id: 25)
    LSP: cpptools/getCodeActions: file:///workspaces/app1/core/include/core/tasks/a_session_task.h (id: 29)
    Attempting to get defaults from C++ compiler in "compilerPath" property: '/usr/bin/g++'
    Checking for syntax errors: /workspaces/app1/core/include/core/tasks/a_session_task.h
    sending compilation args for /workspaces/app1/core/test/googletest/src/tasks/b_task.cc
      include: /workspaces/app1/core/include
       ...
      define: __UINT_LEAST16_MAX__=0xffff
      define: ...
       ...
      other: --g++
      other: --gnu_version=110301
      stdver: c++17
      intelliSenseMode: linux-gcc-x64
    Queueing IntelliSense update for files in translation unit of: /workspaces/app1/core/test/googletest/src/tasks/b_task.cc
    Shutting down IntelliSense server: /workspaces/app1/core/test/googletest/src/utl/arch.cc
    LSP: cpptools/getInlayHints: file:///workspaces/app1/core/include/core/tasks/a_session_task.h (id: 30)
    LSP: $/cancelRequest (cpptools/getInlayHints, id: 30)
    LSP: cpptools/getInlayHints: file:///workspaces/app1/core/include/core/tasks/a_session_task.h (id: 31)
    LSP: Request canceled by server (<unknown/completed>, code: -32802, id: 30)
    LSP: $/cancelRequest (cpptools/getInlayHints, id: 31)
    LSP: cpptools/getInlayHints: file:///workspaces/app1/core/include/core/tasks/a_session_task.h (id: 32)
    LSP: Request canceled by server (<unknown/completed>, code: -32802, id: 31)
    Update IntelliSense time (sec): 5.529
    

• This triggers the first exception in cpptool-srv:

  • Debug Console:

    Thread 40 "cpptools-srv" hit Catchpoint 2 (exception thrown), 0x00000000009f4d0e in __cxa_throw ()
    

  • Call Stack:

    __cxa_throw (Unknown Source:0)
    cfe_throw() (Unknown Source:0)
    exit_compilation(an_error_severity) (Unknown Source:0)
    a_compiler_thread::fetch_fragment() (Unknown Source:0)
    before_tu_wrapup() (Unknown Source:0)
    process_translation_unit(char const*, int, an_exported_template_file*) (Unknown Source:0)
    cfe_main(int, char**) (Unknown Source:0)
    cfe_main_exception_handler(int, char**) (Unknown Source:0)
    edg_main(int, char**) (Unknown Source:0)
    edge_compiler_main(int, char const**) (Unknown Source:0)
    preparse(int, char const**, a_scout_store*, edge::translation_unit*) (Unknown Source:0)
    a_compiler_thread::compiler_thread_routine(a_compiler_thread*) (Unknown Source:0)
    msvc::thread_helper_t::thread_entry(void*) (Unknown Source:0)
    libc.so.6!start_thread (Unknown Source:0)
    libc.so.6!clone3 (Unknown Source:0)
    

• On continue, cpptool-srv exits

[Colengms]

Hi @daniel-trnk . That stack appears to be an internal 'first chance exception' throw that is caught elsewhere in the code, and doesn't look like a crash. Could you try disabling 'first-chance' exceptions?

When an IntelliSense process crashes, there should something in the C/C++ output log: IntelliSense process crash detected. . (Though, sometimes that may not be shown until the next attempt to communicate with the process, which can be triggered by hovering the cursor over some code in the file). I don't see that in the segment of log you posted. It looks like it executed as expected and indicated, Update IntelliSense time (sec): 5.529 when the update was complete.

Would it be possible to isolate, reduce and share the code that's triggering a crash? That would be the best possible information for us to investigate further.

[daniel-trnk]

Hi @Colengms, thanks for looking into this and sorry for time incomplete information.

When an IntelliSense process crashes, there should something in the C/C++ output log: IntelliSense process crash detected. . (Though, sometimes that may not be shown until the next attempt to communicate with the process, which can be triggered by hovering the cursor over some code in the file). I don't see that in the segment of log you posted. It looks like it executed as expected and indicated, Update IntelliSense time (sec): 5.529 when the update was complete.

Indeed when I hover after the first exception (and exiting of cpptool-server) described above, I see this as the next output:

...
Update IntelliSense time (sec): 5.48
LSP: cpptools/textEditorSelectionChange
LSP: textDocument/hover: file:///workspaces/app1/core/include/core/tasks/a_session_task.h (id: 26)
IntelliSense process crash detected.
IntelliSense engine is not responding. Using the Tag Parser instead.
using Tag Parser for quick info
IntelliSense process crash detected.
LSP: $/cancelRequest (<unknown/completed>, id: 26)
LSP: textDocument/hover: file:///workspaces/app1/core/include/core/tasks/a_session_task.h (id: 27)
Resetting IntelliSense server: /workspaces/app1/core/test/googletest/src/tasks/b_task.cc
sending compilation args for ...

(This is a new run, so the IntelliSense time is slightly different.)

That stack appears to be an internal 'first chance exception' throw that is caught elsewhere in the code, and doesn't look like a crash. Could you try disabling 'first-chance' exceptions?

I am using gdb, and it doesn't have that option. When I continue after the first exception, the cpptools-srv process just exits. I tried the same with gdb from the terminal - same behaviour.

In the cli I do see this after the call stack from my last comment:

[Thread 0x7f99797fa640 (LWP 3323) exited]
...
[Thread 0x7f9979ffb640 (LWP 3321) exited]
[Inferior 1 (process 3283) exited normally]

So maybe I am on the wrong trail.

Another observation: if I keep the benign source file open (which wasn't the default as VSCode automatically closed the unchanged file when I opened my test file), the server does not actually exit but I do see the following debug output:

...
IntelliSense client is currently disconnected
IntelliSense client not available, using Tag Parser for quick info.
...

When I start to edit the file, I get cpptools exceptions. I can post them if you think they are useful.

Maybe this is a timeout issue rather than a crash? The client disconnecting, then the server just exiting if this was the only file open?

Would it be possible to isolate, reduce and share the code that's triggering a crash? That would be the best possible information for us to investigate further.

I'll try but so far I had not much luck. It seems to happen fro complex files...

[github-actions]

Hey @sean-mcmanus, this issue might need further attention.

@dmitttri, you can help us out by closing this issue if the problem no longer exists, or adding more information.

[github-actions]

This issue has been closed because it needs more information and has not had recent activity.