Enable a way to limit the max thread pool size (100% cpu in cpptools-srv for days followed by a crash)

[arieleiz]

Environment

• OS and Version: Windows 11 22H2
• VS Code Version: 1.77.3
• C/C++ Extension Version: ms-vscode.cpptools-1.14.5-linux-x64
• If using SSH remote, specify OS of remote machine: Ubuntu 22.04.2 5.19.0-1022-aws

cpp-tools is at 100% cpu for days (reproducable).
Flamegraph attached, it is in strncasecmp_l_sse2 blocking on a futex spin lock.
after a few days it always crashes with this stack:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000814226 in proc_ifdef(int) ()
[Current thread is 1 (Thread 0x7f7f707f8640 (LWP 389364))]
(gdb) bt
#0  0x0000000000814226 in proc_ifdef(int) ()
#1  0x0000000000812f8c in pp_directive() ()
#2  0x000000000078f385 in get_token() ()
#3  0x00000000004b9b29 in compile_fragment((anonymous namespace)::a_fragment_buffer const&, a_canonicalized_file_name, msvc::strong_typedef_t<msvc::offset_traits>, msvc::strong_typedef_t<msvc::offset_traits>, a_declaration_context const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool, a_region_scope_info const&, msvc::strong_typedef_t<msvc::offset_traits>, bool, bool*, bool, bool, bool) ()
#4  0x00000000004b6ec9 in before_tu_wrapup() ()
#5  0x00000000008927c9 in process_translation_unit(char const*, int, an_exported_template_file*) ()
#6  0x00000000005298b2 in cfe_main(int, char**) ()
#7  0x00000000008d11d6 in cfe_main_exception_handler(int, char**) ()
#8  0x00000000005299a6 in edg_main(int, char**) ()
#9  0x00000000008ca6f9 in edge_compiler_main(int, char const**) ()
#10 0x00000000004b7696 in preparse(int, char const**, a_scout_store*, edge::translation_unit*) ()
#11 0x000000000047c71f in a_compiler_thread::compiler_thread_routine(a_compiler_thread*) ()
#12 0x00000000009a647f in msvc::thread_helper_t::thread_entry(void*) ()
#13 0x00007f7f7b494b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#14 0x00007f7f7b526a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

We can provide a coredump.

Bug Summary and Steps to Reproduce

Reproduces 100% on our development server using Remote SSH.
We cannot reproduce on small VMs.
Please note this server has 448 cores.
We are willing to run experiments to debug.

Configuration and Logs

-------- Diagnostics - 4/24/2023, 11:51:50 PM
Version: 1.14.5
Current Configuration:
{
    "name": "Linux",
    "includePath": [
        "/<redacted>/**"
    ],
    "defines": [],
    "cStandard": "c17",
    "cppStandard": "c++14",
    "intelliSenseMode": "linux-clang-x64",
    "intelliSenseModeIsExplicit": false,
    "cStandardIsExplicit": false,
    "cppStandardIsExplicit": false,
    "mergeConfigurations": false,
    "compilerPath": "/usr/bin/clang-15",
    "compilerPathIsExplicit": false,
    "browse": {
        "limitSymbolsToIncludedHeaders": true,
        "path": [
            "${workspaceFolder}/**"
        ]
    }
}
Translation Unit Mappings:
[ redacted ]:
    redacted *
Translation Unit Configurations:
[ redacted ]:
    Process ID: 2389042
    Memory Usage: 142 MB
    Compiler Path: /usr/bin/clang-15
    Includes:
        /usr/include/c++/12
        /usr/include/x86_64-linux-gnu/c++/12
        /usr/include/c++/12/backward
        /usr/lib/llvm-15/lib/clang/15.0.7/include
        /usr/local/include
        /usr/include/x86_64-linux-gnu
        /usr/include
    Standard Version: c++14
    IntelliSense Mode: linux-clang-x64
    Other Flags:
        --clang
        --clang_version=150007
Total Memory Usage: 142 MB

------- Workspace parsing diagnostics -------
Number of files discovered (not excluded): 29178

Other Extensions

No response

Additional context

No response

[sean-mcmanus]

Is this with cpptools or cpptools-srv. You description says cpptools, but the call stack indicates cpptools-srv. While it's using the 100% CPU, can you break into the debugger and provide a call stack of the thread that appears to be doing the work?

[arieleiz]

Thank you for your response. We have continued debugging the issue and found out it stems from cpptools creating too many threads, in our case 448 (the number of cores).
Debugging through cpptools we see that thread_pool::thread_pool() initializes the thread count to be std::thread::hardware_concurrency(). This number is not controlled by anything (e.g. "C_Cpp.maxConcurrentThreads" which is on the client side and not the server side).
Each thread takes about 5% cpu waking up periodically times 448 cores brings us to a constant 2 CPU load.
This means that if we have many concurrent users working on the server, each one will have 448 threads allocated per project and 2 CPUs loaded per project.

Is it possible to fix this issue in thread_pool by capping the thread count? for example when an environment variable is set in the user's .bashrc?

Thanks

T

[sean-mcmanus]

Yeah, the thread pool uses hardware_concurrency.

[michelleangela]

This is resolved in https://github.com/microsoft/vscode-cpptools/releases/tag/1.16.0.

[michelleangela]

Fix is now available in release version 1.16.2.