-
Notifications
You must be signed in to change notification settings - Fork 30.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
investigate flaky test-inspect-async-hook-setup-at-inspect on Windows 2008r2 in CI #26798
Comments
I have seen this multiple times in the last 48h but it's a flake: specifics: |
Ping @nodejs/v8-inspector |
test-rackspace-win2008r2-x64-6 00:17:11 not ok 314 parallel/test-inspect-async-hook-setup-at-inspect
00:17:11 ---
00:17:11 duration_ms: 0.614
00:17:11 severity: fail
00:17:11 exitcode: 1
00:17:11 stack: |-
00:17:11 [test] Connecting to a child Node process
00:17:11 [test] Testing /json/list
00:17:11 [err] Debugger listening on ws://127.0.0.1:57632/cfedc474-b361-4187-80ed-1b68bdbd10cd
00:17:11 [err] For help, see: https://nodejs.org/en/docs/inspector
00:17:11 [err]
00:17:11 [err] Debugger attached.
00:17:11 [err]
00:17:11 [err] FATAL ERROR: node::inspector::Agent::ToggleAsyncHook Cannot toggle Inspector's AsyncHook, please report this.
00:17:11 [err]
00:17:11 [err] 1: 000000013F44A0AF napi_wrap+84447
00:17:11 [err] 2: 000000013F4018C6 uv_loop_fork+44166
00:17:11 [err] 3: 000000013F402416 uv_loop_fork+47062
00:17:11 [err] 4: 000000013F401FF9 uv_loop_fork+46009
00:17:11 [err] 5: 000000013F3692C3 std::basic_ios<char,std::char_traits<char> >::imbue+76067
00:17:11 [err]
00:17:11 [err] 6: 000000013F97F7F7 v8_inspector::protocol::Debugger::API::SearchMatch::fromJSONString+242135
00:17:11 [err] 7: 000000013F95E21C v8_inspector::protocol::Debugger::API::SearchMatch::fromJSONString+105468
00:17:11 [err] 8: 000000013F9478CF v8_inspector::protocol::Debugger::API::SearchMatch::fromJSONString+12975
00:17:11 [err] 9: 000000013F940BCF v8_inspector::protocol::Runtime::API::StackTraceId::fromJSONString+33039
00:17:11 [err] 10: 000000013F9316ED v8::internal::Factory::zero_string+23837
00:17:11 [err] 11: 000000013F735C33 v8_inspector::V8InspectorSession::canDispatchMethod+675
00:17:11 [err] 12: 000000013F369EFA std::basic_ios<char,std::char_traits<char> >::imbue+79194
00:17:11 [err] 13: 000000013F367ED6 std::basic_ios<char,std::char_traits<char> >::imbue+70966
00:17:11 [err] 14: 000000013F357F3E std::basic_ios<char,std::char_traits<char> >::imbue+5534
00:17:11 [err] 15: 000000013F356F9E std::basic_ios<char,std::char_traits<char> >::imbue+1534
00:17:11 [err] 16: 000000013F357ABB std::basic_ios<char,std::char_traits<char> >::imbue+4379
00:17:11 [err] 17: 000000013F3580C7 std::basic_ios<char,std::char_traits<char> >::imbue+5927
00:17:11 [err] 18: 000000013F805035 v8::internal::Isolate::InvalidatePromiseHookProtector+1029
00:17:11 [err]
00:17:11 [err] 19: 000000013F7E1DEE v8::internal::StackGuard::HandleInterrupts+398
00:17:11 [err] 20: 000000013FAD4985 v8::internal::OptimizingCompileDispatcher::Unblock+53381
00:17:11 [err] 21: 000000013FF6DF1F v8::internal::interpreter::BytecodeRegisterOptimizer::RegisterTransfer+1271791
00:17:11 [err]
00:17:11 { Error: read ECONNRESET
00:17:11 at TCP.onStreamRead (internal/stream_base_commons.js:182:27) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
00:17:11 1
00:17:11 ... |
Running 100 x
/CC @nodejs/testing @nodejs/n-api |
Bisecting got me to the range of commits that landed on 2019-03-18 I'm off, if anyone want to continue. |
pinning this since it has ~50% probability of failing for a CI job. |
I cannot seem to identify the first CI run this appeared (some historical jobs are cleaned). It might be related to #26602 |
We could maybe try using |
Trying to show more information in #26859 |
a91d36f parent commit before #26602 landed (https://ci.nodejs.org/job/node-stress-single-test/2179/) still repros |
diff --git a/lib/internal/bootstrap/node.js b/lib/internal/bootstrap/node.js
index f14000f626..516c0e48cd 100644
--- a/lib/internal/bootstrap/node.js
+++ b/lib/internal/bootstrap/node.js
@@ -150,6 +150,9 @@ if (isMainThread) {
setupProcessStdio(getStdout, getStdin, getStderr);
}
+require('internal/async_hooks');
+// XXX(joyeecheung): this has to be done after the initial load of
+// internal/async_hooks?
if (config.hasInspector) {
const {
enable, This fixes the test though I have no idea why |
I opened a fix in #26866 though it's a bit magical.. |
test-rackspace-win2008r2-x64-5 11:52:06 not ok 314 parallel/test-inspect-async-hook-setup-at-inspect
11:52:06 ---
11:52:06 duration_ms: 0.615
11:52:06 severity: fail
11:52:06 exitcode: 1
11:52:06 stack: |-
11:52:06 [test] Connecting to a child Node process
11:52:06 [test] Testing /json/list
11:52:06 [err] Debugger listening on ws://127.0.0.1:49981/48ed0c36-b59f-4d0c-8606-23323334a3b5
11:52:06 [err] For help, see: https://nodejs.org/en/docs/inspector
11:52:06 [err]
11:52:06 [err] Debugger attached.
11:52:06 [err]
11:52:06 [err] FATAL ERROR: node::inspector::Agent::ToggleAsyncHook Cannot toggle Inspector's AsyncHook, please report this.
11:52:06 [err]
11:52:06 [err] 1: 000000013F8C993F napi_wrap+82655
11:52:06 [err]
11:52:06 [err] 2: 000000013F881876 uv_loop_fork+44166
11:52:06 [err] 3: 000000013F8823C6 uv_loop_fork+47062
11:52:06 [err] 4: 000000013F881FA9 uv_loop_fork+46009
11:52:06 [err] 5: 000000013F7E92C3 std::basic_ios<char,std::char_traits<char> >::imbue+76067
11:52:06 [err] 6: 000000013FDFF447 v8_inspector::protocol::Debugger::API::SearchMatch::fromJSONString+242135
11:52:06 [err] 7: 000000013FDDDE6C v8_inspector::protocol::Debugger::API::SearchMatch::fromJSONString+105468
11:52:06 [err] 8: 000000013FDC751F v8_inspector::protocol::Debugger::API::SearchMatch::fromJSONString+12975
11:52:06 [err] 9: 000000013FDC081F v8_inspector::protocol::Runtime::API::StackTraceId::fromJSONString+33039
11:52:06 [err] 10: 000000013FDB133D v8::internal::Factory::zero_string+23837
11:52:06 [err] 11: 000000013FBB5663 v8_inspector::V8InspectorSession::canDispatchMethod+675
11:52:06 [err] 12: 000000013F7E9EFA std::basic_ios<char,std::char_traits<char> >::imbue+79194
11:52:06 [err] 13: 000000013F7E7ED6 std::basic_ios<char,std::char_traits<char> >::imbue+70966
11:52:06 [err] 14: 000000013F7D7F3E std::basic_ios<char,std::char_traits<char> >::imbue+5534
11:52:06 [err] 15: 000000013F7D6F9E std::basic_ios<char,std::char_traits<char> >::imbue+1534
11:52:06 [err] 16: 000000013F7D7ABB std::basic_ios<char,std::char_traits<char> >::imbue+4379
11:52:06 [err] 17: 000000013F7D80C7 std::basic_ios<char,std::char_traits<char> >::imbue+5927
11:52:06 [err] 18: 000000013FC84A65 v8::internal::Isolate::InvalidatePromiseHookProtector+1029
11:52:06 [err] 19: 000000013FC6181E v8::internal::StackGuard::HandleInterrupts+398
11:52:06 [err] 20: 000000013FF545D5 v8::internal::OptimizingCompileDispatcher::Unblock+53381
11:52:06 [err] 21: 00000001403EDB6F v8::internal::interpreter::BytecodeRegisterOptimizer::RegisterTransfer+1271791
11:52:06 [err]
11:52:06 { Error: read ECONNRESET
11:52:06 at TCP.onStreamRead (internal/stream_base_commons.js:182:27) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
11:52:06 1
11:52:06 ... |
test-rackspace-win2008r2-x64-3 00:18:42 not ok 314 parallel/test-inspect-async-hook-setup-at-inspect
00:18:42 ---
00:18:42 duration_ms: 0.516
00:18:42 severity: fail
00:18:42 exitcode: 1
00:18:42 stack: |-
00:18:42 [test] Connecting to a child Node process
00:18:42 [test] Testing /json/list
00:18:42 [err] Debugger listening on ws://127.0.0.1:52798/c0a28fb7-706f-4d6f-b310-39e2eac816c6
00:18:42 [err] For help, see: https://nodejs.org/en/docs/inspector
00:18:42 [err]
00:18:42 [err] Debugger attached.
00:18:42 [err]
00:18:42 [err] FATAL ERROR: node::inspector::Agent::ToggleAsyncHook Cannot toggle Inspector's AsyncHook, please report this.
00:18:42 [err]
00:18:42 [err] 1: 000000013F67976F napi_wrap+82655
00:18:42 [err]
00:18:42 [err] 2: 000000013F631856 uv_loop_fork+44166
00:18:42 [err] 3: 000000013F6323A6 uv_loop_fork+47062
00:18:42 [err] 4: 000000013F631F89 uv_loop_fork+46009
00:18:42 [err] 5: 000000013F5992A3 std::basic_ios<char,std::char_traits<char> >::imbue+76051
00:18:42 [err] 6: 000000013FBAF277 v8_inspector::protocol::Debugger::API::SearchMatch::fromJSONString+242135
00:18:42 [err] 7: 000000013FB8DC9C v8_inspector::protocol::Debugger::API::SearchMatch::fromJSONString+105468
00:18:42 [err] 8: 000000013FB7734F v8_inspector::protocol::Debugger::API::SearchMatch::fromJSONString+12975
00:18:42 [err] 9: 000000013FB7064F v8_inspector::protocol::Runtime::API::StackTraceId::fromJSONString+33039
00:18:42 [err] 10: 000000013FB6116D v8::internal::Factory::zero_string+23837
00:18:42 [err] 11: 000000013F965493 v8_inspector::V8InspectorSession::canDispatchMethod+675
00:18:42 [err] 12: 000000013F599EDA std::basic_ios<char,std::char_traits<char> >::imbue+79178
00:18:42 [err] 13: 000000013F597EB6 std::basic_ios<char,std::char_traits<char> >::imbue+70950
00:18:42 [err] 14: 000000013F587F2E std::basic_ios<char,std::char_traits<char> >::imbue+5534
00:18:42 [err]
00:18:42 [err] 15: 000000013F586F8E std::basic_ios<char,std::char_traits<char> >::imbue+1534
00:18:42 [err] 16: 000000013F587AAB std::basic_ios<char,std::char_traits<char> >::imbue+4379
00:18:42 [err] 17: 000000013F5880B7 std::basic_ios<char,std::char_traits<char> >::imbue+5927
00:18:42 [err] 18: 000000013FA34895 v8::internal::Isolate::InvalidatePromiseHookProtector+1029
00:18:42 [err] 19: 000000013FA1164E v8::internal::StackGuard::HandleInterrupts+398
00:18:42 [err] 20: 000000013FD04405 v8::internal::OptimizingCompileDispatcher::Unblock+53381
00:18:42 [err] 21: 000000014019D99F v8::internal::interpreter::BytecodeRegisterOptimizer::RegisterTransfer+1271791
00:18:42 [err]
00:18:42 { Error: read ECONNRESET
00:18:42 at TCP.onStreamRead (internal/stream_base_commons.js:182:27) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
00:18:42 1
00:18:42 ... |
Otherwise the exports of `internal/async_hooks` may be undefined when the inspector async hooks are registered. PR-URL: nodejs#26866 Fixes: nodejs#26798 Refs: nodejs#26859 Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com>
If Debugger.setAsyncCallStackDepth is sent during bootstrap, we cannot immediately call into JS to enable the hooks, which could interrupt the JS execution of bootstrap. So instead we save the notification in the inspector agent if it's sent in the middle of bootstrap, and process the notification later here. Refs: nodejs#26798
FWIW I think I found the real cause, see #26935 |
Otherwise the exports of `internal/async_hooks` may be undefined when the inspector async hooks are registered. PR-URL: nodejs#26866 Fixes: nodejs#26798 Refs: nodejs#26859 Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Refael Ackermann <refack@gmail.com>
If Debugger.setAsyncCallStackDepth is sent during bootstrap, we cannot immediately call into JS to enable the hooks, which could interrupt the JS execution of bootstrap. So instead we save the notification in the inspector agent if it's sent in the middle of bootstrap, and process the notification later here. Refs: #26798 PR-URL: #26935 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com> Reviewed-By: Minwoo Jung <minwoo@nodesource.com>
If Debugger.setAsyncCallStackDepth is sent during bootstrap, we cannot immediately call into JS to enable the hooks, which could interrupt the JS execution of bootstrap. So instead we save the notification in the inspector agent if it's sent in the middle of bootstrap, and process the notification later here. Refs: #26798 PR-URL: #26935 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com> Reviewed-By: Minwoo Jung <minwoo@nodesource.com>
If Debugger.setAsyncCallStackDepth is sent during bootstrap, we cannot immediately call into JS to enable the hooks, which could interrupt the JS execution of bootstrap. So instead we save the notification in the inspector agent if it's sent in the middle of bootstrap, and process the notification later here. Refs: #26798 PR-URL: #26935 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Eugene Ostroukhov <eostroukhov@google.com> Reviewed-By: Minwoo Jung <minwoo@nodesource.com> Signed-off-by: Beth Griggs <Bethany.Griggs@uk.ibm.com>
Not sure if this is 100% related but I kept getting this on version: 11.13.0.
|
Getting this non-stop in win2008r2 VS 2017 in CI.
https://ci.nodejs.org/job/node-test-binary-windows/24664/COMPILED_BY=vs2017,RUNNER=win2008r2-vs2017,RUN_SUBSET=3/console
test-rackspace-win2008r2-x64-3
The text was updated successfully, but these errors were encountered: