Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SIGNAL USERS READ THIS FIRST: code: 'ERR_INTERNAL_ASSERTION' in internalConnectMultiple #47644

Closed
yuki12321 opened this issue Apr 20, 2023 · 54 comments · Fixed by #48464
Closed
Labels
confirmed-bug Issues with confirmed bugs. net Issues and PRs related to the net subsystem.

Comments

@yuki12321
Copy link

Version

20.0.0

Platform

Darwin MacBookPro.local 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:42:11 PST 2023; root:xnu-8792.81.3~2/RELEASE_X86_64 x86_64

Subsystem

No response

What steps will reproduce the bug?

I ran the Storybook installation on Next.js 13.3.0 and this is what I see.

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior? Why is that the expected behavior?

No response

What do you see instead?

$ npx sb init

 storybook init - the simplest way to add a Storybook to your project. 

 • Detecting project type. ✓
 • Adding Storybook support to your "Next" app
node:internal/assert:14
    throw new ERR_INTERNAL_ASSERTION(message);
    ^

Error [ERR_INTERNAL_ASSERTION]: This is caused by either a bug in Node.js or incorrect usage of Node.js internals.
Please open an issue with this stack trace at https://github.com/nodejs/node/issues

    at new NodeError (node:internal/errors:399:5)
    at assert (node:internal/assert:14:11)
    at internalConnectMultiple (node:net:1106:3)
    at Timeout.internalConnectMultipleTimeout (node:net:1637:3)
    at listOnTimeout (node:internal/timers:575:11)
    at process.processTimers (node:internal/timers:514:7) {
  code: 'ERR_INTERNAL_ASSERTION'
}

Node.js v20.0.0

Additional information

No response

@tniessen
Copy link
Member

Based on the stack trace, it looks similar to #46669 and #46670, so possibly related to #44731, #46587. cc @ShogunPanda

@tniessen tniessen added the net Issues and PRs related to the net subsystem. label Apr 24, 2023
@ShogunPanda
Copy link
Contributor

@tniessen It seems so. I'll take a look soon

@kamagatos
Copy link
Contributor

I'm also seeing a similar issue with ssh2 after upgrading to Node 20.

Repro steps:

  1. Connect to an invalid host
  2. Add an error event handler
client.connect({
  host: 'yahoo.com',
  port: 22,
  username: 'bob',
  password: 'secret'
})

// Adding this line crashes the process.
client.on('error', () => {})

@asp3
Copy link

asp3 commented Apr 29, 2023

seeing this issue as well on production

Error [ERR_INTERNAL_ASSERTION]: This is caused by either a bug in Node.js or incorrect usage of Node.js internals.

  | 2023-04-29T13:32:31.670-04:00 | Please open an issue with this stack trace at https://github.com/nodejs/node/issues
  | 2023-04-29T13:32:31.670-04:00 | at new NodeError (node:internal/errors:399:5)
  | 2023-04-29T13:32:31.670-04:00 | at assert (node:internal/assert:14:11)
  | 2023-04-29T13:32:31.670-04:00 | at internalConnectMultiple (node:net:1106:3)
  | 2023-04-29T13:32:31.670-04:00 | at Timeout.internalConnectMultipleTimeout (node:net:1637:3)
  | 2023-04-29T13:32:31.670-04:00 | at listOnTimeout (node:internal/timers:575:11)
  | 2023-04-29T13:32:31.670-04:00 | at process.processTimers (node:internal/timers:514:7)

Our old build from 10 days ago still works, but all new builds seem to run into this issue.

@MikeRalphson
Copy link
Contributor

Just a note, subscribing as still present in v20.1.0:

In a net and async/await heavy program:

node:internal/assert:14
    throw new ERR_INTERNAL_ASSERTION(message);
    ^

Error [ERR_INTERNAL_ASSERTION]: This is caused by either a bug in Node.js or incorrect usage of Node.js internals.
Please open an issue with this stack trace at https://github.com/nodejs/node/issues

    at new NodeError (node:internal/errors:399:5)
    at assert (node:internal/assert:14:11)
    at internalConnectMultiple (node:net:1107:3)
    at Timeout.internalConnectMultipleTimeout (node:net:1638:3)
    at listOnTimeout (node:internal/timers:575:11)
    at process.processTimers (node:internal/timers:514:7) {
  code: 'ERR_INTERNAL_ASSERTION'
}

Node.js v20.1.0

@ShogunPanda
Copy link
Contributor

ShogunPanda commented May 9, 2023

This might be fixed by #47860.
Will keep you posted on this.

@ShogunPanda
Copy link
Contributor

@kamagatos @yuki12321 The PR above has landed in master. If you can compile Node locally, do you mind checking it if solves your issues as well?

@MikeRalphson
Copy link
Contributor

I now see the following on master:

/Users/mikeralphson/c/node/node[73535]: ../../src/crypto/crypto_tls.cc:1233:static void node::crypto::TLSWrap::GetServername(const FunctionCallbackInfo<v8::Value> &): Assertion `(wrap->ssl_) != nullptr' failed.
 1: 0x100bc1cf0 node::Abort() [/Users/mikeralphson/c/node/out/Release/node]
 2: 0x100bc1a30 node::PrintCaughtException(v8::Isolate*, v8::Local<v8::Context>, v8::TryCatch const&) [/Users/mikeralphson/c/node/out/Release/node]
 3: 0x100d14700 node::crypto::TLSWrap::GetServername(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/mikeralphson/c/node/out/Release/node]
 4: 0x100dca0d4 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, unsigned long*, int) [/Users/mikeralphson/c/node/out/Release/node]
 5: 0x100dc9928 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/Users/mikeralphson/c/node/out/Release/node]
 6: 0x10164cb24 Builtins_CEntry_Return1_ArgvOnStack_BuiltinExit [/Users/mikeralphson/c/node/out/Release/node]
 7: 0x1015c43e4 Builtins_InterpreterEntryTrampoline [/Users/mikeralphson/c/node/out/Release/node]
 8: 0x10664d2cc
 9: 0x1015c250c Builtins_JSEntryTrampoline [/Users/mikeralphson/c/node/out/Release/node]
10: 0x1015c21f4 Builtins_JSEntry [/Users/mikeralphson/c/node/out/Release/node]
11: 0x100eb082c v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/mikeralphson/c/node/out/Release/node]
12: 0x100eb00a8 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/Users/mikeralphson/c/node/out/Release/node]
13: 0x100d77e80 v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/Users/mikeralphson/c/node/out/Release/node]
14: 0x100afdf84 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/Users/mikeralphson/c/node/out/Release/node]
15: 0x100b120b8 node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/Users/mikeralphson/c/node/out/Release/node]
16: 0x100b292dc node::ConnectionWrap<node::TCPWrap, uv_tcp_s>::AfterConnect(uv_connect_s*, int) [/Users/mikeralphson/c/node/out/Release/node]
17: 0x100c917b0 node::MakeLibuvRequestCallback<uv_connect_s, void (*)(uv_connect_s*, int)>::Wrapper(uv_connect_s*, int) [/Users/mikeralphson/c/node/out/Release/node]
18: 0x1015acf94 uv__stream_io [/Users/mikeralphson/c/node/out/Release/node]
19: 0x1015b5650 uv__io_poll [/Users/mikeralphson/c/node/out/Release/node]
20: 0x1015a2e94 uv_run [/Users/mikeralphson/c/node/out/Release/node]
21: 0x100afe7d0 node::SpinEventLoopInternal(node::Environment*) [/Users/mikeralphson/c/node/out/Release/node]
22: 0x100c03504 node::NodeMainInstance::Run() [/Users/mikeralphson/c/node/out/Release/node]
23: 0x100b899d0 node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResultImpl const*) [/Users/mikeralphson/c/node/out/Release/node]
24: 0x100b89c24 node::Start(int, char**) [/Users/mikeralphson/c/node/out/Release/node]
25: 0x1a2ad3f28 start [/usr/lib/dyld]
zsh: abort      

@ShogunPanda
Copy link
Contributor

@MikeRalphson Can you please provide a repro file, along with the dig query of all the hosts you are trying to reach?

@tniessen
Copy link
Member

The same internal assertion is causing CI to fail on Fedora machines, see #48000. However, it is not the ERR_INTERNAL_ASSERTION that this issue was about originally.

Side note: the main branch is called main. We intentionally abandoned the previous branch name.

@bradleybeighton
Copy link

This randomly started happening on my production build after a deployment. I'm using lts-alpine image in my docker file, has there been a recent change that could be causing this?

@ShogunPanda
Copy link
Contributor

@brandon-beacher It's the network family auto selection which was enabled by default in 20.0.0.
I fixed this issue (already merged in main) and once I'll fix some other problems it should go out on 20.2.0 or 20.3.0.

@tniessen
Copy link
Member

This is still an issue in Node.js 20.2.0. As a workaround, you can try restoring the more predictable pre-20 behavior by setting this environment variable:

NODE_OPTIONS="--no-network-family-autoselection"

@arllop
Copy link

arllop commented Jun 3, 2023

this happened when i open signal app on mac laptop. has this been fixed? thanks.

@tniessen
Copy link
Member

tniessen commented Jun 3, 2023

@arllop If Signal does not override NODE_OPTIONS, the workaround in #47644 (comment) might work.

@tniessen tniessen added the confirmed-bug Issues with confirmed bugs. label Jun 3, 2023
@dmcr
Copy link

dmcr commented Aug 4, 2023

Have been redirected here as per my closed duplicated ticket.

So as I understand it this is still an active issue in v20.4 and is related to the network area with the current solutions being to:
Apply NODE_OPTIONS="--no-network-family-autoselection" or to use node ^v19.

I have been redirected here but this is also closed so presumably the best place to follow as to when ^v20 is usable is now #48763 #request/request#3458

I can confirm that going back to v19 resolves this issue in production for me.

@ShogunPanda
Copy link
Contributor

@dmcr can you please provide a repro repo or at least which host are you trying to connect to and how your DNS resolves such host?

brimonk added a commit to brimonk/rollycubes that referenced this issue Aug 6, 2023
cgsdev0 pushed a commit to cgsdev0/rollycubes that referenced this issue Aug 6, 2023
Ceres6 pushed a commit to Ceres6/node that referenced this issue Aug 14, 2023
PR-URL: nodejs#48464
Fixes: npm/cli#6409
Fixes: KararTY/dank-twitch-irc#13
Fixes: nodejs#47644
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Ceres6 pushed a commit to Ceres6/node that referenced this issue Aug 14, 2023
PR-URL: nodejs#48464
Fixes: npm/cli#6409
Fixes: KararTY/dank-twitch-irc#13
Fixes: nodejs#47644
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
@Rand0mF
Copy link

Rand0mF commented Aug 30, 2023

Issue still occurring for node 20.5.1 with the same stacktrace.
Randomly starts occurring and keeps happening for multiple hours after which it randomly stopped.
Node is running as a kubernetes service and not using workers or websockets (like #48763).
Using the docker image node:20.5.1

@ShogunPanda
Copy link
Contributor

@Rand0mF In order to address this, I need to know which host your node app is connecting to and how your local (or Kubernetes system) is resolving it. Can you provide me such info?

@Rand0mF
Copy link

Rand0mF commented Sep 13, 2023

@ShogunPanda Unfortunately I don't know.. The app is doing many queries to all kinds of hosts. The error logs don't provide any useful information for that, that's the only thing which is printed. Reverse engineering is difficult as it seems to occur completely at random, I didn't yet find a way to reproduce it.

Error [ERR_INTERNAL_ASSERTION]: This is caused by either a bug in Node.js or incorrect usage of Node.js internals. Please open an issue with this stack trace at https://github.com/nodejs/node/issues
    at new NodeError (node:internal/errors:405:5)
    at assert (node:internal/assert:14:11)
    at internalConnectMultiple (node:net:1118:3)
    at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
    at listOnTimeout (node:internal/timers:575:11)
    at process.processTimers (node:internal/timers:514:7)

@ainsleyclark
Copy link

Very sporadic error which seems to be difficult to debug locally as @Rand0mF mentioned.
export NODE_OPTIONS=--no-network-family-autoselection works.
All v20.x versions are producing the same error.

@ShogunPanda
Copy link
Contributor

@ainsleyclark Is the same for you? Can you provide a list of which hosts the production system is connecting to and how these are resolved by the production system DNS servers?

@pimterry
Copy link
Member

I'm seeing lots of these errors too, on Node v20.8.0. Similar to other reports, it seems to be very intermittent and only happening under load. I can't confirm the addresses involved either unfortunately (software is running on end user machines, I just see exception reports).

@Rand0mF
Copy link

Rand0mF commented Nov 16, 2023

@ShogunPanda
list of hosts will still be difficult, here's the dig output from within the container for

cluster-external hosts
dig google.com

; <<>> DiG 9.18.19 <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7886
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com.			IN	A

;; ANSWER SECTION:
google.com.		30	IN	A	142.250.185.142

;; Query time: 2 msec
;; SERVER: 10.251.0.10#53(10.251.0.10) (UDP)
;; WHEN: Thu Nov 16 14:03:26 UTC 2023
;; MSG SIZE  rcvd: 55

cluster-internal hosts
dig hello-world.default.svc.cluster.local

; <<>> DiG 9.18.19 <<>> hello-world.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13182
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;hello-world.default.svc.cluster.local. IN A

;; ANSWER SECTION:
hello-world.default.svc.cluster.local. 30 IN A	10.251.3.203

;; Query time: 8 msec
;; SERVER: 10.251.0.10#53(10.251.0.10) (UDP)
;; WHEN: Thu Nov 16 14:08:01 UTC 2023
;; MSG SIZE  rcvd: 71

any other flags which would be helpful?

@ShogunPanda
Copy link
Contributor

Nope, that should be it. I will keep you posted.

@stefanrows
Copy link

Still experiencing the issue with Node 20^. Switching to Node 18 solves the problem for me.

@ShogunPanda
Copy link
Contributor

This should be solved by #51045. Will keep you as soon as this reaches 20.x.

@guillenotfound
Copy link

Any updates on when this is going to be fixed for 20.x @ShogunPanda ?

@brian6932
Copy link

It has been for a long time @guillenotfound https://nodejs.org/en/blog/release/v20.12.0#new-connection-attempt-events.
Latest node release's v21.7.1...

@ShogunPanda
Copy link
Contributor

@guillenotfound It has been release in 20.12.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
confirmed-bug Issues with confirmed bugs. net Issues and PRs related to the net subsystem.
Projects
None yet
Development

Successfully merging a pull request may close this issue.