Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race condition leading to hanging process in test-http2-reset-flood.js #34825

Open
trevnorris opened this issue Aug 18, 2020 · 1 comment
Open
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI. http2 Issues or PRs related to the http2 subsystem.

Comments

@trevnorris
Copy link
Contributor

  • Version: v12.18.3
  • Platform: Linux 4.19.127 x86_64
  • Subsystem: test

What steps will reproduce the bug?

Need to run the test a lot. I use this and wait for the process to hang.

for i in $(seq 1 1000)
  do echo -ne "$i\r"
  ./node test/parallel/test-http2-reset-flood.js
  if [[ $? -ne 0 ]]; then
    break
  fi
done

How often does it reproduce? Is there a required condition?

Can usually cause the test to hang within running the test 1000 times.

Additional information

Seems two handles stay alive after the test has finished. Below is the best debugging information I could get from a Release build, since I wasn't able to reproduce with a Debug build.

(lldb) bt
* thread #1, name = 'node', stop reason = signal SIGSTOP
    frame #0: 0x00007efe5a41dd67 libc.so.6`epoll_wait(epfd=13, events=0x00007fffc970aee0, maxevents=1024, timeout=-1) at epoll_wait.c:30
    frame #1: 0x0000000001234d35 node`uv__io_poll(loop=0x0000000002a2a350, timeout=-1) at linux-core.c:309:14
  * frame #2: 0x0000000001225558 node`uv_run(loop=0x0000000002a2a350, mode=UV_RUN_DEFAULT) at core.c:381:5
    frame #3: 0x0000000000a0b4e5 node`node::NodeMainInstance::Run() + 389
    ...

(lldb) p loop->nfds
(unsigned int) $18 = 2

(lldb) expr *((uv_handle_t *) ((char *) (loop->handle_queue[1]) - __builtin_offsetof(uv_handle_t, handle_queue)))
(uv_handle_t) $19 = {
  data = 0x0000000002f50960
  loop = 0x0000000002a2a350
  type = UV_ASYNC
  close_cb = 0x006f00200064006e
  handle_queue = ([0] = 0x0000000002a2a360, [1] = 0x0000000002f507e0)
  u = {
    fd = 4522070
    reserved = ([0] = 0x0020005200450056, [1] = 0x00540020004f0044, [2] = 0x002e005300490048, [3] = 0x0069006800540020)
  }
  next_closing = 0x0000000000000000
  flags = 12
}

(lldb) expr *((uv_handle_t *) ((char *) (0x0000000002f507e0) - __builtin_offsetof(uv_handle_t, handle_queue)))
(uv_handle_t) $20 = {
  data = 0x0000000002f50720
  loop = 0x0000000002a2a350
  type = UV_TTY
  close_cb = 0x0000000000000000
  handle_queue = ([0] = 0x0000000002f509e8, [1] = 0x0000000002f4be48)
  u = {
    fd = 6619236
    reserved = ([0] = 0x0072007200650064, [1] = 0x007300650064002e, [2] = 0x0079006f00720074, [3] = 0x006e006f006f0053)
  }
  next_closing = 0x0000000000000000
  flags = 49160
}

Not even sure if this is worth fixing, but leaving it here for reference. I've had a couple other tests do something similar, but it's harder to reproduce with those.

NOTE: Can't reproduce this in v14.8.0.

@jasnell
Copy link
Member

jasnell commented Aug 18, 2020

This test is notoriously flaky... the reason is that the conditions necessary to trigger the test condition are difficult to reliably reproduce and depend entirely on operating system timing differences. I've been thinking that we should just remove this test from parallel and move it into one of the optionally run buckets. I'm not entirely sure how we can fix it to run reliably.

@lpinca lpinca added the http2 Issues or PRs related to the http2 subsystem. label Aug 19, 2020
@aduh95 aduh95 added the flaky-test Issues and PRs related to the tests with unstable failures on the CI. label Dec 23, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI. http2 Issues or PRs related to the http2 subsystem.
Projects
None yet
Development

No branches or pull requests

4 participants