Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate flaky test - parallel/test-fs-rmdir-recursive #41201

Closed
mhdawson opened this issue Dec 16, 2021 · 3 comments
Closed

Investigate flaky test - parallel/test-fs-rmdir-recursive #41201

mhdawson opened this issue Dec 16, 2021 · 3 comments
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI. fs Issues and PRs related to the fs subsystem / file system.

Comments

@mhdawson
Copy link
Member

Test

parallel/test-fs-rmdir-recursive

Platform

Windows, arm/linux

Console output

Can't clean tmpdir: C:\workspace\node-test-binary-windows-js-suites\node\test\.tmp.236
Files blocking: [ 'rmdir-recursive-0', 'rmdir-recursive-4', 'rmdir-recursive-5' ]

assert.js:982
    throw newErr;
    ^

AssertionError [ERR_ASSERTION]: ifError got unwanted exception: ENOTEMPTY: directory not empty, rmdir 'C:\workspace\node-test-binary-windows-js-suites\node\test\.tmp.236\rmdir-recursive-0\d-3-2\d-2-2\d-1-2'
    at C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:338:12
    at C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:375:15
    at fs.js:179:23
    at CB (internal/fs/rimraf.js:59:5)
    at internal/fs/rimraf.js:151:18
    at CB (internal/fs/rimraf.js:59:5)
    at internal/fs/rimraf.js:151:18
    at CB (internal/fs/rimraf.js:59:5)
    at internal/fs/rimraf.js:151:18
    at CB (internal/fs/rimraf.js:59:5)
 {
  generatedMessage: false,
  code: 'ERR_ASSERTION',
  actual: [Error: ENOTEMPTY: directory not empty, rmdir 'C:\workspace\node-test-binary-windows-js-suites\node\test\.tmp.236\rmdir-recursive-0\d-3-2\d-2-2\d-1-2'] {
    errno: -4051,
    code: 'ENOTEMPTY',
    syscall: 'rmdir',
    path: 'C:\\workspace\\node-test-binary-windows-js-suites\\node\\test\\.tmp.236\\rmdir-recursive-0\\d-3-2\\d-2-2\\d-1-2'
  },
  expected: null,
  operator: 'ifError'
}

Build links

From latest reliability report

9 (#41083, #41022, #40980, #40899, #36696, #41046, #41146, #41149, #41173)

Additional information

There are also 5 instances of parallel/test-fs-rm failing in the reliability report. Often but not always in the same run as test-fs-rmdir-recursive. Maybe the tests have similar timing issues?

@mhdawson mhdawson added the flaky-test Issues and PRs related to the tests with unstable failures on the CI. label Dec 16, 2021
@Mesteery Mesteery added the fs Issues and PRs related to the fs subsystem / file system. label Dec 16, 2021
mhdawson added a commit to mhdawson/io.js that referenced this issue Jan 14, 2022
Refs: nodejs#41201

From recent reliability reports this is now the most
common failure by far in CI runs. Mark the test as
flaky until the issue is resolved.

Signed-off-by: Michael Dawson <mdawson@devrus.com>
bcoe added a commit to bcoe/node-1 that referenced this issue Jan 15, 2022
Noticed that a few assertions were not being awaited, this could
potentially be leading to flakiness in tmp cleanup.

Refs: nodejs#41201
bcoe added a commit to bcoe/node-1 that referenced this issue Jan 16, 2022
Noticed that a few assertions were not being awaited, this could
potentially be leading to flakiness in tmp cleanup.

Refs: nodejs#41201
mhdawson pushed a commit that referenced this issue Jan 17, 2022
Noticed that a few assertions were not being awaited, this could
potentially be leading to flakiness in tmp cleanup.

Refs: #41201

PR-URL: #41545
Refs: #41201
Reviewed-By: Ian Sutherland <ian@iansutherland.ca>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Mestery <mestery@protonmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: Michael Dawson <midawson@redhat.com>
thedull pushed a commit to thedull/node that referenced this issue Jan 18, 2022
Noticed that a few assertions were not being awaited, this could
potentially be leading to flakiness in tmp cleanup.

Refs: nodejs#41201

PR-URL: nodejs#41545
Refs: nodejs#41201
Reviewed-By: Ian Sutherland <ian@iansutherland.ca>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Mestery <mestery@protonmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: Michael Dawson <midawson@redhat.com>
BethGriggs pushed a commit that referenced this issue Jan 25, 2022
Noticed that a few assertions were not being awaited, this could
potentially be leading to flakiness in tmp cleanup.

Refs: #41201

PR-URL: #41545
Refs: #41201
Reviewed-By: Ian Sutherland <ian@iansutherland.ca>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Mestery <mestery@protonmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: Michael Dawson <midawson@redhat.com>
Linkgoron pushed a commit to Linkgoron/node that referenced this issue Jan 31, 2022
Noticed that a few assertions were not being awaited, this could
potentially be leading to flakiness in tmp cleanup.

Refs: nodejs#41201

PR-URL: nodejs#41545
Refs: nodejs#41201
Reviewed-By: Ian Sutherland <ian@iansutherland.ca>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Mestery <mestery@protonmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: Michael Dawson <midawson@redhat.com>
mhdawson added a commit that referenced this issue Jan 31, 2022
Refs: #41201

From recent reliability reports this is now the most
common failure by far in CI runs. Mark the test as
flaky until the issue is resolved.

Signed-off-by: Michael Dawson <mdawson@devrus.com>

PR-URL: #41533
Reviewed-By: Ben Coe <bencoe@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
ruyadorno pushed a commit that referenced this issue Feb 8, 2022
Refs: #41201

From recent reliability reports this is now the most
common failure by far in CI runs. Mark the test as
flaky until the issue is resolved.

Signed-off-by: Michael Dawson <mdawson@devrus.com>

PR-URL: #41533
Reviewed-By: Ben Coe <bencoe@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
@iansu
Copy link
Contributor

iansu commented Feb 18, 2022

@mhdawson did the fix @bcoe made in #41545 solve this problem?

@bcoe
Copy link
Contributor

bcoe commented Feb 21, 2022

@iansu mentioned in our tooling meeting, I think it might be worth serializing the test-fs-rmdir-recursive cb tests, as it might make it easier to determine where exactly the failure is happening ... I also have a hunch this might make the problem go away, if the underlying problem is two many fs operations happening in parallel on Windows.

danielleadams pushed a commit that referenced this issue Feb 26, 2022
Noticed that a few assertions were not being awaited, this could
potentially be leading to flakiness in tmp cleanup.

Refs: #41201

PR-URL: #41545
Refs: #41201
Reviewed-By: Ian Sutherland <ian@iansutherland.ca>
Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com>
Reviewed-By: Richard Lau <rlau@redhat.com>
Reviewed-By: Mestery <mestery@protonmail.com>
Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Darshan Sen <raisinten@gmail.com>
Reviewed-By: Michael Dawson <midawson@redhat.com>
danielleadams pushed a commit that referenced this issue Mar 2, 2022
Refs: #41201

From recent reliability reports this is now the most
common failure by far in CI runs. Mark the test as
flaky until the issue is resolved.

Signed-off-by: Michael Dawson <mdawson@devrus.com>

PR-URL: #41533
Reviewed-By: Ben Coe <bencoe@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
danielleadams pushed a commit that referenced this issue Mar 3, 2022
Refs: #41201

From recent reliability reports this is now the most
common failure by far in CI runs. Mark the test as
flaky until the issue is resolved.

Signed-off-by: Michael Dawson <mdawson@devrus.com>

PR-URL: #41533
Reviewed-By: Ben Coe <bencoe@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
danielleadams pushed a commit that referenced this issue Mar 14, 2022
Refs: #41201

From recent reliability reports this is now the most
common failure by far in CI runs. Mark the test as
flaky until the issue is resolved.

Signed-off-by: Michael Dawson <mdawson@devrus.com>

PR-URL: #41533
Reviewed-By: Ben Coe <bencoe@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
@marco-ippolito
Copy link
Member

lpinca added a commit to lpinca/node that referenced this issue Jan 25, 2024
nodejs-github-bot pushed a commit that referenced this issue Feb 1, 2024
Refs: #41201 (comment)
PR-URL: #51566
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
rdw-msft pushed a commit to rdw-msft/node that referenced this issue Feb 9, 2024
Refs: nodejs#41201 (comment)
PR-URL: nodejs#51566
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
targos pushed a commit that referenced this issue Feb 15, 2024
Refs: #41201 (comment)
PR-URL: #51566
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
marco-ippolito pushed a commit to marco-ippolito/node that referenced this issue Feb 19, 2024
Refs: nodejs#41201 (comment)
PR-URL: nodejs#51566
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
richardlau pushed a commit that referenced this issue Mar 25, 2024
Refs: #41201 (comment)
PR-URL: #51566
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
richardlau pushed a commit that referenced this issue Mar 25, 2024
Refs: #41201 (comment)
PR-URL: #51566
Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
Reviewed-By: Yagiz Nizipli <yagiz.nizipli@sentry.io>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI. fs Issues and PRs related to the fs subsystem / file system.
Projects
None yet
Development

No branches or pull requests

5 participants