Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: do not set concurrency on parallelized runs #52177

Merged
merged 2 commits into from
Mar 23, 2024
Merged

test: do not set concurrency on parallelized runs #52177

merged 2 commits into from
Mar 23, 2024

Conversation

aduh95
Copy link
Contributor

@aduh95 aduh95 commented Mar 21, 2024

Our CI already run test files in parallel, having node:test spawns child processes concurrently could lead to oversubscribing the CI machine. This commit sets the concurrency depending on the presence of TEST_PARALLEL in the env, so running the test file individually still spawns child processes concurrently, and running the whole test suite does not oversubscribe the machine.

@aduh95
test: do not set concurrency on parallelized runs
a221126 
Our CI already run test files in parallel, having `node:test` spawns
child processes concurrently could lead to oversubscribing the CI
machine. This commit sets the `concurrency` depending
on the presence of `TEST_PARALLEL` in the env, so running the test
file individually still spawns child processes concurrently, and
running the whole test suite does not oversubscribe the machine.
@nodejs-github-bot nodejs-github-bot added esm Issues and PRs related to the ECMAScript Modules implementation. needs-ci PRs that need a full CI run. test Issues and PRs related to the tests. labels Mar 21, 2024
H4ad
H4ad approved these changes Mar 22, 2024
View reviewed changes
Copy link
Member

@H4ad H4ad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non blocking comment: maybe exporting a function like shouldEnableConcurrency() on common could be better than duplicate the same verification every time.

H4ad
H4ad reviewed Mar 22, 2024
View reviewed changes
test/es-module/test-esm-detect-ambiguous.mjs
describe('--experimental-detect-module', { concurrency: true }, () => {
describe('string input', { concurrency: true }, () => {
describe('--experimental-detect-module', { concurrency: !process.env.TEST_PARALLEL }, () => {
describe('string input', { concurrency: !process.env.TEST_PARALLEL }, () => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since concurrency is inherited from the parent, do we need to enable it twice here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll be honest, I only did a quick search and replace, I didn't look closely at the code. I'm tempted to not bother, even if it's redundant, it doesn't harm.

@aduh95 aduh95 added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. request-ci Add this label to start a Jenkins CI on a PR. labels Mar 22, 2024
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Mar 22, 2024
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/57909/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/57922/

@aduh95 aduh95 added the commit-queue Add this label to land a pull request using GitHub Actions. label Mar 23, 2024
@nodejs-github-bot nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Mar 23, 2024
@nodejs-github-bot nodejs-github-bot merged commit bae14b7 into nodejs:main Mar 23, 2024
58 checks passed
@nodejs-github-bot
Copy link
Collaborator

Landed in bae14b7

@aduh95 aduh95 deleted the TEST_PARALLEL branch March 23, 2024 21:12
anonrig pushed a commit to anonrig/node that referenced this pull request Mar 25, 2024
@aduh95 @anonrig
test: do not set concurrency on parallelized runs
32d4c66 
Our CI already run test files in parallel, having `node:test` spawns
child processes concurrently could lead to oversubscribing the CI
machine. This commit sets the `concurrency` depending
on the presence of `TEST_PARALLEL` in the env, so running the test
file individually still spawns child processes concurrently, and
running the whole test suite does not oversubscribe the machine.

PR-URL: nodejs#52177
Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
rdw-msft pushed a commit to rdw-msft/node that referenced this pull request Mar 26, 2024
@aduh95 @rdw-msft
test: do not set concurrency on parallelized runs
dda083b 
Our CI already run test files in parallel, having `node:test` spawns
child processes concurrently could lead to oversubscribing the CI
machine. This commit sets the `concurrency` depending
on the presence of `TEST_PARALLEL` in the env, so running the test
file individually still spawns child processes concurrently, and
running the whole test suite does not oversubscribe the machine.

PR-URL: nodejs#52177
Reviewed-By: Vinícius Lourenço Claro Cardoso <contact@viniciusl.com.br>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com>
@RafaelGSS RafaelGSS mentioned this pull request Apr 15, 2024
Merged
@marco-ippolito
Copy link
Member

I'm not sure we want to land this on v20.x but if we do requires manual backport

@marco-ippolito marco-ippolito added the backport-requested-v20.x PRs awaiting manual backport to the v20.x-staging branch. label May 21, 2024
codebytere added a commit to electron/electron that referenced this pull request Oct 16, 2024
@codebytere
test: do not set concurrency on parallelized runs
adc2dcd 
nodejs/node#52177
codebytere added a commit to electron/electron that referenced this pull request Oct 21, 2024
@codebytere
test: do not set concurrency on parallelized runs
133768a 
nodejs/node#52177
codebytere added a commit to electron/electron that referenced this pull request Oct 21, 2024
@codebytere
test: do not set concurrency on parallelized runs
4af3f54 
nodejs/node#52177
codebytere added a commit to electron/electron that referenced this pull request Oct 22, 2024
@codebytere
test: do not set concurrency on parallelized runs
19a1e6d 
nodejs/node#52177
codebytere added a commit to electron/electron that referenced this pull request Oct 28, 2024
@codebytere
test: do not set concurrency on parallelized runs
ba6d883 
nodejs/node#52177
codebytere added a commit to electron/electron that referenced this pull request Oct 29, 2024
@codebytere
test: do not set concurrency on parallelized runs
484b1ae 
nodejs/node#52177
codebytere added a commit to electron/electron that referenced this pull request Oct 31, 2024
@codebytere
test: do not set concurrency on parallelized runs
45d569b 
nodejs/node#52177
jkleinsc pushed a commit to electron/electron that referenced this pull request Nov 4, 2024
@codebytere
chore: bump Node.js to v22.9.0 (#44281)
c63d0d6 
* chore: bump Node.js to v22.9.0

* build: drop base64 dep in GN build

nodejs/node#52856

* build,tools: make addons tests work with GN

nodejs/node#50737

* fs: add fast api for InternalModuleStat

nodejs/node#51344

* src: move package_json_reader cache to c++

nodejs/node#50322

* crypto: disable PKCS#1 padding for privateDecrypt

nodejs-private/node-private#525

* src: move more crypto code to ncrypto

nodejs/node#54320

* crypto: ensure valid point on elliptic curve in SubtleCrypto.importKey

nodejs/node#50234

* src: shift more crypto impl details to ncrypto

nodejs/node#54028

* src: switch crypto APIs to use Maybe<void>

nodejs/node#54775

* crypto: remove DEFAULT_ENCODING

nodejs/node#47182

* deps: update libuv to 1.47.0

nodejs/node#50650

* build: fix conflict gyp configs

nodejs/node#53605

* lib,src: drop --experimental-network-imports

nodejs/node#53822

* esm: align sync and async load implementations

nodejs/node#49152

* esm: remove unnecessary toNamespacedPath calls

nodejs/node#53656

* module: detect ESM syntax by trying to recompile as SourceTextModule

nodejs/node#52413

* test: adapt debugger tests to V8 11.4

nodejs/node#49639

* lib: update usage of always on Atomics API

nodejs/node#49639

* test: adapt test-fs-write to V8 internal changes

nodejs/node#49639

* test: adapt to new V8 trusted memory spaces

nodejs/node#50115

* deps: update libuv to 1.47.0

nodejs/node#50650

* src: use non-deprecated v8::Uint8Array::kMaxLength

nodejs/node#50115

* src: update default V8 platform to override functions with location

nodejs/node#51362

* src: add missing TryCatch

nodejs/node#51362

* lib,test: handle new Iterator global

nodejs/node#51362

* src: use non-deprecated version of CreateSyntheticModule

nodejs/node#50115

* src: remove calls to recently deprecated V8 APIs

nodejs/node#52996

* src: use new V8 API to define stream accessor

nodejs/node#53084

* src: do not use deprecated V8 API

nodejs/node#53084

* src: do not use soon-to-be-deprecated V8 API

nodejs/node#53174

* src: migrate to new V8 interceptors API

nodejs/node#52745

* src: use supported API to get stalled TLA messages

nodejs/node#51362

* module: print location of unsettled top-level await in entry points

nodejs/node#51999

* test: make snapshot comparison more flexible

nodejs/node#54375

* test: do not set concurrency on parallelized runs

nodejs/node#52177

* src: move FromNamespacedPath to path.cc

nodejs/node#53540

* test: adapt to new V8 trusted memory spaces

nodejs/node#50115

* build: add option to enable clang-cl on Windows

nodejs/node#52870

* chore: fixup patch indices

* chore: add/remove changed files

* esm: drop support for import assertions

nodejs/node#54890

* build: compile with C++20 support

nodejs/node#52838

* deps: update nghttp2 to 1.62.1

nodejs/node#52966

* src: parse inspector profiles with simdjson

nodejs/node#51783

* build: add GN build files

nodejs/node#47637

* deps,lib,src: add experimental web storage

nodejs/node#52435

* build: add missing BoringSSL dep

* src: rewrite task runner in c++

nodejs/node#52609

* fixup! build: add GN build files

* src: stop using deprecated fields of v8::FastApiCallbackOptions

nodejs/node#54077

* fix: shadow variable

* build: add back incorrectly removed SetAccessor patch

* fixup! fixup! build: add GN build files

* crypto: fix integer comparison in crypto for BoringSSL

* src,lib: reducing C++ calls of esm legacy main resolve

nodejs/node#48325

* src: move more crypto_dh.cc code to ncrypto

nodejs/node#54459

* chore: fixup GN files for previous commit

* src: move more crypto code to ncrypto

nodejs/node#54320

* Fixup Perfetto ifdef guards

* fix: missing electron_natives dep

* fix: node_use_node_platform = false

* fix: include src/node_snapshot_stub.cc in libnode

* 5507047: [import-attributes] Remove support for import assertions

https://chromium-review.googlesource.com/c/v8/v8/+/5507047

* fix: restore v8-sandbox.h in filenames.json

* fix: re-add original-fs generation logic

* fix: ngtcp2 openssl dep

* test: try removing NAPI_VERSION undef

* chore(deps): bump @types/node

* src: move more crypto_dh.cc code to ncrypto

nodejs/node#54459

* esm: remove unnecessary toNamespacedPath calls

nodejs/node#53656

* buffer: fix out of range for toString

nodejs/node#54553

* lib: rewrite AsyncLocalStorage without async_hooks

nodejs/node#48528

* module: print amount of load time of a cjs module

nodejs/node#52213

* test: skip reproducible snapshot test on 32-bit

nodejs/node#53592

* fixup! src: move more crypto_dh.cc code to ncrypto

* test: adjust emittedUntil return type

* chore: remove redundant wpt streams patch

* fixup! chore(deps): bump @types/node

* fix: gn executable name on Windows

* fix: build on Windows

* fix: rename conflicting win32 symbols in //third_party/sqlite

On Windows otherwise we get:

lld-link: error: duplicate symbol: sqlite3_win32_write_debug
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:47987
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_sleep
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48042
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_is_nt
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48113
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_unicode
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48470
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_unicode_to_utf8
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48486
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_mbcs_to_utf8
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48502
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_mbcs_to_utf8_v2
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48518
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_mbcs
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48534
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_mbcs_v2
>>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48550
>>>            obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj
>>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj

* docs: remove unnecessary ts-expect-error after types bump

* src: move package resolver to c++

nodejs/node#50322

* build: set ASAN detect_container_overflow=0

nodejs/node#55584

* chore: fixup rebase

* test: disable failing ASAN test

* win: almost fix race detecting ESRCH in uv_kill

libuv/libuv#4341
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@H4ad H4ad H4ad approved these changes

@lpinca lpinca lpinca approved these changes

@marco-ippolito marco-ippolito marco-ippolito approved these changes

Assignees
No one assigned
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. backport-requested-v20.x PRs awaiting manual backport to the v20.x-staging branch. esm Issues and PRs related to the ECMAScript Modules implementation. needs-ci PRs that need a full CI run. test Issues and PRs related to the tests.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@aduh95 @nodejs-github-bot @marco-ippolito @lpinca @H4ad