More benchmarks for the node:test module.

[avivkeller]

The benchmark/test_runner folder currently contains benchmarks for it and describe functions. I suggest we expand these benchmarks to cover additional test runner features, including mocks, coverage, and various test modes.

Here are the functions that (IMO) should be benchmarked:

Basic Testing

These tests should run with a custom reporter without any special logic to make the tests as accurate as possible.

• test
  • Create a test
  • Create a test when it's not running due to only
  • Add a subtest
  • Skip a test
    • Via skip: true
    • Via t.skip()
    • Via t.skip(...)
  • TODO tests
    • Via todo: true
    • Via t.todo()
    • Via t.todo(...)

Hooks

• beforeEach
• afterEach
• before
• after

Reporters (#55757)

• dot
• junit
• spec
• tap
• lcov

Mocking

• mock.fn (benchmark: add test_runner/mock-fn #55771)
• mock.timers for each API, and each sub-function
• mock.module

Snapshots

• snapshot.setDefaultSnapshotSerializers(serializers)
• snapshot.setResolveSnapshotPath(fn)
• t.assert.snapshot

Coverage

Use --expose-internals to exclusively test the coverage part

• Basic
• Excluding files
• Including files
• With source maps

[cu8code]

@redyetidev I'd love to work on this! Is there a guide or info page on how to create benchmarks? The issue also feels a bit vague — could we get an explicit list of functions to test?

[avivkeller]

IMHO we should start off simple.

How long does it take for a basic test to execute? What about a skipped test? A failing test? Etc.

For this to be as accurate as possible, we would probably need the benchmark to contain a custom reporter.

We could probably replace the current (limited) benchmark with this better idea I described above

---

From there, we could move on how long it takes specific parts, such as mocks, etc.

[pmarchini]

I was planning to start something related to coverage to enhance performance, and to do so, benchmarking is definitely needed. It would be great to use this issue to discuss and select a way forward(possibly with a division of the features to cover).

Do you have any ideas on how you would structure this? I think it's non-trivial to benchmark the test runner and coverage with precision, and I'm not sure if we already have anything similar in place.

[avivkeller]

My idea for coverage is to use the --expose-internals flag. We can calculate different forms of coverage, and just test the coverage function.

(I'll update the PR with an explicit list of functions/components later today–it's not a quick process)

[avivkeller]

I've added the list. If you think something else should be added, feel free to edit the issue

[cu8code]

@RedYetiDev, I'm not sure if I should open an issue for this, but I was attempting to create a benchmark for mock.module, and it seems like the functionality is broken. When I try to create a simple mock module, it throws an error saying: Cannot find package which it should not because I am trying to create a mock module at the first place!

benchmark
Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'axios' imported from /home/ankan/Documents/git/me/node/benchmark/test_runner/mock-module.js
    at Object.getPackageJSONURL (node:internal/modules/package_json_reader:267:9)
    at packageResolve (node:internal/modules/esm/resolve:768:81)
    at moduleResolve (node:internal/modules/esm/resolve:854:18)
    at defaultResolve (node:internal/modules/esm/resolve:984:11)
    at nextResolve (node:internal/modules/esm/hooks:748:28)
    at resolve (node:internal/test_runner/mock/loader:78:35)
    at nextResolve (node:internal/modules/esm/hooks:748:28)
    at Hooks.resolve (node:internal/modules/esm/hooks:240:30)
    at handleMessage (node:internal/modules/esm/worker:199:24)
    at Immediate.checkForMessages (node:internal/modules/esm/worker:141:28) {
  code: 'ERR_MODULE_NOT_FOUND'
}
end
✔ <anonymous> (42.524333ms)
(node:91728) ExperimentalWarning: Module mocking is an experimental feature and might change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 48.749785

I'm running the file using:

./node --experimental-test-module-mocks benchmark/test_runner/mock-module.js

Here’s the script I’m using:

"use strict";

const { test } = require("node:test");

function main() {
  test(async (t) => {
    console.log("benchmark");

    try {
      // Create a mock module
      t.mock.module('axios', {
        namedExports: {
          get: (url) => url,
        },
      });
    } catch (e) {
      console.error(e);
    }

    console.log("end");
  });
}

main();

I plan to open an issue but wanted to check with you first to confirm if this is a consistent problem or a mistake on my part. Let me know your thoughts.

[avivkeller]

Any issues should be opened separately, and from there, if it's an issue, it'll be evaluated

[cjihrig]

My guess is that we may need additional logic for mocking modules that don't actually exist.

[avivkeller]

Possibly, but that should be tracked seperately from this.

[pmarchini]

Hey @redyetidev:

Regarding coverage, I think it would be great to add some benchmarks for the entire "e2e" process too.
I mean run with coverage: true.
I'm planning to work on improving performance for coverage collection in the future, and I think this type of benchmark could be very useful.
WDYT?

[avivkeller]

SGTM, I'll be happy to help eventually, although probably not until March (sorry! Busy times, among other things)