chore: implement additional test cases for the node test reporter (part 2)

[galargh]

• Because this PR includes a bug fix, relevant tests have been included.
• Because this PR includes a new feature, the change was previously discussed on an Issue or with someone from the team.
• I didn't do anything of this.

---

This is a follow-up to #5669 and it is related to #5437

This PR covers adding more test cases that we chatted about. I'll be proposing changes to the reporter's output in a separate PR(s).

Newly added test cases:

• an error thrown at the top level of a test file
• a describe.only and it.only executed without the --test-only flag
• a test.only, describe.only and it.only executed with the --test-only flag
• nested its
• an error with nested causes
• a test.todo without a callback

The IT frameworks improvements that support the newly added test cases:

• support for custom RunOptions per fixture test - This was needed to be able to test test.only with the --test-only flag enabled. If one puts a options.json file inside the fixture test, it is going to be parsed by both the IT test runner, as well as our script that regenerates the fixtures.
• saving the actual output produced by the test when ran by the IT test runner - I found it useful to have the output saved to a file on a disk. I save it as result.actual.txt inside the fixture test and I gitignore these files because they do not need to be checked in.
• accept a fixture name as an input to regenerate fixtures script - I found it helpful to be able to regenerate fixtures selectively.
• normalize lines like at async node:... - We missed that flavour of node internal traces before.
• remove all the lines from the output until we encounter one starting with Node.js - This, currently, only affects the test case where we throw an error at the top level of the test file. In such a case, the output first lists the error details, then prints the node version and then continues to print the output from the reporter. I decided to propose removing that output from the checked-in results instead of just skipping it during normalization because we do not have control over that part in the reporter.
  • before:

    /Users/galargh/GitHub/NomicFoundation/hardhat/v-next/hardhat-node-test-reporter/integration-tests/fixture-tests/execution-failure-test/test.ts:1
    throw new Error("error");
          ^
    
    
    Error: error
        at <anonymous> (/Users/galargh/GitHub/NomicFoundation/hardhat/v-next/hardhat-node-test-reporter/integration-tests/fixture-tests/execution-failure-test/test.ts:1:7)
        at ModuleJob.run (node:internal/modules/esm/module_job:262:25)
        at async onImport.tracePromise.__proto__ (node:internal/modules/esm/loader:485:26)
        at async asyncRunEntryPointWithESMLoader (node:internal/modules/run_main:109:5)
    
    Node.js v22.4.1
    
      �[31m1) integration-tests/fixture-tests/execution-failure-test/test.ts�[39m �[31m�[3m(98ms)�[23m�[39m
    
    
    �[32m0 passing�[39m�[90m (130ms)�[39m�[31m�[39m
    �[31m1 failing�[39m
    
      1) integration-tests/fixture-tests/execution-failure-test/test.ts:
    
      �[31mTest file execution failed (exit code 1).�[39m
      �[90m    Did you forget to await a promise?�[39m
    
    

  • after:

    
      �[31m1) integration-tests/fixture-tests/execution-failure-test/test.ts�[39m �[31m�[3m(98ms)�[23m�[39m
    
    
    �[32m0 passing�[39m�[90m (130ms)�[39m�[31m�[39m
    �[31m1 failing�[39m
    
      1) integration-tests/fixture-tests/execution-failure-test/test.ts:
    
      �[31mTest file execution failed (exit code 1).�[39m
      �[90m    Did you forget to await a promise?�[39m
    
    

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
hardhat	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 30, 2024 11:01am

[changeset-bot]

⚠️ No Changeset found

Latest commit: afe66f4

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[alcuadrado]

Note for a future pr: we need to change the output generated in this case, as it has nothing to do with promises.

[alcuadrado]

Looks good! I like that options idea.

I just left a comment about one test.

[galargh]

I also added a test case for an unawaited it inside an it - 473cb03

It does seem to work just fine, though.

[alcuadrado]

It does seem to work just fine, though.

I think we need to force it to go through the event loop. I somehow was getting this error, that's why I added an error message related to promises when you throw from the top-level 😅

I think we can merge this and tackle that on a follow up pr when fixing that error message.