feat(cli): run test modules in parallel

[caspervonb]

This runs test modules in parallel when the number of concurrent jobs passed via --jobs=<N> is greater than 1.
The default is to run with a limit of 1 to retain backwards compatibility, as not all test suites in the wild are concurrency safe.

Closes #6559
Closes #9284

[bartlomieju]

I've noticed two quirks that I'm not sure how big of an impact they'll have:

1. There's Check <module name> for as many modules as there are discovered

Check file:///Users/biwanczuk/dev/deno_std/textproto/test.ts
Check file:///Users/biwanczuk/dev/deno_std/_util/deep_assign_test.ts
Check file:///Users/biwanczuk/dev/deno_std/_util/assert_test.ts
Check file:///Users/biwanczuk/dev/deno_std/flags/test.ts
Check file:///Users/biwanczuk/dev/deno_std/archive/tar_test.ts
Check file:///Users/biwanczuk/dev/deno_std/bytes/test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/_fnv/util_test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/sha512_test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/fnv_test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/sha256_test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/sha1_test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/md5_test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/sha3_test.ts
Check file:///Users/biwanczuk/dev/deno_std/datetime/test.ts
Check file:///Users/biwanczuk/dev/deno_std/fmt/colors_test.ts
Check file:///Users/biwanczuk/dev/deno_std/fmt/printf_test.ts
....

2. For each test module (supposedly) there's running X tests printed:

running 4 tests
test [encoding/base64] testBase64EncodeString ... ok (2ms)
test [encoding/base64] testBase64EncodeBinary ... ok (2ms)
test [encoding/base64] testBase64EncodeBinaryBuffer ... ok (2ms)
test [encoding/base64] testBase64DecodeBinary ... ok (2ms)
running 4 tests
test [encoding.base32] encode ... ok (4ms)
test [encoding.base32] decode ... ok (2ms)
test [encoding.base32] decode bad length ... ok (1ms)
test [encoding.base32] decode bad padding ... ok (1ms)

I don't mind it, but maybe we could change it to running X tests in <module name>

[lucacasonato]

There's Check <module name> for as many modules as there are discovered

This suggests each module is being type checked individually. AFAIK this is way slower than type-checking them all at once. cc @kitsonk

[caspervonb]

1. There's Check <module name> for as many modules as there are discovered

Check file:///Users/biwanczuk/dev/deno_std/textproto/test.ts
Check file:///Users/biwanczuk/dev/deno_std/_util/deep_assign_test.ts
Check file:///Users/biwanczuk/dev/deno_std/_util/assert_test.ts
Check file:///Users/biwanczuk/dev/deno_std/flags/test.ts
Check file:///Users/biwanczuk/dev/deno_std/archive/tar_test.ts
Check file:///Users/biwanczuk/dev/deno_std/bytes/test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/_fnv/util_test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/sha512_test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/fnv_test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/sha256_test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/sha1_test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/md5_test.ts
Check file:///Users/biwanczuk/dev/deno_std/hash/sha3_test.ts
Check file:///Users/biwanczuk/dev/deno_std/datetime/test.ts
Check file:///Users/biwanczuk/dev/deno_std/fmt/colors_test.ts
Check file:///Users/biwanczuk/dev/deno_std/fmt/printf_test.ts
....

Doing the type checking and compilation before attempting to run is intentional.
Type errors occurs immediatly after the check statement so, it looks nice imo and accurately reflects what is going on.

This suggests each module is being type checked individually. AFAIK this is way slower than type-checking them all at once.

Yeah by one literally takes hours in the debug build @lucacasonato, but I do them all in one graph.

1. For each test module (supposedly) there's running X tests printed:

running 4 tests
test [encoding/base64] testBase64EncodeString ... ok (2ms)
test [encoding/base64] testBase64EncodeBinary ... ok (2ms)
test [encoding/base64] testBase64EncodeBinaryBuffer ... ok (2ms)
test [encoding/base64] testBase64DecodeBinary ... ok (2ms)
running 4 tests
test [encoding.base32] encode ... ok (4ms)
test [encoding.base32] decode ... ok (2ms)
test [encoding.base32] decode bad length ... ok (1ms)
test [encoding.base32] decode bad padding ... ok (1ms)

I don't mind it, but maybe we could change it to running X tests in

I've got a branch with slightly nicer test output to address that after this.

[caspervonb]

Ran into some caching issues after introducing the fake module stub that calls await Deno[Deno.internal].runTests.
For example switching from unit tests to std tests:

error: Module "file:///home/caspervonb/deno/test_util/std/fmt/printf_test.ts" is missing from the graph.
  From: file:///home/caspervonb/deno/$deno$test.ts

Also a few runs definitively used old cached versions of "$deno$test.ts" from disk.

[bartlomieju]

@piscisaureus please take a look at this part

[bartlomieju]

LGTM 🚀 thank you Casper, this is a great feature to have