Feature request: Run async tests in parallel within the same file

[voltrevo]

Previous related issues:

• Request: Add --parallel to deno test #6559
• Option/flag to run async tests in parallel  #7615 (marked as duplicate of 6559)

These seem to be considered resolved via the -j flag implemented by #9815.

-j is definitely a welcome improvement. However, it's only enabling separate test files to run parallel. I believe instead the expectation is that parallelism be supported within the same file. In any case, I would like to request this feature.

For example:

// test/blah.ts

Deno.test("foo", async () => {
  console.log("start foo");
  await new Promise((resolve) => setTimeout(resolve, 1000));
  console.log("end foo");
});

Deno.test("bar", async () => {
  console.log("start bar");
  await new Promise((resolve) => setTimeout(resolve, 1000));
  console.log("end bar");
});

Running with -j4:

$ deno test -j4 test/blah.test.ts
running 2 tests from file:///home/voltrevo/workspaces/ethf/aggregator/test/blah.test.ts
start foo
end foo
test foo ... ok (1008ms)
start bar
end bar
test bar ... ok (1006ms)

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out (2031ms)

If parallelism was enabled between these tests, we should expect output more like this:

running 2 tests from file:///home/voltrevo/workspaces/ethf/aggregator/test/blah.test.ts
start foo
start bar
end foo
test foo ... ok (1008ms)
end bar
test bar ... ok (1006ms)

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out (1031ms)

(Implementing this might be considered a breaking change, requiring a new flag, perhaps -jj.)

Can you confirm that this kind of parallelism is not yet supported or advise how to enable it?

If it's not supported, are there any intentions to implement this?

(It's also possible to enable this kind of concurrency for synchronous tests, by running a first pass to identify test names and then running separate processes with different filters. Can this also be considered?)

[caspervonb]

-j is definitely a welcome improvement. However, it's only enabling separate test files to run parallel. I believe instead the expectation is that parallelism be supported within the same file. In any case, I would like to request this feature.

The design I've had in mind for a while now is adding a parallel boolean to TestDefinition which is a hint to the runner that the test can be ran in parallel.

That way async tests marked as safe for parallel execution can simply be partitioned off into a group and awaited on.

It would also backwards compatible.

Can you confirm that this kind of parallelism is not yet supported or advise.

Indeed this is not supported tests with within a module are run in serial by design because most users will have some global shared state.

If it's not supported, are there any intentions to implement this?

We're thinking about it, in particular if we were to implement groups then that's a good entry point to enable concurrency for an entire group of tests.

(It's also possible to enable this kind of concurrency for synchronous tests, by running a first pass to identify test names and then running separate processes with different filters. Can this also be considered?)

Not sure what you are proposing here.

[dsherret]

I think this is covered by the proposals in #10771 and #11235 -- See those to leave any feedback.

[voltrevo]

I think this is covered by the proposals in #10771 and #11235 -- See those to leave any feedback.

@dsherret

Hmm, I don't think they are the same feature though. They're nice proposals for other reasons, but if the goal is just to add more parallelism, it's much simpler for the user to just enable this via a flag.

I understand every feature has a cost ofc. If it's not considered worth it then I support closing this issue, I don't think there is anything here to add to #10771, #11235 since they both already enable the user to configure parallelism within a file.

(It's also possible to enable this kind of concurrency for synchronous tests, by running a first pass to identify test names and then running separate processes with different filters. Can this also be considered?)

Not sure what you are proposing here.

@caspervonb

Ah, to elaborate, I mean create an alternative execution environment where tests don't run but instead the test names are emitted for collection. Then you can spin up workers that run tests in a more normal way and feed them test names as needed. This enables parallelism of synchronous tests.

Obviously adds significant complexity though, so I just wanted to throw it out there.

[caspervonb]

I think this is covered by the proposals in #10771 and #11235 -- See those to leave any feedback.

Not really, you'd have to create a group just to run in them concurrently.
Would be better to just mark something as being concurrency safe imo.

Thinking ahead to when ES decorators are a thing, I'd prefer simply doing this:

@test({ concurrent: true })
async function testFooPrototypeSetBar() {
}

Ah, to elaborate, I mean create an alternative execution environment where tests don't run but instead the test names are emitted for collection. Then you can spin up workers that run tests in a more normal way and feed them test names as needed. This enables parallelism of synchronous tests.

I tried spinning up isolates per tests, was neat but it breaks user expectations of modules only being instantiated once.

[bartlomieju]

I discussed this with @dsherret and we feel that Deno.test() (the declarative approach) should be kept as is, without adding concurrency option. If the user wants to run tests concurrently they should use "test steps" API.

I'm gonna close this issue for now.