Allow hierarchical parameterized/dynamic tests

[nipafx]

However an extension point that allows dynamic or parameterized tests turns out (see for example #14, #371, #395 ), it would be great if it allowed to structure tests hierarchically. If, for example, two test methods testA and testB were run three times each, the test tree should be able to look as follows:

testA
    - run #1
    - run #2
    - run #3
testB
    - run #1
    - run #2
    - run #3

Allowing extensions to create a plan like that gives them the ability to better structure the generated tests, giving better feedback to developers.

This does not mean that a particular implementation for that extension point (let's say Jupiter's support for parameterized tests) should organize tests like that but that other implementations have the ability to do so.

[sormuras]

Like turning test methods into containers? Why not annotated testA as a TestFactory? That already leads to trees like:

...
│  ├─ DynamicTestsDemo
│  │  ├─ dynamicTestsFromCollection()
│  │  │  ├─ 1st dynamic test ✔
│  │  │  ├─ 2nd dynamic test ✔
│  │  ├─ dynamicTestsFromIterable()
│  │  │  ├─ 3rd dynamic test ✔
│  │  │  ├─ 4th dynamic test ✔
│  │  ├─ dynamicTestsFromIterator()
│  │  │  ├─ 5th dynamic test ✔
│  │  │  ├─ 6th dynamic test ✔
│  │  ├─ dynamicTestsFromStream()
│  │  │  ├─ testA ✔
│  │  │  ├─ testB ✔
│  │  │  ├─ testC ✔
│  │  ├─ dynamicTestsFromIntStream()
│  │  │  ├─ test0 ✔
│  │  │  ├─ test2 ✔
│  │  │  ├─ test4 ✔
│  │  │  ├─ test6 ✔
│  │  │  ├─ test8 ✔
│  │  │  ├─ test10 ✔
│  │  │  ├─ test12 ✔
│  │  │  ├─ test14 ✔
│  │  │  ├─ test16 ✔
│  │  │  ├─ test18 ✔
│  │  ├─ generateRandomNumberOfTests()
│  │  │  ├─ input:27 ✔
│  │  │  ├─ input:48 ✔
│  │  │  ├─ input:83 ✔
│  │  │  ├─ input:47 ✔
│  │  │  ├─ input:18 ✔
│  │  │  ├─ input:43 ✔
│  │  │  ├─ input:90 ✔

[nipafx]

But in your example it is not the extension that creates the hierarchy but the engine. If as an extension developer I want my own hierarchy, I can't do it.

[smoyer64]

@nicolaiparlog That's why I proposed allowing an extension to alter the test plan after discovery. See #354 and the associated PR #577. When the test plan is passed into the engine to be executed, it's effectively immutable.

[nipafx]

See my comment there why I am not convinced that sufficiently addresses this use case.

[sormuras]

Addressed by 1a57e4f on master