Support for phpunit --order-by

[Petah]

Does paratest support functionality like phpunit's order-by? I would like my tests to run in a random order, fastest first, etc

  --order-by=<order>          Run tests in order: default|defects|duration|no-depends|random|reverse

https://phpunit.readthedocs.io/en/9.2/textui.html?highlight=%22order-by%22#command-line-options

Thanks.

[Slamdunk]

As of yet, there's no plan to support --order-by: some order requires the PHPUnit cache, which we cannot easily support, some order requires a SUT listener, which we cannot support at all, other orders are weird to me - what would be the benefit or random order in the parallelization context where tests are already started with no particular consequentiality?

[weirdan]

Random order is useful to ensure that tests have no hidden dependencies between them.

[Slamdunk]

What we can do is running each PHPUnit process with --cache-result --cache-result-file=/tmp/random, merge back together the results and in subsequent ParaTest runs use it to provide almost all the order-by PHPUnit gives.

What I don't like is that it would reflect PHPUnit behavior only in ParaTest --functional mode, which is supported only by Runner: WrapperRunner could use only a part of PHPUnit behavior since it can only run a full test class on its default order, unless adding some logic to store the specific cache file and give it back to the subprocess.

The only easy order to implement is random one since it doesn't need a previous run: few randomizer on SuiteLoader class and a --order-by random passthru should suffice. This implementation by the way is only needed to uncover hidden dependencies between different test classes: you are already able to --passthru="--order-by random" to uncover hidden dependencies within test cases of the same test class.

All in all, I see a quite lot of code to add and maintain for a little gain.

[proggga]

I think we can close in some way this task, We added default, random, reversed types, and --random-order-seed for failure reproduce

[Slamdunk]

Closed by #539