Run tests in random order on nodes

[timj]

Sorry if this has been covered elsewhere or is obvious, but I would like to be able to run our test suite with the tests running in a random order where the order is controlled by a random seed to allow reproducibility. At the moment I can do that with pytest-random-order or pytest-randomly but as soon as I use -n everything gets confused because it seems the randomization messes with the test to node allocation and the subprocesses complain that they have been scheduled the same tests or that tests are missing.

Is there any way the xdist scheduler can play nicely with pytest-random-order or any way to control the ordering in xdist itself?

My motivation for this is that we are finding that some of the tests are sensitive to test execution order and fail if run first or on their own.

[RonnyPfannschmidt]

currently this is not possible at all - based on the work in the loadscope sheduler it should be possible to randomize better

the main problem is, that xdist is incompatible with worker side collection reordering and the same plug-ins cant nicely put it in place for the schedulers

[ssbarnea]

I would be quite curious to find out if someone found a way to randomize test order while running xdist as from time to time I endup facing testing issues related to parallel runs that are hard to reproduce.

[nicoddemus]

I understand https://pypi.org/project/pytest-randomly/ supports xdist?

[nicoddemus]

Yes, looking at the code seems like pytest-randomly does support xdist properly:

https://github.com/pytest-dev/pytest-randomly/blob/f646c188bb70660d74b81388094bfd038f5650f9/src/pytest_randomly/__init__.py#L127-L147

Closing this then. 👍