using a PEX with pytest-xdist

[tostasmistas]

Hi,

I am trying to use a PEX to run tests from the pytest framework while using the pytest-xdist plugin for distributed testing.

However I believe I am facing an issue similar to this one, as my worker processes are not recognizing packages that are packaged with pex.

My pex build is configured to have pytest as a module entry point (-m pytest) and from this main process I launch the workers in python subprocesses that are instantiated with the following command pytest -d --tx popen//python=python3.

When the workers begin executing I get the error ModuleNotFoundError: No module named '_pytest' which leads me to believe that the pytest package (contained in my pex file) is not being recognized by the subprocesses.

I've tried to bootstrap the pex environment by configuring the workers before their execution in the pytest_configure_node xdist hook with a call to pex_bootstrapper.bootstrap_pex_env('my.pex') but the problem remains.

Any ideas? Thanks

[jsirois]

This sort of issue was solved with Pex introduction of --venv mode in #1153.

For example, with this project setup:

:; cat foo.py
BAR = 42

:; cat test_foo.py
import foo


def test():
    assert 42 == foo.BAR

A standard PEX still does not work with pytest-xdist:

:; pex pytest pytest-xdist -cpytest -o pex.pex

:; ./pex.pex -n 2
========================================================================================= test session starts =========================================================================================
platform linux -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0
rootdir: /tmp/project
plugins: xdist-3.6.1
initialized: 2/2 workers[gw0] node down: Traceback (most recent call last):
  File "/home/jsirois/.pex/installed_wheels/26dee51f1b80cebd6d0ca8e74dd8745419761d3bef34163928cbebbdc4749fdc/execnet-2.1.1-py3-none-any.whl/execnet/gateway_base.py", line 1291, in executetask
    exec(co, loc)
  File "/home/jsirois/.pex/installed_wheels/9ed4adfb68a016610848639bb7e02c9352d5d9f03d04809919e2dafc3be4cca7/pytest_xdist-3.6.1-py3-none-any.whl/xdist/remote.py", line 23, in <module>
    from _pytest.config import _prepareconfig
ModuleNotFoundError: No module named '_pytest'


replacing crashed worker gw0
initialized: 3/3 workers[gw1] node down: Traceback (most recent call last):
  File "/home/jsirois/.pex/installed_wheels/26dee51f1b80cebd6d0ca8e74dd8745419761d3bef34163928cbebbdc4749fdc/execnet-2.1.1-py3-none-any.whl/execnet/gateway_base.py", line 1291, in executetask
    exec(co, loc)
  File "/home/jsirois/.pex/installed_wheels/9ed4adfb68a016610848639bb7e02c9352d5d9f03d04809919e2dafc3be4cca7/pytest_xdist-3.6.1-py3-none-any.whl/xdist/remote.py", line 23, in <module>
    from _pytest.config import _prepareconfig
ModuleNotFoundError: No module named '_pytest'

...

maximum crashed workers reached: 8

============================================================================== xdist: maximum crashed workers reached: 8 ==============================================================================
======================================================================================== no tests ran in 0.28s ========================================================================================

But telling the PEX to lay itself out in a traditional venv gets things working:

# Note the addition of `--venv`:
:; pex pytest pytest-xdist -cpytest -o pex.pex --venv

:; ./pex.pex -n 2
========================================================================================= test session starts =========================================================================================
platform linux -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0
rootdir: /tmp/project
plugins: xdist-3.6.1
2 workers [1 item]
.                                                                                                                                                                                               [100%]
========================================================================================== 1 passed in 0.17s ==========================================================================================

# /tmp/project via  v3.11.9
:;

There are several Pex --ven* options to investigate for more difficult cases than pytest-xdist, like --venv-site-packages-copies and --non-hermetic-venv-scripts.

[jsirois]

@tostasmistas I'm going to close this as an answered question since Pex grew --venv to help solve cases like these back at the end of 2020.