Session fixtures invoked multiple times with xdist

[pytestbot]

Originally reported by: Andreas Pelme (BitBucket: pelme, GitHub: pelme)

---

Since https://bitbucket.org/hpk42/pytest-xdist/commits/05aa4f48306c905ae4d183a4b337f63cc4490466 session scoped fixtures runs multiple times.

To trace this, I have added this code to the problematic fixture:

class Count(object):
    counter = 0
counter = Count()

@pytest.fixture(scope='session')
def _django_db_setup(request, _django_runner, _django_cursor_wrapper):
    """Session-wide database setup, internal to pytest-django"""
    counter.counter += 1
    if counter.counter >= 2:
        print '_django_db_setup called %d times!' % counter.counter
        assert 0

    # ... no other changes

This is what happends:

$ py.test functional_tests/test_pdf_generation.py --capture=fd -v -n1
======================================================== test session starts ========================================================
platform darwin -- Python 2.7.5 -- py-1.4.21.dev1 -- pytest-2.6.0.dev1
plugins: django, xdist, cache, instafail
gw0 [3]
scheduling tests via LoadScheduling
.E
_________________________________________ ERROR at setup of test_kontrolluppgift_pdf_staff __________________________________________
[gw0] darwin -- Python 2.7.5 /Users/andreas/code/personalkollen/venv-2.7/bin/python
../pytest_django/pytest_django/plugin.py:177: in _django_db_marker
>               request.getfuncargvalue('db')
../pytest/_pytest/python.py:1319: in getfuncargvalue
>       return self._get_active_fixturedef(argname).cached_result[0]
../pytest/_pytest/python.py:1333: in _get_active_fixturedef
>           result = self._getfuncargvalue(fixturedef)
../pytest/_pytest/python.py:1385: in _getfuncargvalue
>           val = fixturedef.execute(request=subrequest)
../pytest/_pytest/python.py:1802: in execute
>           fixturedef = request._get_active_fixturedef(argname)
../pytest/_pytest/python.py:1333: in _get_active_fixturedef
>           result = self._getfuncargvalue(fixturedef)
../pytest/_pytest/python.py:1385: in _getfuncargvalue
>           val = fixturedef.execute(request=subrequest)
../pytest/_pytest/python.py:1835: in execute
>                                      self.yieldctx)
../pytest/_pytest/python.py:1761: in call_fixture_func
>           res = fixturefunc(**kwargs)
../pytest_django/pytest_django/fixtures.py:32: in _django_db_setup
>           assert 0
E           assert 0
------------------------------------------------------- Captured stdout setup -------------------------------------------------------
_django_db_setup called 2 times!
E
_________________________________________ ERROR at setup of test_kontrolluppgift_pdf_manual _________________________________________
[gw0] darwin -- Python 2.7.5 /Users/andreas/code/personalkollen/venv-2.7/bin/python
../pytest_django/pytest_django/plugin.py:177: in _django_db_marker
>               request.getfuncargvalue('db')
../pytest/_pytest/python.py:1319: in getfuncargvalue
>       return self._get_active_fixturedef(argname).cached_result[0]
../pytest/_pytest/python.py:1333: in _get_active_fixturedef
>           result = self._getfuncargvalue(fixturedef)
../pytest/_pytest/python.py:1385: in _getfuncargvalue
>           val = fixturedef.execute(request=subrequest)
../pytest/_pytest/python.py:1802: in execute
>           fixturedef = request._get_active_fixturedef(argname)
../pytest/_pytest/python.py:1333: in _get_active_fixturedef
>           result = self._getfuncargvalue(fixturedef)
../pytest/_pytest/python.py:1385: in _getfuncargvalue
>           val = fixturedef.execute(request=subrequest)
../pytest/_pytest/python.py:1835: in execute
>                                      self.yieldctx)
../pytest/_pytest/python.py:1761: in call_fixture_func
>           res = fixturefunc(**kwargs)
../pytest_django/pytest_django/fixtures.py:32: in _django_db_setup
>           assert 0
E           assert 0
------------------------------------------------------- Captured stdout setup -------------------------------------------------------
_django_db_setup called 3 times!

================================================= 1 passed, 2 error in 1.78 seconds =================================================

The same test works fine without xdist:

$ py.test functional_tests/test_pdf_generation.py --capture=fd -v
======================================================== test session starts ========================================================
platform darwin -- Python 2.7.5 -- py-1.4.21.dev1 -- pytest-2.6.0.dev1
plugins: django, xdist, cache, instafail
collected 3 items

functional_tests/test_pdf_generation.py ...

===================================================== 3 passed in 2.09 seconds ======================================================

This also happends with -n2, -n3 etc. It happens with pytest-xdist from revision 169 to latest tip, and pytest 2.5.2 to latest tip.

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/503

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

the problem very likely lies in SlaveInteractor.runtestloop/runtests and the interaction with nextitem,
i'd guess each chunk gets a new session scope

[pytestbot]

Original comment by BitBucket: davidkr, GitHub: davidkr:

---

wasn't sure it was a bug, but I see what you mean now. I got the impression from http://holgerkrekel.net/2013/11/12/running-tests-against-multiple-devicesresources-in-parallel/ it was for the master and each of the slaves. I misunderstood how things work clearly. ..this stackoverflow post http://stackoverflow.com/questions/12586489/pytest-are-pytest-sessionstart-and-pytest-sessionfinish-valid-hooks clarified that when using xdist session-scoped fixtures are supposed to be only created once per test-slave.

[pytestbot]

Original comment by Jeff Smith (BitBucket: ascarel, GitHub: ascarel):

---

Is it possible that the following problem is exactly the same as the one described here?

Here is some test code. This mimicks the behaviour of my test setup (it records REST API requests made throughout a series of tests and saves that into a file at the end, to avoid overwriting).

If I run that without xdist, I get one file.

If I run that with -n 2, I get a 38 files. I would expect only one file per test slave/process (that's why I tag the filename with a random part).

#!python

from uuid import uuid4

from pytest import yield_fixture, mark

_huge_list_of_numbers = []


@yield_fixture(scope="session")
def save_my_list():
    filename = "{name}-{tag}.txt".format(
        name="my_list_of_numbers",
        tag=str(uuid4()).split("-")[0])

    yield

    # Teardown code
    with open(filename, "w+") as f:
        f.write("\n".join(map(str, _huge_list_of_numbers)))


@mark.usefixtures("save_my_list")
class TestDoStuffAndSaveTheNumbers():

    @yield_fixture(params=range(1000))
    def a_number_in_a_long_list(self, request):
        yield request.param

    def test_first_class(self, a_number_in_a_long_list):
        global _huge_list_of_numbers
        _huge_list_of_numbers.append(a_number_in_a_long_list)
        assert True

This seemed related, so I'm posting this here. I apologize in advance if this should be filed in a separate issue.

[pytestbot]

Original comment by Andreas Pelme (BitBucket: pelme, GitHub: pelme):

---

I can reliably reproduce this with latest pytest-xdist tip and have written a failing test:

https://bitbucket.org/pelme/pytest-xdist/commits/36ca679583b06157d5c5aef00df73b5dc94dfd35?at=default

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

On Tue, Jul 08, 2014 at 20:42 -0000, Andreas Pelme wrote:

Thanks, Andreas for the excellent work on pinpointing the issue and commit.
The relevant code executing on the slave side is found in xdist/remove.py's
"def pytest_runtest_mainloop". In the commit you reference, the
deleted line 54 is of importance: it makes sure that we only
call pytest_runtest_item when we know both an item and a next item.

In your failing test, "item" is the first test function, "nextitem"
is the second before the commit. If nextitem is None (which happens
after the commit was applied) then the first item execution will
teardown the session fixture because the next item is None.

We need to re-instate the old behaviour that "pytest_runtest_item" is
always called with a nextitem unless we have received a "shutdown" signal
in which case it's fine (and required) that nextitem is None.

It is thus important that the "master" distributing the tests to the nodes
guarantes to allocate at least two tests to each node unless we have no
items left. Otherwise the slave would wait for the second item indefinitely.
That code is in xdist/dsession.py's LoadSchedule methods.

If you feel like working on an according fix, you'd be most welcome to
deliver a PR to complete your work!

best,
holger

[pytestbot]

Original comment by Andreas Pelme (BitBucket: pelme, GitHub: pelme):

---

Thanks for the detailed explanation, Holger. I will continue working on a fix and try to resolve it fully.

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

On Sun, Jul 13, 2014 at 14:37 -0000, Andreas Pelme wrote:

I fixed it already, on my still ongoing train ride to Berlin.
see you soon! See the according commit in pytest-xdist.

cheers,
holger

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

fixed on pytest-xdist.

[pytestbot]

Original comment by Anatoly Bubenkov (BitBucket: bubenkoff, GitHub: bubenkoff):

---

Hi @hpk42
this fix which is mentioned, it's only actual if there are not much tests to fulfill the initial chunk of the tests as i understand
but if the number of tests is more than one chunk for same node, then it's possible that certain session-scoped fixture will be executed multiple times, just because of the nature of the pytest-xdist and pytest itself
or im wrong in my understanding?
here's my article about this problem
http://developer.paylogic.com/articles/pytest-xdist-and-session-scoped-fixtures.html

[dusans]

This doesn't work for me if I'm using autouse=True

@pytest.fixture(scope="session", autouse=True)
@memoize

The same fixture is called multiple times.

[RonnyPfannschmidt]

@dusans each process xdist makes is a own session, so thats currently working as intended - there is not yet a mechanism to inject new scopes and handling them (a "control-session" one might be needed

[trbedwards]

I should mention that there is a new command-line option --dist=loadscope for pytest-xdist which allows tests to be divided among workers such that module and class scoped fixtures are only executed once. Not sure if it works for session scoped fixtures, but correct me if I'm wrong?

[RonnyPfannschmidt]

@trbedwards it shouldnt, since session ones would be process global, and each process owuld need one

the loadscope details handle seregation by file by avoiding certain files, that cant be done for session scope