spurious test failures

[orbeckst]

Tests started failing somewhat randomly as mentioned in the post mdnalysis-devel: spurious test failures

Without reliable unit tests, our development workflow is severely hampered. This needs to be fixed but I have no idea how.

Details

See https://travis-ci.org/MDAnalysis/mdanalysis/builds/94270229 for an example. All failures are in the coordinate tests and are of the form

  File "/home/travis/miniconda/envs/pyenv/lib/python2.7/site-packages/MDAnalysisTests/coordinates/base.py", line 198, in test_get_writer_container 
    with tempdir.in_tempdir(): 
  File "/home/travis/miniconda/envs/pyenv/lib/python2.7/contextlib.py", line 17, in __enter__ 
    return self.gen.next() 
  File "/home/travis/miniconda/envs/pyenv/lib/python2.7/site-packages/tempdir.py", line 44, in in_tempdir 
    old_path = os.getcwd() 

OSError: [Errno 2] No such file or directory 

and the common theme appears to be tempdir.in_tempdir().

Initial discussion

Initial reports/discussion about these test failures started at #551 (comment) and should continue here.

On 1 Dec, 2015, at 17:10, @hainm wrote:

it's better to run the test in single core to debug.

Maybe we should try --processes=1 just to avoid any opportunity for race conditions. It is suspicious that tempdir.in_tempdir() appears to be always involved.

How's about

cd testsuite/MDAnalysisTests/coordinates
python test_netcdf.py

python test_netcdf.py does not do anything.

./mda_nosetests coordinates/test_netcdf.py works fine in serial or parallel. The whole 3300+ tests pass locally, with up to 24 threads... most of the time. When it fails for a few tests then these failures are not necessarily the same. See #551 (comment) .

[orbeckst]

@hainm reported that a potentially similar error was reported in frescobaldi/frescobaldi#554 and frescobaldi/frescobaldi#776 although no solution was presented apart from the hint of "problems when creating a tempdir inside a tempdir".

[richardjgowers]

If we've accidentally nested tempdirs (calling Tempdir while inside a Tempdir decorated class?) that might be a way for that to happeb

[kain88-de]

But the errors seem to be random and I can't find such a nested pattern in the code where it fails.

[hainm]

@orbeckst should buy me a beer.

nosetests -vs testsuite/MDAnalysisTests/analysis/test_helanal.py testsuite/MDAnalysisTests/analysis/test_psa.py

will cause error because someone went to temp folder in test_helanal.py but has never come back to working dir.

MDAnalysis.analysis.helanal: Check for resolution of Issue 188. ... ok test_hausdorff_bound (MDAnalysisTests.analysis.test_psa.TestPSAnalysis) ... ERROR test_reversal_frechet (MDAnalysisTests.analysis.test_psa.TestPSAnalysis) ... ERROR test_reversal_hausdorff (MDAnalysisTests.analysis.test_psa.TestPSAnalysis) ... ERROR

[hainm]

it's better to always use contextmanager.

[orbeckst]

On 2 Dec, 2015, at 16:53, Hai Nguyen wrote:

@orbeckst should give beer to me.

nosetests -vs testsuite/MDAnalysisTests/analysis/test_helanal.py testsuite/MDAnalysisTests/analysis/
test_psa.py

will cause error because someone go to temp folder in test_helanal.py but has never come back to working dir.

I just heard from @dldotson and @richardjgowers that this fixes the problem: I am very happy, so yes: I will gladly buy you a 🍺 - find me at ACS (or find @dldotson or @slseyler at BPS who can buy you a beer in my name ;-) ).

[orbeckst]

I added a note to ban os.chdir() when writing testcases.

(And yes, the wiki page on testing needs a major clean-up.)

[orbeckst]

$ git grep chdir
testsuite/MDAnalysisTests/analysis/test_helanal.py:        os.chdir(self.tempdir)
testsuite/MDAnalysisTests/analysis/test_hole.py:        os.chdir(self.dir_name)
testsuite/ez_setup.py:        os.chdir(tmpdir)
testsuite/ez_setup.py:        os.chdir(subdir)
testsuite/ez_setup.py:        os.chdir(old_wd)
testsuite/ez_setup.py:        os.chdir(tmpdir)
testsuite/ez_setup.py:        os.chdir(subdir)
testsuite/ez_setup.py:        os.chdir(old_wd)

#558 should take care of all of this (and of course we should eliminate ez_setup.py etc from the testsuite....)

[orbeckst]

@hainm do you also want to fix test_hole.py with #558? Otherwise I can clean up my own mess in test_hole.py (which, by the way, is only run when you have hole available so it won't trigger the bug on travis).

[hainm]

@orbeckst I am leaving test_hole.py to you since I don't have HOLE installed (so not sure if the change is good or not).