Test failing locally: test_parallel_calculation

[abhiShandy]

Expected behavior

No failed test

Actual behavior

platform darwin -- Python 3.8.2, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /Users/abhishandy/Code/mdanalysis/fork/testsuite, inifile: setup.cfg
plugins: forked-1.1.2, xdist-1.31.0, hypothesis-5.6.0

___________________________________________________________________________________ TestEncore.test_parallel_calculation ___________________________________________________________________________________
[gw0] darwin -- Python 3.8.2 /Users/abhishandy/opt/miniconda3/envs/mda/bin/python

self = <MDAnalysisTests.analysis.test_encore.TestEncore object at 0x133984400>

    @pytest.mark.xfail(os.name == 'nt',
                       strict=True,
                       reason="Not yet supported on Windows.")
    def test_parallel_calculation(self):
    
        def function(x):
            return x**2
    
        arguments = [tuple([i]) for i in np.arange(0,100)]
    
        parallel_calculation = encore.utils.ParallelCalculation(function=function,
                                                                n_jobs=4,
                                                                args=arguments)
>       results = parallel_calculation.run()

analysis/test_encore.py:132: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../package/MDAnalysis/analysis/encore/utils.py:315: in run
    w.start()
../../../../../opt/miniconda3/envs/mda/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
../../../../../opt/miniconda3/envs/mda/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
../../../../../opt/miniconda3/envs/mda/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
../../../../../opt/miniconda3/envs/mda/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
../../../../../opt/miniconda3/envs/mda/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
../../../../../opt/miniconda3/envs/mda/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = <Process name='Process-2' parent=32939 initial>, file = <_io.BytesIO object at 0x139388180>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'TestEncore.test_parallel_calculation.<locals>.function'

../../../../../opt/miniconda3/envs/mda/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
====================================================== 1 failed, 15980 passed, 121 skipped, 1 xfailed, 2 xpassed, 33071 warnings in 691.83s (0:11:31) ======================================================

Code to reproduce the behavior

pytest --disable-pytest-warnings --numprocesses 4

Current version of MDAnalysis

• Which version are you using? 0.20.1
• Which version of Python? 3.8.2
• Which operating system? MacOS Catalina 10.15.3

[shfrz]

Hi @orbeckst
I ain't getting any error. I think @abhiShandy might have changed something locally or not set up a virtual environment properly!

[abhiShandy]

@shfrz I also suspect that my local environment is faulty. Did you test it with the same Python and OS as me?

• Which version of Python? 3.8.2
• Which operating system? MacOS Catalina 10.15.3

[IAlibay]

So I can't reproduce the error using py 3.8.1 (conda hasn't released a main package of 3.8.2 yet). That being said unless someone else tests things with 3.8.2 (particularly on Catalina), it's not possible to tell if this is a reproducible issue or not.

Pinging @RMeli here possibly, as known MacOS user 😉

[RMeli]

I can confirm this issue on macOS 10.15.3 with Python 3.8.2 and on the developer branch (a2951ee).

All dependencies apart from gsd and hypothesis were installed with conda using the conda-forge channel.

[RMeli]

To add on this, this issue is not related to --numprocesses 4.

[IAlibay]

@RMeli does #2615 fix things for you?

[shfrz]

@RMeli @IAlibay
I think this is not an isolated issue, it also happens the other way round : You get no errors locally but Travis Cl shows up errors.
This happens with windows as well.

[abhiShandy]

Yes, Travis CI didn't catch this issue. I don't know how it works exactly, but it may not be testing with this particular case of OS - Python combination.

[IAlibay]

It looks like our Travis test matrix only covers py3.6 for osx (linux covers the whole range), which is probably why this didn't get found earlier.

[mtiberti]

Thanks for the report- I can reproduce on macOS Catalina and Python 3.8.2 - @IAlibay @abhiShandy #2615 effectively fixes it for me, thanks for this!