RemovedInPytest4Warning causes errors at collection time

[bsipocz]

I assume the deprecation of marks applied directly to parameters should only generate a warning rather than errors at collection time. We run into it both in astropy and photutils: astropy/astropy#6418

While the fix is trivial in our tests, we may still need to stick with the old version and pytest <3.2 for a while as some of our CI setups still uses pytest 3.0.x. (Also our LTS version requires pytest>=2.8).

============================= test session starts ==============================
platform linux2 -- Python 2.7.6, pytest-3.2.0, py-1.4.34, pluggy-0.4.0

Running tests with Astropy version 3.0.dev.
Running tests in lib.linux-i686-2.7/astropy docs.

Date: 2017-08-02T01:53:28

Platform: Linux-3.13.0-125-generic-i686-with-Ubuntu-14.04-trusty

Executable: /usr/bin/python

Full Python Version: 
2.7.6 (default, Oct 26 2016, 20:32:47) 
[GCC 4.8.4]

encodings: sys: ascii, locale: UTF-8, filesystem: UTF-8, unicode bits: 20
byteorder: little
float info: dig: 15, mant_dig: 15

Numpy: 1.12.1
Scipy: not available
Matplotlib: not available
h5py: not available
Pandas: not available
Cython: 0.25.2
Using Astropy options: remote_data: none.

rootdir: /tmp/astropy-test-lYT0mu, inifile: setup.cfg
collected 12700 items / 4 errors / 35 skipped

==================================== ERRORS ====================================
 ERROR collecting lib.linux-i686-2.7/astropy/coordinates/tests/test_finite_difference_velocities.py 
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/runner.py:191: in __init__
    self.result = func()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/runner.py:370: in <lambda>
    lambda: list(collector.collect()),
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:389: in collect
    return super(Module, self).collect()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:330: in collect
    res = self.makeitem(name, obj)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:342: in makeitem
    collector=self, name=name, obj=obj)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:745: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:339: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:334: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:613: in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:250: in _wrapped_call
    wrap_controller.send(call_outcome)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:188: in pytest_pycollect_makeitem
    res = list(collector._genfunctions(name, obj))
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:362: in _genfunctions
    self.ihook.pytest_generate_tests(metafunc=metafunc)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:745: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:339: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:334: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:614: in execute
    res = hook_impl.function(*args)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:113: in pytest_generate_tests
    metafunc.parametrize(*marker.args, **marker.kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:781: in parametrize
    for x in argvalues]
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/mark.py:66: in extract_from
    warnings.warn(MARK_PARAMETERSET_UNPACKING)
E   RemovedInPytest4Warning: Applying marks directly to parameters is deprecated, please use pytest.param(..., marks=...) instead.
E   For more details, see: https://docs.pytest.org/en/latest/parametrize.html
_ ERROR collecting lib.linux-i686-2.7/astropy/io/ascii/tests/test_c_reader.py __
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/runner.py:191: in __init__
    self.result = func()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/runner.py:370: in <lambda>
    lambda: list(collector.collect()),
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:389: in collect
    return super(Module, self).collect()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:330: in collect
    res = self.makeitem(name, obj)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:342: in makeitem
    collector=self, name=name, obj=obj)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:745: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:339: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:334: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:613: in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:250: in _wrapped_call
    wrap_controller.send(call_outcome)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:188: in pytest_pycollect_makeitem
    res = list(collector._genfunctions(name, obj))
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:362: in _genfunctions
    self.ihook.pytest_generate_tests(metafunc=metafunc)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:745: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:339: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:334: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:614: in execute
    res = hook_impl.function(*args)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:113: in pytest_generate_tests
    metafunc.parametrize(*marker.args, **marker.kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:781: in parametrize
    for x in argvalues]
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/mark.py:66: in extract_from
    warnings.warn(MARK_PARAMETERSET_UNPACKING)
E   RemovedInPytest4Warning: Applying marks directly to parameters is deprecated, please use pytest.param(..., marks=...) instead.
E   For more details, see: https://docs.pytest.org/en/latest/parametrize.html
_____ ERROR collecting lib.linux-i686-2.7/astropy/utils/tests/test_data.py _____
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/runner.py:191: in __init__
    self.result = func()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/runner.py:370: in <lambda>
    lambda: list(collector.collect()),
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:389: in collect
    return super(Module, self).collect()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:330: in collect
    res = self.makeitem(name, obj)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:342: in makeitem
    collector=self, name=name, obj=obj)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:745: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:339: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:334: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:613: in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:250: in _wrapped_call
    wrap_controller.send(call_outcome)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:188: in pytest_pycollect_makeitem
    res = list(collector._genfunctions(name, obj))
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:362: in _genfunctions
    self.ihook.pytest_generate_tests(metafunc=metafunc)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:745: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:339: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:334: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:614: in execute
    res = hook_impl.function(*args)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:113: in pytest_generate_tests
    metafunc.parametrize(*marker.args, **marker.kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:781: in parametrize
    for x in argvalues]
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/mark.py:66: in extract_from
    warnings.warn(MARK_PARAMETERSET_UNPACKING)
E   RemovedInPytest4Warning: Applying marks directly to parameters is deprecated, please use pytest.param(..., marks=...) instead.
E   For more details, see: https://docs.pytest.org/en/latest/parametrize.html
_____ ERROR collecting lib.linux-i686-2.7/astropy/utils/tests/test_data.py _____
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/runner.py:191: in __init__
    self.result = func()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/runner.py:370: in <lambda>
    lambda: list(collector.collect()),
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:389: in collect
    return super(Module, self).collect()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:330: in collect
    res = self.makeitem(name, obj)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:342: in makeitem
    collector=self, name=name, obj=obj)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:745: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:339: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:334: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:613: in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:250: in _wrapped_call
    wrap_controller.send(call_outcome)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:188: in pytest_pycollect_makeitem
    res = list(collector._genfunctions(name, obj))
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:362: in _genfunctions
    self.ihook.pytest_generate_tests(metafunc=metafunc)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:745: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:339: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:334: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/vendored_packages/pluggy.py:614: in execute
    res = hook_impl.function(*args)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:113: in pytest_generate_tests
    metafunc.parametrize(*marker.args, **marker.kwargs)
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/python.py:781: in parametrize
    for x in argvalues]
/usr/local/lib/python2.7/dist-packages/pytest-3.2.0-py2.7.egg/_pytest/mark.py:66: in extract_from
    warnings.warn(MARK_PARAMETERSET_UNPACKING)
E   RemovedInPytest4Warning: Applying marks directly to parameters is deprecated, please use pytest.param(..., marks=...) instead.
E   For more details, see: https://docs.pytest.org/en/latest/parametrize.html
=============================== warnings summary ===============================
None
  [pytest] section in setup.cfg files is deprecated, use [tool:pytest] instead.

-- Docs: http://doc.pytest.org/en/latest/warnings.html
!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!
=============== 35 skipped, 1 warnings, 4 error in 23.28 seconds ===============

./.run_docker_tests.sh returned exit code 2

• Include a detailed description of the bug or suggestion
• pip list of the virtual environment you are using
• pytest and operating system versions
• Minimal example if possible

[nicoddemus]

I believe astropy and photutils both turn deprecation warnings into errors, correct? If so, then warnings raising an error is expected.

But I agree that we should not raise that warning during collection time otherwise it will stop the entire suite. Ideally we should issue the warning during testing instead. @RonnyPfannschmidt is that doable you think?

[bsipocz]

Oh, indeed that's it. I've just never seen an error due to this during collection time.

[RonnyPfannschmidt]

@nicoddemus it will be a mess to track this as of now and imho its correct behaviour if warnigns are turned into errors

imho we should have a mode where warnings are tracked always but dont error out pytest itself
but thats a different kind of solution

[nicoddemus]

it will be a mess to track this as of now and imho its correct behaviour if warnigns are turned into errors

Yeah, seems like a problem given that at that point we are only dealing with ParameterSet.

imho we should have a mode where warnings are tracked always but dont error out pytest itself
but thats a different kind of solution

Hmm not sure I would like to complicate things more; one of the advantages of just using the builtin warnings module is to keep things simple on our side. Users which turn warnings into errors are free to use the warnings filter system to filter out pytest-specific warnings if they want to.

Well given all this should we close this issue as "working as intended"?

[RonnyPfannschmidt]

then we should have a followup on pytest warnings in the documentation to help those users avoid the problems

i propose to make a ticket for introducing a pytest.PyTestWarning class anensuring we keep it in always warn mode

we might want to investigate into having a pytest option that considers warning as errors but ensures the actual erroring happens at the end ofthe test

[nicoddemus]

i propose to make a ticket for introducing a pytest.PyTestWarning class anensuring we keep it in always warn mode

That's #2452 I believe. 😉

we might want to investigate into having a pytest option that considers warning as errors but ensures the actual erroring happens at the end ofthe test

Perhaps, although I would like to avoid adding more complexity to the warnings system in general as I said earlier unless absolutely necessary.

[RonnyPfannschmidt]

good call, i added a note to #2452

im closing this one as works as intended, enhancement upcoming