Migrate from nose to py.test

[kain88-de]

See: https://nose.readthedocs.io/en/latest/

Alternatives are py.test and nose2

http://nose2.readthedocs.io/en/latest/

http://pytest.org/latest/

I've only have additional experience with py.test and actually like it very much.

This should be done in one go as test-fixtures are handled differently in py.test for example.

[richardjgowers]

Yeah I used py.test a little too and it seems nice, I think @dotsdl is also a fan.

As a little test, I ran test_atomgroup with py.test and got 16 fails and 328 passes, so it looks like there will be a little effort required, but not a crazy amount.

[orbeckst]

What is numpy going to do?

Changing the whole test suite over to something else is going to be a nightmare. And there are all the plugins that we have for our custom checks that we would need to rewrite. What do we gain by doing this? What is the opportunity cost?

Parenthetically, I admit that I found py.test confusing and never figured out how to do generators (so I use the nose syntax that is somewhat supported) but I am sure that someone who spent a bit more time with py.test will be able to convince me that this is a fine piece of software.

[kain88-de]

This is nothing we have to do quick. We can also wait to see what others are doing. Currently everything works so no hurry.

[mnmelo]

@dotsdl also commented (personal communication) that he had good experience with py.test (as it allows for linter plugins and the like).

Personally, I'm happy to change, but agree it'll be a hassle. As for plugins, maybe some of the functionality already exists and saves us some trouble.

[orbeckst]

With all the hoozah! for py.test, I'd say we start thinking about transitioning to py.test but monitor what numpy is doing, given that we extensively rely on their array comparisons. Or does py.test already include equivalent tests?

[kain88-de]

Nope I can ask on the numpy dev list

[dotsdl]

+1 for py.test, and most things we currently do should "just work" without changing anything. To eliminate nosetests as a dependency we will mainly need to replace decorators with py.test equivalents.

[jbarnoud]

Just for the record, I ran py.test with the xdist plug-in for parallelization and I got:

651 failed, 2781 passed, 5 skipped, 46 error in 127.21 seconds

I find the error messages very verbose, and I do not like that the traceback is not displayed. Yet, I guess it can be configures.

I do not like either how the tests coming from test generators are displayed in verbose mode. Instead of the very informative list of arguments passed to the test, py.test only displays the index of the test in the generator. Once again, it is hopefully configurable.

[kain88-de]

test generators work different in py.test, you decorate the function with @pytest.mark.parametrize This is all stuff that would need to be ported. For example to test multiple inputs for a function even you write:

@pytest.mark.parametrize('input, expected', [(1, False) , (2, True), (3, False)])
def test_even(input, expected):
     assert even(input) == expected

That is the most basic approach. With fixtures you can do even more advanced things and even completely change the default behavior.

Tracebacks are configurable, they are shown in single-process mode and by default more verbose then nosetests.

[kain88-de]

numpy is doing, given that we extensively rely on their array comparisons.

I just had a look into the array comparison code from numpy. They only use numpy internal functions for that. No need for us to pay any attention to them since it would affect our own choice of testing frame work. They do use nose in assert_raise and raises but these would be replaced with pytest equivalents.

I also have the feeling that @dotsdl is anxious to move away from nose.

[orbeckst]

On 8 Aug, 2016, at 14:25, Max Linke notifications@github.com wrote:

numpy is doing, given that we extensively rely on their array comparisons.

I just had a look into the array comparison code from numpy. They only use numpy internal functions for that. No need for us to pay any attention to them since it would affect our own choice of testing frame work.

So we would write our own array_equal, array_almost_equal etc?

That would make the transition easier… just import from our own testing helper module instead of numpy.testing

They do use nose in assert_raise and raises but these would be replaced with pytest equivalents.

Seems easy enough (famous last words).

[dotsdl]

So we would write our own array_equal, array_almost_equal etc?

@orbeckst, no, we could continue using the numpy functions as we already are. They work just fine with py.test as written as far as I can tell. The only things that change are the nose things we're using.

[orbeckst]

So this looks feasible, doesn't it?

So I would suggest, in preparation for the eventual move to py.tests, do not use nose-test specific constructs in tests anymore. Which ones are the ones to avoid?

[kain88-de]

yes it's feasible. One construct we should avoid are test-generators that work with yield. There are much better alternatives available in pytest.

A short list at the top post of test refactors that will make the transition easier would be helpful. @dotsdl do you where this could be done?

On the side of actually doing this transition. This is going to be a huge change. Does anyone have a plan how to best do this? At the top of my head there are 3 things we should check/do.

• port our plugins if necessary
• do small refactorings that makes tests nose/pytest compatible
• make a list of tests that won't run under pytest right now.

[orbeckst]

On 9 Aug, 2016, at 02:11, Max Linke notifications@github.com wrote:

yes it's feasible. One construct we should avoid are test-generators that work with yield. There are much better alternatives available in pytest.

Can you give an example for how to transform one of the tests that is currently written with yield? That would make it easier.

(Also, doesn't py.test happily work with generators? – http://doc.pytest.org/en/latest/nose.html#supported-nose-idioms)

[kain88-de]

As you link generators are supported nose idioms and pytest has much better alternatives with pytest.mark.parametrize.

But your link already gives one item. All tests classes should inherit from unittest.TestCase for setup to work.

[richardjgowers]

@kain88-de the generators don't work inside a TestCase class, see the last line here:

http://nose.readthedocs.io/en/latest/writing_tests.html#test-generators

But yeah, rewriting the test generators will be required for pytest

[richardjgowers]

Ok I've changed this to a migrate to py.test issue, nobody seemed to want/know about nose2 and py.test seems to do everything we need.

[vedantrathore]

I would like to work on this, I have some experience in writing tests with py.test and would love to contribute to mdanalysis repository. Can anyone guide me on where to start with this?

[richardjgowers]

Before any coding on this starts, I think we need to compile a list of the nosetests features we currently use that are incompatible with pytest. (the plugins and test generators are two of the top of my head) There might be a migrating help page out there if we're lucky..

…

On Sat, 4 Feb 2017, 8:40 p.m. Vedant Rathore, ***@***.***> wrote: I would like to work on this, I have some experience in writing tests with py.test and would love to contribute to mdanalysis repository. Can anyone guide me on where to start with this? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#884 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AI0jB8h-JU7rxmZRNT-mah9Uvlk_9gvwks5rZOHSgaJpZM4JChX_> .

[utkbansal]

This might be useful. http://docs.pytest.org/en/latest/nose.html#nosestyle

[kain88-de]

I think the best start here is to create a wiki-page where we document several things

• the plugins we have and how they could be ported to py.test
• how many tests pass now with pytests
• what fails
• what idoms with examples in our testsuite fail
• where can we replace nose with pytest idoms

After this initial research is complete we can look into further work.

[vedantrathore]

@kain88-de @richardjgowers I have created a wiki page. If there is anything you'd like to add or edit do tell me.

Link to wiki page : here
Link to google group discussion : here

[jbarnoud]

@utkbansal Any update on the performances of the memory plugin?

[utkbansal]

@jbarnoud Yes, I'm working on one alternative. Should I open a separate issue for the same and continue discussions on it?

[jbarnoud]

On 05/11/2017 06:54 PM, Utkarsh Bansal wrote: @jbarnoud <https://github.com/jbarnoud> Yes, I'm working on one alternative. Should I open a separate issue for the same and continue discussions on it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#884 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUWuqw7_kgBkuIcue0u6sQ5qAUI2AeGks5r4z1CgaJpZM4JChX_>.

Yes please.

[orbeckst]

Awesome, congratulations @utkbansal to this milestone! Well done.