Tests fail with pytest 3.5.0 DOCTESTS

[dotlambda]

After updating from pytest from 3.4.2 to 3.5.0, pytest-sugar's testsuite fails:

============================= test session starts ==============================
platform linux -- Python 3.6.5, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
rootdir: /build/pytest-sugar-0.9.1, inifile:
plugins: sugar-0.9.1
collected 27 items

test_sugar.py \x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m [ 11%]
\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m [ 22%]
\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m [ 33%]
\x1b[32m\u2713\x1b[0ms\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m [ 44%]
\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m [ 55%]
\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m [ 66%]
\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m [ 77%]
\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0ms\x1b[32m\u2713\x1b[0m [ 88%]
\x1b[32m\u2713\x1b[0ms\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[32m\u2713\x1b[0m\x1b[31m\u2a2f\x1b[0m\x1b[32m\u2713\x1b[0m [100%]

=================================== FAILURES ===================================
___________________ TestTerminalReporter.test_doctest_lineno ___________________

self = <test_sugar.TestTerminalReporter object at 0x7ffff0a768d0>
testdir = <Testdir local('/build/pytest-of-nixbld/pytest-0/test_doctest_lineno0')>

    def test_doctest_lineno(self, testdir):
        """ Test location reported for doctest-modules """

        testdir.makepyfile(
            """
                def foobar():
                    '''
                    >>> foobar()
                    '''
                    raise NotImplementedError
                """
        )
        result = testdir.runpytest('--force-sugar', '--doctest-module')

        assert result.ret == 1, result.stderr.str()
        result.stdout.fnmatch_lines([
            'UNEXPECTED EXCEPTION: NotImplementedError()',
            '*test_doctest_lineno.py:3: UnexpectedException',
            'Results*:',
>           '*-*test_doctest_lineno.py*:3*',
        ])

/build/pytest-sugar-0.9.1/test_sugar.py:514:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_pytest.pytester.LineMatcher object at 0x7ffff0ccb3c8>
lines2 = ['UNEXPECTED EXCEPTION: NotImplementedError()', '*test_doctest_lineno.py:3: UnexpectedException', 'Results*:', '*-*test_doctest_lineno.py*:3*']

    def fnmatch_lines(self, lines2):
        """Search captured text for matching lines using ``fnmatch.fnmatch``.

            The argument is a list of lines which have to match and can use glob
            wildcards.  If they do not match a pytest.fail() is called.  The
            matches and non-matches are also printed on stdout.

            """
>       self._match_lines(lines2, fnmatch, 'fnmatch')
E       Failed: nomatch: 'UNEXPECTED EXCEPTION: NotImplementedError()'
E           and: 'Test session starts (platform: linux, Python 3.6.5, pytest 3.5.0, pytest-sugar 0.9.1)'
E           and: 'rootdir: /build/pytest-of-nixbld/pytest-0/test_doctest_lineno0, inifile:'
E           and: 'plugins: sugar-0.9.1'
E           and: ''
E           and: ''
E           and: '\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015 [doctest] test_doctest_lineno.foobar \u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015'
E           and: '002 '
E           and: '003                 >>> foobar()'
E       exact match: 'UNEXPECTED EXCEPTION: NotImplementedError()'
E       nomatch: '*test_doctest_lineno.py:3: UnexpectedException'
E           and: 'Traceback (most recent call last):'
E           and: ''
E           and: '  File "/nix/store/33v7rg3qmwfj3smvhna5634n7x286wl6-python3-3.6.5/lib/python3.6/doctest.py", line 1330, in __run'
E           and: '    compileflags, 1), test.globs)'
E           and: ''
E           and: '  File "<doctest test_doctest_lineno.foobar[0]>", line 1, in <module>'
E           and: ''
E           and: '  File "/build/pytest-of-nixbld/pytest-0/test_doctest_lineno0/test_doctest_lineno.py", line 5, in foobar'
E           and: '    raise NotImplementedError'
E           and: ''
E           and: 'NotImplementedError'
E           and: ''
E       fnmatch: '*test_doctest_lineno.py:3: UnexpectedException'
E          with: '/build/pytest-of-nixbld/pytest-0/test_doctest_lineno0/test_doctest_lineno.py:3: UnexpectedException'
E       nomatch: 'Results*:'
E           and: '\r'
E           and: '\\r \\x1b[36m\\x1b[0mtest_doctest_lineno.py\\x1b[0m \\x1b[31m\u2a2f\\x1b[0m                                        \\x1b[31m100% \\x1b[0m\\x1b[40m\\x1b[31m\u2588\\x1b[0m\\x1b[40m\\x1b[31m\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\\x1b[0m'
E           and: ''
E       fnmatch: 'Results*:'
E          with: 'Results (0.01s):'
E       nomatch: '*-*test_doctest_lineno.py*:3*'
E           and: '\x1b[31m       1 failed\x1b[0m'
E           and: '         - \x1b[36m\x1b[0mtest_doctest_lineno.py\x1b[0m:2 \x1b[31m[doctest] test_doctest_lineno.foobar\x1b[0m'
E           and: ''
E       remains unmatched: '*-*test_doctest_lineno.py*:3*'

/nix/store/chvgi987hbqcw7zx0q8qzbk4d1dg9mnd-python3.6-pytest-3.5.0/lib/python3.6/site-packages/_pytest/pytester.py:1175: Failed
----------------------------- Captured stdout call -----------------------------
Test session starts (platform: linux, Python 3.6.5, pytest 3.5.0, pytest-sugar 0.9.1)\nrootdir: /build/pytest-of-nixbld/pytest-0/test_doctest_lineno0, inifile:\nplugins: sugar-0.9.1\n\n\n\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015 [doctest] test_doctest_lineno.foobar \u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\u2015\n002 \n003                 >>> foobar()\nUNEXPECTED EXCEPTION: NotImplementedError()\nTraceback (most recent call last):\n\n  File "/nix/store/33v7rg3qmwfj3smvhna5634n7x286wl6-python3-3.6.5/lib/python3.6/doctest.py", line 1330, in __run\n    compileflags, 1), test.globs)\n\n  File "<doctest test_doctest_lineno.foobar[0]>", line 1, in <module>\n\n  File "/build/pytest-of-nixbld/pytest-0/test_doctest_lineno0/test_doctest_lineno.py", line 5, in foobar\n    raise NotImplementedError\n\nNotImplementedError\n\n/build/pytest-of-nixbld/pytest-0/test_doctest_lineno0/test_doctest_lineno.py:3: UnexpectedException\n\r\n\r \x1b[36m\x1b[0mtest_doctest_lineno.py\x1b[0m \x1b[31m\u2a2f\x1b[0m                                        \x1b[31m100% \x1b[0m\x1b[40m\x1b[31m\u2588\x1b[0m\x1b[40m\x1b[31m\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\x1b[0m\n\nResults (0.01s):\n\x1b[31m       1 failed\x1b[0m\n         - \x1b[36m\x1b[0mtest_doctest_lineno.py\x1b[0m:2 \x1b[31m[doctest] test_doctest_lineno.foobar\x1b[0m
================ 1 failed, 76 passed, 3 skipped in 2.16 seconds ================

[EdwardBetts]

git bisect suggests this is the first commit that caused the test to fail: pytest-dev/pytest@fbc45be

[EdwardBetts]

The test is failing because the line number of the failing test changed from 3 to 2.

Results (0.02s):
       1 failed
         - test_doctest_lineno.py:3 [doctest] test_doctest_lineno.foobar

Results (0.03s):
       1 failed
         - test_doctest_lineno.py:2 [doctest] test_doctest_lineno.foobar

[EdwardBetts]

See pytest-dev/pytest#3463

[Teemu]

It seems it always points out to the beginning of doctests with the new pytest.

# Doctest example

def foobar(i):
    '''
    >>> foobar(2)
    2
    >>> foobar(3)
    10
    '''
    return i

Anyway, I disabled the test for now until this gets dealt with.