Allow reporting of skipped optional tests

[roed314]

From a discussion on sage-combinat-devel:

Optionally print extra information after doctesting: how many optional doctests were skipped, and how they were labeled ("skipped 5 doctests marked "optional -- bug").

See also #14276.

Apply attachment: 13278.2.patch

Apply attachment: 13278_script.patch to the script repo

Depends on #12415
Depends on #14331

Component: doctest framework

Author: David Roe

Reviewer: Jeroen Demeyer

Merged: sage-5.9.beta2

Issue created by migration from https://trac.sagemath.org/ticket/13278

[roed314]

Author: David Roe

[roed314]

comment:1

It was necessary to do some cleanup in sage.doctest.sources.DocTestSource: now it just stores an options object rather than separate options that it cares about.

There will be some double counting if optional tests are marked with multiple tags.

For ReST files with inner sources we'll also miss the optional tests in the inner sources.

Both issues seem minor enough that they're not worth the extra code complexity.

[roed314]

Attachment: 13278_script.patch.gz

[jdemeyer]

comment:4

Could you add a test to sage/doctest/test.py to test a complete sage -t --show-skipped run (also checking that optional tests which are run are not reported).

And I would enable --show-skipped automatically when --verbose is given.

[roed314]

Attachment: 13278.patch.gz

[roed314]

comment:5

Replying to @jdemeyer:

Could you add a test to sage/doctest/test.py to test a complete sage -t --show-skipped run (also checking that optional tests which are run are not reported).

And I would enable --show-skipped automatically when --verbose is given.

Done.

[jdemeyer]

comment:6

Why still use useabspath here (but remove it from the documentation)?

def __init__(self, path, options, useabspath=False)

especially since you replace

if self.useabspath: 

by

if self.options.abspath: 

[jdemeyer]

comment:7

And why isn't --long displayed in the output here:

sage: subprocess.call(["sage", "-t", "--long", "--show-skipped", "--optional=sage,gap", "show_skipped.rst"], cwd=tests_dir) 
Running doctests ... 
Doctesting 1 file. 
sage -t show_skipped.rst 

[jdemeyer]

comment:8

Line 405, you could replace

untested = optionals["not tested"] + optionals["not implemented"]

by

untested = optionals.pop("not tested") + optionals.pop("not implemented")

Then you don't need to check for "not tested" or "not implemented" further.

And I think the translation "known bug" -> "bug" should better be moved inside the parse_optional_tags() function itself.

[jdemeyer]

Reviewer: Jeroen Demeyer

[roed314]

comment:10

Fixed.

[jdemeyer]

comment:11

Attachment: 13278_review.patch.gz

**********************************************************************
File "devel/sage/sage/doctest/test.py", line 240, in sage.doctest.test
Failed example:
    subprocess.call(["sage", "-t", "--long", "--show-skipped", "--optional=sage,gap", "show_skipped.rst"], cwd=tests_dir)
Expected:
    Running doctests ...
    Doctesting 1 file.
    sage -t --long show_skipped.rst
        1 unlabeled test not run
        2 tests not run due to known bugs
        1 other test skipped
        [3 tests, ... s]
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    ...
    0
Got:
    Running doctests with ID 2013-03-27-00-20-38-1d0ae1fc.
    Doctesting 1 file.
    sage -t --long show_skipped.rst
        1 unlabeled test not run
        2 tests not run due to known bugs
        0 not tested tests not run
        1 other test skipped
        [3 tests, 0.0 s]
    ----------------------------------------------------------------------
    All tests passed!
    ----------------------------------------------------------------------
    Total time for all tests: 0.0 seconds
        cpu time: 0.0 seconds
        cumulative wall time: 0.0 seconds
    0
**********************************************************************

I'll have a look.

[jdemeyer]

Changed dependencies from #12415 to #12415, #14331

[jdemeyer]

Attachment: 13278_review2.patch.gz

[jdemeyer]

comment:13

attachment: 13278_review2.patch (which replaces attachment: 13278_review.patch) needs review.

[roed314]

comment:14

Works for me.

[jdemeyer]

comment:15

This conflicts massively with #14331, rebasing... :-(

[jdemeyer]

comment:16

Attachment: 13278.2.patch.gz

Rebased and squashed the two patches.

[jdemeyer]

Merged: sage-5.9.beta2