Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added doctest encoding command line option #2101

Merged
merged 8 commits into from
Nov 30, 2016
6 changes: 5 additions & 1 deletion _pytest/doctest.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ def pytest_addoption(parser):
action="store_true", default=False,
help="ignore doctest ImportErrors",
dest="doctest_ignore_import_errors")
group.addoption("--doctest-encoding",
Copy link
Member

@nicoddemus nicoddemus Nov 29, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would be better off being a ini option instead:

  1. If one has to use this option, then usually all doctest.txt files will have the same encoding (if the developers want to keep their sanity that is) so it makes sense to configure that in pytest.ini;
  2. If really desired, one can use the -o option to override ini options in the command-line.
  3. It is usually not a good idea to require users to pass certain command-line options to make the test suite pass, and in which case the users don't pass the correct option the test-suite will fail in mysterious ways;

type=str.lower, default="utf-8",
help="choose the encoding to use for doctest files",
dest="doctestencoding")


def pytest_collect_file(path, parent):
Expand Down Expand Up @@ -171,7 +175,7 @@ def collect(self):

# inspired by doctest.testfile; ideally we would use it directly,
# but it doesn't support passing a custom checker
text = self.fspath.read()
text = self.fspath.read_text(self.config.getoption("doctestencoding"))
filename = str(self.fspath)
name = self.fspath.basename
globs = {'__name__': '__main__'}
Expand Down
4 changes: 2 additions & 2 deletions _pytest/pytester.py
Original file line number Diff line number Diff line change
Expand Up @@ -471,7 +471,7 @@ def chdir(self):
if not hasattr(self, '_olddir'):
self._olddir = old

def _makefile(self, ext, args, kwargs):
def _makefile(self, ext, args, kwargs, encoding="utf-8"):
items = list(kwargs.items())
if args:
source = py.builtin._totext("\n").join(
Expand All @@ -490,7 +490,7 @@ def my_totext(s, encoding="utf-8"):

source_unicode = "\n".join([my_totext(line) for line in source.lines])
source = py.builtin._totext(source_unicode)
content = source.strip().encode("utf-8") # + "\n"
content = source.strip().encode(encoding) # + "\n"
#content = content.rstrip() + "\n"
p.write(content, "wb")
if ret is None:
Expand Down
11 changes: 9 additions & 2 deletions doc/en/doctest.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,13 @@ can change the pattern by issuing::
on the command line. Since version ``2.9``, ``--doctest-glob``
can be given multiple times in the command-line.

You can specify the encoding that will be used for those doctest files
using the ``--doctest-encoding`` command line option::
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the docs to mention the doctest_encoding ini option instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, forgot about that.


pytest --doctest-encoding='ascii'

The default encoding is UTF-8.

You can also trigger running of doctests
from docstrings in all python modules (including regular
python test modules)::
Expand Down Expand Up @@ -52,9 +59,9 @@ then you can just invoke ``pytest`` without command line options::
platform linux -- Python 3.5.2, pytest-3.0.4, py-1.4.31, pluggy-0.4.0
rootdir: $REGENDOC_TMPDIR, inifile: pytest.ini
collected 1 items

mymodule.py .

======= 1 passed in 0.12 seconds ========

It is possible to use fixtures using the ``getfixture`` helper::
Expand Down
46 changes: 46 additions & 0 deletions testing/test_doctest.py
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,52 @@ def test_multiple_patterns(self, testdir):
'*1 passed*',
])

def test_encoding_ascii(self, testdir):
Copy link
Member

@nicoddemus nicoddemus Nov 29, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you could use @pytest.mark.parametrize on the tests to avoid repeating yourself. Something like (already incorporating my request to change to an ini option):

import pytest

@pytest.mark.parametrize('string, encoding', [
    (u'foo', 'ascii'),
    (u'üäö', 'latin1'),
    (u'üäö', 'utf-8'),
])
def test_encoding_latin1(testdir, string, encoding):
    """Test support for doctest_encoding option.
    """
    testdir.makeini("""
        [pytest]
        doctest_encoding={0}
    """.format(encoding))
    testdir._makefile(".txt", u"""
        >>> len(u'{0}')
        3
    """.format(string), encoding=encoding)

    result = testdir.runpytest()
    result.stdout.fnmatch_lines([
        '*1 passed*',
    ])

EDIT: fixed a comment left by accident

"""Test support for --doctest-encoding option.
"""
testdir._makefile(".txt", ["""
>>> 1
1
"""], {}, encoding='ascii')

result = testdir.runpytest("--doctest-encoding=ascii")

result.stdout.fnmatch_lines([
'*1 passed*',
])

def test_encoding_latin1(self, testdir):
"""Test support for --doctest-encoding option.
"""
testdir._makefile(".txt", ["""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at first glance i wonder if this shouldn't use unicode marked strings like u"..."

i believe the actual encoding can be a parametrize parameter, and we might want to add an exotic one just to be on the paranoid side

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would break the code for Python 3.0 I guess because iirc it didn't have the unicode prefix anymore.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wheerd python 3.3 added it back for more pleasant dual codebases

>>> 'üäö'
'üäö'
"""], {}, encoding='latin1')

result = testdir.runpytest("--doctest-encoding=latin1")

result.stdout.fnmatch_lines([
'*1 passed*',
])

def test_encoding_utf8(self, testdir):
"""Test support for --doctest-encoding option.
"""
testdir.maketxtfile("""
>>> 'üäö'
'üäö'
""")

result = testdir.runpytest()
result.stdout.fnmatch_lines([
'*1 passed*',
])

result = testdir.runpytest("--doctest-encoding=utf-8")
result.stdout.fnmatch_lines([
'*1 passed*',
])

def test_doctest_unexpected_exception(self, testdir):
testdir.maketxtfile("""
>>> i = 0
Expand Down