DOC/CI: run doctests on travis

[jorisvandenbossche]

Trying to see if we can check the doctests on travis. We were already running them, but at this point they were not failing travis if they failed.

Closes #19934
Closes #3439

[codecov]

Codecov Report

Merging #19952 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #19952   +/-   ##
=======================================
  Coverage   92.05%   92.05%           
=======================================
  Files         169      169           
  Lines       50709    50709           
=======================================
  Hits        46679    46679           
  Misses       4030     4030

Flag	Coverage Δ
#multiple	90.46% <ø> (ø)	⬆️
#single	42.25% <ø> (ø)	⬆️

Impacted Files	Coverage Δ
pandas/core/generic.py	96.44% <ø> (ø)	⬆️
pandas/core/frame.py	97.24% <ø> (ø)	⬆️
pandas/util/testing.py	85.85% <ø> (ø)	⬆️
pandas/core/config.py	87.17% <ø> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b5d81cf...25bc158. Read the comment docs.

[TomAugspurger]

There were failures https://travis-ci.org/pandas-dev/pandas/jobs/347769808#L6516, but they build still passed.

[jorisvandenbossche]

Hmm, that is because the script is still called itself from another script that does exit 0. Added a commit to try to progagate this, but not sure that will work with this 'wait' stuff.

On the other hand, it is maybe better to move this to a separate script anyhow.

[jorisvandenbossche]

OK, this is working now, in the sense that travis is failing because of the failing doctests.

The question is now how to list the ones we want to test:

• manually list the files we want to test (and for some files, manually list the functions we want to test)
• or run all the doctests, and manually (for now) list the ones we want to skip

Checking how it would look like with the second way

[jorisvandenbossche]

Hmm, so running the doctests on travis gives segfaults for some of them (don't have this locally):

...
pandas/core/frame.py::pandas.core.frame.DataFrame.count PASSED           [  4%]
pandas/core/frame.py::pandas.core.frame.DataFrame.cov PASSED             [  4%]
pandas/core/frame.py::pandas.core.frame.DataFrame.cummax *** Error in `/home/travis/miniconda3/envs/pandas/bin/python': munmap_chunk(): invalid pointer: 0x00000000027217b0 ***
ci/doctests.sh: line 57:  5790 Aborted                 (core dumped) pytest --doctest-modules --ignore=pandas/tests -v pandas

Somebody any idea?

[jorisvandenbossche]

So this still has a segfault when running the doctests.

I cannot reproduce it locally with pytest (pytest --doctest-modules --ignore=pandas/tests -v pandas/core/frame.py).
But I actually can reproduce it when calling it more manually

import pandas as pd
import numpy as np
import doctest
doctest.testmod(pd.core.frame, globs={'pd': pd, 'np': np})

[jorisvandenbossche]

@jreback @TomAugspurger @datapythonista @WillAyd any comments?

This enable doctests on travis, and for now runs tests for frame/generic/series.py.
I took the approach of skipping those docstrings that don't work yet. We can then do follow-up PRs gradually fixing the others + removing its skip, but this at least ensures the others keep working or newly added will work.

[jreback]

small comments. generally +1 on this. Is there a doc about this in contributing? (certainly can do in later PRs), in particular formatting gotchas? Also list a couple of gold-star examples (in our current docs) that show good contstruction of these tests, so we can simply point to them and say, like this!

Once this is merged we can look to add these on new PRs and pls open a tracking issue for existing doc-tests to add to the tester

[jreback]

are these just not-working tests?

[jorisvandenbossche]

Well, they are failing, and don't want to start fixing all of them here. Will remove or either skip the failing ones

[datapythonista]

I think it'll be great to have this, but it'll take a while until we have all doctests passing (it's in my TODO list, together with fixing all warnings from sphinx).

I'm happy to merge this PR as it is, and keep adding docstrings and files as we fix the tests.

[jorisvandenbossche]

Is there a doc about this in contributing? (certainly can do in later PRs), in particular formatting gotchas? Also list a couple of gold-star examples (in our current docs) that show good contstruction of these tests, so we can simply point to them and say, like this!

Yes, we have a very elaborate docstring guide: http://pandas.pydata.org/pandas-docs/stable/contributing_docstring.html#docstring, which also contains some tips to get the doctests passing

(it might need some better linking in the contributing docs though)

[jorisvandenbossche]

@datapythonista eventually we should rework the documentation guidelines to include more of sprint content like https://python-sprints.github.io/pandas/guide/pandas_pr.html, but for now added short section more clearly mentioning the validation script and that the doctests need to pass

[datapythonista]

Looks great, there is just a small typo.

[datapythonista]

s/to/the