Doctest manual pages too with doctest?

[mortenpi]

#774 brought the doctest function, but it can only check the doctests in docstrings at the moment. It would be worth looking into whether there are any issues with trying to doctest manual pages too in this lightweight manner.

[mortenpi]

@davidanthoff wrote in #1053 (comment):

I.e. ideally I don't really want to deal with any subtleties at all, but would just like to call one function, and have it sort out everything.

That would be ideal. So, there is the doctest function which gets you half-way there at the moment. You can run doctest(MyPackage) (MyPackage being a Module) and it will test all the doctests in docstrings.

What is missing, though, is the ability to run doctests on manual pages in a simple way. We could have some automagic behaviour:

• By default, also doctest all .md files in joinpath(pathof(MyPackage), "..", "docs", "src"))
• If the directory is missing, throw an error. You can get around this by passing e.g. doctest(...; manual = false), which disables doctesting of .md pages.
• The manual keyword could also be a path, in case your doctests are in a non-standard directory..
• Also, have it potentially be a list of paths, for cases when you have some other collection of .md files with doctests?

I was slightly worried that there might be some state in make.jl that will affect the doctests.. but now that I've thought about it a bit more, I think it would be fine.

The only thing to be aware of is when using DocMeta to attach :DocTestSetups to modules, this needs to be done in both runtests.jl and make.jl (probably by having e.g. a test/doctestsetup.jl file, which needs to be included in both).

Instead of doing everything in a single doctest call, we could also provide separate functions for doctesting docstrings and doctests, e.g.:

@test doctest(MyPackage)
@test doctest_pages("../docs/src")

But probably, if you do doctest(MyPackage), you want the manual pages to be tested too.

[davidanthoff]

Sounds good!

Not in the short term, but would it make sense to move some of the info that is in make.jl into a toml config file? For example which pages to include in the docs? That might make it easier for something like doctest to pick that info up?