Fail docs build if there are doctest errors

[ericphanson]

I think Documenter should (at least have the option to) fail makedocs if there are doctest errors, otherwise they need to either be tested elsewhere or they slip through. In my opinion if you’re using doctests you generally want them to act like tests (ie red X in CI not green check).

I don’t really think they should be tested as part of the usual tests since those are run on multiple Julia versions leading to annoying failures from printing changes, whereas the docs build is usually done on a single version of Julia.

I know strict=true does this but it also requires a lot of other things. Perhaps strict=:doctest could be an intermediate level which just requires doctests to pass?

I’m hoping to make seeing and fixing doctests more ergonomic (you’ll get a tiny PR to fix the logging statement so doctest @errors are more actionable downstream edit: #1687) but I think easily failing on them is an important step.

[mortenpi]

Yep, I think more flexibility with the strict keyword would be perfectly fine. There is some earlier discussion here: #1504 (comment)

[ericphanson]

Ok great, I’ll try to make a PR soon. Would it be ok to just start with doctests or should such a PR cover other subsets of what strict does?

[mortenpi]

I think starting with just doctests is fine. But the design should keep in mind that we probably want to add other options in the future.