Generalize doctest() API

CHANGELOG.md

@@ -14,7 +14,7 @@
 
 * ![Feature][badge-feature] `makedocs` now accepts the `doctest = :only` keyword, which allows doctests to be run while most other build steps, such as rendering, are skipped. This makes it more feasible to run doctests as part of the test suite (see the manual for more information). ([#198][github-198], [#535][github-535], [#756][github-756], [#774][github-774])
 
-* ![Feature][badge-feature] Documenter now exports the `doctest` function, which verifies the doctests in all the docstrings of a given module. This can be used to verify docstring doctests as part of test suite. ([#198][github-198], [#535][github-535], [#756][github-756], [#774][github-774])
+* ![Feature][badge-feature] Documenter now exports the `doctest` function, which verifies the doctests in all the docstrings of a given module. This can be used to verify docstring doctests as part of test suite, or to fix doctests right in the REPL. ([#198][github-198], [#535][github-535], [#756][github-756], [#774][github-774], [#1054][github-1054])
 
 * ![Feature][badge-feature] `makedocs` now accepts the `expandfirst` argument, which allows specifying a set of pages that should be evaluated before others. ([#1027][github-1027], [#1029][github-1029])
 
@@ -30,6 +30,10 @@
 
 * ![Experimental][badge-experimental] ![Feature][badge-feature] The current working directory when evaluating `@repl` and `@example` blocks can now be set to a fixed directory by passing the `workdir` keyword to `makedocs`. _The new keyword and its behaviour are experimental and not part of the public API._ ([#1013][github-1013], [#1025][github-1025])
 
+## Version `v0.22.5`
+
+* ![Maintenance][badge-maintenance] Fix a test dependency problem revealed by a bugfix in Julia / Pkg. ([#1037][github-1037])
+
 ## Version `v0.22.4`
 
 * ![Bugfix][badge-bugfix] Documenter no longer crashes if the build includes doctests from docstrings that are defined in files that do not exist on the file system (e.g. if a Julia Base docstring is included when running a non-source Julia build). ([#1002][github-1002])
@@ -358,7 +362,9 @@
 [github-1029]: https://github.com/JuliaDocs/Documenter.jl/pull/1029
 [github-1031]: https://github.com/JuliaDocs/Documenter.jl/issues/1031
 [github-1034]: https://github.com/JuliaDocs/Documenter.jl/pull/1034
+[github-1037]: https://github.com/JuliaDocs/Documenter.jl/pull/1037
 [github-1047]: https://github.com/JuliaDocs/Documenter.jl/pull/1047
+[github-1054]: https://github.com/JuliaDocs/Documenter.jl/pull/1054
 
 [documenterlatex]: https://github.com/JuliaDocs/DocumenterLaTeX.jl
 [documentermarkdown]: https://github.com/JuliaDocs/DocumenterMarkdown.jl
@@ -372,6 +378,7 @@
 [badge-bugfix]: https://img.shields.io/badge/bugfix-purple.svg
 [badge-security]: https://img.shields.io/badge/security-black.svg
 [badge-experimental]: https://img.shields.io/badge/experimental-lightgrey.svg
+[badge-maintenance]: https://img.shields.io/badge/maintenance-gray.svg
 
 <!--
 # Badges
@@ -383,4 +390,5 @@
 ![Bugfix][badge-bugfix]
 ![Security][badge-security]
 ![Experimental][badge-experimental]
+![Maintenance][badge-maintenance]
 -->

Project.toml

@@ -11,6 +11,7 @@ LibGit2 = "76f85450-5226-5b5a-8eaa-529ad045b433"
 Logging = "56ddb016-857b-54e1-b83d-db4d58db5568"
 Markdown = "d6f4376e-aef5-505a-96c1-9c027394607a"
 REPL = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"
+Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 Unicode = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"
 
 [compat]
@@ -22,7 +23,6 @@ julia = "1"
 DocumenterMarkdown = "997ab1e6-3595-5248-9280-8efb232c3433"
 DocumenterTools = "35a29f4d-8980-5a13-9543-d66fff28ecb8"
 Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
-Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
 [targets]
-test = ["Test", "Random", "DocumenterMarkdown", "DocumenterTools"]
+test = ["Random", "DocumenterMarkdown", "DocumenterTools"]

docs/src/man/doctests.md

@@ -311,81 +311,66 @@ julia> @time [1,2,3,4]
     keyword argument are all applied to each doctest.
 
 
-## Doctesting Without Building the Docs
+## Doctesting as Part of Testing
 
-Documenter has a few ways to verify the doctests without having to run a potentially
-expensive full documentation build.
+Documenter provides the [`doctest`](@ref) function which can be used to verify all doctests
+independently of manual builds. It behaves like a `@testset`, so it will return a testset
+if all the tests pass or throw a `TestSetException` if it does not.
 
-### Doctesting docstrings only
-
-An option for doctesting just the docstrings of a particular module (and all its submodules)
-is to use the [`doctest`](@ref) function. It takes a list of modules as an argument and runs
-all the doctests in all the docstrings it finds. This can be handy for quick tests when
-writing docstrings of a package.
-
-[`doctest`](@ref) will return `true` or `false`, depending on whether the doctests pass or
-not, making it easy to include a doctest of all the docstrings in the package test suite:
-
-```julia
-using MyPackage, Documenter, Test
-@test doctest([MyPackage])
-```
-
-Note that you still need to make sure that all the necessary [Module-level metadata](@ref)
-for the doctests is set up before [`doctest`](@ref) is called.
-
-### Doctesting without a full build
-
-An alternative, which also runs doctests on the manual pages, but still skips most other
-build steps, is to pass `doctest = :only` to [`makedocs`](@ref).
-
-This also makes it more practical to include doctests as part of the normal test suite of a
-package. One option to set it up is to make the `doctest` keyword depend on command line
-arguments passed to the `make.jl` script:
+For example, it can be used to verify doctests as part of the normal test suite by having
+e.g. the following in `runtests.jl`:
 
 ```julia
-makedocs(...,
-    doctest = ("doctest-only" in ARGS) ? :only : true
-)
+using Test, Documenter, MyPackage
+doctest(MyPackage)
 ```
 
-Now, the `make.jl` script can be run on the command line as `julia docs/make.jl
-doctest-only` and it will only run the doctests. On doctest failure, the `makedocs` throws
-an error and `julia` exits with a non-zero exit code.
+By default, it will also attempt to verify all the doctests on manual `.md` files, which it
+assumes are located under `docs/src`. This can be configured or disabled with the `manual`
+keyword (see [`doctest`](@ref) for more information).
 
-For running the doctests as part of the standard test suite, the  `docs/make.jl` can simply
-be `include`d in the `test/runtest.jl` file:
+It can also be included in another testset, in which case it gets incorporated into the
+parent testset. So, as another example, to test a package that does have separate manual
+pages, just docstrings, and also collects all the tests into a single testset, the
+`runtests.jl` might look as follows:
 
 ```julia
-push!(ARGS, "doctest-only")
-include(joinpath(@__DIR__, "..", "docs", "make.jl"))
+using Test, Documenter, MyPackage
+@testset "MyPackage" begin
+    ... # other tests & testsets
+    doctest(MyPackage; manual = false)
+    ... # other tests & testsets
+end
 ```
 
-The `push!` to `ARGS` emulates the passing of the `doctest-only` command line argument.
-
-Note that, for this to work, you need to add Documenter and all the other packages that get
-loaded in `make.jl`, or in the doctest, as test dependencies.
+Note that you still need to make sure that all the necessary [Module-level metadata](@ref)
+for the doctests is set up before [`doctest`](@ref) is called. Also, you need to add
+Documenter and all the other packages you are loading in the doctests as test dependencies.
 
 
 ## Fixing Outdated Doctests
 
-To fix outdated doctests, the `doctest` flag to [`makedocs`](@ref) can be set to
-`doctest = :fix`. This will run the doctests, and overwrite the old results with
-the new output.
+To fix outdated doctests, the [`doctest`](@ref) function can be called with `fix = true`.
+This will run the doctests, and overwrite the old results with the new output. This can be
+done just in the REPL:
 
-!!! note
+```julia-repl
+julia> using Documenter, MyPackage
+julia> doctest(MyPackage, fix=true)
+```
 
-    The `:fix` option currently only works for LF line endings (`'\n'`)
+Alternatively, you can also pass the `doctest = :fix` keyword to [`makedocs`](@ref).
 
 !!! note
 
-    It is recommended to `git commit` any code changes before running the doctest fixing.
-    That way it is simple to restore to the previous state if the fixing goes wrong.
+    * The `:fix` option currently only works for LF line endings (`'\n'`)
 
-!!! note
+    * It is recommended to `git commit` any code changes before running the doctest fixing.
+      That way it is simple to restore to the previous state if the fixing goes wrong.
 
-    There are some corner cases where the fixing algorithm may replace the wrong code snippet.
-    It is therefore recommended to manually inspect the result of the fixing before committing.
+    * There are some corner cases where the fixing algorithm may replace the wrong code
+      snippet. It is therefore recommended to manually inspect the result of the fixing
+      before committing.
 
 
 ## Skipping Doctests

docs/src/showcase.md

@@ -0,0 +1,33 @@
+# Hidden showcase page
+
+Currently exists just so that there would be doctests to run in manual pages of Documenter's
+manual. This page does not show up in navigation.
+
+```jldoctest
+julia> 2 + 2
+4
+```
+
+The following doctests needs doctestsetup:
+
+```jldoctest; setup=:(using Documenter)
+julia> Documenter.Utilities.splitexpr(:(Foo.Bar.baz))
+(:(Foo.Bar), :(:baz))
+```
+
+Let's also try `@meta` blocks:
+
+```@meta
+DocTestSetup = quote
+  f(x) = x^2
+end
+```
+
+```jldoctest
+julia> f(2)
+4
+```
+
+```@meta
+DocTestSetup = nothing
+```

src/Documenter.jl

@@ -13,6 +13,7 @@ $(EXPORTS)
 """
 module Documenter
 
+using Test: @testset, @test
 using DocStringExtensions
 import Base64: base64decode
 
@@ -826,29 +827,95 @@ function getenv(regex::Regex)
 end
 
 """
-    doctest(modules::AbstractVector{Module}) -> Bool
+    doctest(package::Module; kwargs...)
 
-Runs all the doctests in the given modules. Returns `true` if the doctesting was successful
-and false if any error occurred.
+Convenience method that runs and checks all the doctests for a given Julia package.
+`package` must be the `Module` object corresponding to the top-level module of the package.
+Behaves like an `@testset` call, returning a testset if all the doctests are successful or
+throwing a `TestSetException` if there are any failures. Can be included in other testsets.
+
+# Keywords
+
+**`manual`** controls how manual pages are handled. By default (`manual = true`), `doctest`
+assumes that manual pages are located under `docs/src`. If that is not the case, the
+`manual` keyword argument can be passed to specify the directory. Setting `manual = false`
+will skip doctesting of manual pages altogether.
+
+Additional keywords are passed on to the main [`doctest`](@ref) method.
 """
-function doctest(modules::AbstractVector{Module})
-    dir = mktempdir()
-    @debug "Doctesting in temporary directory: $(dir)" modules
-    mkdir(joinpath(dir, "src"))
-    r = try
-        makedocs(
-            root = dir,
-            sitename = "",
-            doctest = :only,
-            modules = modules,
-        )
-        true
-    catch e
-        @error "Doctesting failed" e
-        false
+function doctest(package::Module; manual=true, testset=nothing, kwargs...)
+    if pathof(package) === nothing
+        throw(ArgumentError("$(package) is not a top-level package module."))
+    end
+    source = nothing
+    if manual === true
+         source = normpath(joinpath(dirname(pathof(package)), "..", "docs", "src"))
+         isdir(source) || throw(ArgumentError("""
+         Package $(package) does not have a documentation source directory at standard location.
+         Searched at: $(source)
+         If ...
+         """))
+    end
+    testset = (testset === nothing) ? "Doctests: $(package)" : testset
+    doctest(source, [package]; testset=testset, kwargs...)
+end
+
+"""
+    doctest(source, modules; kwargs...)
+
+Runs all the doctests in the given modules and on manual pages under the `source` directory.
+Behaves like an `@testset` call, returning a testset if all the doctests are successful or
+throwing a `TestSetException` if there are any failures. Can be included in other testsets.
+
+The manual pages are searched recursively in subdirectories of `source` too. Doctesting of
+manual pages can be disabled if `source` is set to `nothing`.
+
+# Keywords
+
+**`testset`** specifies the name of test testset (default `Doctests`).
+
+**`fix`**, if set to `true`, updates all the doctests that fail with the correct output
+(default `false`).
+
+!!! warning
+    When running `doctest(...; fix=true)`, Documenter will modify the Markdown and Julia
+    source files. It is strongly recommended that you only run it on packages in Pkg's
+    develop mode and commit any staged changes. You should also review all the changes made
+    by `doctest` before committing them, as there may be edge cases when the automatic
+    fixing fails.
+"""
+function doctest(
+        source::Union{AbstractString,Nothing},
+        modules::AbstractVector{Module};
+        fix = false,
+        testset = "Doctests",
+    )
+    function all_doctests()
+        dir = mktempdir()
+        try
+            @debug "Doctesting in temporary directory: $(dir)" modules
+            if source === nothing
+                source = joinpath(dir, "src")
+                mkdir(source)
+            end
+            makedocs(
+                root = dir,
+                source = source,
+                sitename = "",
+                doctest = fix ? :fix : :only,
+                modules = modules,
+            )
+            true
+        catch e
+            @error "Doctesting failed" e
+            false
+        finally
+            rm(dir; recursive=true)
+        end
+    end
+    @testset "$testset" begin
+        @test all_doctests()
     end
-    rm(dir; recursive=true)
-    return r
 end
 
 end # module