Skip to content

Commit

Permalink
Update doctest API
Browse files Browse the repository at this point in the history 
doctest() can now also doctest manual pages, which makes it easier to
include doctesting as part of the test suite, and behaves like a
testset.
  • Loading branch information
@mortenpi
mortenpi committed Jul 7, 2019
1 parent 807cce6 commit 258e0ca
Show file tree
Hide file tree
Showing 8 changed files with 219 additions and 100 deletions.
3 changes: 2 additions & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

* ![Feature][badge-feature] `makedocs` now accepts the `doctest = :only` keyword, which allows doctests to be run while most other build steps, such as rendering, are skipped. This makes it more feasible to run doctests as part of the test suite (see the manual for more information). ([#198][github-198], [#535][github-535], [#756][github-756], [#774][github-774])

* ![Feature][badge-feature] Documenter now exports the `doctest` function, which verifies the doctests in all the docstrings of a given module. This can be used to verify docstring doctests as part of test suite. ([#198][github-198], [#535][github-535], [#756][github-756], [#774][github-774])
* ![Feature][badge-feature] Documenter now exports the `doctest` function, which verifies the doctests in all the docstrings of a given module. This can be used to verify docstring doctests as part of test suite, or to fix doctests right in the REPL. ([#198][github-198], [#535][github-535], [#756][github-756], [#774][github-774], [#1054][github-1054])

* ![Feature][badge-feature] `makedocs` now accepts the `expandfirst` argument, which allows specifying a set of pages that should be evaluated before others. ([#1027][github-1027], [#1029][github-1029])

Expand Down Expand Up @@ -364,6 +364,7 @@
[github-1034]: https://github.com/JuliaDocs/Documenter.jl/pull/1034
[github-1037]: https://github.com/JuliaDocs/Documenter.jl/pull/1037
[github-1047]: https://github.com/JuliaDocs/Documenter.jl/pull/1047
[github-1054]: https://github.com/JuliaDocs/Documenter.jl/pull/1054

[documenterlatex]: https://github.com/JuliaDocs/DocumenterLaTeX.jl
[documentermarkdown]: https://github.com/JuliaDocs/DocumenterMarkdown.jl
Expand Down
4 changes: 2 additions & 2 deletions Project.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ LibGit2 = "76f85450-5226-5b5a-8eaa-529ad045b433"
Logging = "56ddb016-857b-54e1-b83d-db4d58db5568"
Markdown = "d6f4376e-aef5-505a-96c1-9c027394607a"
REPL = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
Unicode = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"

[compat]
Expand All @@ -22,7 +23,6 @@ julia = "1"
DocumenterMarkdown = "997ab1e6-3595-5248-9280-8efb232c3433"
DocumenterTools = "35a29f4d-8980-5a13-9543-d66fff28ecb8"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["Test", "Random", "DocumenterMarkdown", "DocumenterTools"]
test = ["Random", "DocumenterMarkdown", "DocumenterTools"]
91 changes: 38 additions & 53 deletions docs/src/man/doctests.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -311,81 +311,66 @@ julia> @time [1,2,3,4]
keyword argument are all applied to each doctest.


## Doctesting Without Building the Docs
## Doctesting as Part of Testing

Documenter has a few ways to verify the doctests without having to run a potentially
expensive full documentation build.
Documenter provides the [`doctest`](@ref) function which can be used to verify all doctests
independently of manual builds. It behaves like a `@testset`, so it will return a testset
if all the tests pass or throw a `TestSetException` if it does not.

### Doctesting docstrings only

An option for doctesting just the docstrings of a particular module (and all its submodules)
is to use the [`doctest`](@ref) function. It takes a list of modules as an argument and runs
all the doctests in all the docstrings it finds. This can be handy for quick tests when
writing docstrings of a package.

[`doctest`](@ref) will return `true` or `false`, depending on whether the doctests pass or
not, making it easy to include a doctest of all the docstrings in the package test suite:

```julia
using MyPackage, Documenter, Test
@test doctest([MyPackage])
```

Note that you still need to make sure that all the necessary [Module-level metadata](@ref)
for the doctests is set up before [`doctest`](@ref) is called.

### Doctesting without a full build

An alternative, which also runs doctests on the manual pages, but still skips most other
build steps, is to pass `doctest = :only` to [`makedocs`](@ref).

This also makes it more practical to include doctests as part of the normal test suite of a
package. One option to set it up is to make the `doctest` keyword depend on command line
arguments passed to the `make.jl` script:
For example, it can be used to verify doctests as part of the normal test suite by having
e.g. the following in `runtests.jl`:

```julia
makedocs(...,
doctest = ("doctest-only" in ARGS) ? :only : true
)
using Test, Documenter, MyPackage
doctest(MyPackage)
```

Now, the `make.jl` script can be run on the command line as `julia docs/make.jl
doctest-only` and it will only run the doctests. On doctest failure, the `makedocs` throws
an error and `julia` exits with a non-zero exit code.
By default, it will also attempt to verify all the doctests on manual `.md` files, which it
assumes are located under `docs/src`. This can be configured or disabled with the `manual`
keyword (see [`doctest`](@ref) for more information).

For running the doctests as part of the standard test suite, the `docs/make.jl` can simply
be `include`d in the `test/runtest.jl` file:
It can also be included in another testset, in which case it gets incorporated into the
parent testset. So, as another example, to test a package that does have separate manual
pages, just docstrings, and also collects all the tests into a single testset, the
`runtests.jl` might look as follows:

```julia
push!(ARGS, "doctest-only")
include(joinpath(@__DIR__, "..", "docs", "make.jl"))
using Test, Documenter, MyPackage
@testset "MyPackage" begin
... # other tests & testsets
doctest(MyPackage; manual = false)
... # other tests & testsets
end
```

The `push!` to `ARGS` emulates the passing of the `doctest-only` command line argument.

Note that, for this to work, you need to add Documenter and all the other packages that get
loaded in `make.jl`, or in the doctest, as test dependencies.
Note that you still need to make sure that all the necessary [Module-level metadata](@ref)
for the doctests is set up before [`doctest`](@ref) is called. Also, you need to add
Documenter and all the other packages you are loading in the doctests as test dependencies.


## Fixing Outdated Doctests

To fix outdated doctests, the `doctest` flag to [`makedocs`](@ref) can be set to
`doctest = :fix`. This will run the doctests, and overwrite the old results with
the new output.
To fix outdated doctests, the [`doctest`](@ref) function can be called with `fix = true`.
This will run the doctests, and overwrite the old results with the new output. This can be
done just in the REPL:

!!! note
```julia-repl
julia> using Documenter, MyPackage
julia> doctest(MyPackage, fix=true)
```

The `:fix` option currently only works for LF line endings (`'\n'`)
Alternatively, you can also pass the `doctest = :fix` keyword to [`makedocs`](@ref).

!!! note

It is recommended to `git commit` any code changes before running the doctest fixing.
That way it is simple to restore to the previous state if the fixing goes wrong.
* The `:fix` option currently only works for LF line endings (`'\n'`)

!!! note
* It is recommended to `git commit` any code changes before running the doctest fixing.
That way it is simple to restore to the previous state if the fixing goes wrong.

There are some corner cases where the fixing algorithm may replace the wrong code snippet.
It is therefore recommended to manually inspect the result of the fixing before committing.
* There are some corner cases where the fixing algorithm may replace the wrong code
snippet. It is therefore recommended to manually inspect the result of the fixing
before committing.


## Skipping Doctests
Expand Down
33 changes: 33 additions & 0 deletions docs/src/showcase.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Hidden showcase page

Currently exists just so that there would be doctests to run in manual pages of Documenter's
manual. This page does not show up in navigation.

```jldoctest
julia> 2 + 2
4
```

The following doctests needs doctestsetup:

```jldoctest; setup=:(using Documenter)
julia> Documenter.Utilities.splitexpr(:(Foo.Bar.baz))
(:(Foo.Bar), :(:baz))
```

Let's also try `@meta` blocks:

```@meta
DocTestSetup = quote
f(x) = x^2
end
```

```jldoctest
julia> f(2)
4
```

```@meta
DocTestSetup = nothing
```
107 changes: 87 additions & 20 deletions src/Documenter.jl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ $(EXPORTS)
"""
module Documenter

using Test: @testset, @test
using DocStringExtensions
import Base64: base64decode

Expand Down Expand Up @@ -826,29 +827,95 @@ function getenv(regex::Regex)
end

"""
doctest(modules::AbstractVector{Module}) -> Bool
doctest(package::Module; kwargs...)
Runs all the doctests in the given modules. Returns `true` if the doctesting was successful
and false if any error occurred.
Convenience method that runs and checks all the doctests for a given Julia package.
`package` must be the `Module` object corresponding to the top-level module of the package.
Behaves like an `@testset` call, returning a testset if all the doctests are successful or
throwing a `TestSetException` if there are any failures. Can be included in other testsets.
# Keywords
**`manual`** controls how manual pages are handled. By default (`manual = true`), `doctest`
assumes that manual pages are located under `docs/src`. If that is not the case, the
`manual` keyword argument can be passed to specify the directory. Setting `manual = false`
will skip doctesting of manual pages altogether.
Additional keywords are passed on to the main [`doctest`](@ref) method.
"""
function doctest(modules::AbstractVector{Module})
dir = mktempdir()
@debug "Doctesting in temporary directory: $(dir)" modules
mkdir(joinpath(dir, "src"))
r = try
makedocs(
root = dir,
sitename = "",
doctest = :only,
modules = modules,
)
true
catch e
@error "Doctesting failed" e
false
function doctest(package::Module; manual=true, testset=nothing, kwargs...)
if pathof(package) === nothing
throw(ArgumentError("$(package) is not a top-level package module."))
end
source = nothing
if manual === true
source = normpath(joinpath(dirname(pathof(package)), "..", "docs", "src"))
isdir(source) || throw(ArgumentError("""
Package $(package) does not have a documentation source directory at standard location.
Searched at: $(source)
If ...
"""))
end
testset = (testset === nothing) ? "Doctests: $(package)" : testset
doctest(source, [package]; testset=testset, kwargs...)
end

"""
doctest(source, modules; kwargs...)
Runs all the doctests in the given modules and on manual pages under the `source` directory.
Behaves like an `@testset` call, returning a testset if all the doctests are successful or
throwing a `TestSetException` if there are any failures. Can be included in other testsets.
The manual pages are searched recursively in subdirectories of `source` too. Doctesting of
manual pages can be disabled if `source` is set to `nothing`.
# Keywords
**`testset`** specifies the name of test testset (default `Doctests`).
**`fix`**, if set to `true`, updates all the doctests that fail with the correct output
(default `false`).
!!! warning
When running `doctest(...; fix=true)`, Documenter will modify the Markdown and Julia
source files. It is strongly recommended that you only run it on packages in Pkg's
develop mode and commit any staged changes. You should also review all the changes made
by `doctest` before committing them, as there may be edge cases when the automatic
fixing fails.
"""
function doctest(
source::Union{AbstractString,Nothing},
modules::AbstractVector{Module};
fix = false,
testset = "Doctests",
)
function all_doctests()
dir = mktempdir()
try
@debug "Doctesting in temporary directory: $(dir)" modules
if source === nothing
source = joinpath(dir, "src")
mkdir(source)
end
makedocs(
root = dir,
source = source,
sitename = "",
doctest = fix ? :fix : :only,
modules = modules,
)
true
catch e
@error "Doctesting failed" e
false
finally
rm(dir; recursive=true)
end
end
@testset "$testset" begin
@test all_doctests()
end
rm(dir; recursive=true)
return r
end

end # module
Loading

0 comments on commit 258e0ca

Please sign in to comment.