Exit code for makedocs is 0 when doctest error

[albheim]

It seems like the exit code for julia -e 'using Documenter, Example; makedocs(modules=[Example])' is 0 even when there are errors in the doctests while julia -e 'using Documenter, Example; doctest(Example) will have an exit code based on if the tests errored or not.

This could be nice to use in a CI chain. Currently I first run doctest and then make.jl to make sure I catch errors, but would be nice if the doctest that is run while making the docs could be used to check for errors, skipping one unnecessary run.

[fredrikekre]

Try passing strict = true to makedocs.

[albheim]

Oh...

I thought I scrolled through all options and didn't find one, but that seems exactly like what it want.

Thanks!

[albheim]

I realised this also did one more thing that I didn't notice at first, it seems to make what was previously warnings result in errors. I know I should probably fix the warnings instead of ignoring them, but is there any way to have that errors stop the program but warnings are still allowed?

[mortenpi]

Depending on what errors you're getting, you might be able to disable them by passing checkdocs = :none -- this should at least make Documenter ignore all the "docstring missing" errors.

[albheim]

Yeah, I did find that when I looked through and it allows me to do what I originally asked for.

I don't care too much about loosing the warnings, but it can still be nice to see them. So it seems to me it would be reasonable to have a flag like strict but the only thing it does is that on error it gives an exit code of 1 instead of 0 so that we can use makedocs doctest with CI but still print everything including warnings as normal.

But that may be just me, and I'm also fine with the solution as it is now. Just giving my two cents.

[mortenpi]

It should be relatively easy to generalize strict to e.g. accept a vector of Symbols, which would specify a subset of classes of errors that would make makedocs throw.

So it seems to me it would be reasonable to have a flag like strict but the only thing it does is that on error it gives an exit code of 1 instead of 0 so that we can use makedocs doctest with CI but still print everything including warnings as normal.

The exit code is caused by makedocs throwing (when strict = true) in global scope. You can always try ... catch the makedocs call yourself to handle the errors (although you won't be able to know exactly why makedocs threw).

[albheim]

The exit code is caused by makedocs throwing (when strict = true) in global scope. You can always try ... catch the makedocs call yourself to handle the errors (although you won't be able to know exactly why makedocs threw).

Yeah, I guessed that was how it worked but the try ... catch it does not really help for my case though since, as you mention, I won't be able to know why makedocs threw.

It should be relatively easy to generalize strict to e.g. accept a vector of Symbols, which would specify a subset of classes of errors that would make makedocs throw.

That sound like a nice solution to me. So you would have strict=[] for the current strict=false, strict=[:error, :warning] for the current strict=true and strict=[:error] for what I currently want.

[mortenpi]

I'll close this in favour of #1686.