Tests don't fail when exception is thrown

[kenahoo]

It appears that if a @testitem in runtests.jl throws an exception, the overall test process still returns status 0, i.e., success.

Here's a minimal working example:

% cat test/runtests.jl 
using Test
using ReTestItems

@testitem "Example tests" begin
    error("Fail")
end


% julia --project=. -e 'using Pkg; Pkg.test()'
...
     Testing Running tests...
12:31:04 | maxrss  0.9% | mem 100.0% | START test item "Example tests" at runtests.jl:4
Example tests: Error During Test at /Users/.../Inflect.jl/test/runtests.jl:4
  Got exception outside of a @test
...
Test Summary: | Error  Total  Time
Example tests |     1      1  1.5s
12:31:06 | maxrss  1.2% | mem 99.9% | DONE test item "Example tests" 0 secs
     Testing Inflect tests passed 


% echo $?                                     
0

Notice the Testing Inflect tests passed message and the exit status.

Thoughts on why this is happening and how to fix it?

[nickrobinson251]

thanks for using the package!

@testitem cannot be places in runtests.jl, only in a file named with the suffix *_test.jl or *_tests.jl -- note runtests.jl does not meet this convention. runtests.jl is the script used to launch tests (it's the script that Pkg.test() runs), and the design of ReTestItems.jl is for no @testitems live in this file.

Intended usage would look like

% cat test/runtests.jl 
using ReTestItems
runtests()

% cat test/example_tests.jl 
@testitem "Example tests" begin
    error("Fail")
end

[kenahoo]

Aha, thanks. I indeed get better luck with that setup.

It does seem like something, somewhere, should be noticing that tests failed under the setup I had, though - is that Test.jl maybe?

[nickrobinson251]

hmmm, yeah, good point

Does this branch give you the behaviour you expect: #178 ?

You can try it out if using Pkg.add(name="ReTestItems", rev="npr-propogate-test-error")

[kenahoo]

For some reason I can't seem to try that out, I keep getting ERROR: can not merge projects when I try to run tests with that branch installed. I did some googling about the issue but couldn't find a solution.

[nickrobinson251]

looks like a known issue with using test/Project.toml files (in general i'd advise against doing that)
https://discourse.julialang.org/t/can-not-merge-projects-error-when-dev-ing-sister-package-from-test-project/115575/3?