Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test_warn macro to Base.Test for checking for warnings #19903

Merged
merged 15 commits into from
Jan 7, 2017
Merged

Add test_warn macro to Base.Test for checking for warnings #19903

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
ee012bd
add test_warn macro to Base.Test for verifying warning messages
stevengj Jan 6, 2017
5103a1c
use test_warn in compilation tests
stevengj Jan 6, 2017
63c55d0
add test for #18725
stevengj Jan 6, 2017
5ac63ab
doc links for test_warn
stevengj Jan 6, 2017
fc2ece2
compile test no longer needs to save STDERR
stevengj Jan 6, 2017
89257bd
rewrite pkg tests in terms of test_nowarn
stevengj Jan 6, 2017
ee53a54
don't strip trailing newline in test_warn, don't allow trailing newli…
stevengj Jan 6, 2017
6febcfa
check for specific branch in pkg pinning test
stevengj Jan 6, 2017
0d8447a
rm extraneous parens
stevengj Jan 6, 2017
f99c96a
restore pkg test for lack of output
stevengj Jan 6, 2017
d981d39
return something useful from test_warn
stevengj Jan 6, 2017
6ae2e67
small fixes
stevengj Jan 6, 2017
3ca3209
support arbitrary boolean function in test_warn, fix Upgrading Colors…
stevengj Jan 6, 2017
3e7ca3a
delete blank line in docstring
stevengj Jan 6, 2017
28c208c
eval 18725 in Main
stevengj Jan 7, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions NEWS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,9 @@ Library improvements
implemented; the semantics are as if the `Nullable` were a container with
zero or one elements ([#16961]).

* New `@test_warn` and `@test_nowarn` macros in the `Base.Test` module to
test for the presence or absence of warning messages ([#19903]).

* `logging` can be used to redirect `info`, `warn`, and `error` messages
either universally or on a per-module/function basis ([#16213]).

Expand Down
53 changes: 52 additions & 1 deletion base/test.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ and summarize them at the end of the test set with `@testset`.
"""
module Test

export @test, @test_throws, @test_broken, @test_skip
export @test, @test_throws, @test_broken, @test_skip, @test_warn, @test_nowarn
export @testset
# Legacy approximate testing functions, yet to be included
export @test_approx_eq_eps, @inferred
Expand Down Expand Up @@ -356,6 +356,57 @@ function do_test_throws(result::ExecutionResult, orig_expr, extype)
record(get_testset(), testres)
end

#-----------------------------------------------------------------------
# Test for warning messages

ismatch_warn(s::AbstractString, output) = contains(output, s)
ismatch_warn(s::Regex, output) = ismatch(s, output)
ismatch_warn(s::Function, output) = s(output)
ismatch_warn(S::Union{AbstractArray,Tuple}, output) = all(s -> ismatch_warn(s, output), S)

"""
@test_warn msg expr

Test whether evaluating `expr` results in [`STDERR`](@ref) output that contains
the `msg` string or matches the `msg` regular expression. If `msg` is
a boolean function, tests whether `msg(output)` returns `true`. If `msg` is a
tuple or array, checks that the error output contains/matches each item in `msg`.
Returns the result of evaluating `expr`.

See also [`@test_nowarn`](@ref) to check for the absence of error output.
"""
macro test_warn(msg, expr)
quote
let fname = tempname(), have_color = Base.have_color
try
eval(Base, :(have_color = false))
ret = open(fname, "w") do f
redirect_stderr(f) do
$(esc(expr))
end
end
@test ismatch_warn($(esc(msg)), readstring(fname))
ret
finally
eval(Base, Expr(:(=), :have_color, have_color))
rm(fname, force=true)
end
end
end
end

"""
@test_nowarn expr

Test whether evaluating `expr` results in empty [`STDERR`](@ref) output
(no warnings or other messages). Returns the result of evaluating `expr`.
"""
macro test_nowarn(expr)
quote
@test_warn r"^(?!.)"s $expr
end
end

#-----------------------------------------------------------------------

# The AbstractTestSet interface is defined by two methods:
Expand Down
2 changes: 2 additions & 0 deletions doc/src/stdlib/test.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,8 @@ checks using either `@test a ≈ b` (where `≈`, typed via tab completion of `\

```@docs
Base.Test.@inferred
Base.Test.@test_warn
Base.Test.@test_nowarn
```

## Broken Tests
Expand Down
56 changes: 5 additions & 51 deletions test/compile.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,6 @@

using Base.Test

function redirected_stderr(expected)
rd, wr = redirect_stderr()
t = @async begin
read = readstring(rd) # also makes sure the kernel isn't being forced to buffer the output
if !contains(read, expected)
@show expected
@show read
@test false
end
nothing
end
return t
end

Foo_module = :Foo4b3a94a1a081a8cb
FooBase_module = :FooBase4b3a94a1a081a8cb
@eval module ConflictingBindings
Expand All @@ -29,7 +15,6 @@ using .ConflictingBindings
# so we disable it for the tests below
withenv( "JULIA_DEBUG_LOADING" => nothing ) do

olderr = STDERR
dir = mktempdir()
dir2 = mktempdir()
insert!(LOAD_PATH, 1, dir)
Expand Down Expand Up @@ -136,14 +121,9 @@ try

# use _require_from_serialized to ensure that the test fails if
# the module doesn't reload from the image:
t = redirected_stderr("WARNING: replacing module Foo4b3a94a1a081a8cb.\nWARNING: Method definition ")
try
@test_warn "WARNING: replacing module Foo4b3a94a1a081a8cb.\nWARNING: Method definition " begin
@test isa(Base._require_from_serialized(myid(), Foo_module, cachefile, #=broadcast-load=#false), Array{Any,1})
finally
close(STDERR)
redirect_stderr(olderr)
end
Copy link
Contributor

@tkelman tkelman Jan 7, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is now misleadingly ending a much-earlier try

misread, didn't see the begin

wait(t)

let Foo = getfield(Main, Foo_module)
@test_throws MethodError Foo.foo(17) # world shouldn't be visible yet
Expand Down Expand Up @@ -217,17 +197,13 @@ try
end
""")

t = redirected_stderr("ERROR: LoadError: Declaring __precompile__(false) is not allowed in files that are being precompiled.\nStacktrace:\n [1] __precompile__")
try
@test_warn "ERROR: LoadError: Declaring __precompile__(false) is not allowed in files that are being precompiled.\nStacktrace:\n [1] __precompile__" try
Base.compilecache("Baz") # from __precompile__(false)
error("__precompile__ disabled test failed")
catch exc
close(STDERR)
redirect_stderr(olderr)
isa(exc, ErrorException) || rethrow(exc)
!isempty(search(exc.msg, "__precompile__(false)")) && rethrow(exc)
end
wait(t)

# Issue #12720
FooBar1_file = joinpath(dir, "FooBar1.jl")
Expand Down Expand Up @@ -273,14 +249,7 @@ try
fb_uuid1 = Base.module_uuid(Main.FooBar1)
@test fb_uuid != fb_uuid1

t = redirected_stderr("WARNING: replacing module FooBar.")
try
reload("FooBar")
finally
close(STDERR)
redirect_stderr(olderr)
end
wait(t)
@test_warn "WARNING: replacing module FooBar." reload("FooBar")
@test fb_uuid != Base.module_uuid(Main.FooBar)
@test fb_uuid1 == Base.module_uuid(Main.FooBar1)
fb_uuid = Base.module_uuid(Main.FooBar)
Expand All @@ -289,14 +258,7 @@ try
@test !Base.stale_cachefile(FooBar1_file, joinpath(dir2, "FooBar1.ji"))
@test !Base.stale_cachefile(FooBar_file, joinpath(dir2, "FooBar.ji"))

t = redirected_stderr("WARNING: replacing module FooBar1.")
try
reload("FooBar1")
finally
close(STDERR)
redirect_stderr(olderr)
end
wait(t)
@test_warn "WARNING: replacing module FooBar1." reload("FooBar1")
@test fb_uuid == Base.module_uuid(Main.FooBar)
@test fb_uuid1 != Base.module_uuid(Main.FooBar1)

Expand All @@ -314,22 +276,14 @@ try
error("break me")
end
""")
t = redirected_stderr("ERROR: LoadError: break me\nStacktrace:\n [1] error")
try
@test_warn "ERROR: LoadError: break me\nStacktrace:\n [1] error" try
Base.require(:FooBar)
error("\"LoadError: break me\" test failed")
catch exc
close(STDERR)
redirect_stderr(olderr)
isa(exc, ErrorException) || rethrow(exc)
!isempty(search(exc.msg, "ERROR: LoadError: break me")) && rethrow(exc)
end
wait(t)
finally
if STDERR != olderr
close(STDERR)
redirect_stderr(olderr)
end
splice!(Base.LOAD_CACHE_PATH, 1:2)
splice!(LOAD_PATH, 1)
rm(dir, recursive=true)
Expand Down
11 changes: 11 additions & 0 deletions test/core.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4822,6 +4822,17 @@ end
@test f14893() == 14893
@test M14893.f14893() == 14893

# issue #18725
@test_nowarn begin
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't tests run in a non-Main anonymous module now?

Copy link
Member Author

@stevengj stevengj Jan 7, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Weird, right: the warning is getting output (correctly), but for some reason it is not captured by redirect_stderr.

For example, if I do the following in test/core.jl:

let fname = tempname()
    try
        open(fname, "w") do f
            redirect_stderr(f) do
                begin
                    f18725x(x) = 1
                    f18725x(x) = 2
                    @test f18725x(0) == 2
                end
            end
        end
        println("GOT WARNING: \"", readstring(fname), '"')
    finally
        rm(fname, force=true)
    end
end

it outputs

WARNING: Method definition f18725x(Any) in module TestMain_core at /Users/stevenj/Documents/Code/julia/test/core.jl:4830 overwritten at /Users/stevenj/Documents/Code/julia/test/core.jl:4831.
GOT WARNING: ""

(And yet, the other @test_warn passes, indicating that the warning is sometimes being captured by redirect_stderr)

Any idea why the warning would not be captured here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I change the begin...end to

                eval(current_module(), :(begin
                    f18725x(x) = 1
                    f18725x(x) = 2
                    @test f18725x(0) == 2
                end))
            end

then it correctly captures the warning.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somehow, it seems that the "Method definition overwritten" error is being printed at some other stage of code evaluation, so that it occurs before the redirect_stderr is executed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've pushed a workaround, and verified that it is not printing any warning that fails to get captured. (The issue of when the method-override warning gets printed seems unrelated to this PR.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(My workaround now evals the expression in Main, so it correctly gets the case that is not supposed to have a warning.)

f18725(x) = 1
f18725(x) = 2
@test f18725(0) == 2
end
@test_warn "WARNING: Method definition f18725(Any) in module Module18725" eval(Main, :(module Module18725
f18725(x) = 1
f18725(x) = 2
end))

# issue #19599
f19599{T}(x::((S)->Vector{S})(T)...) = 1
@test f19599([1],[1]) == 1
Expand Down
Loading