Regression: Base.@doc problem with :toplevel #246

cstjean · 2019-03-04T13:22:21Z

On Revise 1.0.3, given this package definition:

module TestPkg

macro toplev(x)
    esc(Expr(:toplevel, quote
             Base.@__doc__ $x
             end))
end

""" Problem """
@toplev struct Foo end

end # module

Then:

julia> using Revise, TestPkg
[ Info: Recompiling stale cache file /home/cst-jean/.julia/compiled/v1.1/TestPkg/9LPSD.ji for TestPkg [9db6ce8c-046d-11e9-3d6b-c5272c17d34a]

julia> 1   # touch src/TestPkg.jl
┌ Warning: expected block expression got $(Expr(:toplevel, quote
│     #= /home/cst-jean/.julia/dev/TestPkg/src/TestPkg.jl:5 =#
│     begin
│         struct Foo
│             #= /home/cst-jean/.julia/dev/TestPkg/src/TestPkg.jl:10 =#
│         end
│         (Base.Docs.doc!)(TestPkg, (Base.Docs.Binding)(TestPkg, :Foo), (Base.Docs.docstr)((Core.svec)(" Fine "), (Dict)(:path => "", :linenumber => 0, :module => TestPkg, (Pair)(:fields, (Dict)()))), Union{})
│     end
│ end)) from #= /home/cst-jean/.julia/dev/TestPkg/src/TestPkg.jl:9 =# Core.:(@doc) " Fine " #= /home/cst-jean/.julia/dev/TestPkg/src/TestPkg.jl:10 =# @toplev(struct Foo
│             #= /home/cst-jean/.julia/dev/TestPkg/src/TestPkg.jl:10 =#
│         end)
└ @ Revise ~/.julia/packages/Revise/yp5KG/src/parsing.jl:141
┌ Warning: expected block expression got $(Expr(:toplevel, quote
│     #= /home/cst-jean/.julia/dev/TestPkg/src/TestPkg.jl:5 =#
│     begin
│         struct Foo
│             #= /home/cst-jean/.julia/dev/TestPkg/src/TestPkg.jl:10 =#
│         end
│         (Base.Docs.doc!)(TestPkg, (Base.Docs.Binding)(TestPkg, :Foo), (Base.Docs.docstr)((Core.svec)(" Fine "), (Dict)(:path => "", :linenumber => 0, :module => TestPkg, (Pair)(:fields, (Dict)()))), Union{})
│     end
│ end)) from #= /home/cst-jean/.julia/dev/TestPkg/src/TestPkg.jl:9 =# Core.:(@doc) " Fine " #= /home/cst-jean/.julia/dev/TestPkg/src/TestPkg.jl:10 =# @toplev(struct Foo
│             #= /home/cst-jean/.julia/dev/TestPkg/src/TestPkg.jl:10 =#
│         end)
└ @ Revise ~/.julia/packages/Revise/yp5KG/src/parsing.jl:141
1

It fails on Revise v1.0.3, but worked fine on v0.7.12. I can bisect it if that helps. It's a problem when using QuickTypes.jl, because it has to generate toplevel struct definitions.

cstjean · 2019-03-04T13:51:11Z

I tried bisecting, but I ran into ERROR: AssertionError: mex.head == :block, which I had to bisect skip. End result was:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
09a3d35739e8e7d75e3dc021da1be7b6c361ee7c
72b4597c32adab175cf0aafecb575e53152a9371
c1151992ebc7f0d68a40fb7111c220c134f67939
a25a9d89b9cb276d40966441bc33bcf34c45bb28
eae92013f176b5fb2be41b204b1a9ea41755120b
0071d5f6ffc215b2a62ee5b39656a447ccf63261
b76ae7c840699feebd08182192de2f93de5961e6
d892d77f9804e494fe0042e7916fec6933795ce0
e09be05d5f625032343043f3ca22ebcc492e793a
0aa44233ffb4c8a067d3c35e9abbc56a71765ef4
e57797a1455aecd16b13a74b0ef95b24fcbf385a
45782642ac0576cc7f77e1b9076d979bc9bf29f1
75ee835b7ec941961d71877ba52831c5166a5ff0
7be2cdaab520922d6aff7ad193fc4b018fdad73d
d53db38d03d5e2af6e38c4e4f1443b91ea078593
5b38ceed85ed464090f85fbade35308de4f5c62a
f4dbc9664d06ddf6e3d60f8438f20edbe0c3d332
cfc8fc7bd858760785d1dc5085f5d9077f7194ec
f59b6437fb59bba54f2cffa7060abbc05ac08766
2cfdde2fb758bcaeff60147a1ffe11d92ad71c6b
We cannot bisect more!

v0.7.14 is fine, v0.7.15 has the mex error.

timholy · 2019-03-04T19:12:58Z

The ~~stacktrace~~warning location information appears to indicate an easy fix,

Revise.jl/src/parsing.jl

Line 140 in a2c0fd6

if mex.head != :block

Maybe try generalizing that to allow :toplevel?

timholy · 2019-03-21T14:07:30Z

Fixed by #243

cstjean · 2019-03-21T14:42:44Z

Thank you! Sorry that I haven't been able to contribute of late, life has been very busy.

@warn

There can be cases where a callback key is removed from `user_callbacks_by_key` but it still remains in `user_callbacks_queue`. The problem may be illustrated with the following MRE. Say, with the following diff ```diff diff --git a/src/callbacks.jl b/src/callbacks.jl index bff5256..865bb24 100644 --- a/src/callbacks.jl +++ b/src/callbacks.jl @@ -159,6 +159,7 @@ function entr(f::Function, files, modules=nothing; all=false, postpone=false, pa sleep(pause) f() end + @info "added $(key)" try while true wait(revision_event) @@ -168,7 +169,7 @@ function entr(f::Function, files, modules=nothing; all=false, postpone=false, pa isa(err, InterruptException) || rethrow(err) finally remove_callback(key) + @info "removed $(key)" end nothing end - ``` and we run the following script: > entr.jl ```julia using Revise const TARGET_FILE = "watched.jl" let Revise.includet(TARGET_FILE) interrupted = false while !interrupted try entr([TARGET_FILE]) do # do something ... end interrupted = true catch err # handle expected errors, keep running # sync errors if isa(err, LoadError) || (isa(err, ErrorException) && startswith(err.msg, "lowering returned an error")) || isa(err, Revise.ReviseEvalException) @warn "handling sync error" err continue end # async errors if isa(err, CompositeException) errs = err.exceptions i = findfirst(e->isa(e, TaskFailedException), errs) if i !== nothing tfe = errs[i]::TaskFailedException res = tfe.task.result if isa(res, LoadError) || (isa(res, ErrorException) && startswith(res.msg, "lowering returned an error")) || isa(res, Revise.ReviseEvalException) @warn "handling async error" res continue end end end rethrow(err) end end end ``` where `watched.jl` has something like below: ```julia foo(a) = a # first state # foo(a) = end # second state ``` Now, boot up REPL and `include("entr.jl")`, and then modify `watch.jl` so that we alternately comment-in/out the lines of first/second state. It will result in something like below on the current master: ```julia julia> include("entr.jl") [ Info: added #timholy#245 [ Info: removed #timholy#245 ┌ Warning: handling sync error │ err = │ LoadError: "unexpected \"end\"" │ in expression starting at /Users/aviatesk/julia/packages/Revise/watched.jl:2 └ @ Main ~/julia/packages/Revise/entr.jl:22 [ Info: added #timholy#246 [ Info: removed #timholy#246 ERROR: LoadError: KeyError: key Symbol("#timholy#245") not found Stacktrace: [1] getindex(h::Dict{Any, Any}, key::Symbol) @ Base ./dict.jl:482 [2] macro expansion @ ~/julia/packages/Revise/src/callbacks.jl:116 [inlined] [3] macro expansion @ ./task.jl:382 [inlined] [4] process_user_callbacks!(keys::Set{Any}; throw::Bool) @ Revise ~/julia/packages/Revise/src/callbacks.jl:115 [5] revise(; throw::Bool) @ Revise ~/julia/packages/Revise/src/packagedef.jl:816 [6] entr(f::var"timholy#3#5", files::Vector{String}, modules::Nothing; all::Bool, postpone::Bool, pause::Float64) @ Revise ~/julia/packages/Revise/src/callbacks.jl:166 [7] entr(f::Function, files::Vector{String}, modules::Nothing) @ Revise ~/julia/packages/Revise/src/callbacks.jl:156 [8] top-level scope @ ~/julia/packages/Revise/entr.jl:11 [9] include(fname::String) @ Base.MainInclude ./client.jl:451 [10] top-level scope @ REPL[1]:1 in expression starting at /Users/aviatesk/julia/packages/Revise/entr.jl:5 ``` , which will be fixed with this PR.

@warn

…#591) There can be cases where a callback key is removed from `user_callbacks_by_key` but it still remains in `user_callbacks_queue`. The problem may be illustrated with the following MRE. Say, with the following diff ```diff diff --git a/src/callbacks.jl b/src/callbacks.jl index bff5256..865bb24 100644 --- a/src/callbacks.jl +++ b/src/callbacks.jl @@ -159,6 +159,7 @@ function entr(f::Function, files, modules=nothing; all=false, postpone=false, pa sleep(pause) f() end + @info "added $(key)" try while true wait(revision_event) @@ -168,7 +169,7 @@ function entr(f::Function, files, modules=nothing; all=false, postpone=false, pa isa(err, InterruptException) || rethrow(err) finally remove_callback(key) + @info "removed $(key)" end nothing end - ``` and we run the following script: > entr.jl ```julia using Revise const TARGET_FILE = "watched.jl" let Revise.includet(TARGET_FILE) interrupted = false while !interrupted try entr([TARGET_FILE]) do # do something ... end interrupted = true catch err # handle expected errors, keep running # sync errors if isa(err, LoadError) || (isa(err, ErrorException) && startswith(err.msg, "lowering returned an error")) || isa(err, Revise.ReviseEvalException) @warn "handling sync error" err continue end # async errors if isa(err, CompositeException) errs = err.exceptions i = findfirst(e->isa(e, TaskFailedException), errs) if i !== nothing tfe = errs[i]::TaskFailedException res = tfe.task.result if isa(res, LoadError) || (isa(res, ErrorException) && startswith(res.msg, "lowering returned an error")) || isa(res, Revise.ReviseEvalException) @warn "handling async error" res continue end end end rethrow(err) end end end ``` where `watched.jl` has something like below: ```julia foo(a) = a # first state # foo(a) = end # second state ``` Now, boot up REPL and `include("entr.jl")`, and then modify `watch.jl` so that we alternately comment-in/out the lines of first/second state. It will result in something like below on the current master: ```julia julia> include("entr.jl") [ Info: added ##245 [ Info: removed ##245 ┌ Warning: handling sync error │ err = │ LoadError: "unexpected \"end\"" │ in expression starting at /Users/aviatesk/julia/packages/Revise/watched.jl:2 └ @ Main ~/julia/packages/Revise/entr.jl:22 [ Info: added ##246 [ Info: removed ##246 ERROR: LoadError: KeyError: key Symbol("##245") not found Stacktrace: [1] getindex(h::Dict{Any, Any}, key::Symbol) @ Base ./dict.jl:482 [2] macro expansion @ ~/julia/packages/Revise/src/callbacks.jl:116 [inlined] [3] macro expansion @ ./task.jl:382 [inlined] [4] process_user_callbacks!(keys::Set{Any}; throw::Bool) @ Revise ~/julia/packages/Revise/src/callbacks.jl:115 [5] revise(; throw::Bool) @ Revise ~/julia/packages/Revise/src/packagedef.jl:816 [6] entr(f::var"#3#5", files::Vector{String}, modules::Nothing; all::Bool, postpone::Bool, pause::Float64) @ Revise ~/julia/packages/Revise/src/callbacks.jl:166 [7] entr(f::Function, files::Vector{String}, modules::Nothing) @ Revise ~/julia/packages/Revise/src/callbacks.jl:156 [8] top-level scope @ ~/julia/packages/Revise/entr.jl:11 [9] include(fname::String) @ Base.MainInclude ./client.jl:451 [10] top-level scope @ REPL[1]:1 in expression starting at /Users/aviatesk/julia/packages/Revise/entr.jl:5 ``` , which will be fixed with this PR.

timholy closed this as completed Mar 21, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regression: Base.@doc problem with :toplevel #246

Regression: Base.@doc problem with :toplevel #246

cstjean commented Mar 4, 2019 •

edited

Loading

cstjean commented Mar 4, 2019 •

edited

Loading

timholy commented Mar 4, 2019 •

edited

Loading

timholy commented Mar 21, 2019

cstjean commented Mar 21, 2019

Regression: Base.__@doc__ problem with :toplevel #246

Regression: Base.__@doc__ problem with :toplevel #246

Comments

cstjean commented Mar 4, 2019 • edited Loading

cstjean commented Mar 4, 2019 • edited Loading

timholy commented Mar 4, 2019 • edited Loading

timholy commented Mar 21, 2019

cstjean commented Mar 21, 2019

Regression: Base.@doc problem with :toplevel #246

Regression: Base.@doc problem with :toplevel #246

cstjean commented Mar 4, 2019 •

edited

Loading

cstjean commented Mar 4, 2019 •

edited

Loading

timholy commented Mar 4, 2019 •

edited

Loading