JET doesn't find errors when run in the tests

[JamesWrigley]

With this module in a package:

module JetDebug

greet() = Base.blah()

end # module JetDebug

JET finds the problem when run in the REPL:

julia> using JET                                                                                                                                                                                                                               
                                                                                                                                                                                                                                               
julia> import JetDebug                                                                                                                                                                                                                         
                                                                                                                                                                                                                                               
julia> report_package(JetDebug)                                                                                                                                                                                                                
[toplevel-info] virtualized the context of Main (took 0.022 sec)                                                                                                                                                                               
[toplevel-info] entered into /home/james/git/scratch/JetDebug/src/JetDebug.jl                                                                                                                                                                  
[toplevel-info]  exited from /home/james/git/scratch/JetDebug/src/JetDebug.jl (took 0.142 sec)                                                                                                                                                 
[toplevel-info] analyzing from top-level definitions (1/1)                                                                                                                                                                                     
[toplevel-info] analyzed 1 top-level definitions (took 2.595 sec)                                                                                                                                                                              
[ Info: tracking Base                                                                                                                                                                                                                          
═════ 1 possible error found ═════                                                                                                                                                                                                             
┌ greet() @ JetDebug /home/james/git/scratch/JetDebug/src/JetDebug.jl:3                                                                                                                                                                        
│┌ getproperty(x::Module, f::Symbol) @ Base ./Base.jl:42                                                                                                                                                                                       
││ `Base.blah` is not defined: Base.getglobal(x::Module, f::Symbol)                                                                                                                                                                            
│└────────────────────                                                      

But with this in my runtests.jl:

using JET
using JetDebug

report_package(JetDebug)

It does not find the problem:

Precompiling project for configuration --code-coverage=none --color=yes --check-bounds=yes --warn-overwrite=yes --depwarn=yes --inline=yes --startup-file=no --track-allocation=none...
  ✓ JetDebug
  1 dependency successfully precompiled in 1 seconds. 33 already precompiled.
  1 dependency precompiled but a different version is currently loaded. Restart julia to access the new version
     Testing Running tests...
[toplevel-info] virtualized the context of Main (took 0.026 sec)
[toplevel-info] entered into /home/james/git/scratch/JetDebug/src/JetDebug.jl
[toplevel-info]  exited from /home/james/git/scratch/JetDebug/src/JetDebug.jl (took 0.164 sec)
[toplevel-info] analyzing from top-level definitions (1/1)
[toplevel-info] analyzed 1 top-level definitions (took 3.472 sec)
     Testing JetDebug tests passed 

I noticed that when run in the REPL it also logged that it was tracking Base, but the same thing occurs if I have greet() = blah(). Any ideas about what could be causing that? I'm on 1.11:

julia> versioninfo()
Julia Version 1.11.0-rc1
Commit 3a35aec36d1 (2024-06-25 10:23 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × Intel(R) Core(TM) i7-10850H CPU @ 2.70GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, skylake)
Threads: 1 default, 0 interactive, 1 GC (on 12 virtual cores)
Environment:
  LD_LIBRARY_PATH = :/opt/zfp/lib64:/opt/adios2/lib:/usr/lib64/openmpi/lib:/usr/local/lib
  JULIA_EDITOR = vim

And using the latest version of JET.

[aviatesk]

This is because what shows the JET's error message (i.e. ═════ 1 possible error found ═════ ...) is the return value of report_package and report_package itself doesn't show the result into stdout or the other output stream. When you want to use report_package for test, I'd recommend using test_package instead, which runs report_package in a way that is integrated with the Test.jl infrastructure.

[JamesWrigley]

Ahhh I'm an idiot, thank you very much.