Conditional tests (skipping testitems)

[maleadt]

Some tests of a project of mine can only run on specific Julia versions. Currently, I have:

if LLVM.has_orc()
@testitem "orc" begin
...
end
end

ReTestItems.jl does not like this: LoadError: Test files must only include `@testitem` and `@testsetup` calls.

Putting the conditional within the testitem doesn't always work, e.g,. when using types that are only conditionally defined:

@testitem "orc" begin
if has_orc()

function do_something(x::SomeOrcType)
...
end

end
end

I could make the test @static, but that's not really intuitive to most users. Furthermore, it still results in the skipped test item being shown on stdout, with a warning that it doesn't contain any tests:

22:22:26 | maxrss  1.0% | mem  4.7% | START (18/26) test item "orc" at test/orc_tests.jl:1
22:22:26 | maxrss  1.0% | mem  4.7% | DONE  (18/26) test item "orc" <0.1 secs (85.4% compile), 1.88 K allocs (113.380 KB)
┌ Warning: Test item "orc" at test/orc_tests.jl:1 contains test sets without tests:
│ "orc"

Is there another workaround? Maybe @testitem should support passing-in a skip boolean or so?

[nickrobinson251]

yeah, this sounds like a nice feature! So far, in the couple cases something like this could have been used, i've just seen the if ... block put inside the @testitem, which as you say has the downside of noisy logs in stdout

i guess we'd want the skip keyword to take an expression (evaluating to a boolean) that's evaluated on the worker immediately before the testitem would have been run?

Perhaps, we'd log still something like

22:22:26 | maxrss  1.0% | mem  4.7% | SKIPPED (18/26) test item "orc" at test/orc_tests.jl:1

and still have this test item show up in the test results summary table (as skipped)?

[Drvi]

One workaround would be to use @static if instead of if, but I'm not sure if that works in your example.

Another workaround would be to give each test item a tag (like :needs_orc) and then implement the filtering logic in the runtests a la:

tags_to_filter_out = get_tags_to_filter_out(...)
runtests(PkgName) do ti
    isdisjoint(ti.tags, tags_to_filter_out)
end

[nickrobinson251]

another use-case for this we just ran into: skipping certain tests (e.g. JET) on pre-release versions on Julia e.g. skip=!isempty(VERSION.prerelease)