Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do all filtering of testitems at AST level #179

Merged
merged 5 commits into from
Sep 11, 2024

Conversation

nickrobinson251
Copy link
Collaborator

@nickrobinson251 nickrobinson251 commented Sep 9, 2024

Follow-up to #170 that allows us to remove the FilteredVector and do all filtering of testitems before eval'ing any code, this is possible (after #170) if we special-case the codepath we have to support RAI usage. This adds a bit more RAI-specific code to this package, but (i) speeds up RAI's use-case a lot and (ii) tidies up the filtering story in this package (since we can drop the codepaths that were doing filtering after evaling code).

On the RAI test codebase this speeds up filtering of test-items considerably. Here's how long it takes to filter out all the tests (e.g. with runtests(; name="doesnotexist", tags=[:nosuchtag]))

  • on v1.25.1: ~10 seconds
  • after PR#170: ~8 seconds
  • with this PR: ~1.3 seconds

Closes #41

@nickrobinson251 nickrobinson251 changed the title WIP do all filtering at AST level Do all filtering of testitems at AST level Sep 11, 2024
@nickrobinson251 nickrobinson251 marked this pull request as ready for review September 11, 2024 11:17
@nickrobinson251 nickrobinson251 added the perf About improving the performance of the package label Sep 11, 2024
Copy link
Collaborator

@Drvi Drvi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

People are going to notice this one:) Very nice, Nick!!

I tried it on RAICode and it was much faster! I did run into this one issue when I tried to run a single top-level @test_rel:

julia> runtests("test"; name="constant-folding-old-periods-old-1")
  Activating project at `~/_proj/raicode_master3`
[ Info: Scanning for test items in project `RAICode` at paths: test
┌ Warning: Encountered duplicate @testsetup with name: `LoadJSONSetup`. Replacing...
└ @ ReTestItems ~/.julia/packages/ReTestItems/NieGI/src/ReTestItems.jl:663
ERROR: TaskFailedException

    nested task error: LoadError: AssertionError: `tags` keyword must be passed a collection of `Symbol`s, got: $(Expr(:escape, :([:ring1, :integration]))) (Expr)
    Stacktrace:
      [1] var"@testitem"(__source__::LineNumberNode, __module__::Module, nm::Any, exs::Vararg{Any})
        @ ReTestItems ~/.julia/packages/ReTestItems/NieGI/src/macros.jl:264
...

(I has to comment out the top-level evals first)

@nickrobinson251
Copy link
Collaborator Author

Thanks for pushing us to do this!

one issue when I tried to run a single top-level @test_rel

yeah, this is a bug in RAI's @test_rel that i'll fix when updating to this new ReTestItems.jl release

@nickrobinson251 nickrobinson251 merged commit e8295b0 into main Sep 11, 2024
7 checks passed
@nickrobinson251 nickrobinson251 deleted the npr-only-ast-filtering branch September 11, 2024 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
perf About improving the performance of the package
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Statically parse, rather than run, test files to extract @testitems
2 participants