Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky test attribute not working #6253

Merged
merged 5 commits into from
Apr 19, 2024
Merged

Conversation

smoogipoo
Copy link
Contributor

@smoogipoo smoogipoo commented Apr 18, 2024

There's a few issues here:

  1. We were throwing arbitrary exceptions, not AssertionExceptions. The latter are handled specially by NUnit to allow retries.
  2. If an assertion happens inside [TearDown], it does not re-run the test at all.

Using TestActionAttribute allows us to do basically the same thing without running into the second issue. Note however that this does not do anything like trying to recover the GameHost after a non-assertion exception, so those will still fail immediately and not retry. I think that this is fine and "intentional".

Until steps now throw an AssertionException instead of a TimeoutException. I don't believe we rely on this anywhere. Messaging has also changed a bit:

Previous:

TearDown : System.TimeoutException : "aaaaa" timed out
--TearDown
   at osu.Framework.Testing.Drawables.Steps.UntilStepButton.<>c__DisplayClass11_0.<.ctor>b__0() in /Users/smgi/Repos/osu-framework/osu.Framework/Testing/Drawables/Steps/UntilStepButton.cs:line 65

Now:

"aaaaa" timed out
   at osu.Framework.Testing.Drawables.Steps.UntilStepButton.<>c__DisplayClass11_0.<.ctor>b__0() in /Users/smgi/Repos/osu-framework/osu.Framework/Testing/Drawables/Steps/UntilStepButton.cs:line 66

I've run a full osu! test run with this without issue.

I've had to disable Roslyn analysers because there's a pretty bad memory
leak somewhere :(
@peppy peppy self-requested a review April 19, 2024 05:22
Copy link
Member

@peppy peppy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks promising.

@peppy peppy merged commit af8bbd2 into ppy:master Apr 19, 2024
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants