Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed flaky tests #581

Merged
merged 3 commits into from
Jun 7, 2021
Merged

Fixed flaky tests #581

merged 3 commits into from
Jun 7, 2021

Conversation

TonioGela
Copy link
Member

Closes #455.

See this comment for details.

@TonioGela TonioGela marked this pull request as draft June 6, 2021 19:13
@TonioGela TonioGela marked this pull request as ready for review June 6, 2021 19:33
@TonioGela
Copy link
Member Author

ofc the "Fixed flaky tests" PR has flaky tests

@TonioGela
Copy link
Member Author

I would really like to spot the problem, but testing locally 100 times with jdk 11 I haven't been able to reproduce it. Any idea @eed3si9n?

@eed3si9n
Copy link
Member

eed3si9n commented Jun 6, 2021

Maybe the property based testing doesn't need to consider version numbers with hanzi/kanji characters in them?

[info] ! StableVersion.ordering: Falsified after 943 passed tests.
[info] > Labels of failing property: 
[info] Reversing and sorting should be the same as sorting. Expected version: 365.181.24.0-臿揧氛-䗶蒳쬍㣴ꂦ-弮堢䋊弑⯄೏즃㴼㼺䀧᪟⿽, but got: 365.181.24-㦧녦-㓒ꑡ-Ყ懸喺݄辺芭ᤀ

@TonioGela
Copy link
Member Author

Wow, how did you picked that line in that huge log? Btw, I think you're right, in a few I'll push a fix for that

@TonioGela
Copy link
Member Author

Is not so trivial, after some iterations I get errors like this

failing seed for StableVersion.ordering is 7cjg5dZl9xbfN9xQFEJ2dWFH63NCIWZuWFMC77gKS2C=
[info] ! StableVersion.ordering: Falsified after 20155 passed tests.
[info] > Labels of failing property:
[info] Reversing and sorting should be the same as sorting. Expected version: 319.975.54.0-J4-im1-iquf4jladvx, but got: 319.975.54-V50uE2WjP1
[info] > ARG_0: List(319.975.54.0-J4-im1-iquf4jladvx, 319.975.54-V50uE2WjP1)

even if I set something strict like

private val genChar = Gen.alphaNumChar

I'll investigate further

@TonioGela
Copy link
Member Author

TonioGela commented Jun 6, 2021

I don't know why but in this commit this line was added:

numbers = prefix.toSeq ++: vn.numbers.drop(prefix.length)

To the best of my understanding, this should concatenate and empty list to prefix.toSeq, but sometimes the net effect is that lists like List(83, 39, 374) became List(83, 39, 374, 0) and so the test fails.

Removing this map the test passed flawlessly

genVersionNumber.map { vn =>
          VersionNumber(
            numbers = prefix.toSeq ++: vn.numbers.drop(prefix.length),
            tags = vn.tags,
            extras = vn.extras
          )
        }

@eed3si9n
Copy link
Member

eed3si9n commented Jun 6, 2021

In general feel free to remove any flaky tests too if it's not delivering values. It's not like we have a ton of bug reports complaining about version number parsing.

@TonioGela
Copy link
Member Author

In general, feel free to remove any flaky tests too if it's not delivering values. It's not like we have a ton of bug reports complaining about version number parsing.

Okay 👍 This time, I think I fixed it in a meaningful way btw :)

justfile Outdated Show resolved Hide resolved
@eed3si9n eed3si9n merged commit 7a472d4 into foundweekends:develop Jun 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

intermittent test failures in community build (Format, StableVersion)
2 participants