Performance Degradation with 1.5.4

[tmohme]

Testing Problem

I'm experiencing a significant performance degradation with 1.5.4 (compared to 1.5.3, 1.5.2).
While the overall runtime of the complete test suite increased by roughly 25% compared to 1.5.3 and 100% compared to 1.5.2 (760 tests, 1.5.2: 8:22 => 1.5.3: 13:58 => 1.5.4: 17:27), some tests are hit exceptionally hard.

The current situation is probably a mix of not having completely addressed the problem in #206 and some new things.
That does not at all mean the fixes done in #206 were useless.
The package (28 tests) that saw the largest increase in runtime from 1.5.2 to 1.5.3 still/again looks good: 0:55 => 4:36 => 1:03

This time a different package (229 tests) got hit hardest: 3:06 => 3:49 => 6:27

The test class (16 tests) with the biggest degradation: 0:04 => 0:10 => 1:48

I'll continue my investigation, trying to identify some hot spots and post the results here.

[jlink]

@tmohme If you find a way to extract the specific test class that would be very helpful. Most probably the secret is in the arbitraries, so just posting the provider methods might already be a first step towards identifying the root cause.

When you collect additional evidence, switching off edge cases can also provide additional data points.

[tmohme]

Without edge case generation, the time for the complete test suite goes down from 17:27 to 7:56 - even fasten than 1.5.2

Edit:
I tested all possible edge case generation settings with the most affected class (16 tests). NONE brings the runtime down to 4s, every other setting gives pretty much the same result 1:04 ~ 1:07.
The measured times absolute time (1:04~1:07) is less is less than in my first post (1:48) because initially I ran the whole suite n-way parallel and now I ran only the tests of this class. So the absolute values are not comparable.
The relation within one test series nevertheless should be usable.

[tmohme]

While in #206 the most affected test used our most complicated arbitrary, this time it is our third largest arbitrary . . . it will take some time to extract all the required pieces into a reproducer project.

[jlink]

Since it is obviously edge case related just a screenshot of the provider methods (or arbitrary provider classes if there are any) could already help me to form a theory and try a few things.

[jlink]

@tmohme Do you use Combinators a lot in the affected properties?

[tmohme]

Yes, combinators are used heavily.
I finally managed to extract the test for a reproducer, but all in all, there are still too many classes involved.
I'll give it another try, synthesizing a test until it shows the problematic behavior.

[tmohme]

After all back to the strip-down approach: jqwik-214-demo.zip

[jlink]

That was very helpful. The bug (well, there may be more than one) is an oversight by me which can be easily fixed. I'll keep you posted.

[jlink]

Initial fix available in 1.5.5-SNAPSHOT

[tmohme]

The fix looks good!
Our whole test suite now took 9:20 which is only a moderate increase compared to 1.5.2 with 8:22.
While faster tests are always welcome, this improvement should be enough for us.