fix flaky test org.aim42.htmlsanitycheck.check.BrokenCrossReferencesCheckerTest.testTwoBrokenLinks

[hofi1]

Problem

The test org.aim42.htmlsanitycheck.check.BrokenCrossReferencesCheckerTest.testTwoBrokenLinks asserts the finding of the SingleCheckResults. These findings are stored in a Set. But the order, in which the two given findings are returned by the set, is not deterministic. The test expects the values to be in certain order. This leads to a flaky test.

htmlSanityCheck/src/test/groovy/org/aim42/htmlsanitycheck/check/BrokenCrossReferencesCheckerTest.groovy

Lines 91 to 112 in fa8b073

	void testTwoBrokenLinks() {
	String HTML_WITH_TWO_LINKS_NO_ID = '''
	<a href="#aim42">link-to-aim42</a>
	<a href="#arc42">non-existing-link</a>
	'''
	check HTML_WITH_TWO_LINKS_NO_ID
	assertEquals( "expected two findings", 2, collector.nrOfProblems())
	assertEquals( "expected four checks", 4, collector.nrOfItemsChecked)
	// first finding: aim42 link missing
	String actual = collector.findings.first()
	String expected = "link target \"aim42\" missing"
	String message = "expected $expected"
	assertEquals(message, expected, actual)
	// second finding: arc42 link missing
	actual = collector.findings[1]
	expected = "link target \"arc42\" missing"
	assertEquals(message, expected, actual)
	}

This problem was found by the NonDex Engine.

Solution

Sort the set, to make sure, it is returned in predefined order.

htmlSanityCheck/src/main/groovy/org/aim42/htmlsanitycheck/check/BrokenCrossReferencesChecker.groovy

Line 41 in fa8b073

hrefSet = hrefList.toSet().sort()

Reproduce

To reproduce follow the steps:

1. ./gradlew build -x test
2. Add the following text to the top of the build.gradle file in $PROJ_DIR.

buildscript {
    repositories {
      maven {
        url = uri('https://plugins.gradle.org/m2/')
      }
    }
    dependencies {
      classpath('edu.illinois:plugin:2.1.1')
    }
}

3. Add the following line to the end of the build.gradle file in $PROJ_DIR.

apply plugin: 'edu.illinois.nondex'

4. Run

./gradlew --info nondexTest --tests=org.aim42.htmlsanitycheck.check.BrokenCrossReferencesCheckerTest.testTwoBrokenLinks --nondexRuns=50

[gernotstarke]

thanx, good improvement!