introduce back go-junit-report for parsing go test output

[marcuscaisey]

Stacked on please-build#81

The test output from go_test is currently quite hard to read and also can be inaccurate.

For example, given this test file:

package foo_test

import (
	"fmt"
	"os"
	"testing"

	"github.com/stretchr/testify/assert"
)

func Test(t *testing.T) {
	fmt.Fprintln(os.Stdout, "Test stdout")
	fmt.Fprintln(os.Stderr, "Test stderr")
	assert.Equal(t, 1, 2, "Test failure")

	t.Run("Subtest1", func(t *testing.T) {
		fmt.Fprintln(os.Stdout, "Subtest1 stdout")
		fmt.Fprintln(os.Stderr, "Subtest1 stderr")
		assert.Equal(t, 1, 2, "Subtest1 failure")

		t.Run("NestedSubtest", func(t *testing.T) {
			fmt.Fprintln(os.Stdout, "NestedSubtest stdout")
			fmt.Fprintln(os.Stderr, "NestedSubtest stderr")
			assert.Equal(t, 1, 2, "NestedSubtest failure")
		})
	})

	t.Run("Subtest2", func(t *testing.T) {
		fmt.Fprintln(os.Stdout, "Subtest2 stdout")
		fmt.Fprintln(os.Stderr, "Subtest2 stderr")
		assert.Equal(t, 1, 2, "Subtest2 failure")
	})
}

we get the following output:

00:11:06.488   ERROR: //foo:test failed
Fail: //foo:test   0 passed   0 skipped   4 failed   0 errored Took 60ms
Failure:  in Test
Subtest2 stdout
Subtest2 stderr
    foo_test.go:31: 
        	Error Trace:	foo_test.go:31
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	Test/Subtest2
        	Messages:   	Subtest2 failure
Standard error:
Test stdout
Test stderr
    foo_test.go:14: 
        	Error Trace:	foo_test.go:14
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	Test
        	Messages:   	Test failure
Subtest1 stdout
Subtest1 stderr
    foo_test.go:19: 
        	Error Trace:	foo_test.go:19
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	Test/Subtest1
        	Messages:   	Subtest1 failure
NestedSubtest stdout
NestedSubtest stderr
    foo_test.go:24: 
        	Error Trace:	foo_test.go:24
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	Test/Subtest1/NestedSubtest
        	Messages:   	NestedSubtest failure
Subtest2 stdout
Subtest2 stderr
    foo_test.go:31: 
        	Error Trace:	foo_test.go:31
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	Test/Subtest2
        	Messages:   	Subtest2 failure

Failure:  in Test/Subtest1
Subtest2 stdout
Subtest2 stderr
    foo_test.go:31: 
        	Error Trace:	foo_test.go:31
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	Test/Subtest2
        	Messages:   	Subtest2 failure
--- FAIL: Test (0.00s)
Failure:  in Test/Subtest1/NestedSubtest
Subtest2 stdout
Subtest2 stderr
    foo_test.go:31: 
        	Error Trace:	foo_test.go:31
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	Test/Subtest2
        	Messages:   	Subtest2 failure
--- FAIL: Test (0.00s)
    --- FAIL: Test/Subtest1 (0.00s)
Failure:  in Test/Subtest2
Subtest2 stdout
Subtest2 stderr
    foo_test.go:31: 
        	Error Trace:	foo_test.go:31
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	Test/Subtest2
        	Messages:   	Subtest2 failure
--- FAIL: Test (0.00s)
    --- FAIL: Test/Subtest1 (0.00s)
        --- FAIL: Test/Subtest1/NestedSubtest (0.00s)
//foo:test 4 tests run in 62ms; 0 passed, 4 failed
    Test                         FAIL  0s
    Test/Subtest1                FAIL  0s
    Test/Subtest1/NestedSubtest  FAIL  0s
    Test/Subtest2                FAIL  0s
1 test target and 4 tests run; 0 passed, 4 failed.
Total time: 100ms real, 60ms compute.

There are 4 assertions which fail, however 8 assertion failures are shown. All 4 expected assertions are output under Standard error header and then an assertion failure is also output for each failing test / subtest. However, the assertion failures under each Failure: in XXX header are not the correct failures. Instead, the failure from Test/Subtest2 is output each time.

I can see that go-junit-report v0.9.0 (technically a couple of commits after this tag) was introduced to the Please repo in 2020 and then reverted due to it not being able to handle more complex input (thought-machine/please#995). This PR introduces it back, now that the tool seems to have matured a bit. Between v0.9.0 and v2.0.0, 81 test cases have been added and it seems to pass the eye test on the above test file:

00:47:04.082   ERROR: //foo:test failed: Failed
Fail: //foo:test   0 passed   0 skipped   4 failed   0 errored Took 70ms
Failure:  in Test
Failed
Test stdout
Test stderr
    foo_test.go:14: 
        	Error Trace:	foo_test.go:14
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	Test
        	Messages:   	Test failure
Failure:  in Test/Subtest1
Failed
Subtest1 stdout
Subtest1 stderr
    foo_test.go:19: 
        	Error Trace:	foo_test.go:19
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	Test/Subtest1
        	Messages:   	Subtest1 failure
Failure:  in Test/Subtest1/NestedSubtest
Failed
NestedSubtest stdout
NestedSubtest stderr
    foo_test.go:24: 
        	Error Trace:	foo_test.go:24
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	Test/Subtest1/NestedSubtest
        	Messages:   	NestedSubtest failure
Failure:  in Test/Subtest2
Failed
Subtest2 stdout
Subtest2 stderr
    foo_test.go:31: 
        	Error Trace:	foo_test.go:31
        	Error:      	Not equal: 
        	            	expected: 1
        	            	actual  : 2
        	Test:       	Test/Subtest2
        	Messages:   	Subtest2 failure
//foo:test 4 tests run in 68ms; 0 passed, 4 failed
    Test                         FAIL  0s
    Test/Subtest1                FAIL  0s
    Test/Subtest1/NestedSubtest  FAIL  0s
    Test/Subtest2                FAIL  0s
1 test target and 4 tests run; 0 passed, 4 failed.
Total time: 110ms real, 70ms compute.

Now, only 4 assertion failures are output and each one is under the correct Failure: in XXX heading.

The version that i've added to this repo is my fork where i've set the message attribute of the <skipped> node correctly so that the reason is output correctly by Please (without this change, the skipped test output looks like Reason: Skipped vs Reason: external_test.go:32: Failing on Alpine currently). I've made a PR to upstream this: jstemmer/go-junit-report#158.

I'm happy to throw any other test files you can think of at go-junit-report to make sure it handles them properly 🙏

[marcuscaisey]

wrong repo lol