Append test output when a test fails

[klaaspieter]

Motivation

When a test fails assertTestPasses throws an error with the stdout of the command. This wasn't added to the test's output in the catch handler which made VSCode report the test run as not recording any output.

Closes #3090

Implementation

Automated Tests

The current tests for test controller are minimal and I don't know where to get started to implement a test for this specific behavior.

Manual Tests

I ran the extension and with a project where my (failed) tests weren't showing any output. This PR fixed it.

[graphite-app]

How to use the Graphite Merge Queue

Add the label graphite-merge to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

[klaaspieter]

I have signed the CLA!

[vinistock]

This is appending the output twice. We already do that here.

I suspect the issue is with how we attempt to parse the output, which starts on this line and may not be catching the right output for your tests.

[klaaspieter]

I responded in more detail on the issue but I think what's happening is that run.failed reports the message on the test case but not the entire test run.

When I appendOutput in the catch block I'm not seeing the output twice.

What I am seeing is that when running multiple tests at once that the failing ones now show up on the test run whereas previously those would only show on the test case. The output is a bit messy though because even though it's testing an entire file each test command is run individually:

Running 2 tests in a single process (parallelization threshold is 50)
Run options: --seed 4866

# Running:

F

Failure:
RubyLSPTest#test_failure [test/ruby_lsp_test.rb:9]:
Expected false to be truthy.

bin/rails test test/ruby_lsp_test.rb:8



Finished in 0.033372s, 29.9652 runs/s, 29.9652 assertions/s.
1 runs, 1 assertions, 1 failures, 0 errors, 0 skips
Running 2 tests in a single process (parallelization threshold is 50)
Run options: --seed 42880

# Running:

.

Finished in 0.032442s, 30.8242 runs/s, 30.8242 assertions/s.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips

[vinistock]

I can't reproduce what you mentioned using the LSP's own codebase. For example, if I force a test to fail in store_test.rb and run the entire file through the explorer, all outputs are properly attributed.

I think the issue may be related to parallelization, which might potentially be printing multiple test results all at once.

Do you have parallelization in the project that fails? How is that configured?