Test reporting in CI

[kylefleming]

Motivation

CircleCI has a feature that allows a project to display test results in the Tests tab for a given CI run. We're reporting our test results, but not doing so in the way that CircleCI expects. This PR addresses that, as well as a number of edge cases that cause the tests to not be fully reported.

In this PR

• Calls store_test_results with the directory containing the tests results, instead of the test result file itself. See https://support.circleci.com/hc/en-us/articles/360021624194-Test-summary-troubleshooting
• Ensures that test results get converted to junit format on test failure (the bash script that runs the tests is configured with -e, so curly brackets have been added to make sure the command for converting gets executed prior to bash exiting the script early)
• Version printing of various tools during CI was mistakenly removed. This adds it back in, along with printing the jq version.
• Fail-fast is now disabled for tests during CI so that CI can report all the failed tests, not just the first one. There is a tradeoff between showing the test failure sooner and showing all the test failures. In my opinion, fail fast is most useful for local development, where development speed is more important, and that showing all failed tests during CI runs is more important than showing that the CI run has failed sooner.

Future Work

• There is currently a bug with Rust's libtest where log output that gets printed to stdout from a thread other than the main thread is not captured by libtest's stdout log capturing mechanism. This causes a problem with test reporting because the logs are not saved by Rust in a way that allows them to be show later as part of a particular test run, and additionally, it makes it so that the test report parsing tool has to separate out the json log results from the application log printing, which is prone to error. Additionally, sometimes stdout flushing can cause the json log results to get intermingled with the application log output, thus resulting in corrupted test reports. The bug report for this issue can be found here: cargo test doesn't capture print from threads rust-lang/rust#42474. At present the only easy workaround is to disable all application logging, but this is not a viable solution since it would make debugging failed tests on CI significantly more difficult.

Demo

Here is what the Tests tab of a CI run currently looks like:

Here is what the Tests tab of a successful CI run will look like after this PR:

Here is what the Tests tab of a failed CI run will look like after this PR: