Fail the build only after all test runners have run

[lukesmith]

In some cases you may have multiple test runners, nunit, jslint, FxCop and you would only want to fail the build after all tests have run failing or not.

[gshutler]

I'm not sure this makes sense for test runners as I think I mentioned before. For test cases I would lean towards failing fast and generally putting unit tests before integration. If integration tests take 10+ minutes (quite common once you start using them) you wouldn't want to wait for them to run if the unit tests have failed.

However, I see there being a difference for "code analysis" tools such as jsLint, FxCop, NCover, etc. Once your tests have passed, or your code compiled not sure which would be best or if they are separate, it makes sense for all of these to run even if one of them trigger a failure by violating some sort of threshold.

What do you think?

[lukesmith]

When a test fails, what causes bozo to terminate? The post test hooks would still need to run so to get the results into teamcity/jenkins.

If you have a number of test runners present (say you have javascript tests and nunit tests) i'd expect bozo to be able to say "hey, something failed" but continue running all unit test runners so you know where all the issues are for the build rather than just the first (Integration tests shouldn't run if unit tests fail.)

Maybe bozo needs to be able to distinguish between integration test runners and unit test runners?