stack test should not run concurrently by default

[pseudonom]

As far I understand things, in a multi-project stack directory, tests are run concurrently (using getNumCapabilities). This has the potential to cause spurious errors for test suites which are not concurrency safe (and has for a stack directory I'm working with). That is,stack test fails where stack test foo && stack test bar succeeds.

Setting jobs: 1 in stack.yaml resolves the problem, but I didn't find this in the documentation.

[mboes]

I consider this a doc problem, rather than a defaults problem: if test suites claim exclusive access to system resources, then arguably this is a bug in the test suites, or if test suites really must claim exclusive access to system resources, then they should learn to block rather than fail waiting for the resource to become available.

I think it's important to make the common case predictable and efficient. The common case is that test suites don't claim global resources, hence making things single-threaded by default would be an unnecessary pessimization for the common case. Running test suites concurrently is also consistent with stack build behaviour.

[pseudonom]

I agree that better documentation would go a long way.

Our use case doesn't seem so exotic. We're doing integration tests with a database. At the end of each test suite, the suite cleans up after itself by truncating the tables it touched. When run concurrently, this clean-up is likely to break any other executing suites touching those tables.

I think it's important to make the common case predictable

I think the current behavior is rather unexpected. It took me quite a while to figure out why cd foo; stack test and stack test foo worked where stack test did not.

Running test suites concurrently is also consistent with stack build behaviour.

From the outside, it's not apparent to me why stack build and stack test should be consistent with each other. They seem like very different operations.

[snoyberg]

For the record, we've had users request specifically to have their tests run in parallel, so it's impossible to have a default that satisfies everyone's expected default behavior. The current behavior though does provide consistency with both cabal and the rest of the build experience (which is in parallel), and for most use cases will work and be faster.

In other words: I agree with @mboes here about keeping the default as-is.

[gregwebs]

Regardless of the default, is there a way to configure the number of jobs differently for building and testing (or otherwise indicate to only run tests in serial) ?

[snoyberg]

No, not currently. Added a lock such that only one test job can run at a time is straightforward though.

[gregwebs]

since it seems unlikely that the default will be changed, I opened #492. We can close this issue after updating the docs

[gregwebs]

I put a blurb about project testing at the bottom of the Architecture page. Feel free to edit/move it

[mboes]

I don't think caveats about project testing and further info about usage have their place in the Architecture doc, which documents the design. Moving to the FAQ.

[mboes]

Moved here: https://github.com/commercialhaskell/stack/wiki/FAQ#my-tests-are-failing-what-should-i-do. @pseudonom please confirm.

[gregwebs]

great! I removed a few superfluous sentences from the entry.

[pseudonom]

Thanks, looks good to me.