test.pytest shows more tests failed than actually failed

[kwlzn]

seen here: https://travis-ci.org/pantsbuild/pants/jobs/252109723

the pytest summary indicates a single test failure:

 1 failed, 1036 passed, 13 skipped, 1 xfailed, 2 xpassed in 342.35 seconds 

but according to the log output, all tests in the shard report as FAILURE:

tests.python.pants_test.build_graph.target                                      .....   FAILURE
tests.python.pants_test.build_graph.build_file_parser                           .....   FAILURE
tests.python.pants_test.build_graph.build_file_aliases                          .....   FAILURE
tests.python.pants_test.build_graph.address                                     .....   FAILURE
tests.python.pants_test.build_graph.build_file_address_mapper                   .....   FAILURE
tests.python.pants_test.build_graph.build_configuration                         .....   FAILURE
tests.python.pants_test.build_graph.source_mapper                               .....   FAILURE
tests.python.pants_test.build_graph.build_graph                                 .....   FAILURE
tests.python.pants_test.build_graph.scopes                                      .....   FAILURE
tests.python.pants_test.backend.codegen.antlr.java.java                         .....   FAILURE
tests.python.pants_test.backend.jvm.subsystems.jar_dependency_management        .....   FAILURE
tests.python.pants_test.backend.jvm.subsystems.shader                           .....   FAILURE
tests.python.pants_test.backend.jvm.subsystems.custom_scala                     .....   FAILURE
tests.python.pants_test.backend.project_info.tasks.depmap                       .....   FAILURE
tests.python.pants_test.backend.project_info.tasks.dependencies                 .....   FAILURE
tests.python.pants_test.backend.project_info.tasks.filedeps                     .....   FAILURE
tests.python.pants_test.backend.project_info.tasks.ide_gen                      .....   FAILURE
tests.python.pants_test.backend.project_info.tasks.export                       .....   FAILURE
...

cc @jsirois

[jsirois]

A known compromise in the invalidation work. I'll think about how to make this better without stepping into results merging, which I had a strong preference for avoiding from the get go.

[stuhood]

@jsirois : If it makes things any easier: I don't think there is any need to cache anything for failing tests... I wouldn't expect a run containing a failing test to have hit the cache at all.

[jsirois]

No cache here. This is just a run on N targets in fast mode that, w/o calculating target->test ownership, cannot say more than the whole run failed. The solution will just be making the code more complex to deal with fast vs no fast differently for display purposes.

[jsirois]

Addressing this before finishing off caching for pytest in #4734.

[jsirois]

OK - I think I have a decent compromise. The summary is now only output for --no-fast as was the case in the past. So:

1. --fast, 1 partition with all targets: no summary
2. --no-fast, 1 partition per-target: summary
3. multiple partitions with multiple targets each: no summary

The 3rd case is not good, but we also don't actually have this partitioning case today, so TODO added.

[jsirois]

Erm, except that means no output at all for --fast on "cached" success, ie:

...
08:50:11 00:01   [test]
08:50:11 00:01     [test-jvm-prep-command]
08:50:11 00:01       [jvm_prep_command]
08:50:11 00:01     [test-prep-command]
08:50:11 00:01     [test]
08:50:11 00:01     [pytest-prep]
08:50:11 00:01     [pytest]
08:50:12 00:02     [junit]
08:50:12 00:02     [go]
08:50:12 00:02     [node]
08:50:12 00:02   [complete]
               SUCCESS

Which looks alot like a faulty noop. May re-introduce the summary for --fast, but only on success.