Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JUnit test failure summary prints too many tests #2310

Closed
ericzundel opened this issue Oct 3, 2015 · 1 comment
Closed

JUnit test failure summary prints too many tests #2310

ericzundel opened this issue Oct 3, 2015 · 1 comment
Assignees
@gmalmquist
Labels
bug

Comments

@ericzundel
Copy link
Member

In https://rbcommons.com/s/twitter/r/2916/ we added a summary of failures at the end of the junit run.

If there is a single test failure, it is printing out more tests than just the one that actually failed.

                     There was 1 failure:
                     1) testBlockingGeocoder(com.squareup.geocoding.actions.ReverseGeocodeActionTest)
                     java.lang.AssertionError: expected:<true> but was:<false>
                        at org.junit.Assert.fail(Assert.java:88)
                        at org.junit.Assert.failNotEquals(Assert.java:834)
                        at org.junit.Assert.assertEquals(Assert.java:118)
                        at org.junit.Assert.assertEquals(Assert.java:144)
                        at com.squareup.geocoding.actions.ReverseGeocodeActionTest.testBlockingGeocoder(ReverseGeocodeActionTest.java:447)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        at java.lang.reflect.Method.invoke(Method.java:497)
                        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
                        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
                        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
                        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
                        at com.squareup.testing.runners.BaseTestRunner$3.evaluate(BaseTestRunner.java:78)
                        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
                        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
                        at com.squareup.testing.guice.ComponentTestSupportRule$1.evaluate(ComponentTestSupportRule.java:26)
                        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
                        at com.squareup.testing.runners.BaseTestRunner$2.evaluate(BaseTestRunner.java:58)
                        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
                        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
                        at com.squareup.testing.guice.AbstractGuiceTestRunner.runChild(AbstractGuiceTestRunner.java:51)
                        at com.squareup.testing.guice.AbstractGuiceTestRunner.runChild(AbstractGuiceTestRunner.java:40)
                        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
                        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
                        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
                        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
                        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
                        at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
                        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
                        at com.squareup.testing.runners.BaseTestRunner$1.evaluate(BaseTestRunner.java:44)
                        at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
                        at __shaded_by_pants__.org.pantsbuild.tools.junit.impl.CompositeRequest.runChild(CompositeRequest.java:66)
                        at __shaded_by_pants__.org.pantsbuild.tools.junit.impl.ConcurrentCompositeRequest$1$1.run(ConcurrentCompositeRequest.java:36)
                        at __shaded_by_pants__.org.pantsbuild.tools.junit.impl.ConcurrentRunnerScheduler.finished(ConcurrentRunnerScheduler.java:85)
                        at __shaded_by_pants__.org.pantsbuild.tools.junit.impl.ConcurrentCompositeRequest$1.evaluate(ConcurrentCompositeRequest.java:45)
                        at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
                        at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
                        at __shaded_by_pants__.org.pantsbuild.tools.junit.impl.ConsoleRunnerImpl.run(ConsoleRunnerImpl.java:334)
                        at __shaded_by_pants__.org.pantsbuild.tools.junit.impl.ConsoleRunnerImpl.main(ConsoleRunnerImpl.java:660)
                        at org.pantsbuild.tools.junit.ConsoleRunner.main(ConsoleRunner.java:12)

                     FAILURES!!!
                     Tests run: 157,  Failures: 1


FAILURE: 
    geocoding/src/test/java:test
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testBlockingGeocoder
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testCacheAuthoritativeAddresses
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testCacheAuthoritativeNoResultAnswers
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testCacheWriteFailure
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testDontCacheNonAuthoritativeAddresses
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testDontCacheNonAuthoritativeNoResultAnswers
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testDontCacheUnknownResultAnswers
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testDontUseImpreciseMethodWhenNotRequested
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testFastTimeoutBlockingGeocoder
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testQueryGeocoders
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testSimulateRedisBlockAndQueryFail
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testUseContingencyAfterDelay
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testUseImpreciseMethodWhenRequested
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testUseImpreciseMethodWithRestrictionWhenRequested
        com.squareup.geocoding.actions.ReverseGeocodeActionTest#testUseNoResultAnswers

The workaround is to turn off this summary:

[test.junit]
failure-summary: False
@ericzundel ericzundel added the bug label Oct 3, 2015
@gmalmquist
Copy link
Contributor

I guess that flag came in handy after all. I must have made some incorrect assumptions regarding how the test report xml is formatted, I'll look into it.

gmalmquist added a commit to gmalmquist/pants that referenced this issue Oct 5, 2015
@gmalmquist
Test-cases that passed are now properly omitted the junit summary.
107ce0f 
I introduced a bug in https://rbcommons.com/s/twitter/r/2916/
wherein all test-cases in a class were printed if any of the
test-cases in the class had failed. This occurred due to my
incorrect assumption that only testcases that failed were listed
in the .xml report from the junit runner.

Now junit_run.py only prints a summary of those test-cases that
actually failed.

This fixes pantsbuild#2310.
@gmalmquist gmalmquist mentioned this issue Oct 5, 2015
Closed
gmalmquist added a commit to gmalmquist/pants that referenced this issue Oct 6, 2015
@gmalmquist
Test-cases that passed are now properly omitted the junit summary.
c8477c7 
I introduced a bug in https://rbcommons.com/s/twitter/r/2916/
wherein all test-cases in a class were printed if any of the
test-cases in the class had failed. This occurred due to my
incorrect assumption that only testcases that failed were listed
in the .xml report from the junit runner.

Now junit_run.py only prints a summary of those test-cases that
actually failed.

This fixes pantsbuild#2310.
@asherf asherf mentioned this issue Aug 16, 2020
Merged
Eric-Arellano pushed a commit that referenced this issue Aug 16, 2020
@asherf
Upgrade setuptools from 49.2.0 to 49.60 (#10626)
8c958a1 
https://github.com/pypa/setuptools/releases/tag/v49.6.0

https://github.com/pypa/setuptools/blob/master/CHANGES.rst

* #2129: In pkg_resources, no longer detect any pathname ending in .egg as a Python egg. Now the path must be an unpacked egg or a zip file.
* #2306: When running as a PEP 517 backend, setuptools does not try to install
  ``setup_requires`` itself. They are reported as build requirements for the
  frontend to install.
* #2310: Updated vendored packaging version to 20.4.
* #2300: Improve the ``safe_version`` function documentation
* #2297: Once again, in stubs prefer exec_module to the deprecated load_module.
* #2316: Removed warning when ``distutils`` is imported before ``setuptools`` when ``distutils`` replacement is not enabled.
* #2259: Setuptools now provides a .pth file (except for editable installs of setuptools) to the target environment to ensure that when enabled, the setuptools-provided distutils is preferred before setuptools has been imported (and even if setuptools is never imported). Honors the SETUPTOOLS_USE_DISTUTILS environment variable.
pypa/setuptools#2257: Fixed two flaws in distutils._msvccompiler.MSVCCompiler.spawn.

instead of #10561

[ci skip-build-wheels]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gmalmquist gmalmquist

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Test-cases that passed are now properly omitted the junit summary. gmalmquist/pants
2 participants
@ericzundel @gmalmquist