Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unsupported or missing version of JUnit 4 should fail the build #1944

Closed
nielsbasjes opened this issue Jul 3, 2019 · 15 comments
Closed

Unsupported or missing version of JUnit 4 should fail the build #1944

nielsbasjes opened this issue Jul 3, 2019 · 15 comments

Comments

@nielsbasjes
Copy link

I'm using junit and vintage from the bom 5.5.0 in a Maven based project.
https://github.com/nielsbasjes/yauaa

In this project I have created a udf for Apache Pig.
In the pig dependency for this udf I also get junit 4.11.

The consequence of this older version of junit is that I get this perfectly fine stack trace during my build:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
Jul 02, 2019 11:42:33 PM org.junit.platform.launcher.core.DefaultLauncher handleThrowable
WARNING: TestEngine with ID 'junit-vintage' failed to discover tests
org.junit.platform.commons.JUnitException: Unsupported version of junit:junit: 4.11. Please upgrade to version 4.12 or later.
	at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:39)
	at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:32)
	at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)

and later on

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

The bug I'm reporting is that I think this is should be a fatal error and not just a warning.

None of the tests I have ran and the ONLY reason I noticed this is because I saw a sudden drop in the code coverage of the tests.

So if this problem happens then please fail the build.

@sbrannen
Copy link
Member

sbrannen commented Jul 3, 2019

Tentatively slated for 5.6 M1 for team discussion

@marcphilipp
Copy link
Member

Related to #242.

@sbrannen sbrannen changed the title "Unsupported version of junit:junit:" should fail the build. "Unsupported version of junit:junit:" should fail the build Jul 3, 2019
@marcphilipp
Copy link
Member

Team Decision: We agree that this should fail during test execution. 🙂

@marcphilipp marcphilipp modified the milestones: 5.6 M1, 5.6 Backlog Jul 5, 2019
@marcphilipp marcphilipp modified the milestones: 5.6 Backlog, 5.6 M1 Jul 15, 2019
@sbrannen sbrannen changed the title "Unsupported version of junit:junit:" should fail the build Unsupported version of JUnit 4 should fail the build Jul 16, 2019
@sbrannen sbrannen changed the title Unsupported version of JUnit 4 should fail the build Unsupported or missing version of JUnit 4 should fail the build Sep 23, 2019
@sbrannen
Copy link
Member

I updated the title to point out that a missing JUnit 4 artifact should also fail the build for use cases where someone excluded JUnit 4 from the classpath but left junit-vintage-engine in the classpath.

The logic in JUnit4VersionCheck should already handle this use case as well, but I wanted to ensure we cover both use cases explicitly: missing and unsupported JUnit 4 artifacts.

@marcphilipp
Copy link
Member

This was fixed in #1909.

@nielsbasjes Would you care to verify using the latest snapshot?

@nielsbasjes
Copy link
Author

nielsbasjes commented Sep 24, 2019

The problem has not been fixed in junit-vintage-engine-5.6.0-20190924.094617-81.jar.

I get

WARNING: TestEngine with ID 'junit-vintage' failed to discover tests
org.junit.platform.commons.JUnitException: Unsupported version of junit:junit: 4.11. Please upgrade to version 4.12 or later.

then

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

and

[INFO] BUILD SUCCESS

To facilitate you guys repeating what I have I created a testing branch on my project
https://github.com/nielsbasjes/yauaa/tree/ReproduceJunit5Issue1944

I have the sonatype snapshot repo enabled using my ~/.m2/settings.xml

To limit the time needed to reproduce this (some of my tests take a while) do:

mvn clean install -DskipTests=true

verify I actually use the right version in the pom.xml

fgrep junit5.version ~/.m2/repository/nl/basjes/parse/useragent/yauaa-parent/5.13-SNAPSHOT/yauaa-parent-5.13-SNAPSHOT.pom 
    <junit5.version>5.6.0-SNAPSHOT</junit5.version>
    <version>${junit5.version}</version>

then

cd udfs/pig/

verify the actual dependency versions for this module

$ mvn dependency:list | fgrep junit
[INFO]    org.junit.jupiter:junit-jupiter-api:jar:5.6.0-SNAPSHOT:test
[INFO]    org.junit.platform:junit-platform-commons:jar:1.6.0-SNAPSHOT:test
[INFO]    org.junit.jupiter:junit-jupiter-engine:jar:5.6.0-SNAPSHOT:test
[INFO]    org.junit.platform:junit-platform-engine:jar:1.6.0-SNAPSHOT:test
[INFO]    org.junit.vintage:junit-vintage-engine:jar:5.6.0-SNAPSHOT:test
[INFO]    junit:junit:jar:4.11:provided

then

mvn clean test

I see this (which says BUILD SUCCESS):

$ mvn clean test
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building Yauaa : UDF : Apache Pig 5.13-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ yauaa-pig ---
[INFO] Deleting /home/nbasjes/workspace/Prive/yauaa_/udfs/pig/target
[INFO] 
[INFO] --- git-commit-id-plugin:3.0.0:revision (get-the-git-infos) @ yauaa-pig ---
[INFO] 
[INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (enforce-maven) @ yauaa-pig ---
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.4:prepare-agent (pre-unit-test) @ yauaa-pig ---
[INFO] jacoco.surefireArgLine set to -javaagent:/home/nbasjes/.m2/repository/org/jacoco/org.jacoco.agent/0.8.4/org.jacoco.agent-0.8.4-runtime.jar=destfile=/home/nbasjes/workspace/Prive/yauaa_/udfs/pig/target/jacoco.exec
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ yauaa-pig ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/nbasjes/workspace/Prive/yauaa_/udfs/pig/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ yauaa-pig ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/nbasjes/workspace/Prive/yauaa_/udfs/pig/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ yauaa-pig ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ yauaa-pig ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/nbasjes/workspace/Prive/yauaa_/udfs/pig/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:3.0.0-M3:test (default-test) @ yauaa-pig ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
Sep 24, 2019 12:24:27 PM org.junit.platform.launcher.core.DefaultLauncher handleThrowable
WARNING: TestEngine with ID 'junit-vintage' failed to discover tests
org.junit.platform.commons.JUnitException: Unsupported version of junit:junit: 4.11. Please upgrade to version 4.12 or later.
	at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:39)
	at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:32)
	at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)
	at org.junit.platform.launcher.core.DefaultLauncher.discoverEngineRoot(DefaultLauncher.java:168)
	at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:155)
	at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:120)
	at org.apache.maven.surefire.junitplatform.TestPlanScannerFilter.accept(TestPlanScannerFilter.java:56)
	at org.apache.maven.surefire.util.DefaultScanResult.applyFilter(DefaultScanResult.java:102)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.scanClasspath(JUnitPlatformProvider.java:143)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

Sep 24, 2019 12:24:27 PM org.junit.platform.launcher.core.DefaultLauncher handleThrowable
WARNING: TestEngine with ID 'junit-vintage' failed to discover tests
org.junit.platform.commons.JUnitException: Unsupported version of junit:junit: 4.11. Please upgrade to version 4.12 or later.
	at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:39)
	at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:32)
	at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)
	at org.junit.platform.launcher.core.DefaultLauncher.discoverEngineRoot(DefaultLauncher.java:168)
	at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:155)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] --- maven-checkstyle-plugin:3.1.0:check (checkstyle-check) @ yauaa-pig ---
[INFO] Starting audit...
Audit done.
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.4:report (post-unit-test) @ yauaa-pig ---
[INFO] Skipping JaCoCo execution due to missing execution data file.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5.658 s
[INFO] Finished at: 2019-09-24T12:24:28+02:00
[INFO] Final Memory: 48M/599M
[INFO] ------------------------------------------------------------------------

@marcphilipp
Copy link
Member

@nielsbasjes Thanks for testing! Must be Surefire swallowing the error. I'll take a closer look myself.

@marcphilipp marcphilipp self-assigned this Sep 24, 2019
@nielsbasjes
Copy link
Author

Just by looking at the messages it seems the JUnitException is classified as a WARNING.

And a warning should not fail the build...

@marcphilipp
Copy link
Member

@nielsbasjes We still log this as a warning when Surefire calls discover(). However, if Surefire called execute() with the same TestPlan, it would throw an exception. I'll add an integration test.

@marcphilipp
Copy link
Member

@nielsbasjes The missing piece was that Surefire bundles an old version of junit-platform-launcher. If you add the snapshot version to the test scoped dependencies, it fails.

I've created #2028 to ensure Surefire and other build tools and IDEs upgrade their bundled Launcher once 5.6 is released.

@nielsbasjes
Copy link
Author

@marcphilipp Thanks. I've check it and when I make my maven look like this

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.junit</groupId>
        <artifactId>junit-bom</artifactId>
        <version>5.6.0-SNAPSHOT</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>

    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-engine</artifactId>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.junit.vintage</groupId>
      <artifactId>junit-vintage-engine</artifactId>
      <scope>test</scope>
    </dependency>

    <dependency>
      <groupId>org.junit.platform</groupId>
      <artifactId>junit-platform-launcher</artifactId>
      <scope>test</scope>
    </dependency>

  </dependencies>

it correctly fails with

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
Sep 29, 2019 4:12:16 PM org.junit.platform.launcher.core.DefaultLauncher discoverEngineRoot
SEVERE: TestEngine with ID 'junit-vintage' failed to discover tests
org.junit.platform.commons.JUnitException: Unsupported version of junit:junit: 4.11. Please upgrade to version 4.12 or later.
	at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:39)
	at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:32)
	at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)
	at org.junit.platform.launcher.core.DefaultLauncher.discoverEngineRoot(DefaultLauncher.java:177)
	at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:166)
	at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:122)
	at org.apache.maven.surefire.junitplatform.TestPlanScannerFilter.accept(TestPlanScannerFilter.java:56)
	at org.apache.maven.surefire.util.DefaultScanResult.applyFilter(DefaultScanResult.java:102)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.scanClasspath(JUnitPlatformProvider.java:143)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

Sep 29, 2019 4:12:16 PM org.junit.platform.launcher.core.DefaultLauncher discoverEngineRoot
SEVERE: TestEngine with ID 'junit-vintage' failed to discover tests
org.junit.platform.commons.JUnitException: Unsupported version of junit:junit: 4.11. Please upgrade to version 4.12 or later.
	at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:39)
	at org.junit.vintage.engine.JUnit4VersionCheck.checkSupported(JUnit4VersionCheck.java:32)
	at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)
	at org.junit.platform.launcher.core.DefaultLauncher.discoverEngineRoot(DefaultLauncher.java:177)
	at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:166)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:130)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
	at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

[INFO] Running junit-vintage
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.006 s <<< FAILURE! - in junit-vintage
[ERROR] junit-vintage  Time elapsed: 0.003 s  <<< ERROR!
org.junit.platform.commons.JUnitException: TestEngine with ID 'junit-vintage' failed to discover tests
	at org.junit.vintage.engine.VintageTestEngine.discover(VintageTestEngine.java:61)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   VintageTestEngine » JUnit TestEngine with ID 'junit-vintage' failed to discove...
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.972 s
[INFO] Finished at: 2019-09-29T16:12:17+02:00
[INFO] Final Memory: 24M/396M
[INFO] ------------------------------------------------------------------------

Thanks for fixing this.

@marcphilipp
Copy link
Member

@nielsbasjes Thanks for double-checking! 👍

@raul1991
Copy link

raul1991 commented Apr 26, 2020

I had this problem when using junit 4.8.2 and then changing it to 4.13 worked (see doc). I am just commenting it here so as to validate my understanding that vintage engine requires junit 4.13 and not older to be able to detect tests.

Please correct me if I am wrong.

@sbrannen
Copy link
Member

Please correct me if I am wrong.

JUnit Vintage requires JUnit 4.12 or higher.

@raul1991
Copy link

Please correct me if I am wrong.

JUnit Vintage requires JUnit 4.12 or higher.

I directly used 4.13 so would have missed that. Thanks for clearing it out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants