Maven Surefire Provider should support configuration/excludeGroups directly

[ttrelle]

Overview

I'd like to report a possible bug with the JUnit 5 vintage engine.

Versions:

• JUnit4: 4.12
• JUnit5: 5.0.3
• Maven Surefire Plugin: 2.21.0

Observed behaviour: In a project with mixed JUnit4 and JUnit5 tests, I am using the maven-surefire-plugin together with the junit-platform-surefire-provider to run both kind of tests. Under these circumstances the @Category annotation from JUnit 4 is ignored. The test runs although the category is excluded in the configuration of the plugin:

		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-surefire-plugin</artifactId>
				<version>2.21.0</version>
				<configuration>
					<useSystemClassLoader>false</useSystemClassLoader>
					<excludedGroups>tests.IntegrationTest</excludedGroups>
				</configuration>
<!-- ... -->

Desired behaviour: the excludedGroups configuration is honored and tests annotated with @Category(IntegrationTest.class) are skipped.

If I delete all dependencies of the maven-surefire-plugin (so that the old JUnit4 test runner kicks in), the @Category annotation is processed as desired.

Steps to Reproduce

I set up a minimal project that demonstrates the bug here: ttrelle/junit5-vintage-junit4-category

[sbrannen]

Thanks for raising the issue and providing an example project!

[sbrannen]

In your example project, the fully-qualified class name for the category interface is actually junit4.IntegrationTest instead of tests.IntegrationTest, but making that correction doesn't change anything.

So it appears this is a real bug.

We'll look into it...

[sbrannen]

FYI: it would appear that Jupiter's @Tag annotation is also not supported.

[marcphilipp]

Like <excludeTags/> (see User Guide), the <excludedGroups/> element currently needs to be nested inside a <properties/> element when using the junit-platform-surefire-provider:

<plugin>
	<artifactId>maven-surefire-plugin</artifactId>
	<version>2.21.0</version>
	<configuration>
		<useSystemClassLoader>false</useSystemClassLoader>
		<properties>
			<excludedGroups>junit4.IntegrationTest</excludedGroups>
		</properties>
	</configuration>
	<dependencies>
		<dependency>
			<groupId>org.junit.platform</groupId>
			<artifactId>junit-platform-surefire-provider</artifactId>
			<version>1.2.0</version>
		</dependency>
		...
	</dependencies>
</plugin>

[sormuras]

This feature will be realized by the provider shipping with Surefire 2.22.0 -- see https://github.com/apache/maven-surefire/pull/184/files#diff-794c5135083cc0a67561991ccd77fa43R144

Keeping this issue open until Surefire version 2.22.0 or later is available that includes the support for <groups> and <excludedGroups>

[sormuras]

As Surefire 2.22.0 is released you now may use <groups> and <excludedGroups> directly to configure which JUnit Platform Tags are included and excluded. Copied from http://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html

Filtering by Tags

You can use JUnit5 Tags and filter tests by tags or tag expressions.

To include tags or tag expressions, use groups.
To exclude tags or tag expressions, use either excludedGroups.

...
<build>
    <plugins>
        ...
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.22.0</version>
            <configuration>
                <groups>acceptance | !feature-a</groups>
                <excludedGroups>integration, regression</excludedGroups>
            </configuration>
        </plugin>
    </plugins>
</build>
...

Thus, closing this issue.

If you find any related (or new) problems please report them here: https://issues.apache.org/jira/projects/SUREFIRE/issues