Use the JUnit (5) Platform to execute Cucumber scenarios.
Add the cucumber-junit-platform-engine dependency to your pom.xml and use
the cucumber-bom for dependency management:
<dependency>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-junit-platform-engine</artifactId>
<scope>test</scope>
</dependency>This will allow IntelliJ IDEA, Eclipse, Maven, Gradle, etc, to discover, select and execute Cucumber scenarios.
The JUnit Platform provides a single interface for tools and IDE's to discover, select and execute tests from different test engines. Conceptually this looks like this:
erDiagram
"IDE, Maven, Gradle or Console Launcher" ||--|{ "JUnit Platform" : "requests discovery and execution"
"Console Launcher" ||--|{ "JUnit Platform" : "requests discovery and execution"
"JUnit Platform" ||--|{ "Cucumber Test Engine": "forwards request"
"JUnit Platform" ||--|{ "Jupiter Test Engine": "forwards request"
"Cucumber Test Engine" ||--|{ "Feature Files": "discovers and executes"
"Jupiter Test Engine" ||--|{ "Test Classes": "discovers and executes"
Maven Surefire and Gradle do not yet support discovery of non-class based tests (see: gradle/#4773, SUREFIRE-1724). As a workaround, you can either use:
- the JUnit Platform Suite Engine;
- the JUnit Platform Console Launcher or;
- the Gradle Cucumber-Companion plugins for Gradle and Maven.
- the Cucable plugin for Maven.
The JUnit Platform Suite Engine can be used to run Cucumber. See Suites with different configurations for a brief how to.
Because Surefire and Gradle reports provide the results in a <Class Name> - <Method Name>
format, only scenario names or example numbers are reported. This
can make for hard to read reports. To improve the readability of the reports
provide the cucumber.junit-platform.naming-strategy=long configuration
parameter. This will include the feature name as part of the test name.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.2.5</version>
<configuration>
<properties>
<configurationParameters>
cucumber.junit-platform.naming-strategy=long
</configurationParameters>
</properties>
</configuration>
</plugin>tasks.test {
useJUnitPlatform()
systemProperty("cucumber.junit-platform.naming-strategy", "long")
}You can integrate the JUnit Platform Console Launcher in your build by using either the Maven Antrun plugin or the Gradle JavaExec task.
Add the following to your pom.xml:
<dependencies>
....
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-console</artifactId>
<version>${junit-platform.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<!--Work around. Surefire does not use JUnit's Test Engine discovery functionality -->
<id>CLI-test</id>
<phase>integration-test</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<echo message="Running JUnit Platform CLI"/>
<java classname="org.junit.platform.console.ConsoleLauncher"
fork="true"
failonerror="true"
newenvironment="true"
maxmemory="512m"
classpathref="maven.test.classpath">
<arg value="--include-engine"/>
<arg value="cucumber"/>
<arg value="--scan-classpath"/>
<arg value="${project.build.testOutputDirectory}"/>
</java>
</target>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>Add the following to your build.gradle.kts:
tasks {
val consoleLauncherTest by registering(JavaExec::class) {
dependsOn(testClasses)
val reportsDir = file("$buildDir/test-results")
outputs.dir(reportsDir)
classpath = sourceSets["test"].runtimeClasspath
main = "org.junit.platform.console.ConsoleLauncher"
args("--scan-classpath")
args("--include-engine", "cucumber")
args("--reports-dir", reportsDir)
}
test {
dependsOn(consoleLauncherTest)
exclude("**/*")
}
}To select a single scenario or feature the cucumber.features property can be
used. Because this property will cause Cucumber to ignore any other selectors
from JUnit, it is prudent to execute only the Cucumber engine.
To select the scenario on line 10 of the example.feature file use:
mvn test -Dsurefire.includeJUnit5Engines=cucumber -Dcucumber.plugin=pretty -Dcucumber.features=path/to/example.feature:10 TODO: (I don't know how. Feel free to send a pull request. ;))
The JUnit Platform Suite Engine can be used to run Cucumber multiple times with different configurations. Conceptually this looks like this:
erDiagram
"IDE, Maven, Gradle or Console Launcher" ||--|{ "JUnit Platform" : "requests discovery and execution"
"JUnit Platform" ||--|{ "Suite Test Engine": "forwards request"
"Suite Test Engine" ||--|{ "@Suite annotated class A" : "discovers and executes"
"Suite Test Engine" ||--|{ "@Suite annotated class B" : "discovers and executes"
"@Suite annotated class A" ||--|{ "JUnit Platform (A)" : "requests discovery and execution"
"@Suite annotated class B" ||--|{ "JUnit Platform (B)" : "requests discovery and execution"
"JUnit Platform (A)" ||--|{ "Cucumber Test Engine (A)": "forwards request"
"JUnit Platform (B)" ||--|{ "Cucumber Test Engine (B)": "forwards request"
"Cucumber Test Engine (A)" ||--|{ "Feature Files (A)": "discovers and executes"
"Cucumber Test Engine (B)" ||--|{ "Feature Files (B)": "discovers and executes"
To use, add the junit-platform-suite dependency and use
the junit-bom for dependency management:
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-suite</artifactId>
<scope>test</scope>
</dependency>Then define suites as needed using the annotation from the
org.junit.platform.suite.api
package:
package com.example;
import org.junit.platform.suite.api.ConfigurationParameter;
import org.junit.platform.suite.api.IncludeEngines;
import org.junit.platform.suite.api.SelectPackages;
import org.junit.platform.suite.api.Suite;
import static io.cucumber.junit.platform.engine.Constants.GLUE_PROPERTY_NAME;
@Suite
@IncludeEngines("cucumber")
@SelectPackages("com.example")
@ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "com.example")
public class RunCucumberTest {
}By default, Cucumber runs tests sequentially in a single thread. Running tests
in parallel is available as an opt-in feature. To enable parallel execution, set
the cucumber.execution.parallel.enabled configuration parameter to true,
e.g., in junit-platform.properties.
To control properties such as the desired parallelism and maximum parallelism,
Cucumber supports JUnit 5s ParallelExecutionConfigurationStrategy. Cucumber
provides two implementations: dynamic and fixed that can be set through
cucumber.execution.parallel.config.strategy. You may also implement a custom
strategy.
-
dynamic: Computes the desired parallelism as<available cores>*cucumber.execution.parallel.config.dynamic.factor. -
fixed: Setcucumber.execution.parallel.config.fixed.parallelismto the desired parallelism andcucumber.execution.parallel.config.fixed.max-pool-sizeto the maximum pool size of the underlying ForkJoin pool. -
custom: Specify a customParallelExecutionConfigurationStrategyimplementation throughcucumber.execution.parallel.config.custom.class.
If no strategy is specified Cucumber will use the dynamic strategy with a
factor of 1.
Note: While .fixed.max-pool-size effectively limits the maximum number of
concurrent threads, Cucumber does not guarantee that the number of concurrently
executing scenarios will not exceed this. See junit5/#3108
for details.
To avoid flaky tests when multiple scenarios manipulate the same resource, tests can be synchronized on that resource.
To synchronize a scenario on a specific resource, the scenario must be tagged and this tag mapped to a lock for the specific resource. A resource is identified by an arbitrary string and can be either locked with a read-write-lock, or a read-lock.
For example, the following tags:
Feature: Exclusive resources
@reads-and-writes-system-properties
Scenario: first example
Given this reads and writes system properties
When it is executed
Then it will not be executed concurrently with the second example
@reads-system-properties
Scenario: second example
Given this reads system properties
When it is executed
Then it will not be executed concurrently with the first examplewith this configuration:
cucumber.execution.exclusive-resources.reads-and-writes-system-properties.read-write=java.lang.System.properties
cucumber.execution.exclusive-resources.reads-system-properties.read=java.lang.System.propertieswhen executing the first scenario tagged with
@reads-and-writes-system-properties will lock the java.lang.System.properties
resource with a read-write lock and will not be concurrently executed with the
second scenario that locks the same resource with a read lock.
Note: The @ from the tag is not included in the property name.
Note: For canonical resource names see junit5/Resources.java
To ensure that a scenario runs while no other scenarios are running the global
resource org.junit.platform.engine.support.hierarchical.ExclusiveResource.GLOBAL_KEY
can be used.
Feature: Isolated scenarios
@isolated
Scenario: isolated example
Given this scenario runs isolated
When it is executed
Then it will not be executed concurrently with the second or third example
Scenario: second example
When it is executed
Then it will not be executed concurrently with the isolated example
And it will be executed concurrently with the third example
Scenario: third example
When it is executed
Then it will not be executed concurrently with the isolated example
And it will be executed concurrently with the second examplewith this configuration:
cucumber.execution.exclusive-resources.isolated.read-write=org.junit.platform.engine.support.hierarchical.ExclusiveResource.GLOBAL_KEYBy default, when parallel execution is enabled, scenarios and examples are
executed in parallel. Due to limitations, JUnit 4 could only execute features in
parallel. This behaviour can be restored by setting the configuration parameter
cucumber.execution.execution-mode.feature to same_thread.
Cucumber receives its configuration from the JUnit Platform. To see how these can be supplied; see the JUnit documentation 4.5. Configuration Parameters. For documentation on Cucumber properties, see Constants.
cucumber.ansi-colors.disabled= # true or false.
# default: false
cucumber.filter.name= # a regular expression.
# only scenarios with matching names are executed.
# combined with cucumber.filter.tags using "and" semantics.
# example: ^Hello (World|Cucumber)$
# note: To ensure consistent reports between Cucumber and
# JUnit 5 prefer using JUnit 5s discovery request filters
# or JUnit 5 tag expressions instead.
cucumber.features= # comma separated paths to feature files.
# example: path/to/example.feature, path/to/other.feature
# note: When used any discovery selectors from the JUnit
# Platform will be ignored. This may lead to multiple
# executions of Cucumber. For example when used in
# combination with the JUnit Platform Suite Engine.
# When using Cucumber through the JUnit Platform
# Launcher API or the JUnit Platform Suite Engine, it is
# recommended to use JUnit's DiscoverySelectors or
# Junit Platform Suite annotations.
cucumber.filter.tags= # a cucumber tag expression.
# only scenarios with matching tags are executed.
# combined with cucumber.filter.name using "and" semantics.
# example: @Cucumber and not (@Gherkin or @Zucchini)
# note: To ensure consistent reports between Cucumber and
# JUnit 5 prefer using JUnit 5s discovery request filters
# or JUnit 5 tag expressions instead.
cucumber.glue= # comma separated package names.
# example: com.example.glue
cucumber.junit-platform.naming-strategy= # long or short.
# default: short
# include parent descriptor name in test descriptor.
cucumber.junit-platform.naming-strategy.short.example-name= # number or pickle.
# default: number
# Use example number or pickle name for examples when
# short naming strategy is used
cucumber.junit-platform.naming-strategy.long.example-name= # number or pickle.
# default: number
# Use example number or pickle name for examples when
# long naming strategy is used
cucumber.plugin= # comma separated plugin strings.
# example: pretty, json:path/to/report.json
cucumber.uuid-generator # uuid generator class name of a registered service provider.
# default: io.cucumber.core.eventbus.RandomUuidGenerator
# example: com.example.MyUuidGenerator
cucumber.object-factory= # object factory class name.
# example: com.example.MyObjectFactory
cucumber.publish.enabled # true or false.
# default: false
# enable publishing of test results
cucumber.publish.quiet # true or false.
# default: false
# suppress publish banner after test execution.
cucumber.publish.token # any string value.
# publish authenticated test results.
cucumber.snippet-type= # underscore or camelcase.
# default: underscore
cucumber.execution.dry-run= # true or false.
# default: false
cucumber.execution.execution-mode.feature= # same_thread or concurrent
# default: concurrent
# same_thread - executes scenarios sequentially in the
# same thread as the parent feature
# concurrent - executes scenarios concurrently on any
# available thread
cucumber.execution.parallel.enabled= # true or false.
# default: false
cucumber.execution.parallel.config.strategy= # dynamic, fixed or custom.
# default: dynamic
cucumber.execution.parallel.config.fixed.parallelism= # positive integer.
# example: 4
cucumber.execution.parallel.config.fixed.max-pool-size= # positive integer.
# example: 4
cucumber.execution.parallel.config.dynamic.factor= # positive double.
# default: 1.0
cucumber.execution.parallel.config.custom.class= # class name.
# example: com.example.MyCustomParallelStrategy
cucumber.execution.exclusive-resources.<tag-name>.read-write= # a comma separated list of strings
# example: resource-a, resource-b.
cucumber.execution.exclusive-resources.<tag-name>.read= # a comma separated list of strings
# example: resource-a, resource-b
JUnit 5 introduced a test discovery mechanism
as a dedicated feature of the platform itself. This allows IDEs and build tools
to identify tests. Supported DiscoverySelectors are:
ClasspathRootSelectorClasspathResourceSelectorClassSelectorPackageSelectorFileSelectorDirectorySelectorUriSelectorUniqueIdSelector
The only supported DiscoveryFilter is the PackageNameFilter and only when
features are selected from the classpath.
The FileSelector and ClasspathResourceSelector support a FilePosition.
DiscoverySelectors.selectClasspathResource("rule.feature", FilePosition.from(5))DiscoverySelectors.selectFile("rule.feature", FilePosition.from(5))
The UriSelector supports URI's with a line query parameter:
classpath:/com/example/example.feature?line=20file:/path/to/com/example/example.feature?line=20
Any TestDescriptor that matches the line and its descendants will be included in the discovery result. For example,
selecting a Rule will execute all scenarios contained within the Rule.
Cucumber tags are mapped to JUnit tags. Note that the @ symbol is not part of
the JUnit tag. So the scenarios below are tagged with Smoke and Sanity.
@Smoke
@Ignore
Scenario: A tagged scenario
Given I tag a scenario
When I select tests with that tag for execution
Then my tagged scenario is executed
@Sanity
Scenario: Another tagged scenario
Given I tag a scenario
When I select tests with that tag for execution
Then my tagged scenario is executedWhen using Maven, tags can be provided from the CLI using the groups and excludedGroups parameters. These take a
JUnit5 Tag Expression. The example
below will execute Another tagged scenario.
mvn verify -DexcludedGroups="Ignore" -Dgroups="Smoke | Sanity"
For more information on how to select tags, see the relevant documentation:
- JUnit 5 Suite: @Include Tags
- JUnit 5 Suite: @Exclude Tags
- JUnit 5 Console Launcher: Options
- JUnit 5 Tag Expression
- Maven: Filtering by Tags
- Gradle: Test Grouping
It is possible to recreate JUnit Jupiter's @Disabled functionality by
setting the cucumber.filter.tags=not @Disabled property1. Any scenarios
tagged with @Disabled will be skipped. See Configuration Options
for more information.
- Do note that this is a Cucumber Tag Expression rather than a JUnit5 tag expression.
Cucumber supports OpenTest4Js
TestAbortedException. This makes it possible to use JUnit Jupiter's
Assumptions to abort rather than fail a scenario.
package com.example;
import io.cucumber.java.Before;
import org.junit.jupiter.api.Assumptions;
import java.util.List;
public class RpnCalculatorSteps {
@Before
public void before() {
boolean condition = // decide if tests should abort
Assumptions.assumeTrue(condition, "Condition not met");
}
}When using cucumber-junit-platform-engine rerun files are not supported.
Use either Maven, Gradle or the JUnit Platform Launcher API.
When running Cucumber through the JUnit Platform Suite Engine
use rerunFailingTestsCount.
Note: any files written by Cucumber will be overwritten during the rerun.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.2.5</version>
<configuration>
<rerunFailingTestsCount>2</rerunFailingTestsCount>
<properties>
<!-- Work around. Surefire does not include enough
information to disambiguate between different
examples and scenarios. -->
<configurationParameters>
cucumber.junit-platform.naming-strategy=long
</configurationParameters>
</properties>
</configuration>
</plugin>Gradle support for JUnit 5 is rather limited gradle#4773, junit5#2849. As a workaround you can the Gradle Cucumber-Companion plugin in combination with Gradle Test Retry plugin.
Note: any files written by Cucumber will be overwritten while retrying.
package com.example;
import org.junit.platform.engine.discovery.DiscoverySelectors;
import org.junit.platform.engine.discovery.UniqueIdSelector;
import org.junit.platform.launcher.Launcher;
import org.junit.platform.launcher.LauncherDiscoveryRequest;
import org.junit.platform.launcher.TestIdentifier;
import org.junit.platform.launcher.core.LauncherFactory;
import org.junit.platform.launcher.listeners.SummaryGeneratingListener;
import org.junit.platform.launcher.listeners.TestExecutionSummary;
import org.junit.platform.launcher.listeners.TestExecutionSummary.Failure;
import java.util.List;
import java.util.stream.Collectors;
import static org.junit.platform.engine.discovery.DiscoverySelectors.selectDirectory;
import static org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder.request;
public class RunCucumber {
public static void main(String[] args) {
LauncherDiscoveryRequest request = request()
.selectors(
selectDirectory("path/to/features")
)
.build();
Launcher launcher = LauncherFactory.create();
SummaryGeneratingListener listener = new SummaryGeneratingListener();
launcher.registerTestExecutionListeners(listener);
launcher.execute(request);
TestExecutionSummary summary = listener.getSummary();
// Do something with summary
List<UniqueIdSelector> failures = summary.getFailures().stream()
.map(Failure::getTestIdentifier)
.filter(TestIdentifier::isTest)
.map(TestIdentifier::getUniqueId)
.map(DiscoverySelectors::selectUniqueId)
.collect(Collectors.toList());
LauncherDiscoveryRequest rerunRequest = request()
.selectors(failures)
.build();
launcher.execute(rerunRequest);
TestExecutionSummary rerunSummary = listener.getSummary();
// Do something with rerunSummary
}
}