How do we rerun failed scenarios #31
Comments
|
I'd really consider this outside the scope of the parallel plugin. Isn't that more down to the execution mechanism? |
|
@temyers there is a mechanism at the cucumber end. We need to generate the Runner this way true 2 And then if a Failures is present then a new Runner class should be as |
|
You can use TestNG to reRun failed scenarios. https://martinholladay.wordpress.com/2013/11/15/a-better-retrylistener/ https://github.com/NetEase/arrow/blob/master/src/com/netease/qa/testng/TestngRetry.java This may help. |
|
How does this work in rerunning the cucumber features ? |
|
Sorry I meant to say it reruns failed Features, so when one scenario fails in one of the features, it will rerun the entire feature with all the scenarios even if some of the scenarios passed (I am trying to figure out a way to make it work where it only runs the failed scenario instead of the entire feature). So if you are running your features through an XML file from TestNG and if you setup it up something like this. ?xml version="1.0" encoding="UTF-8"? first the RetryListener will check if the retry is needed by checking in with RepeatTestListner (It will be null at first, then it will return true when something fails)The TestNG getResultStatusName(result.getStatus()) function in RepeatTestListner will read the status and see what the status is. if the status is 2 then the feature has failed of course. That it will rerun it once, or how many times you want it reran. This will only work if you run your cucumber test from the TestNG XML file or through a Maven configuration. IT will not work if you are running though the runner directly or if you are running the feature directly. The only way it could maybe work with the cucumber-jvm-parallel-plugin I think is through Maven, because the TestNG XML file needs a specific runner (Although I am not too sure, and have not yet tried it, I will let you know if it works later on). (It is not perfect, but I am trying to find a way to make it work a little better). public class RepeatTestListner implements IRetryAnalyzer { public boolean retry(ITestResult result) { public String getResultStatusName(int status) { public class RetryListener implements IAnnotationTransformer } |
|
@temyers I would also really love to see this feature added, I can try to help and make it possible. We can inject the "rerun:target/rerun.txt" (or wherever you want to failed scenarios to go) in all of the runners in the plugins section, then automatically create an extra runners with the _ at the start of it. and have the runner run like Sai said with features = { "@target/rerun.txt" }. I can for see the issue of many runners and trying to figure out which runners should run on which device if running parallel devices. Let me know thank you. TestExampleCucumberIT.java: test runner that will perform the first test execution and record data in target/rerun.txt package example.cucumber; import org.junit.runner.RunWith; import cucumber.api.CucumberOptions; import cucumber.api.junit.Cucumber; } the _TestExampleCucumberIT.java will perform the second test execution (as maven will run all IT class, the second one start with _ so it will be run after the previous test bulk) package example.cucumber; import org.junit.runner.RunWith; import cucumber.api.CucumberOptions; import cucumber.api.junit.Cucumber; |
|
Hello temyers, |
|
Thinking about this more, I'm actually not sure if it's a desirable feature since it is pointing to an underlying issue with the tests - they are flaky. If you re-run an automated test without making changes, you should really get the same results. Flaky tests are an anti-pattern: |
|
@temyers i feel this a needed feature, let me explain i run my tests on real devices android and iOS at times the server time-out / network issues and at times i have seen USB releasing the devices and replugging and lot more... These tests are not functional failure tests just the env problem, rerunning them would help.. The links which you have mentioned above all talks on website functional tests, but mobile tests are very much different from that .. |
|
They were in relation to web apps, but i think it's still valid for mobile development. Things like server time-outs, etc are real issues that need to be considered when testing and developing the app. For example, adding in appropriate waits etc. Shouldn't the application be able to better handle such environmental changes which are natural on a mobile device? If you've got flaky tests, doesn't that suggest that there is something missing in the setup/interactions? |
|
@temyers i agree.. But when i have drop in network i need to retest the functionality |
|
Just an FYI, maven surefire plugin knows about failing tests as well. However, the maven surefire plugin does not know how to use a cucumber feature file, so we need to use the cucumber solution (in my opinion). Not sure how to configure the junit surefire plugin to use the Cucumber.class file instead of the default JUnit one. |
|
I'm going to add this in with pr #30. |
|
It's a big mistake to accept flaky tests, just ask Facebook. A better approach is to build the required resiliency in to the application and determine transient issues and act accordingly. In Spring, see Spring-retry. If using Java you can introduce your own annotation like this library has: |
|
Re-implemented in PR #89. |
|
I would just like to add a use case to support this feature... I'd like to run a selection of scenario's that cannot be grouped by tags. The only way to do this is produce a file in the rerun format. Using this technique, I can then build a parameterised build on Jenkins to accept these files to run them on an ad-hoc basis. |
|
This plug in only creates the tests. It does not execute them. Running tests and rerunning failing tests is a build-in feature of surefire and failsafe. You should use these. https://maven.apache.org/surefire/maven-failsafe-plugin/examples/rerun-failing-tests.html |
|
Does anyone try it out for Tag level? Pls let me know |
|
Mrph. It appears that using surefire rerun results in a stack trace. Seems to be caused by cucumber not using the runners class name. So that option is out unless some one fixes surefire. |
|
@temyers could you help out in reviewing cucumber/cucumber-jvm#1035? This appears to be the current blocker. For context please see the Cukes mailing list. And perhaps @MagenTys might also be interested. |
…e tests. Surefire currently uses a tuple of (Class, Method) to identify failed tests. Cucumber-jvm uses test suites which do not consist of Classes or methods. Therefore another method is needed. JUnit provides the option to select tests that match a Description. Descriptions are compared using a unique Id. To identify tests between different runs we should provide Descriptions with a predictable unique id. The current implementation did not suffice. While the provided cucumber elements are unique, they are not predictable between runs making it impossible to use their descriptions to rerun a failed test. After this change it will be possible to update Surefire to use descriptions to rerun failed tests. Related Issues: - https://issues.apache.org/jira/browse/SUREFIRE-1372 - #1120 - temyers/cucumber-jvm-parallel-plugin#31 (cherry picked from commit 4cc4439)
Rerunning failed tests with Surefire requires a method to identify the tests. Surefire currently uses a tuple of (Class, Method) to identify failed tests. Cucumber-jvm uses test suites which do not consist of Classes or methods. Therefore another method is needed. JUnit provides the option to select tests that match a Description. Descriptions are compared using an unique Id. To identify tests between different runs we should provide Descriptions with a predictable unique id. The current implementation did not suffice. While the provided cucumber elements are unique, they are not predictable. Each run would create a new instance of the identifier making it impossible to use their descriptions to rerun a failed test. After this change it will be possible to update Surefire to use descriptions to rerun failed tests. Related Issues: - https://issues.apache.org/jira/browse/SUREFIRE-1372 - #1120 - temyers/cucumber-jvm-parallel-plugin#31
Rerunning failed tests with Surefire requires a method to identify the tests. Surefire currently uses a tuple of (Class, Method) to identify failed tests. Cucumber-jvm uses test suites which do not consist of Classes or methods. Therefore another method is needed. JUnit provides the option to select tests that match a Description. Descriptions are compared using an unique Id. To identify tests between different runs we should provide Descriptions with a predictable unique id. The current implementation did not suffice. While the provided cucumber elements are unique, they are not predictable. Each run would create a new instance of the identifier making it impossible to use their descriptions to rerun a failed test. After this change it will be possible to update Surefire to use descriptions to rerun failed tests. Related Issues: - https://issues.apache.org/jira/browse/SUREFIRE-1372 - #1120 - temyers/cucumber-jvm-parallel-plugin#31
Rerunning failed tests with Surefire requires a method to identify the tests. Surefire currently uses a tuple of (Class, Method) to identify failed tests. Cucumber-jvm uses test suites which do not consist of Classes or methods. Therefore another method is needed. JUnit provides the option to select tests that match a Description. Descriptions are compared using an unique Id. To identify tests between different runs we should provide Descriptions with a predictable unique id. The current implementation did not suffice. While the provided cucumber elements are unique, they are not predictable. Each run would create a new instance of the identifier making it impossible to use their descriptions to rerun a failed test. After this change it will be possible to update Surefire to use descriptions to rerun failed tests. Related Issues: - https://issues.apache.org/jira/browse/SUREFIRE-1372 - #1120 - temyers/cucumber-jvm-parallel-plugin#31
Description#createSuiteDescription Filtering tests by the tuple of (Class, Method) does not work when a test suite is used as a suite may not have (Class, Method) . Filtering by the description of the failed test should however work. Related Issues: - cucumber/cucumber-jvm#1134 - temyers/cucumber-jvm-parallel-plugin#31 SUREFIRE-1372 Filter junit4 tests to be rerun by description
|
Anyone is working on "rerun" failed scenario , I have found few old links which have mentioned about "rerun" https://github.com/sugatmankar/cucumber-jvm-parallel-plugin-rerun-example |
|
|
|
Hi Team, Surefire-1372 fixed and released so Can we use Rerun functionality in tymers cucumber parallel plug in to rerun failed cases. |
|
Haven't had a chance to test it yet! |
|
I tested with providing Surefire version 2.21.0 and temyers cucumber parallel plugin 4.2.0. while running the POM it is failing saying Forked VM is crashed. I am able to run by using Surefire 2.20. Please refer below Configuration and Error Message. error Message: |
|
@Shareef112 I just had a chance to test it. I can't reproduce that particular problem. I'm afraid you'll have make an MVCE out of it. This works as expected: <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>io.cucumber</groupId>
<artifactId>cucumber-examples</artifactId>
<version>3.0.0-SNAPSHOT</version>
</parent>
<artifactId>java-calculator</artifactId>
<packaging>jar</packaging>
<name>Examples: Java Calculator</name>
<dependencies>
...
</dependencies>
<properties>
<surefire.rerunFailingTestsCount>2</surefire.rerunFailingTestsCount>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.21.0</version>
</plugin>
...
</build>
</project>Which for a flaky test will output: Also worth noting that the output from json formatter and other cucumber formatters will be overwritten with the latest results from the rerun. This shouldn't be a problem if you're parallelizing by scenario but will cause problems if you're parallelizing by feature (as I've done in this example). The surefire report however is still accurate. |
|
@mpkorstanje can you plese provide me the dependencies which you are used. |
|
It's the cucumber-java-calculator example. The 3.0.0-SNAPSHOT verdion from the cucumber-jvm git. Rigged it with an intermittent failure. Latest 2.x should work too. |
|
You could also use threads instead of forks. Or forbid them all together and see what kills the JVM. |
|
Hi Team , [INFO] Scanning for projects... |
Fix the gherkin syntax. |
|
@laddushashavali in your scenario outline Example you provided More values than columns or vice versa. |
|
@mpkorstanje I tried all the combinations but still I am getting that error. Can you please help me to mitigate that error. if you want I can send my POM file |
|
You can make it easier for me by creating a minimal complete verifiable
example. Preferably in the form of a github repo. See:
https://stackoverflow.com/help/mcve
But first I'd suggest you clone the
https://github.com/cucumber/cucumber-java-skeleton and configure it just
with the retry enabled to confirm it works for you. Then increase the
complexity by adding this plugin until it fails to work.
If that yields you nothing, you'll definitly have have to strip down your
own setup.
…On Tue, Apr 17, 2018, 18:07 Shareef112 ***@***.***> wrote:
@mpkorstanje <https://github.com/mpkorstanje> I tried all the
combinations but still I am getting that error. Can you please help me to
mitigate that error. if you want I can send my POM file
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#31 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AGoAZ73Yqrvz0FVgrgz2XeKS9xmWziFqks5tphNRgaJpZM4IawPO>
.
|
|
While running cucumber parallel plugin with surefire 2.21.0 with TestNG True then I am getting below error message. can anyone Help me to resolve this issue. [**INFO] Running Parallel01IT [ERROR] tearDownClass(Parallel01IT) Time elapsed: 0.81 s <<< FAILURE! [INFO] |
|
Hello, I am trying to run with the following conf: The surefire: org.apache.maven.plugins maven-surefire-plugin 2.20 ${surefireSkip} classes ${surefireThreadCount} 1 ${project.build.directory}/classes **/*IT.java 3 //It doesn't matter what version I have above.I am not able to see the test which is failed to be run again. |
|
Could you add in this repo any complete example about how to implement re-run executions? but how can I use those txt files for rerunning?? I need to create another maven profile? It would be good if the doc was clear about that because it's a feature required for most testers. |
|
Hi Team, when I run tymers cucumber parllel plug in without TestNG i am able to achieve Rerun Functionolity. if I use testNG Rerun functionolity is not working. |
|
Hello Team, I am using testngRunner to run my cucumber features..i am using temyers/cucumber-jvm-parallel-plugin to execute scripts in parallel using selenium grid. I was able to see 1.txt and 2.txt is the test is failed. but i didnt know how to re run failed scripts...could you please help me. Attached are my pom.xml and .vm template. |
|
|
|
#macro( stringArray $array ){#foreach( $element in $array )"$element"#if( $foreach.hasNext ), #end#end}#end @CucumberOptions( } |
@temyers is there a way we can re-run failed scenarios?
The text was updated successfully, but these errors were encountered: