Generic reruner

[sugatmankar]

Hi @temyers ,

I have re implemented rerunner, but only IT failing to compare Actual String and Expected string in verify.groovy in generic-re-runner IT. Do you have any suggestion to compare large text data.

[sugatmankar]

@temyers I need help to compare larger text. Could you please look IT/generic/generic runner build log?

[sugatmankar]

@temyers when im comparing expected with actual travis CI is failing but if i compare by removing \n and " in text file both are same . could you please have a look ?

import com.github.timm.cucumber.options.ExtendedRuntimeOptions;
import cucumber.runtime.Runtime;
import cucumber.runtime.io.MultiLoader;
import cucumber.runtime.io.ResourceLoaderClassFinder;
import net.masterthought.cucumber.Configuration;
import net.masterthought.cucumber.ReportBuilder;
import org.apache.commons.io.FileUtils;
import org.junit.Test;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
public class Parallel01IT {
 private String outputPath = "target/cucumber-parallel/Parallel01IT/Parallel01IT";
 private String glue = "com.foo.bar";
 @Test public void reRun() {
  defaultRun();
  switch (1) {
   case 1:
    firstRun();
    break;
   case 2:
    firstRun();
    secondRun();
    break;
   case 3:
    firstRun();
    secondRun();
    thirdRun();
    break;
   case 4:
    firstRun();
    secondRun();
    thirdRun();
    fourthRun();
    break;
   case 5:
    firstRun();
    secondRun();
    thirdRun();
    fourthRun();
    fifthRun();
    break;
   default:
    break;
  }
  generateAllRunReports();
 }
 private void defaultRun() {
  List < String > arguments = new ArrayList < String > ();
  arguments.add("/home/travis/build/sugatmankar/cucumber-jvm-parallel-plugin/target/it/generic/generic-re-runner/src/test/resources/features/feature1.feature:4");
  arguments.add("--plugin");
  arguments.add("html:target/cucumber-parallel/Parallel01IT/Parallel01IT.html");
  arguments.add("--plugin");
  arguments.add("json:target/cucumber-parallel/Parallel01IT/Parallel01IT.json");
  arguments.add("--plugin");
  arguments.add("rerun:target/cucumber-parallel/Parallel01IT/Parallel01IT.txt");
  String[] gluepackages = glue.split(",");
  for (String packages: gluepackages) {
   if (!packages.contains("none")) {
    arguments.add("--glue");
    arguments.add(packages);
   }
  }
  final String[] argv = arguments.toArray(new String[0]);
  try {
   executeCLIMain(argv);
  } catch (InterruptedException e) {
   e.printStackTrace();
  } catch (IOException e) {
   e.printStackTrace();
  }
 }
 public void firstRun() {
  try {
   if (new File(outputPath + ".txt").exists() && new BufferedReader(new FileReader(outputPath + ".txt")).readLine() != null) {
    executeReRerun("@" + outputPath + ".txt", outputPath + "/cucumber1.json", outputPath + "/rerun1.txt");
   }
  } catch (Exception e) {
   e.printStackTrace();
  }
 }
 public void secondRun() {
  try {
   if (new File(outputPath + "/rerun1.txt").exists() && new BufferedReader(new FileReader(outputPath + "/rerun1.txt")).readLine() != null) {
    executeReRerun("@" + outputPath + "/rerun1.txt", outputPath + "/cucumber2.json", outputPath + "/rerun2.txt");
   }
  } catch (Exception e) {
   e.printStackTrace();
  }
 }
 public void thirdRun() {
  try {
   if (new File(outputPath + "/rerun2.txt").exists() && new BufferedReader(new FileReader(outputPath + "/rerun2.txt")).readLine() != null) {
    executeReRerun("@" + outputPath + "/rerun2.txt", outputPath + "/cucumber3.json", outputPath + "/rerun3.txt");
   }
  } catch (Exception e) {
   e.printStackTrace();
  }
 }
 public void fourthRun() {
  try {
   if (new File(outputPath + "/rerun3.txt").exists() && new BufferedReader(new FileReader(outputPath + "/rerun3.txt")).readLine() != null) {
    executeReRerun("@" + outputPath + "/rerun3.txt", outputPath + "/cucumber4.json", outputPath + "/rerun4.txt");
   }
  } catch (Exception e) {
   e.printStackTrace();
  }
 }
 public void fifthRun() {
  try {
   if (new File(outputPath + "/rerun4.txt").exists() && new BufferedReader(new FileReader(outputPath + "/rerun4.txt")).readLine() != null) {
    executeReRerun("@" + outputPath + "/rerun4.txt", outputPath + "/cucumber5.json", outputPath + "/rerun5.txt");
   }
  } catch (Exception e) {
   e.printStackTrace();
  }
 }
 public void executeReRerun(String rerunFile, String targetJson, String targetRerun) {
  List < String > arguments = new ArrayList < String > ();
  arguments.add(rerunFile);
  arguments.add("--plugin");
  arguments.add("pretty:" + outputPath + "/cucumber-pretty.txt");
  arguments.add("--plugin");
  arguments.add("json:" + targetJson);
  arguments.add("--plugin");
  arguments.add("rerun:" + targetRerun);
  String[] gluepackages = glue.split(",");
  for (String packages: gluepackages) {
   if (!packages.contains("none")) {
    arguments.add("--glue");
    arguments.add(packages);
   }
  }
  final String[] argv = arguments.toArray(new String[0]);
  try {
   executeCLIMain(argv);
  } catch (InterruptedException e) {
   e.printStackTrace();
  } catch (IOException e) {
   e.printStackTrace();
  }
 }
 public byte executeCLIMain(final String[] argv) throws InterruptedException, IOException {
  ExtendedRuntimeOptions runtimeOptions = new ExtendedRuntimeOptions(new ArrayList(Arrays.asList(argv)));
  MultiLoader resourceLoader = new MultiLoader(this.getClass().getClassLoader());
  ResourceLoaderClassFinder classFinder = new ResourceLoaderClassFinder(resourceLoader, this.getClass().getClassLoader());
  Runtime runtime = new Runtime(resourceLoader, classFinder, this.getClass().getClassLoader(), runtimeOptions);
  runtime.run();
  System.out.println(runtime.exitStatus());
  return runtime.exitStatus();
 }
 public static void generateAllRunReports() {
  try {
   List < File > jsons = finder("target/cucumber-parallel/");
   List < File > defaultRunJSONs = new ArrayList < File > ();
   List < File > firstRunJSONs = new ArrayList < File > ();
   List < File > secondRunJSONs = new ArrayList < File > ();
   List < File > thirdRunJSONs = new ArrayList < File > ();
   List < File > fourthRunJSONs = new ArrayList < File > ();
   List < File > fifthRunJSONs = new ArrayList < File > ();
   for (File f: jsons) {
    if (f.getName().contains("cucumber1") && f.getName().endsWith(".json")) {
     firstRunJSONs.add(f);
    } else if (f.getName().contains("cucumber2") && f.getName().endsWith(".json")) {
     secondRunJSONs.add(f);
    } else if (f.getName().contains("cucumber3") && f.getName().endsWith(".json")) {
     thirdRunJSONs.add(f);
    } else if (f.getName().contains("cucumber4") && f.getName().endsWith(".json")) {
     fourthRunJSONs.add(f);
    } else if (f.getName().contains("cucumber5") && f.getName().endsWith(".json")) {
     fifthRunJSONs.add(f);
    } else {
     defaultRunJSONs.add(f);
    }
   }
   if (defaultRunJSONs.size() != 0) {
    generateRunWiseReport(defaultRunJSONs, "Default_Run");
   }
   if (firstRunJSONs.size() != 0) {
    generateRunWiseReport(firstRunJSONs, "First_Re-Run");
   }
   if (secondRunJSONs.size() != 0) {
    generateRunWiseReport(secondRunJSONs, "Second_Re-Run");
   }
   if (thirdRunJSONs.size() != 0) {
    generateRunWiseReport(thirdRunJSONs, "Third_Re-Run");
   }
   if (fourthRunJSONs.size() != 0) {
    generateRunWiseReport(fourthRunJSONs, "Fourth_Re-Run");
   }
   if (fifthRunJSONs.size() != 0) {
    generateRunWiseReport(fifthRunJSONs, "Fifth_Re-Run");
   }
  } catch (Exception e) {
   e.printStackTrace();
  }
 }
 public static List < File > finder(String dirName) {
  return (List < File > ) FileUtils.listFiles(new File(dirName), new String[] {
   "json"
  }, true);
 }
 public static void generateRunWiseReport(List < File > jsons, String run) {
  try {
   File rd = new File("target/cucumber-parallel/Consolidated-Report/" + run);
   List < String > jsonReports = new ArrayList < String > ();
   for (File json: jsons) {
    jsonReports.add(json.getAbsolutePath());
   }
   Configuration configuration = new Configuration(rd, "cucumber-reporting");
   configuration.setBuildNumber(run);
   ReportBuilder reportBuilder = new ReportBuilder(jsonReports, configuration);
   reportBuilder.generateReports();
  } catch (Exception e) {
   e.printStackTrace();
  }
 }
}

[temyers]

I'll try and have a look. Probably a difference between windows/linux line endings/file encoding.

[temyers]

I'm still not convinced that this is the right place for this functionality. Re-run implementation is already supported by the Surefire/Falsafe plugin and should be implemented there in my view.

[temyers]

Also, please can you create an integration test to demonstrate (actually run the generated code) that this runner doesn't force the user to add additional dependencies. Ideally, using this plugin should not enforce the addition of any libraries. If it does, then those dependencies need to be explicit within the README.

[sugatmankar]

@temyers i agree with you about re-run functionality added by Surefire/Failsafe but does they really have functionality to execute cucumbers in generic way ? Instead of depending on JUnit or TestNG why don't we directly interact with Cucumber CLI in generic way ? Why shouldn't we allow our plugin to have execution of only failed runner get trigger instantly ? Why should't our plugin generate consolidate reports run wise ?

All this has been added to this PR. I have also updated README for explicit dependency.

[sugatmankar]

Also there is integration test in cucumber-jvm-parallel-plugin/src/it/generic-re-runner/

[mpkorstanje]

@sugatmankar This plugin generates the unit tests. The actual execution of tests is outside the scope of this plugin so I do not think it should be in here. However running tests and rerunning failing tests is a build-in feature of surefire and failsafe.

https://maven.apache.org/surefire/maven-failsafe-plugin/examples/rerun-failing-tests.html
https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html

@temyers might be best to just close this pull request.

[mpkorstanje]

I'm closing this PR.

A maven test runner that directly interacts with the cucumber command line is a good idea but also well out of scope for this plugin. It would probably be better of as a separate plugin.

[sugatmankar]

@mpkorstanje show me how you can rerun failed scenario with surefire or fail-safe. There is no solution as of now from sure fire. I would have close this so far before and for your info I have contributed too many changes that you weren't aware. By the way if you find solution let me know.

[mpkorstanje]

The main objection is that rerunning tests and more in general the execution of tests is outside the scope of this plugin. That there is no solution is not a strong argument to add such a solution here.

As it currently stands the rerun functionality is almost completely disjoint of the other functionality of this plugin. So if you have a strong and immediate need for this functionality I think the best option would be to create a separate maven plugin. I think you'll find it greatly reduces the complexity of the whole thing.

I also have several lesser but nonetheless severe objections against the specific solution proposed here. By using the cucumber cli directly the rerun functionality does not play well with users who use glue code that depends on static and before class initializers in their customVmTemplate. I also find the addition of a report builder highly inappropriate. This plugin should be completely agnostic towards reporting. Never mind the requirement that end users add several dependencies to their build.

[mpkorstanje]

That being said. I certainly wouldn't mind having some help in reviewing cucumber/cucumber-jvm#1035. It is the main blocker towards fixing surefire.