Allow different sources for configuration

[martinschneider]

Added CucumberOptionsProvider to dynamically create and use CucumberOptions

Usage example:

@RunWith(Cucumber.class)
@CucumberOptionsProvider(PropertiesFileCucumberOptionsProvider.class)
public class JUnitRunnerClass {}

An implementation of AbstractCucumberOptionsProvider can be extended to load or create a configuration from any source.
Options passed via @CucumberOptions will be loaded after the ones from the options-provider.

Summary

A limitation of using @CucumberOptions is that its values are constant (i.e. determined at compile time). This PR introduces an option to contribute these values dynamically during runtime.

Details

An additional annotation CucumberOptionsProvider specifies a class which will be called in RuntimeOptionsFactory to fetch the CucumberOptions to be used.

An example using the JUnit Runner could look like this:

@RunWith(Cucumber.class)
@CucumberOptionsProvider(PropertiesFileCucumberOptionsProvider.class)
public class JUnitRunnerClass {}

Motivation and Context

Every test framework I have written myself or co-worked on which uses Cucumber + JUnit at some point faces the issue described above. I think this PR is a simple enough and useful way to remove this limitation.

It has also been discussed on forums multiple times without a fully satisfying solution. The only option giving the same flexibility as implemented here would be to use a custom JUnit runner instead of cucumber.api.junit.Cucumber.

• https://stackoverflow.com/questions/22774099/pass-cucumber-options-dynamically-in-cucumber-junit
• https://stackoverflow.com/questions/45095557/java-cucumber-take-cucumberoptions-from-external-source-like-a-property-file/52663213#52663213

How Has This Been Tested?

I have added unit tests in cucumber.runtime.RuntimeOptionsFactoryTest.
I have created an additional test class cucumber.runtime.PropertiesFileCucumberOptionsProviderTest
I have tested the change within my own test framework: martinschneider/justtestlah#17

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue).
• New feature (non-breaking change which adds functionality).
• Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist:

• I've added tests for my code.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.

[syamsasi99]

+1

[abhivaikar]

Yes. Much needed. +1

[mpkorstanje]

Hey thanks for looking to contribute. This is a useful feature. However the way in which it is implemented is problematic.

To start the current implementation of RuntimeOptionsFactory is doing the wrong thing. It is transforming a hierarchy of@Cucumber annotations into a command-line arguments to be parsed by RuntimeOptions. This PR would then add to that @CucumberOptionsProvider which then loads a class that will create new instances of the @CucumberOptions annotations .

So in essence we currently have:

@CucumberOptions --> Commandline Arguments --> RuntimeOptions

And that would now become:

@CucumberOptionsProvider --> AbstractCucumberOptionsProvider --> @CucumberOptions --> Commandline Arguments --> RuntimeOptions

In my opinion extending this already poorly designed chain is not sustainable.

Rather each source of options should be transformed directly into a some object. The objects created by all different option sources should then be merged to create the configuration used to execute cucumber.

In essence:

@CucumberOptionsProvider --> AbstractCucumberOptionsProvider --> Map of Options
@CucumberOptions --> Map of Options
Commandline Arguments --> Map of Options

[Map of Options] --> RuntimeOptions

There is a cucumber-config project setup to facilitate this. The project is still in its infancy. And nobody has gotten around refactoring this yet as cucumber is currently mostly developed by people in their spare time.

Would you be willing and able to do this refactoring? Perhaps @syamsasi99 and/or @abhivaikar would also be willing to contribute.

[mpkorstanje]

Cucumber core is currently compiled with at source level 1.7. GeAnTyRef is compiled against 1.8. Cucumber v5 will be compiled at 1.8 so we'd have to wait until then.

[mpkorstanje]

Though given my main objection I doubt GeAnTyRef will be necessary.

[mpkorstanje]

Please rebase on master instead. This will cause merge conflicts down the line. You can ignore the failing semantic versioning check at the moment.

[martinschneider]

Thanks for your comments @mpkorstanje. It makes sense to implement this properly. I will look into your suggestions and get back.

[martinschneider]

Thanks again for your feedback @mpkorstanje. I had a dive into this and updated the PR.

This is far from finished. However, could you have a quick look and tell me if this is the general direction you were looking for.

There is a new annotation called CucumberConfiguration which takes two values type and path.

type can be YAML, JSON, PROPERTIES etc. and ANNOTATION which is the fallback to the current behaviour with @CucumberOptions. I've played around with YAML but it would be easy to implement the others (and I'd be happy to do so).

path is the path to the config file (file or classpath).

The YAML syntax is rather simple:

tags:
  - `@tag1`
  - `@tag2`
glue:
  - io.github.glue1
  - io.github.glue2
plugins:
dry: false
monochrome: false
strict: true
snippets:
junit:
features: src/test/resources/features

When a parameter expects a list, single values are also accepted, so e.g. this will work too:

glue: io.github.glue1

The flow is as follows:

1. RuntimeOptionsFactory evaluates the annotation and delegates parsing of the config file to an implementation of ConfigurationParser (so far I've done OptionsConfigurationParser and YamlConfigurationParser)
2. ConfigurationParser returns Map<String, ?>
3. RuntimeOptionsFactory transforms this map into RuntimeOptions

It works ok so far but tbh I haven't tested much more than the standard cases. It's still a long way and even more testing to go.

Just asking for some guidance before I continue with this.

[martinschneider]

Two more questions:

1. cucumber-config is in the main cucumber repo. Should this be moved to cucumber-jvm? Or can I reference and extend it another way?

2. There's some code in the current RuntimeOptions implementation that prints version or usage info:

if (arg.equals("--help") || arg.equals("-h")) {
  printUsage();
  System.exit(0);
} else if (arg.equals("--version") || arg.equals("-v")) {
  System.out.println(VERSION);
  System.exit(0);
} else if (arg.equals("--i18n")) {
  String nextArg = args.remove(0);
  System.exit(printI18n(nextArg));
}

How should this behave when the configuration is loaded from external sources?

[mpkorstanje]

You're welcome. Happy to see you're willing to solve this problem.

This is far from finished. However, could you have a quick look and tell me if this is the general direction you were looking for.

It is a bit hard to tell from the work what the general direction is but it looks promising. I like the @CucumberConfiguration annotation and the Yaml syntax. And I believe it is something that can be made to work. Does this satisfy your need for a configurable source of properties? I was under the impression that you needed a more general solution to provide properties.

There's some code in the current RuntimeOptions implementation that prints version or usage info. How should this behave when the configuration is loaded from external sources?

The structure of cucumber-jvm features a set of components that when composed together in a runner make Cucumber work. RuntimeOptions is one of these components. Depending on the runner used different configuration sources are applicable and useful. I believe this should be reflected in the code.

Currently JUnit and TestNG take options from the environment and @CucumberOptions. The CLI takes options from the environment and commandline. So when using RuntimeOptions in Junit or TestNG the RuntimeOptions should only be created/configured with ConfigurationParsers for @CucumberOptions and the environment. Likewise the CLI should do the same.

So ideally the problem of receiving --version or --help doesn't occur because we have structured the application such that external sources can't provide those options. So the expected behavior is the same as if an other unknown option had provided. For the CLI it is probably still desirable to exit out as soon as possible when--help or --version is used. But I think we work out how that happens later.

cucumber-config is in the main cucumber repo. Should this be moved to cucumber-jvm? Or can I reference and extend it another way?

For now I would suggest building it locally and importing the snapshot version.

[mpkorstanje]

I am not sure if it is wise to let the RuntimeOptionsFactory handle both the @CucumberConfiguration and @CucumberOptions annotations. They are different sources of configuration that can exist together and should be applied in some order.

[mpkorstanje]

As @CucumberConfigurationand@CucumberOptions` are different sources of configuration and as yaml, json and properties can be derived from the file name I don't think this type is needed.

[martinschneider]

Makes sense. I will change it accordingly.

[mpkorstanje]

One of the downsides of building a testing framework is that you can't use all the popular libraries. There are ways to work around this. We'll come to that later.

[martinschneider]

Understood. However, if I base the PR on cucumber-config instead as suggested these libraries will come as transitive dependencies. What's the reason for not having them (footprint, Java version limitations)? What's the alternative you suggest?

[mpkorstanje]

Cucumber is a test library. Adding dependencies means that those dependencies are also present on the class path when tests are executed. Now depending on the build system used, the order in the dependency declaration it may also occur that our transitive dependencies replace the transitive dependencies of the project we're testing. This is highly undesirable.

We can resolve it by adding the dependency with scope provided and checking if the class we're interested in is available on the class path before creating an instance of it.

[aslakhellesoy]

We’ll shade those deps, created a ticket in the monorepo

[mpkorstanje]

So I would imagine something like this to create the options in JUnit/TestNG

RuntimeOptions runtimeOptions = RuntimeOptions.builder()
   .addRuntimeOptionsProvider(new CucumberConfigurationProvider(clazz))
   .addRuntimeOptionsProvider(new CucumberOptionsFactory(clazz))
   .addRuntimeOptionsProvider(new EnvironmentOptionsProvider(env))
   .build();

And something like this for the CLI:

RuntimeOptions runtimeOptions = RuntimeOptions.builder()
   .addRuntimeOptionsProvider(new CommandlineOptionsProvider(args))
   .addRuntimeOptionsProvider(new EnvironmentOptionsProvider(env))
   .build();

[aslakhellesoy]

I don't think we should merge this as-is.

The plan is to start using cucumber-config. I was about to start that work a while ago, but decided to postpone it since there was too much else going on.

@martinschneider do you want to have a go at it? There should be snapshot builds available for cucumber-config.

[martinschneider]

@aslakhellesoy Sure, I will create a PR using cucumber-config

[mpkorstanje]

I'm closing this issue for now. While useful the way this was originally implemented is not sustainable. I've made several attempts to refactor RuntimeOptions to make this possible but the whole thing is to tangled up to do this with any degree of certainty.

The only way out that I see is to implement new runners (JUnit5, Maven, Gradle, Stable API) that do not use RuntimeOptions to parse their options. Then in due time we can deprecate JUnit and TestNG. To this end RuntimeOptions now implements FeatureOptions, FilterOptions, PluginOptions and RunnerOptions.

[mlvandijk]

(deleted duplicate comments)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.