RESTest configuration files

Table of contents

1. Properties files: configuring RESTest execution
2. Required parameters
   1. oaispecpath
   2. confpath
3. Optional parameters
   1. numtestcases
   2. targetdirjava
   3. packagename
   4. experimentname
   5. testclassname
   6. enableinputcoverage
   7. enableoutputcoverage
   8. enablecsvstats
   9. ignoredependencies
   10. numtotaltestcases
   11. delay
   12. faultyratio
4. Optional parameters if ignoredependencies is false
   1. faultydependencyratio
   2. reloadinputdataevery
   3. inputdatamaxvalues
5. An example: testing Spotify API

Properties files: configuring RESTest execution

RESTest needs a .properties file for setting up the experiment. We recommend to create a properties file for every API you test. This is the basic structure of a properties file:

# Required parameters
oaispecpath=path/to/oas/api/spec/file.yaml
confpath=path/to/test/conf/file.yaml

# Optional parameters
numtestcases=10
targetdirjava=folder/where/test/is/generated
packagename=packageNameOfTheTest
experimentname=nameOfTheAPI
testclassname=ClassNameOfTest
enableinputcoverage=true
enableoutputcoverage=true
enablecsvstats=true
ignoredependencies=false
numtotaltestcases=-1
delay=-1
faultyratio=0.1

#CBT only:
faultydependencyratio=0.1
reloadinputdataevery=100
inputdatamaxvalues=1000

Required parameters

oaispecpath

Path to the OpenAPI specification file.

confpath

Path to the test configuration file generated from the OpenAPI specification file.

Optional parameters

numtestcases

Number of test cases to generate per operation. For instance, if you test 4 operations and set this parameter to 5, 20 test cases will be generated in each iteration.

targetdirjava

Directory where to store Java test classes.

packagename

Package name of the generated test classes.

experimentname

Identifier of the experiment performed. Used for folder names where test reports are generated (under target folder).

testclassname

Name of the generated test classes.

enableinputcoverage

Boolean parameter. Set to true if you want to generate a report of the input coverage. Defaults to true.

enableoutputcoverage

Boolean parameter. Set to true if you want to generate a report of the output coverage. Defaults to true.

enablecsvstats

Boolean parameter. Set to true if you want to generate statistics of the test cases, such as the parameters of every request or the number of nominal and faulty test cases. Defaults to true.

numtotaltestcases

Total number of test cases to generate. Setting it to -1 will make RESTest to never stop.

delay

Delay in milliseconds between iterations. Each iteration executed numtestcases * number of operations tested test cases. Set it to -1 to disable delay.

faultyratio

Ratio (0-1) of faulty test cases to generate. A faulty test case is one that uses invalid request data and therefore expects a client error response from the API (4XX status code).

ignoredependencies

Boolean parameter. Set to false to enable constraint-based testing (CBT). In CBT, inter-parameter dependencies are handled when generating test cases.

Optional parameters for CBT

These parameters are only applicable if ignoredependencies is false. Otherwise they will be ignored.

faultydependencyratio

Ratio (0-1) of faulty test cases to generate due to inter-parameter dependencies. These test cases violate one or more inter-parameter dependencies present in the operation and therefore are considered faulty.

reloadinputdataevery

Number of requests using the same randomly generated input data.

inputdatamaxvalues

Number of values used for each parameter when reloading input data.

An example: testing Spotify API

We have obtained the OAS specification of the Spotify API from here and we have generated a test configuration file with the operations GET of /v1/artists/{id} - get an artist -, GET /v1/albums/{id} - get an album - and GET /v1/albums/{id}/tracks - get an album's tracks -. We have those files in src/test/resources/Spotify directory. For the first test we will only use the required parameters of the properties file:

oaispecpath=src/test/resources/Spotify/swagger.yaml
confpath=src/test/resources/Spotify/testConf.yaml

RESTest will generate and execute 30 test cases, generating input and output coverage reports and statistics in a CSV file in each iteration until we stop the JVM.

We want to change the configuration for the next test. In this case, we want to produce 20 test cases for each operation in each iteration, generating a maximum of 180 test cases. Therefore, the test will make 3 iterations - 3 * 3 * 20 = 180 -, and we want to add a delay of 30 seconds between iterations. We will put enablecsvstats to false as we don't want the CSV statistics file. We will set ignoredependencies to true as the operations we are testing don't have any inter parameter dependency. Besides, we don't want to produce faulty test cases, so we will set the faultyratio parameter to 0. We will also use targetdirjava, packagename, experimentname and testclassname as we want to make it clear that the test suite doesn't have any faulty test case:

oaispecpath=src/test/resources/Spotify/swagger.yaml
confpath=src/test/resources/Spotify/testConf.yaml

numtestcases=20
targetdirjava=src/generation/java/spotify_allNominal
packagename=spotify_allNominal
experimentname=SpotifyAllNominal
testclassname=SpotifyAllNominal
enablecsvstats=false
ignoredependencies=true
numtotaltestcases=180
delay=30
faultyratio=0