Notes on the Python Client Combinatorial Executor

The combinatorial executor creates a set of config.ini files to be passed through the sequential executor.

Initially each [section] of the original ini file are stored in individual files. The sections [MiddlewareGet] and [TestRun] are treated differently than all the other sections. First the [MiddlewareGet] sections are paired with the [TestRun] sections to create the base files to build upon. To get multiples of these sections simply add :id to make each one unique. Now we have the base runLog and the other sections in the iniLog.

The rest of the sections will be checked for comma separated values in all the options. If no comma separated values are found the section is copied into each of the runLog files. For options with comma separated values, each value is added to the files in the runLog individually, looping through the list of values. For each pass through the runLog the previous files are cleared out and the updated files are added to the runLog.

Once the list of new configuration files is complete combinatorial.py loops through the runLog, sends each file to testDef.configNewTest() and then runs testDef.executeTest() which runs the sequential executor on that file.

Only the TestRun, and MiddlewareGet sections can have more than one unique version.

For example, the following entry in the config.ini file would be incorrect:

[MiddlewareBuild:OMPIMaster]
option1 = value1, value2

[MiddlewareBuild:OMPI]
option1 = value3, value4

One of the MiddlewareBuild sections would overwrite the other and this would result in either two runs using value1 and value2, or two runs using value3 and value4 instead of the intended four runs.

NOTE: options are not parsed until testDef.configNewTest() runs. Each file overwrites the options from the previous file.