- Working installation of conda/miniconda (https://docs.conda.io/en/latest/miniconda.html)
-
Create a new conda environment based on the package list at
conda.txt
:conda create --name cubes --file conda.txt
-
Activate the conda environment:
conda activate cubes
From now on all commands should be run with the conda environment activated.
-
Install required Python packages through pip:
pip install psutil z3-solver chromalog pylru frozendict sqlalchemy libs/TestSuiteEval-1.0.3-py3-none-any.whl pebble
Instance files are in the tests-examples
folder.
python ./sequential.py path-to-yaml-file
You can use --help
to see all configuration options available in cubes.
python ./cubes.py path-to-yaml-file
You can use --help
to see all configuration options available in cubes (parallel version).
In order to select which instances you would like to benchmark, you need to create a folder tests
. You can then copy the desired instances from tests-examples
or symlink them, if you desire.
PYTHONPATH=. ./helper-scripts/benchmark.py execution-descriptor
Here execution-descriptor
is any string that is a valid filename and is used to store the results in the folder ./analysis/data/execution-descriptor
.
You can use --help
to see all configuration options available in the benchmark script.
At this point disambiguation is implemented as a post-processing step after cubes has generated all solution under the desired time limit and requires the benchmark script to have been used.
To perform disambiguation you can use the following command:
PYTHONPATH=. ./helper-scripts/disambiguation-post.py execution-descriptor
You can use --help
to see all configuration options available in the disambiguation script.
Like the disambiguation script, accuracy requires instances to have been run using the benchmark script introduced before.
To perform accuracy analysis you can use the following command:
PYTHONPATH=. ./helper-scripts/fuzzy-check.py --run=execution-descriptor
If you want to perform accuracy analysis over the results of a previous disambiguation you can use:
PYTHONPATH=. ./helper-scripts/fuzzy-check.py --from-dis --run=execution-descriptor
You can use --help
to see other configuration options available in the accuracy analysis script.
-
The file
groups/cubes_500.instances
contains the list of instances used for evaluation in the parts where only a subset of instances was used. This file can be passed as the--instances
argument to the benchmark script. -
The folder
analysis
contains all code used to analyse the results obtained and produced all graphs included in the paper. -
The folders
analysis/data
andanalysis/fuzzy
come pre-packaged with all logs used in the evaluation section of the paper.