This repository is the replication package related to the ICSE-NIER 2023 submission "A. Guerriero, R. Pietrantuono and S. Russo, "Iterative Assessment and Improvement of DNN Operational Accuracy," in 2023 IEEE/ACM 45th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), Melbourne, Australia, 2023 pp. 43-48.". In this repository, the code to repeat the experiments is available. We reported the datasets to completely reproduce the experiments on MNIST.
"results.csv" contains all the raw results discussed in the paper.
To run the experiments it is required to install Python 3 and all the requirements reported in the requirement.txt file. DNN-OS requires the installation of KNIME (https://www.knime.com/downloads).
To execute all the experiments run the command 'sh run_complete.sh'. All the pre-trained models are available. The script will stop the execution at each cycle to allow the execution of DNN-OS on KNIME. To execute DNN-OS it is required to set the path to the desired training, validation, and test sets automatically generated by the Python script. The operational accuracy estimate can be read as the output of the last "column rename" block.
The experiments with SelfChecher can be executed by running the code available in the replication package available at https://github.com/self-checker/SelfChecker.