Repository of state-of-the art requirements formalisation approaches, evaluated on common examples
This repository provides a common set of requirements statement cases, so that RF approaches can be compared on their results when applied to the same cases.
Cases are divided into 3 categories: (i) unstructured documents, such as background material on a domain; (ii) semi-structured documents, such as user stories, which have an expected but flexible format; (iii) fully-structured documents according to a grammar or other formal definition.
A set of evaluation tools are also provided to carry out (i) comparison of produced models to a 'gold standard' or reference formalisation carried out by a human expert (accuracy check); (ii) comparison of the produced model to the original document to identify how much information has been captured (completeness check); (iii) internal quality check of the model to determine if it is a valid model and what its quality level is.
Authors of RF approaches are welcome to contribute a description and tooling for their approach, and to apply their approach on the case studies.
The work of assembling cases, implementing approaches and writing reference models was carried out by Dr Kevin Lano (KCL), Dr Sobhan Yassipour-Tehrani (UCL), Yiqi Liu (Sheffield), Dr Shekoufeh Kolahdouz-Rahimi (Roehampton) and Muhammad Aminu Umar (KCL).
The work was funded by the EPSRC in the project "Natural Language Processing for Automated Requirements Formalisation", PI Chenghua Lin, Sheffield University.
Two additional industrial case studies have been added from the paper "Generating Domain Models from Natural Language Text Using NLP: A Benchmark Dataset and Experimental Comparison of Tools", Fatma Bozyigit et al., SoSym, 2024.