Adding check files for verification integration tests #1641
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR extends the verification test suite to optionally validate the VCs results.
For each
.scala
file, a corresponding.check
file dictates the expected output.The latter is a CSV with the following columns:
<line>:<column>
precondition
,assert
and so on)valid
,invalid
,unknown
)Note that VCs are allowed to be undistinguishable (i.e. all 5 properties are equal), for instance
BigIntMonoidLaws.check
has 3 times the entry10:9;append;precondition;valid
because there are 3 corresponding VCs.To opt-in for
.check
files test for benchmarks that are expected to fail, the functionVerificationComponentTestSuite#testNegAllWithCheckFiles
may be used.It is currently only used by
VerificationSuite
to testverification/invalid
. For now, a warning is issued for benchmarks missing a corresponding.check
file.To quickly generate
.check
files, a utilitycheck-files-util
module is provided.It essentially runs Stainless and writes the results.
The utility must be passed a folder or individual files (which must resides in
frontends/benchmarks
).It accepts the following options:
--ignore=<comma-separated-without-spaces-files>
to ignore specific files--parallelism=<n>
sets the number of threads for StainlessOptions can be passed to Stainless by adding a separation
--
.For instances, the following command (in SBT shell):
will:
.check
files for theverification/invalid
benchmarks exceptBadConcRope.scala
andBadOpaque.scala
as indicated by the--ignore
options (note that the files are concatenated with commas without any space!)smt-cvc5
as solverstrict-arithmetic
option)Note that the primary motivation for this is to fix missing positions in VCs (which are probably the second most annoying things after timed-out VCs).
If a benchmark has VCs with missing positions, the utility will issue a warning but it won't be able to know from which file the VC comes (since the position is also responsible to contain the filename).