Nexus: enable command line and add user example tests #875
Conversation
ghost
assigned 
|
Can one of the maintainers verify this patch? |
1 similar comment
|
Can one of the maintainers verify this patch? |
|
Okay to test |
nexus/executables/ntest
Outdated
| if verbose: | ||
| print 'Executing:\n '+command | ||
| #end if | ||
| out,err = Popen(command,shell=True,stdout=PIPE,stderr=PIPE,close_fds=True).communicate() |
There was a problem hiding this comment.
The return value of the executed command should be checked for success or failure (or at least passed to the caller so that function can check)
There was a problem hiding this comment.
Return value added, see commit.
nexus/executables/ntest
Outdated
| #end if | ||
|
|
||
| # copy over nexus user examples into reference generation directory | ||
| execute('rsync -av {0}/* {1}/'.format(example_dir,refgen_example_dir)) |
There was a problem hiding this comment.
Checking for success would be good here. A failure to copy the files here will lead to cascading failures that are harder to track down.
There was a problem hiding this comment.
Agreed. Added a check both here and where a similar command is executed within the testing environment.
|
Better? |
|
Is anything else needed here? I am preparing another PR that depends on this one. |
|
I believe that this is ready, so we will merge unless anyone objects before the CI completes... |
Currently Nexus user examples are run in generate only mode via ctest in the QMCPACK distribution. To do so, each user example input file was modified as a workaround to change the default setting of generate_only from "False" to "True".
This PR extends Nexus to accept command line inputs to override the settings present in any Nexus user script. This facility is used to run all user examples within ntest in generate_only mode without modification to the user-facing files.
In addition to checking for the existence of generated files, the new tests supersede those in ctest by checking that generated files match the contents of a set of reference files. This is done by comparing the nested contents of the Nexus object representation of reference and generated files, which includes comparison of floating point values within a tolerance. Future tests are needed to ensure the continued correct reading and writing of input files based on this object representation. Existing ctest tests for the user examples will be removed in a future PR as these are now duplicated with ctest running ntest directly.
The ntest testing script has been extended to generate and update reference data as needed via two new command line inputs (--generate_reference and --update_reference, see "ntest -h" for command line documentation). The idea here is to first check whether all tests pass. If the current tests fail due to an intentional change (e.g. what default fields are written to QMCPACK input files), then the reference files can be automatically updated to the newly intended reference state.