Unexpected behavior for coverage tests

[pgrete]

The default CODE_COVERAGE option is off, still the test with coverage are created by default (tested on Summit with cmake 3.14.1)

$ ctest -N
Test project /gpfs/alpine/ast146/world-shared/build-cuda
  Test  #1: unit_tests:Just check everything
  Test  #2: unit_tests:Can create a vector-valued face-variable
  Test  #3: unit_tests:Add and Get is called
  Test  #4: unit_tests:reset is called
  Test  #5: unit_tests:when hasKey is called
  Test  #6: unit_tests:Physical constants
  Test  #7: unit_tests:Checking IndexShape indices
  Test  #8: unit_tests:Checking IndexShape cell counts
  Test  #9: unit_tests:par_for loops
  Test #10: unit_tests:nested par_for loops
  Test #11: unit_tests:Overlapping SpaceInstances
  Test #12: unit_tests:Built-in flags are registered
  Test #13: unit_tests:A Metadata flag is allocated
  Test #14: unit_tests:A Metadata struct is created
  Test #15: unit_tests:ParArrayND
  Test #16: unit_tests:ParArrayND with LayoutLeft
  Test #17: unit_tests:Time simple stencil operations
  Test #18: unit_tests:Check registry pressure
  Test #19: unit_tests:Check many arrays
  Test #20: unit_tests:Can pull variables from containers based on Metadata
  Test #21: unit_tests:Test required/desired checking from inputs
  Test #22: unit_tests:Parthenon Error Checking
  Test #23: performance_tests:Catch2 Container Iterator Performance
  Test #24: regression_test:output_hdf5
  Test #25: regression_coverage_test:output_hdf5
  Test #26: regression_test:calculate_pi
  Test #27: regression_coverage_test:calculate_pi
  Test #28: regression_test:advection_convergence
  Test #29: regression_coverage_test:advection_convergence
  Test #30: regression_test:advection_performance
  Test #31: regression_coverage_test:advection_performance

The behavior I expected would be that there are no coverage tests when CODE_COVERAGE=OFF.

The second unexpected behavior pertains to the labels.
Of the 31 tests above the "coverage" labels are assigned as follows:

$ ctest -N -L "coverage"
Test project /gpfs/alpine/ast146/world-shared/build-cuda
  Test  #1: unit_tests:Just check everything
  Test  #2: unit_tests:Can create a vector-valued face-variable
  Test  #3: unit_tests:Add and Get is called
  Test  #4: unit_tests:reset is called
  Test  #5: unit_tests:when hasKey is called
  Test  #7: unit_tests:Checking IndexShape indices
  Test  #8: unit_tests:Checking IndexShape cell counts
  Test  #9: unit_tests:par_for loops
  Test #10: unit_tests:nested par_for loops
  Test #12: unit_tests:Built-in flags are registered
  Test #13: unit_tests:A Metadata flag is allocated
  Test #14: unit_tests:A Metadata struct is created
  Test #15: unit_tests:ParArrayND
  Test #16: unit_tests:ParArrayND with LayoutLeft
  Test #20: unit_tests:Can pull variables from containers based on Metadata
  Test #21: unit_tests:Test required/desired checking from inputs
  Test #22: unit_tests:Parthenon Error Checking
  Test #25: regression_coverage_test:output_hdf5
  Test #27: regression_coverage_test:calculate_pi
  Test #29: regression_coverage_test:advection_convergence
  Test #31: regression_coverage_test:advection_performance

$ ctest -N -LE "coverage"
Test project /gpfs/alpine/ast146/world-shared/build-cuda
  Test  #6: unit_tests:Physical constants
  Test #11: unit_tests:Overlapping SpaceInstances
  Test #17: unit_tests:Time simple stencil operations
  Test #18: unit_tests:Check registry pressure
  Test #19: unit_tests:Check many arrays
  Test #23: performance_tests:Catch2 Container Iterator Performance
  Test #24: regression_test:output_hdf5
  Test #26: regression_test:calculate_pi
  Test #28: regression_test:advection_convergence
  Test #30: regression_test:advection_performance

Total Tests: 10

This is unexpected as in the case of code coverage turned off I had expected all tests to not carry the coverage label.

@JoshuaSBrown is this what was intended to happen or is this potentially a bug
In the latter case we may want to add ifs to the cmake/TestSetup.cmake so that coverage tests are not added if they're turned off.

[JoshuaSBrown]

That is definately not the intended behavior. I will take a look, do you know what flags you built with or are you seeing that for all builds?

[JoshuaSBrown]

So I think there has been some confusion. The coverage label is applied to almost all unit tests, the label does not indicate that the code coverage has been built with the test. As in compiler flags have been set. This is a bit challenging to address because Catch2 sets the labels in the .cpp source files, so there is no way using cmake, that I know of to alter the labels.

What I think you want, and what would be ideal is to only add the coverage label when the tests are built with coverage. This can and has now been done with the regression tests because the coverage label is added in cmake.

I see one possible alternative, that is to write the unit tests as cmake configuration files and let cmake insert the labels in the source files. This actually might work well.

[AndrewGaspar]

I see one possible alternative, that is to write the unit tests as cmake configuration files and let cmake insert the labels in the source files. This actually might work well.

In a shared header file, something like:

#cmakedefine COVERAGE_ENABLED

#ifdef COVERAGE_ENABLED
#define COVERAGE_TEST(tags) tags "[coverage]"
#else
#define COVERAGE_TEST(tags) tags
#endif

Then, e.g.:

TEST_CASE("Can create a vector-valued face-variable",
          COVERAGE_TEST("[FaceVariable][Constructor][Get][Set]"))