sonar.cxx.xunit.reportPaths

Overview

Sensor to read reports from tools creating xUnit files. xUnit is the collective name for several unit testing frameworks that derive their structure and functionality from Smalltalk's SUnit. The names of many of these frameworks are a variation on "SUnit", usually replacing the "S" with the first letter (or letters) in the name of their intended language ("JUnit" for Java, "RUnit" for R etc.). These frameworks and their common architecture are collectively known as "xUnit".

In combination with the XSL transformation almost all unit test frameworks can be integrated by the cxx plugin.

Note: The cxx plugin itself does not run any tools, you have to do that yourself beforehand. The sensor only reads the report generated by a tool!

Supported XML report file versions

xUnit is not really a standard. There are many variants, most of which differ in fine details. The reports supported by this cxx plugin look like the following (for details see xunit.rnc):

...
<testsuite name="ts" filename="some path" ... >

  <!-- optional: the testsuites can be nested: -->
  <testsuite name="nested ts"
           filename="some path" ... >

    <testcase name="tc1" filename="some path" status="run" time="0.05" classname="classA"/>
    <testcase name="tc2" status="run" time="0" classname="classB">
      <failure message="..." type="">
        <![CDATA[test_component1.cc:17
                 Value of: 112
                 Expected: bar.foo()
                 Which is: 111]]>
      </failure>
    </testcase>
  </testsuite>
  ...
</testsuite>
...

Hints:

• the root tag <testsuites> is ignored and the included <testsuite> tags are processed instead

• the <testsuite> tags can be nested

• Within a <testsuite> element, each unit test is represented by a <testcase> tag. It can have the following attributes:

  • <testcase> (mandatory)
    • name (mandatory): name of the test case
    • classname (optional): name of the test case class. If available, the class name is combined with the test case name. The plugin can't derive the the test file name from the classname.
    • time (optional): long value in milliseconds
    • status (optional): failure|error|skipped: Error status of the test case, if not present ok is assumed. Google Test reports mark the skipped tests with status notrun.
    • message (optional): short message describing the cause
    • stacktrace (optional): long message containing details about failure|error|skipped status

• The filename attribute is optional and tool specific. The filename attribute can be added to the <testsuite> or the <testcase> tag. The values of these attributes should contain a path to the corresponding source file. The path can be absolute or relative to the project base directory. The files present in the report must be test file names, not source code files covered by tests (see sonar.tests analysis parameter).

<testsuite name="ts" filename="tests/mytest.cc" ... >
  ... testcases ...
</testsuite>

and/or

...
    <testcase name="tc" filename="tests/myothertest.cc" ... />
...

Test Case metrics in the SonarQube user interface

The image below shows the display of test metrics in SonarQube 8.9 LTS. To do this, select Measures > Coverage > Tests (1 -> 2) in your project.

At the top, the aggregated total number of tests is displayed (3). Below that, the tests per test file are displayed (4). The display per test file only appears if the filename tag is set in the report.

• Hint: The display options are limited, more details are not displayed.

SonarSource comments as follows: I’ll be honest & say that as an organization we’ve never really believed in those test metrics. Why? Because we feel that if you’ve got failing tests, the pipeline should stop then & there. You shouldn’t just count the failures & move on. That’s why we’ve slowly but surely been moving away from these metrics.

Create report

In order to generate a fitting report, make sure:

• that the paths in the report matches the sonar.sources and sonar.tests lists in sonar-project.properties
• the path can be absolute or relative to the project base directory

Sample command lines:

To create xUnit compatible output with Google Test use it this way:

GoogleTestSample.exe --gtest_output=xml:report.xml

With Boost.Test use --log_level=all or --log_level=success:

BoostTestSample.exe --log_format=XML --log_sink=report.xml --log_level=all --report_level=no

CTest sample:

ctest --output-junit report.xml

Google Test Example

This sample creates a test case with Google Test:

#include <gtest/gtest.h>
#include <iostream>

TEST(TestSuiteName, TestName) {
    std::cout << "Hello world!" << std::endl;
    ASSERT_EQ(1+1, 2);
}

Compile and run the Google Test binary to create XUnit compatible output:

HelloWorld.exe --gtest_output=xml:report.xml

If the tool was executed successfully, a report like the example below should be generated:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="1" failures="0" disabled="0" errors="0" timestamp="2015-01-28T23:29:18" time="0" name="AllTests">
  <testsuite name="TestSuiteName" tests="1" failures="0" disabled="0" errors="0" time="0">
    <testcase name="TestName" status="run" time="0" classname="TestSuiteName" />
  </testsuite>
</testsuites>

Boost.Test Example

This sample creates a test case with Boost.Test:

#define BOOST_TEST_MODULE MyModule
#include <boost/test/unit_test.hpp>
#include <iostream>

BOOST_AUTO_TEST_SUITE(TestSuiteName)
    BOOST_AUTO_TEST_CASE(TestName)
    {
        std::cout << "Hello world!" << std::endl;
        BOOST_CHECK_EQUAL(1+1, 2);
    }
BOOST_AUTO_TEST_SUITE_END()

Compile and run the Boost.Test binary to create a report:

HelloWorld.exe --log_format=XML --log_sink=report.xml --log_level=all --report_level=no

If the tool was executed successfully, a report like the example below should be generated:

<TestLog>
  <TestSuite name="MyModule">
    <TestSuite name="TestSuiteName">
      <TestCase name="TestName">
        <Info file="d:/sample/HelloWorld.cpp" line="8"><![CDATA[check 1+1 == 2 passed]]></Info>
        <TestingTime>1</TestingTime>
      </TestCase>
    </TestSuite>
  </TestSuite>
</TestLog>

CTest Example

The CTest executable is the CMake test driver program. CMake-generated build trees created for projects that use the enable_testing() and add_test() commands have testing support. This program will run the tests and report results. For details more, see the cmake documentation.

Run CTest to create a report:

ctest --output-junit report.xml

If the tool was executed successfully, a report like the example below should be generated:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="Linux-amd64-cc-mpi" tests="3" failures="0" disabled="0" skipped="0" hostname="esthar" time="0" timestamp="2022-08-08T09:51:26" >
   <testcase name="heat_seq_usage" classname="heat_seq_usage" time="0.000872568" status="run"></testcase>
   <testcase name="heat_seq_err_10" classname="heat_seq_err_10" time="0.000996883" status="run"></testcase>
   <testcase name="heat_par_4" classname="heat_par_4" time="0.382706" status="run"></testcase>
</testsuite>

• Hint: CTest does not support the filename tag, i.e. you cannot see test metrics per test file.

Configure cxx plugin

1. First check if the file extensions read in by the cxx plugin are set (sonar.cxx.file.suffixes).
2. That the paths in the report matches the sonar.sources and sonar.tests lists in sonar-project.properties.
3. Set the analysis parameter sonar.cxx.xunit.reportPaths in the configuration file sonar-project.properties of your project. The Report Paths link describes the configuration options.
4. If the report needs to be adjusted, use the built-in XLS transformation, see sonar.cxx.xslt for more information.
5. Execute the SonarScanner to transfer the project with the report to the SonarQube Server.

Sample for sonar-project.properties:

In the simplest case, the report can be read in directly:

sonar.cxx.xunit.reportPaths=report.xml

For formats that are not directly supported, an XSL transformation must be performed first. Here is an example with Boost.Test:

# XSL transformation 'sample.xml' => 'sample.after_xslt'
sonar.cxx.xslt.1.stylesheet=boosttest-1.x-to-junit-1.0.xsl
sonar.cxx.xslt.1.inputs=sample.xml
sonar.cxx.xslt.1.outputs=*.after_xslt

# xUnit: read XML after XSL transformation
sonar.cxx.xunit.reportPaths=sample.after_xslt

Troubleshooting

• If scanning is failing, check items listed under Troubleshooting Configuration.
• If only a total metric and no values per test file are displayed, the report file does not contain a filename tag or the paths are invalid.
• To display metrics per test file, sonar.tests must be set correctly and the report must contain filename tags.
• New versions of SonarQube do not display the contents of the message and stacktrace tags in the user interface.