test report formats
- SubUnit
- Test Anything Protocol
- JUnit
- TeamCity CI:
- DejaGnu
- HP QuickTest Professional
- IBM Rational Functional Tester
- Gallio Test Report
- Parasoft C/C++ test
- Ranorex
- TAPOUT - TAP Y J Specification
- Universal test output
- XML-based
- TestRail
- SonarQube
- GTester XML Format (convert to JUnit)
- Allure (XSD Schema)
- xUnit
- Microsoft Test (.TRX)
- YAML-based
- JSON-based
- MozLog
- Language Independent Test Format
- Chromium The JSON Test Results Format
- CMake
ctest(also TAP13) - Go
go test(JSON) - KernelCI JSON schema
- Common Reporter Interface (CRI) for JavaScript testing frameworks
Test Anything Protocol
Grammar in an EBNF (source):
Program ::= Statement
Statement ::= Plan Directive? Description?
| "TAP version" Num
| Status Num? "-"? Description? Directive?
| Comment
| BailOut
| YAML
Status ::= ( "ok" | "not ok" )
Directive ::= "#" ( "TODO" | "SKIP" )
Word ::= '"' .* '"' | [^-] .+
Plan ::= Num ".." Num
Num ::= [0-9]+
Comment ::= [^#]
BailOut ::= "Bail out!"
Description ::= '"' .* '"' | [^-] .+
YAML ::= Start Data End
Start ::= ( "\n---" | "" )
End ::= ( "\n..." | "\n---" | "" )
Data ::= (Scalar | Sequence | Mapping )
Scalar ::= (Number | String | Date | Boolean | Nil)
Sequence ::= ( "[" Data ("," Data)* "]" | OptionalTab "-" Data ("\n" OptionalTab "-" Data)* )
Mapping ::= ( "{" Key ":" Data ("," Key ":" Data)* "}" | Tab Key ":" Data ("\n" Tab Key ":" Data)* )
OptinalTab ::= Space*
Tab ::= Space+
String ::= '"' .* '"' | [^-] .+
Number ::= ("+" | "-")? [0-9]* ("." [0-9]+)?
Date ::= [0-9][0-9][0-9][0-9] "-" [0-1][0-9] "-" [0-3][0-9] ( [0-2][0-9] ":" [0-5][0-9] ":" [0-5][0-9] )?
Boolean ::= "true" | "false"
Nil ::= "~"
Space ::= " "
SubUnit v1
Grammar in an EBNF:
Program ::= Statement
Statement ::= Test ":" "test" Label
| Status ":" "test" Label Details
| "progress" ":" Action
| "tags" ":" Tag
| "time" ":" Date
Test ::= ( "test" | "testing" )
Status ::= ( Test | Success | "failure" | "error" | "skip" | "xfail" | "uxsuccess" )
Test ::= ( "test" | "testing" | "test" | "testing" )
Success ::= ( "success" | "success" | "successful" | "successful" )
Label ::= '"' .* '"' | [^-] .+
Details ::= ( Bracketed | Multipart )
Action ::= ( "push" | "pop" | "[+|-]X" )
Tag ::=
Date ::= [0-9][0-9][0-9][0-9] "-" [0-1][0-9] "-" [0-3][0-9] ( [0-2][0-9] ":" [0-5][0-9] ":" [0-5][0-9] "Z" )?
Name ::= UTF8*
Bracketed ::= '[' CR UTF8-lines ']' "CR"
Multipart ::= '[ multipart' CR PART* ']' "CR"
Part ::= PartType "CR" Name "CR" PartBytes "CR"
PartType ::= "Content-Type: type/sub-type(;parameter=value,parameter=value)"
JUnit
<?xml version="1.0" encoding="UTF-8"?>
<!-- a description of the JUnit XML format and how Jenkins parses it. See also junit.xsd -->
<!-- if only a single testsuite element is present, the testsuites
element can be omitted. All attributes are optional. -->
<testsuites disabled="" <!-- total number of disabled tests from all testsuites. -->
errors="" <!-- total number of tests with error result from all testsuites. -->
failures="" <!-- total number of failed tests from all testsuites. -->
name=""
tests="" <!-- total number of successful tests from all testsuites. -->
time="" <!-- time in seconds to execute all test suites. -->
>
<!-- testsuite can appear multiple times, if contained in a testsuites element.
It can also be the root element. -->
<testsuite name="" <!-- Full (class) name of the test for non-aggregated testsuite documents.
Class name without the package for aggregated testsuites documents. Required -->
tests="" <!-- The total number of tests in the suite, required. -->
disabled="" <!-- the total number of disabled tests in the suite. optional -->
errors="" <!-- The total number of tests in the suite that errored. An errored test is one that had an unanticipated problem,
for example an unchecked throwable; or a problem with the implementation of the test. optional -->
failures="" <!-- The total number of tests in the suite that failed. A failure is a test which the code has explicitly failed
by using the mechanisms for that purpose. e.g., via an assertEquals. optional -->
hostname="" <!-- Host on which the tests were executed. 'localhost' should be used if the hostname cannot be determined. optional -->
id="" <!-- Starts at 0 for the first testsuite and is incremented by 1 for each following testsuite -->
package="" <!-- Derived from testsuite/@name in the non-aggregated documents. optional -->
skipped="" <!-- The total number of skipped tests. optional -->
time="" <!-- Time taken (in seconds) to execute the tests in the suite. optional -->
timestamp="" <!-- when the test was executed in ISO 8601 format (2014-01-21T16:17:18). Timezone may not be specified. optional -->
>
<!-- Properties (e.g., environment settings) set during test
execution. The properties element can appear 0 or once. -->
<properties>
<!-- property can appear multiple times. The name and value attributres are required. -->
<property name="" value=""/>
</properties>
<!-- testcase can appear multiple times, see /testsuites/testsuite@tests -->
<testcase name="" <!-- Name of the test method, required. -->
assertions="" <!-- number of assertions in the test case. optional -->
classname="" <!-- Full class name for the class the test method is in. required -->
status=""
time="" <!-- Time taken (in seconds) to execute the test. optional -->
>
<!-- If the test was not executed or failed, you can specify one
the skipped, error or failure elements. -->
<!-- skipped can appear 0 or once. optional -->
<skipped message="" <!-- message/description string why the test case was skipped. optional -->
/>
<!-- Indicates that the test errored. An errored test is one
that had an unanticipated problem. For example an unchecked
throwable or a problem with the implementation of the
test. Contains as a text node relevant data for the error,
for example a stack trace. optional -->
<error message="" <!-- The error message. e.g., if a java exception is thrown, the return value of getMessage() -->
type="" <!-- The type of error that occured. e.g., if a java execption is thrown the full class name of the exception. -->
></error>
<!-- Indicates that the test failed. A failure is a test which
the code has explicitly failed by using the mechanisms for
that purpose. For example via an assertEquals. Contains as
a text node relevant data for the failure, e.g., a stack
trace. optional -->
<failure message="" <!-- The message specified in the assert. -->
type="" <!-- The type of the assert. -->
></failure>
<!-- Data that was written to standard out while the test was executed. optional -->
<system-out>STDOUT text</system-out>
<!-- Data that was written to standard error while the test was executed. optional -->
<system-err>STDERR text</system-err>
</testcase>
<!-- Data that was written to standard out while the test suite was executed. optional -->
<system-out>STDOUT text</system-out>
<!-- Data that was written to standard error while the test suite was executed. optional -->
<system-err>STDERR text</system-err>
</testsuite>
</testsuites>The following XML Schema Definition is supposed to validate the JUnit format from the previous section.
<?xml version="1.0" encoding="UTF-8" ?>
<!-- from https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd -->
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="failure">
<xs:complexType mixed="true">
<xs:attribute name="type" type="xs:string" use="optional"/>
<xs:attribute name="message" type="xs:string" use="optional"/>
</xs:complexType>
</xs:element>
<xs:element name="error">
<xs:complexType mixed="true">
<xs:attribute name="type" type="xs:string" use="optional"/>
<xs:attribute name="message" type="xs:string" use="optional"/>
</xs:complexType>
</xs:element>
<xs:element name="properties">
<xs:complexType>
<xs:sequence>
<xs:element ref="property" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="property">
<xs:complexType>
<xs:attribute name="name" type="xs:string" use="required"/>
<xs:attribute name="value" type="xs:string" use="required"/>
</xs:complexType>
</xs:element>
<xs:element name="skipped">
<xs:complexType mixed="true">
<xs:attribute name="message" type="xs:string" use="optional"/>
</xs:complexType>
</xs:element>
<xs:element name="system-err" type="xs:string"/>
<xs:element name="system-out" type="xs:string"/>
<xs:element name="testcase">
<xs:complexType>
<xs:sequence>
<xs:element ref="skipped" minOccurs="0" maxOccurs="1"/>
<xs:element ref="error" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="failure" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="system-out" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="system-err" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="name" type="xs:string" use="required"/>
<xs:attribute name="assertions" type="xs:string" use="optional"/>
<xs:attribute name="time" type="xs:string" use="optional"/>
<xs:attribute name="classname" type="xs:string" use="optional"/>
<xs:attribute name="status" type="xs:string" use="optional"/>
</xs:complexType>
</xs:element>
<xs:element name="testsuite">
<xs:complexType>
<xs:sequence>
<xs:element ref="properties" minOccurs="0" maxOccurs="1"/>
<xs:element ref="testcase" minOccurs="0" maxOccurs="unbounded"/>
<xs:element ref="system-out" minOccurs="0" maxOccurs="1"/>
<xs:element ref="system-err" minOccurs="0" maxOccurs="1"/>
</xs:sequence>
<xs:attribute name="name" type="xs:string" use="required"/>
<xs:attribute name="tests" type="xs:string" use="required"/>
<xs:attribute name="failures" type="xs:string" use="optional"/>
<xs:attribute name="errors" type="xs:string" use="optional"/>
<xs:attribute name="time" type="xs:string" use="optional"/>
<xs:attribute name="disabled" type="xs:string" use="optional"/>
<xs:attribute name="skipped" type="xs:string" use="optional"/>
<xs:attribute name="timestamp" type="xs:string" use="optional"/>
<xs:attribute name="hostname" type="xs:string" use="optional"/>
<xs:attribute name="id" type="xs:string" use="optional"/>
<xs:attribute name="package" type="xs:string" use="optional"/>
</xs:complexType>
</xs:element>
<xs:element name="testsuites">
<xs:complexType>
<xs:sequence>
<xs:element ref="testsuite" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="name" type="xs:string" use="optional"/>
<xs:attribute name="time" type="xs:string" use="optional"/>
<xs:attribute name="tests" type="xs:string" use="optional"/>
<xs:attribute name="failures" type="xs:string" use="optional"/>
<xs:attribute name="disabled" type="xs:string" use="optional"/>
<xs:attribute name="errors" type="xs:string" use="optional"/>
</xs:complexType>
</xs:element>
</xs:schema>
Learning
- Glossary
- Books:
- Courses
- Learning Tools
- Bugs And Learned Lessons
- Cheatsheets
Tools / Services / Tests
- Quality Assurance Tools
- Test Runners
- Testing-As-A-Service
- Conformance Test Suites
- Test Infrastructure
- Fault injection
- TTCN-3
- Continuous Integration
- Speedup your CI
- Performance
- Formal Specification
- Toy Projects
- Test Impact Analysis
- Formats
Functional testing
- Automated testing
- By type:
WIP sections
Community
Links