Open Test Reporting

XML and HTML test reporting formats that are agnostic of testing framework and programming language.

Introduction

This repository contains the definition of and tooling for two new XML-based test reporting formats that are agnostic of any testing framework and programming language yet concrete enough to be consumable by a wide range of downstream tools such as IDEs, build tools, and report generators. Moreover, it provides a report generator that produces a standalone HTML file based on the XML formats.

The event-based XML format is suitable for writing test events to a file or streaming them over a local socket or network connection. Instead of having to collect all data for a test only to write it to the report after the test has completed, testing frameworks can emit events as they occur, for example when a test is started or finished.

The hierarchical XML format is meant to be closer to existing representations of test results that users are familiar with. For example, when executing tests in IDEs, one is usually presented with a tree of tests and their results. This same structure can be found in files that use this format. Thus, instead of requiring humans or processing tools to piece together all relevant events for a test from the event-based format, this hierarchical format collects all relevant information in a single place for each test. Besides being human-readable, this format is easier to transform into an HTML report by a downstream reporting tool or CI server. Both XML formats are designed to complement each other such that the event-based format can be mechanically converted into the hierarchical one.

This repository contains a reference implementation of an API for writing event-based XML files (from JVM languages such as Java), as well as a CLI tool for validating them, converting them to the hierarchical format, and generating HTML reports from them. This way, testing frameworks can focus on writing the event-based format; build tools may use the converter to write the hierarchical format or generate HTML reports; and reporting tools can consume whichever format they prefer.

Note	Please refer to the appendix for more information about the motivation behind this project and its design goals.

XML format specification

Both formats share a set of core elements that is defined in core.xsd.

Event-based format

The event-based format is defined in events.xsd.

The following example shows the result of a test run with a single top-level "container" and a "test" child.

<?xml version="1.0" ?>
<e:events
        xmlns="https://schemas.opentest4j.org/reporting/core/0.1.0"
        xmlns:e="https://schemas.opentest4j.org/reporting/events/0.1.0">
    <infrastructure> <!--(1)-->
        <hostName>wonderland</hostName>
        <userName>alice</userName>
    </infrastructure>
    <e:started id="1" name="container" time="2022-02-05T16:30:39.129888Z"/> <!--(2)-->
    <e:started id="2" name="test" parentId="1" time="2022-02-05T16:30:39.137022Z"/> <!--(3)-->
    <e:finished id="2" time="2022-02-05T16:30:39.143013Z"> <!--(4)-->
        <result status="SUCCESSFUL"/>
    </e:finished>
    <e:finished id="1" time="2022-02-05T16:30:39.143292Z"> <!--(5)-->
        <result status="SUCCESSFUL"/>
    </e:finished>
</e:events>

1. attributes of the infrastructure the tests were run on

2. start event of "container" with timestamp

3. start event of "test" with timestamp and reference to its parent "container"

4. finished event of "test" with timestamp and result status

5. finished event of "container" with timestamp and result status

This file contains 4 events: 2 started and 2 finished ones. The test framework reporting these events was able to write them as they occurred. In particular, it did not have to wait for all children of "container" to be finished.

Hierarchical format

The event-based format is defined in hierarchy.xsd.

The following example shows the result of converting the above event-based report into the hierarchical format.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<h:execution xmlns:h="https://schemas.opentest4j.org/reporting/hierarchy/0.1.0"
             xmlns="https://schemas.opentest4j.org/reporting/core/0.1.0">
    <infrastructure> <!--(1)-->
        <hostName>wonderland</hostName>
        <userName>alice</userName>
    </infrastructure>
    <h:root duration="PT0.013404S" name="container" start="2022-02-05T16:30:39.129888Z"> <!--(2)-->
        <result status="SUCCESSFUL"/>
        <h:child duration="PT0.005991S" name="test" start="2022-02-05T16:30:39.137022Z"> <!--(3)-->
            <result status="SUCCESSFUL"/>
        </h:child>
    </h:root>
</h:execution>

1. attributes of the infrastructure the tests were run on (same as in the event-based format)

2. root node of "container" with timestamp, duration, and result status

3. child node of "test" with timestamp, duration, and result status

Schema extensions

All schema definitions mentioned so far are language-agnostic.

Java

In order to report Java-specific attributes, e.g. the class or method name of a test, an extension schema is defined in java.xsd.

Git

In order to include Git-specific metadata, e.g. the commit hash or branch name, an extension schema is defined in git.xsd.

Tip	Test frameworks are encouraged to define their own similar framework-specific extensions if they want to report additional information that is not suitable to be added to the core namespace. See the JUnit project for an example of how to define an extension schema.

Reference implementation

While the reporting formats are language-agnostic, the reference implementation is written in Java due to being the language its authors (the JUnit team) are most familiar with.

This repository contains the following subprojects:

schema

XML schema definitions of both formats

events

Java API for writing the event-based format without having to deal with Java’s XML APIs

cli

Command-line interface (CLI) for validating both formats, converting from the event-based to the hierarchical format, and generating HTML reports

tooling-core

Java API for validating both formats, converting from the event-based to the hierarchical format (suitable for inclusion in build tools and reporting tools), and generating HTML reports

tooling-spi

Java Service Provider Interface (SPI) for the extending the HTML report

html-report

Vue.js-based HTML report single-page application (SPA)

sample-project

Sample project that demonstrates how to enable JUnit’s event-based output and set up Gradle to convert it to the hierarchical format and generate an HTML report

API for writing event-based format

Testing frameworks that run on the JVM can use the API provided by the events subprojects as follows.

import org.opentest4j.reporting.events.api.DocumentWriter;
import org.opentest4j.reporting.events.api.NamespaceRegistry;
import org.opentest4j.reporting.events.core.CoreFactory;
import org.opentest4j.reporting.events.root.Events;
import org.opentest4j.reporting.schema.Namespace;

import java.nio.file.Paths;
import java.time.Instant;

import static org.opentest4j.reporting.events.core.CoreFactory.*;
import static org.opentest4j.reporting.events.core.Result.Status.SUCCESSFUL;
import static org.opentest4j.reporting.events.root.RootFactory.finished;
import static org.opentest4j.reporting.events.root.RootFactory.started;

public class DocumentWriterSample {

    public static void main(String[] args) throws Exception {

        NamespaceRegistry namespaceRegistry = NamespaceRegistry.builder(Namespace.REPORTING_CORE) // (1)
                .add("e", Namespace.REPORTING_EVENTS) //
                .add("java", Namespace.REPORTING_JAVA) //
                .build();

        try (DocumentWriter<Events> writer = Events.createDocumentWriter(namespaceRegistry, Paths.get("events.xml"))) {
            writer.append(infrastructure(), infrastructure -> infrastructure // (2)
                    .append(userName("alice")) //
                    .append(hostName("wonderland")));
            writer.append(started("1", Instant.now(), "container")); // (3)
            writer.append(started("2", Instant.now(), "test"), started -> started.withParentId("1")); // (4)
            writer.append(finished("2", Instant.now()), finished -> finished.append(CoreFactory.result(SUCCESSFUL))); // (5)
            writer.append(finished("1", Instant.now()), finished -> finished.append(CoreFactory.result(SUCCESSFUL))); // (6)
        }
    }
}

1. create a registry of all namespaces used in the document along with their prefixes

2. report infrastructure attributes

3. start event of "container" with timestamp

4. start event of "test" with timestamp and reference to its parent "container"

5. finished event of "test" with timestamp and result status

6. finished event of "container" with timestamp and result status

Tip	See the JUnit project for an example of how to write an event-based report.

CLI tool

The CLI tool provided by the cli subprojects provides subcommands for validating both formats, converting from the event-based to the hierarchical format, and generating an HTML report.

Installation

You can download the CLI tool from Maven Central using the org.opentest4j.reporting:open-test-reporting:0.1.0-M2-cli:standalone coordinates.

Alternatively, you can use JBang:

$ jbang org.opentest4j.reporting:open-test-reporting-cli:0.1.0-M2:standalone --help

Validate XML files

$ java -jar open-test-reporting-cli-0.1.0-M2.jar validate events.xml
ℹ️ Valid: events.xml

Note	Please run validate --help for more information.

Convert event-based to hierarchical format

$ java -jar open-test-reporting-cli-0.1.0-M2.jar convert events.xml
ℹ️ Converted events.xml to file:///some/path/hierarchy.xml

Note	Please run convert --help for more information.

HTML report

The HTML report is a single-page application (SPA) that displays the test results in a tree-like structure and allows users to view details about each test/container. It can be generated as follows using the CLI:

$ java -jar open-test-reporting-cli-0.1.0-M2.jar html-report --output open-test-report.html events-1.xml hierarchical.xml
ℹ️ Wrote HTML report to file:///some/path/open-test-report.html

Note	Please run html-report --help for more information.

The command takes one or multiple XML files in the event-based or hierarchical format as input and generates a single HTML file that can be opened in any modern web browser without requiring a server.

Figure 1. Sample HTML report where all tests pass (light theme)

Figure 2. Sample HTML report with a failing test (dark theme)

Extending the HTML report

Since the XML formats are extensible, the HTML report can be extended to display additional information that is not part of the core schemas. In order to do so, the tooling-spi subproject defines a Java Service Provider Interface (SPI) that allows developers to implement a custom extension that is automatically picked up by the HTML report generator when it is on the classpath/module path. Please refer to the Javadoc of the org.opentest4j.reporting.tooling.spi.htmlreport.Contributor interface for more information.

Tip	See the JUnit project for an example of how to implement a Contributor.

Appendix A: Prior art

In the Java ecosystem, Ant originally defined an XML-based reporting format for tests. Other build tools like Maven and Gradle have adopted the format and, in the case of Maven Surefire, later made changes to add additional data. Many build servers know how to parse the XML-based format, and even non-Java tools sometimes support it. However, it’s based on the concept of test classes and methods, so using it for frameworks and tools where those elements are not present is awkward at best. Moreover, it does not support nested structures beyond a simple parent-child relationship. Finally, it is not extensible: no additional attributes can be added without the risk of breaking existing tools.

For those reasons, many testing frameworks such as TestNG and Spock have defined their own reporting formats. This has given them the flexibility they need, but the number of tools that can parse, display, or transform their custom formats is very limited.

To overcome the limitations described above, this document defines a new format for test reporting. Its goal is to be platform-agnostic so that as many testing frameworks as possible can benefit from it. Moreover, it is designed to be extensible so new data can be added as needed, without breaking consumers. However, all well-known attributes are properly defined so it’s consumable by downstream reporting tools.

Appendix B: Design goals

human-readable

The new format needs to be human-readable so it can be inspected without requiring another tool. A format that represents the test tree via nesting is easier to understand than a flat list of events that reference their parents via IDs.

streamable

Writing report files during test execution should not require keeping state about currently running tests, etc. in memory. Instead, the new format should allow tools to write events to a file as they occur.

machine-readable

At the same time, the format must be machine-readable so it can be easily parsed and transformed on a variety of platforms.

schema-aware

Implementors and tools should be able to validate a given document against a well-known schema.

extensible

Adding additional language- or framework-specific attributes to nodes must be possible without breaking backwards compatibility.

XML vs. JSON

JSON is less verbose than XML, but the latter provides more expressive ways to define schemas. Moreover, XML has typed extensions built-in via the use of multiple schemas. Thus, the new formats use XML with accompanying XML schemas.