How to integrate test results into jenkins

[ncarrier]

Hello,
I'm trying to integrate the xml output of unit tests into jenkins, using the xUnit plugin.
Software versions are:

• xUnit plugin: 2.3.7
• jenkins: 2.204.1
• python: 3.7.4
• xmlrunner: 2.5.1

The relevant configuration is the following.

At the end of the build, I have the following log lines:

16:37:16 INFO: Processing CUnit-2.1 (default)
16:37:16 INFO: [CUnit-2.1 (default)] - 1 test report file(s) were found with the pattern 'test-result-*.xml' relative to '/var/lib/jenkins/workspace/libsc-common-test/PRODUCT/diamond' for the testing framework 'CUnit-2.1 (default)'.
16:37:16 WARNING: All test reports are empty.

Yet the xml files contains:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
	<testsuite errors="0" failures="0" file=".py" name="ConfTests-20200115103230" skipped="0" tests="6" time="0.333" timestamp="2020-01-15T10:32:43">
		<testcase classname="ConfTests" file="libsc-common_tests" line="129" name="test_NWP_BuildTime" time="0.144" timestamp="2020-01-15T10:32:43">
			<system-out>
<![CDATA[]]>			</system-out>
			<system-err>
<![CDATA[test_NWP_BuildTime
Build Time (NWP_BuildTime): Jan 15 2020 11:32:20
]]>			</system-err>
		</testcase>
		<testcase classname="ConfTests" file="libsc-common_tests" line="93" name="test_NWP_GetBuildTime" time="0.045" timestamp="2020-01-15T10:32:43">
...

Can it be related to issue #201 ?
Is there a way I can get this to work?

[ncarrier]

Hum now by using JUnit instead of CUnit report type, I got a bunch of errors of this kind:

Attribute 'timestamp' is not allowed to appear in element 'testcase'.

for attributes timestamp, line and file, which seems to indicate that I'm back to the errors described in #201

[ncarrier]

I've somehow worked around the problem by using the Custom tool Report type of the xUnit plugin, with the following custom stylesheet:

<?xml version='1.0'?>
<!-- This stylesheet allows to filter out the attributes which
     python-xmlrunner but which are not supported by the junkins xUnit plugin
     (because its author broke the retro-compatibility by removing them).
     So it'll remove the file, line and timestamp attributes of all the testcase
     tags. -->
<xsl:stylesheet
	xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">

	<xsl:output method="xml" indent="yes" />

	<xsl:template match="node()|@*">
		<xsl:copy>
			<xsl:apply-templates select="node()|@*" />
		</xsl:copy>
	</xsl:template>

	<xsl:template match="//testcase">
		<testcase>
			<xsl:for-each select="@*[name() != 'line' and name() != 'file' and name() != 'timestamp']|child::*">
					<xsl:copy-of select="."/>
			</xsl:for-each>
		</testcase>
	</xsl:template>
</xsl:stylesheet>

Yet I think that this should be fixed in either unittest-xml-reporting or in the xUnit plugin.

[pbudzon]

@ncarrier Any way to add failed/skipped tests support? For me the "failed" and "skipped" columns in the report show as 0, even if there were such tests.

[ncarrier]

@pbudzon Hello
Locally, I managed to have non-zero failed and skipped tests counts:

<testsuites>
        <testsuite errors="0" failures="1" file=".py" name="StartupTest-20200217032055" skipped="1" tests="12" time="19.168" timestamp="2020-02-17T03:21:14">
                <testcase classname="StartupTest" name="test_ExistingLocalConfig1RemoteRemoved" time="1.723"><system-out>
[...]

On jenkins, I only tested failed tests (which work).

FWIW I changed my method a little bit.
Now I'm using the JUnit report type and I post process the xml inside of my test python script:

import xmlrunner
import unittest
import lxml.etree as etree
import io
import os

...

if __name__ == '__main__':
    out = io.BytesIO()

    # run the tests while temporarily storing them in a memory backed file-like
    # object
    unittest.main(testRunner=xmlrunner.XMLTestRunner(output=out),
                  failfast=False, buffer=False, catchbreak=False, exit=False)

    # transform the XML generated to remove the attributes breaking the
    # compatibility with the jenkins xUnit plugin
    out_xml = etree.XML(out.getvalue())
    xslt = etree.parse(os.path.join(DIRNAME, "xmlunittest.xsl"))
    transform = etree.XSLT(xslt)
    out_processed = transform(out_xml)

    # we're good, we can echo the final XML to stdout
    print(etree.tostring(out_processed, pretty_print=True).decode("utf8"))

[pbudzon]

@ncarrier Thanks for this. We actually decided to move away from xunit completely, as it seems the junit parser that's built-in in Jenkins parses the output from xmlrunner without issues, out of the box.

[ncarrier]

Ha, ok, glad to read you've found a solution.
It's not really possible to me since we need a common solution for a variety of unit test report types.

[dnozay]

Here is a XSL you can use similar to the one @ncarrier posted earlier.

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
    <xsl:output method="xml" indent="yes" />

    <!-- /dev/null for these attributes -->
    <xsl:template match="//testcase/@file" />
    <xsl:template match="//testcase/@line" />
    <xsl:template match="//testcase/@timestamp" />

    <!-- copy the rest -->
    <xsl:template match="node()|@*">
        <xsl:copy>
            <xsl:apply-templates select="node()|@*" />
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>

You'd need to use either Custom tool report type and make sure the XSL file in a proper path, or as mentioned you can do the transformation yourself and chose JUnit report type.

I've amended the readme with the steps which I am going to repeat here:

import io
import unittest
import xmlrunner

# run the tests storing results in memory
out = io.BytesIO()
unittest.main(
    testRunner=xmlrunner.XMLTestRunner(output=out),
    failfast=False, buffer=False, catchbreak=False, exit=False)

Transform the results removing extra attributes.

from xmlrunner.extra.xunit_plugin import transform

with open('TEST-report.xml', 'wb') as report:
    report.write(transform(out.getvalue()))

There is a PR where there is a discussion to either add back the fields as optional on JUnit report type or to add a new report type with the same XSL : jenkinsci/xunit-plugin#81

Yet I think that this should be fixed in either unittest-xml-reporting or in the xUnit plugin.

To go back to this earlier comment; I don't plan on dropping the attributes because they provide good debugging information; testsuites are collection of testcases, and testcases can be defined in different file at a certain line number and based on how tests are run, the start time held in timestamp could be useful to debug timing issues.

[ncarrier]

Hello,
Thank you, I tested both methods and both worked for me.
First I'll use the XSLT transform you provided since it's way clearer than mine ^^

But for the xmlrunner.extra.xunit_plugin, I'll wait until a new version is out.

I don't plan on dropping the attributes because they provide good debugging information [...]

I understand and I think you're right, but is there a way to have these pieces of information correctly used by the xUnit plugin? For example by applying another XSLT transform and maybe, using another report type?

[dnozay]

You could add support directly in xunit by adding a custom type.

e.g. jenkinsci/xunit-plugin@2ccffb7#diff-1f96023169ac3d7dfb33b8a32c703bea