JUnit report taskforce

[francois-normandin]

It seems to me that reporting is the hot topic of the hour in Caraya. Most issues and feature requests currently explore this frontier where an ever increasing share of LabVIEW development is being done with continuous integration in mind.
As a project, we're bound to have more and more requests along those lines, and an even greater responsibility to not break existing deployments.

I'd like to propose a task force (opened topic) dedicated to this subject and keep it alive, while we branch off the tasks we set out to do in separate threads.

I'll start by referencing recent issues #95 , #100 , #104 , #108 , #109 and #110 as a starting point.

[francois-normandin]

I've relabeled 1.1.0 build 118 as pre-release.
It is stable for anyone that want to take it for a spin, but the jUnit format is not resolved, as per discussions referenced above

[ChrisStrykesAgain]

Per our discussion in #109, it seems Caraya doesn't adhere to the generally accepted JUnit schema as rigidly as it could: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd

That's probably a good thing for a lot of LabVIEW developers, since the don't use classes (especially for their tests) while JUnit is predicated on classes. For some of us, however, a more strict interpretation could be useful. I propose that there be another module developed called "strict JUnit" or such that attempts to follow the schema above more closely.

This would allow the existing implementation to continue without breaking things but still fit the needs of those that want/need the more rigid approach. If you are open to this, what's the best mechanism for collaboration on such a thing? (Sorry, I don't use GitHub a lot.)

More importantly, is anyone else interested?

[felipefoz]

I like the idea. If there is anything I can help. @francois-normandin I am planning on installing LabVIEW 2013 SP1 to do some pull requests on other things whenever is possible.

@ChrisStrykesAgain - I found a link with the description of the fields, more straight (not full) version.

https://www.ibm.com/support/knowledgecenter/en/SSUFAU_1.0.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html

[felipefoz]

Can we leverage on some of this code that is already there?

https://www.vipm.io/package-list/53cb20b3-e4ea-4c60-98d6-f99ac00b79a6/

[jimkring]

I'm happy to be part of this task force. I think a good starting point would be to ask the question:

Why we want to support JUnit reports?

I'm most interested in engineering use cases that deliver direct value to people/projects.

Here's my current use case:

Caraya lets you see your test results in GitLab's CI system, so that you can tell when a branch merge breaks tests.

I'm a little less interested in software features like...

Caraya outputs strict JUnit XML reports.

Are there any other use cases? Anyone trying to have their test results pulled into Jenkins? Other CI Systems?

I think it's best for us to support the most popular (specific) systems really well than all of them poorly :)

[francois-normandin]

My gut feeling is that we should come up with Caraya extension packages instead of adding to the core. The Test Report class can be overridden/overwritten. There is already a Tools menu hook to generate new report classes from template.
(shameless plug: https://youtu.be/ZynOnumXgRE)

edit: I agree with you Jim.

[francois-normandin]

@felipefoz It looks like NI's packages are aimed towards Jenkins. It would be worth to look into them and see if there are some nice gems we can leverage.

UTF JUnit project does not seem to install the dependencies correctly (missing some UTF stuff)... and the VI is kind of similar to what Caraya does, so that one will not be much of help.

Now, the JUnit Results Library is promising I think, for its cluster structure. Very much worth a look. It is geared towards writing a XML file to disk, not generating it in-memory, but that just mean we can only find inspiration, not reuse as a dependency.

[ChrisStrykesAgain]

@jimkring re: a strict report:

My primary purpose is integrating with GitLab, but they don't document anywhere what they use (other than linking to the IBM page, which then links to the Windy Road website that I referenced). As such, it makes sense to me that a "strict" implementation would be the safest to get full support; you would also get the benefit that anything that claims JUnit support would also accept it, and thus minimize the need for different flavors of the plugin.

Essentially, I think it would be worth putting potentially unneeded work in to a "universal" plugin over developing multiple plugins for different platforms.

edit: I'm also fine with this being an extension, and not included in core.

[felipefoz]

@jimkring I agree that we should support the most popular and avoid trying to support all of them poorly.
Considering that I would like discuss some points:

First point: How far is Caraya in fulfilling the requirements of the Gitlab Test Report Viewer?
In my opinion (not from an advanced user), Caraya is doing a very good job in generating a report for Gitlab, but that leads to the second point.

Second point: Gitlab is well known for implementing gradually some features, such as these reports. So, currently, it is working fine, but in the next version Gitlab implements a new feature that uses another property or field from the XML, and this report will become deprecated, needing a new revision.

Third point: So, as @ChrisStrykesAgain said, it is not a bad idea at all creating a strict jUnit XML. As this format doesn't change very often, we expect that at least the parsers (gitlab, jenkins) would take this XML Schema into account, so we may never have to worry about modifying the report for any target anymore.

Using the template could be a way to go, I agree with @francois-normandin. And it could not be directly related to this Caraya Github Repository.