In the context of CI, get results of single tests as they are run

[francois-normandin]

@francois-normandin after two years, we finally got around to implement this feature into our own Release Automation Tools.

I had hoped to be able to receive an event for each single test, so we can output what's happening while the whole test suite is executed. Is this something that would fit with the current architecture, and are there any plans to implement this?

Originally posted by @joerghampel in #95 (comment)

[francois-normandin]

@joerghampel The issue #95 was closed, so I'm referencing your ask in this new thread.

[francois-normandin]

It is conceivable, yes. The piping for public events when run as part of a Suite is there, so it's a matter of defining what makes sense for information to be shared as tests are produced.

Currently, we have "Test Suite Result" which is fired at the end, and "suite destroyed".

What granularity to you see as being needed?
We can probably have all these events, potentially: (which ones are important)

• Test Registered
• Assertion (each test can have one or more assertions)
• Assertion Comment (when verbosity = true)

Anything else that you would expect?

[francois-normandin]

I propose the following schemas, events to be published as JSON payloads.

Test Registration Event:

{
   "test-uid": "myTestID",
   "test-name" : "myTestName",
   "timestamp": "ISO timestamp",
   "caller-uid": "myCallerTestID",
   "call-chain": ["", "", ""]
}

Assertion Event:

{
   "assert-uid": "myAssertID",
   "test-uid": "myTestID",
   "assert-name" : "myAssertName",
   "timestamp": "ISO timestamp",
   "passed": true|false,
   "test-error": {"status": false, "code": 0, "source": ""},
   "call-chain": ["", "", ""],
   "execution-time": 0.001
}

Assertion Update Event: (uid + fields that have been updated)

{
   "assert-uid": "myAssertID",
   "explanation": "theVerboseExplanation"
}

[joerghampel]

Ideally, we'd like to be able to reproduce the UI's output in a plain-text form:

So by the looks of it, we would need all three proposed events (test registration, assertion, assertion update), right?

[francois-normandin]

The events are not ordered. Caraya will execute them in different order depending on how you wire the error terminals and how many tests are launched in parallel.
You will be able to reconstruct a plain text report once you get the full report, but at run-time, you will need to handle it as a tree structure, where the uid's are the keys.

<test-uid>   UT_Main.vi
<test-uid>      Test hse-configuration lvlib
<test-uid>            Test extending a config with another...
<assert-uid>               Key in root was added
<assert-uid>               Keys were extended and overwritten
<test-uid>            Test creating a basis-class object
<assert-uid>               Not existent sections raises an error

[joerghampel]

Maybe I'm overthinking it... It's not a hard requirement to recreate the UI's output. The important thing here is to generate some output in case a user either runs our tools manually or watches the CI output while debugging, to show that there is progress, and what the tool currently does.

Seeing as the output is written back to the command line, we can't (easily) update previous lines, so maintaining a tree seems moot. Maybe we should just go with the Test Registration, that seems rather straight forward and would probably give an indication of what is currently happening...?

[francois-normandin]

This is already supported in Caraya 1.3.0, but it will be documented with example in 1.4.0

[francois-normandin]

Featured in 1.4.0 RC1
#172

(Not yet released on VIPM Community.)

There is a new Test Report class of type "Events", which can also be used.
An example will be added to this discussion thread shortly. There is already an example event report in the example folder.

[francois-normandin]

Example code for Test Suite Events are in the Caraya>> Advanced>> Examples palette

Example code for Event-based Report is in Caraya >> Advanced >> Reporting >> Examples

The latter Reporting class can be cloned by developers to provide their own event-based injection mechanisms.