Libtest json output

[Gilnaa]

Added an option to output the tests' summary in Json.
The exact format can use a good bikesheding.

A new "--format" flag is added, with the following options:

• Pretty: The default output format.
• Terse: Like -q
• Json: Json

-q is now an alias to --format=terse, but any --format= overrides -q

Discussed a bit here: https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/12

The actual code generating JSON is a bit messy. I thought about using serde, but I think it's simple enough, and as @matklad said, events should be outputted as they are ready.

Example:

running 2 tests
test t1 ... FAILED
test t2 ... FAILED

failures:

---- t1 stdout ----
	FLO"RP
thread 't1' panicked at 'explicit panic', f.rs:4:1
note: Run with `RUST_BACKTRACE=1` for a backtrace.

---- t2 stdout ----
	BLARG\
thread 't2' panicked at 'assertion failed: 2 != 2', f.rs:10:1


failures:
    t1
    t2

test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out

Turns into

{
	"events": [
		{ "test": "t2", "event": "started" },
		{ "test": "t2", "event": "failed" },
		{ "test": "t1", "event": "started" },
		{ "test": "t1", "event": "failed" }
	],
	"summary": {
		"passed": 0,
		"failed": 2,
		"allowed_fail": 0,
		"ignored": 0,
		"measured": 0,
		"filtered_out": 0,
		"failures": [
			"t2": "BLARG\\
thread 't2' panicked at 'assertion failed: 2 != 2', f.rs:10:1
",
			"t1": "FLO\"RP
thread 't1' panicked at 'explicit panic', f.rs:4:1
note: Run with `RUST_BACKTRACE=1` for a backtrace.
"
		]
	}
}

[shepmaster]

Auto-assigning to...

r? @Kimundi

[matklad]

Related issues are

#44813

rust-lang/rfcs#816

[killercup]

Your JSON example output confuses me. Wasn't there discussion of outputting every event as a JSON document? This was also discussed in rust-lang/rfcs#1284.

[Gilnaa]

Yeah, I might have misunderstood that one.
Why would you want each of them to be separate document?

Is there some consensus about how the output should look like?

Edit: What you linked is a bit in conflict with @matklad request to have separate "events" for tests' start/finish.

Otherwise, if the old RFC is acceptable, I'll close this PR and fix my code to match it.

[killercup]

So that when an event happens (e.g., a test fails), it can be written immediately. This way, it's easy to write a tool that displays the progress interactively and without waiting for the whole test suite to finish. Gilad Naaman <notifications@github.com> schrieb am Sa. 11. Nov. 2017 um 13:32:

…

Yeah, I might have misunderstood that one. Why would you want each of them to be separate document? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#45923 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABOXzru6Fo4jM19Db73IKClcy4dJtkTks5s1ZPZgaJpZM4QaL-C> .

[Gilnaa]

So then I have a few questions:

• Why not use a streaming parser?
• The whole difference is just using a flat structure with no commas, right?
• How should several test threads be handled?

[killercup]

I have no authority to give any advice on how to do this, I'm just someone from the dev tools team who likes tools that offer machine readable output :)

Edit: What you linked is a bit in conflict with @matklad request to have separate "events" for tests' start/finish.
Otherwise, if the old RFC is acceptable, I'll close this PR and fix my code to match it.

I just wanted to point out that there's been discussion around this topic. The RFC was closed as the author wanted to rewrite it IIUC. As such, it was never accepted by the Rust team in charge.

• Why not use a streaming parser?

Is there a good streaming JSON parser for Rust? I don't think serde_json supports this.

• The whole difference is just using a flat structure with no commas, right?

Instead of

{
  "events": [
    { "test": "t2", "event": "started" },
    { "test": "t2", "event": "failed" },
    { "test": "t1", "event": "started" },
    { "test": "t1", "event": "failed" }
  ],
  "summary": { }
}

you'd output something like

{ "type": "test", "event": "start", "name": "foo::t2" }
{ "type": "test", "event": "start", "name": "bar::t1" }
{ "type": "test", "event": "fail", "name": "foo::t2" }
{ "type": "test", "event": "success", "name": "bar:t1" }
{ "type": "final", "summary": {  } }

• How should several test threads be handled?

Same as right now? Does it need to change in any way? When a test is started, you emit an event, when the test finishes (it thread ends) you emit another event depending on the result of the thread (panic/no panic basically). If you really wanted, you could capture stdout/stderr and emit it (line or timeout based) as events in-between, but I haven't really thought about this.

[Gilnaa]

I see. I'll contact the other maintainer to make sure I'm not ruining his work somehow. Thanks

As for streaming parsers, I hacked one using serde https://github.com/Gilnaa/rusty3bar/blob/master/src/infinite_array.rs
But it is not pretty

[dnsco]

I'm excited to see work happening in this direction, I would love to have more easily machine readable test output.