test_runner_describe_it failures break TAP output

[richardlau]

Version

main

Platform

No response

Subsystem

test

What steps will reproduce the bug?

No response

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior?

No response

What do you see instead?

Seen on the CI. When test_runner_describe_it (formerly test_runner_destribe_it (#43764)) fails it breaks the tap2junit TAP parser we have in the CI:
e.g.

• https://ci.nodejs.org/job/node-test-commit-linuxone/32996/nodes=rhel8-s390x/consoleFull
• https://ci.nodejs.org/job/node-test-commit-osx/46125/nodes=osx11-x64/consoleFull

15:47:26 not ok 3127 message/test_runner_desctibe_it
15:47:26   ---
15:47:26   duration_ms: 3.801
15:47:26   severity: fail
15:47:26   exitcode: 1
15:47:26   stack: |-
15:47:26     TAP version 13
15:47:26     # Subtest: sync pass todo
15:47:26     ok 1 - sync pass todo # TODO
15:47:26       ---
15:47:26       duration_ms: 0.017320097
15:47:26       ...
15:47:26     # Subtest: sync pass todo with message
15:47:26     ok 2 - sync pass todo with message # TODO this is a passing todo
15:47:26       ---
15:47:26       duration_ms: 0.00010036
15:47:26       ...
15:47:26     # Subtest: sync fail todo
15:47:26     not ok 3 - sync fail todo # TODO
...
16:00:46 + tap2junit -i test.tap -o test.xml
16:00:48 Traceback (most recent call last):
16:00:48   File "/usr/local/bin/tap2junit", line 10, in <module>
16:00:48     sys.exit(main())
16:00:48   File "/Library/Python/3.8/site-packages/tap2junit/__main__.py", line 64, in main
16:00:48     convert(args.input, args.output)
16:00:48   File "/Library/Python/3.8/site-packages/tap2junit/__main__.py", line 43, in convert
16:00:48     result = parse(input_file, data)
16:00:48   File "/Library/Python/3.8/site-packages/tap2junit/__main__.py", line 35, in parse
16:00:48     tap_parser.parse(data)
16:00:48   File "/Library/Python/3.8/site-packages/tap2junit/tap13.py", line 174, in parse
16:00:48     self._parse(StringIO(source))
16:00:48   File "/Library/Python/3.8/site-packages/tap2junit/tap13.py", line 144, in _parse
16:00:48     raise ValueError("Descending test id on line: %r" % line)
16:00:48 ValueError: Descending test id on line: 'not ok 1 - +sync throw fail'

Such failures are also breaking ncu-ci from node-core-utils (nodejs/node-core-utils#639 (comment)) which breaks the CI Reliability reports in https://github.com/nodejs/reliability.

Additional information

No response

[MoLow]

Taking a look. might take me a few days

[MoLow]

@richardlau do we know if this affects anything else besides node-core-utils?

[richardlau]

@MoLow https://github.com/nodejs/tap2junit, as described above. In the Jenkins CI this is used to convert the TAP output of Node.js' tools/test.py to XML which Jenkins can parse -- we do this because the tap plugin for Jenkins has very bad performance (nodejs/build#453).

[F3n67u]

I experience multiple time when jenkins job say test.xml is empty, but I don't know what is wrong and ignored. If a job say test.xml is empty, then there must be some problem when run tap2junit right?

[richardlau]

@F3n67u Yes, either tap2junit has failed to parse the tap output or the build failed to compile and no tests were run.

[MoLow]

@richardlau please see my PR - the output is valid, it is that tap2junit parser that cannot handle it