The error message in the JUnit xml result is not informative

[chizhg]

When we use kubetest2 to run a test workflow, for each phase, it will add a test in the JUnit xml result if it fails, and include the error message, the main reason for adding this is for integration with Testgrid and Spyglass.

However, the error message for the test is something like exit status 255 and does not contain any information in detail, for example https://prow.knative.dev/view/gs/knative-prow/logs/ci-knative-client-continuous/1308147648712151041.

Two issues related to this:

1. Other phases like Up and Down also have this issue, it would be nice if some detailed information can be added in the error message.
2. Since most of the test flows will generate the JUnit xml files themselves (lots of the projects use gotestsum), is adding the Test phase into the JUnit result a bit noisy? (It is already causing a bit confusion around the Knative community).

/kind bug

[amwat]

Now that we've split the testers into separate binaries, we can probably skip wrapping the Test phase into a Junit error (but that's only for use cases where the tester itself is generating it's own Junit.) For simple cases especially with the --test=exec most commands you'll run would be simple commands that won't generate Junit so we still need an overall pass/fail for the entire Test phase.

For up/down we definitely do need to improve the output there, we should be using metadata.NewJUnitError in the deployers in most places.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[chizhg]

/lifecycle frozen