Supporting junit XML report in rust_test #1303
Labels
Comments
|
Discovered a previous related discussion here. Is this feature still in the roadmap? |
|
The test launcher was actually deleted #1069 But I think it'd be great if |
|
+ for support for |
|
If anyone's got context on this, any PRs for this would be welcomed! 😄 |
|
Note that having a test runner again would potentially help support code coverage as well. Although I'm sure there are downsides |
|
I think my only hesitation to adding a test runner again is breaking the ability to use ps: #690 is I think related to the topic of process wrappers in my comment here. |
|
What orchestration are you thinking for this case? I wonder if another approach would be to provide some library that could do this generation instead? I don't know rust very well but in apple land there is some infra for being able to observe test results in code, so that can actually be used in process to do this. |
|
There's an unstable JUnit formatter built in to libtest. Tracking issue: rust-lang/rust#85563 |
aspotashev
added a commit
to aspotashev/rust
that referenced
this issue
Mar 7, 2024
Rationale for adding this functionality:
* Bazel (build system) doesn't provide a way to process output from a
binary (in this case, Rust test binary's output) other using a wrapper
binary. However, using a wrapper binary potentially breaks debugging,
because Bazel would suggest to debug the wrapper binary rather than
the Rust test itself.
* See bazelbuild/rules_rust#1303.
* Cargo is not used in Rust Bazel rules.
* Although we could wait for
rust-lang#96290 and then modify Rust
Bazel rules to pass `--logfile` on the command line to
provisionally unblock
bazelbuild/rules_rust#1303, that
solution still wouldn't allow advanced test output postprocessing
such as changing JSON/XML schema and injecting extra JUnit
properties.
* Due to limitations of Rust libtest formatters, Rust developers often
use a separate tool to postprocess the test results output (see
comments to rust-lang#85563).
* Examples of existing postprocessing tools:
* https://crates.io/crates/cargo2junit
* https://crates.io/crates/gitlab-report
* https://crates.io/crates/cargo-suity
* For these use cases, it might be helpful to use the new flags
`--output_postprocess_executable`, `--output_postprocess_args`
instead of piping the test output explicitly, e.g. to more
reliably separate test results from other output.
Rationale for implementation details:
* Use platform-dependent scripts (.sh, .cmd) because it doesn't seem to
be possible to enable unstable feature `bindeps`
(https://rust-lang.github.io/rfcs/3028-cargo-binary-dependencies.html)
in "x.py" by default.
* Here's a preexisting test that also uses per-platform
specialization: `library/std/src/process/tests.rs`.
How to use:
```
./testbinary --format=junit -Zunstable-options --output_postprocess_executable=/usr/bin/echo --output_postprocess_args=abc123
```
aspotashev
added a commit
to aspotashev/rust
that referenced
this issue
Mar 7, 2024
Rationale for adding this functionality:
* Bazel (build system) doesn't provide a way to process output from a
binary (in this case, Rust test binary's output) other using a wrapper
binary. However, using a wrapper binary potentially breaks debugging,
because Bazel would suggest to debug the wrapper binary rather than
the Rust test itself.
* See bazelbuild/rules_rust#1303.
* Cargo is not used in Rust Bazel rules.
* Although we could wait for
rust-lang#96290 and then modify Rust
Bazel rules to pass `--logfile` on the command line to
provisionally unblock
bazelbuild/rules_rust#1303, that
solution still wouldn't allow advanced test output postprocessing
such as changing JSON/XML schema and injecting extra JUnit
properties.
* Due to limitations of Rust libtest formatters, Rust developers often
use a separate tool to postprocess the test results output (see
comments to rust-lang#85563).
* Examples of existing postprocessing tools:
* https://crates.io/crates/cargo2junit
* https://crates.io/crates/gitlab-report
* https://crates.io/crates/cargo-suity
* For these use cases, it might be helpful to use the new flags
`--output_postprocess_executable`, `--output_postprocess_args`
instead of piping the test output explicitly, e.g. to more
reliably separate test results from other output.
Rationale for implementation details:
* Use platform-dependent scripts (.sh, .cmd) because it doesn't seem to
be possible to enable unstable feature `bindeps`
(https://rust-lang.github.io/rfcs/3028-cargo-binary-dependencies.html)
in "x.py" by default.
* Here's a preexisting test that also uses per-platform
specialization: `library/std/src/process/tests.rs`.
How to use:
```
./testbinary --format=junit -Zunstable-options --output_postprocess_executable=/usr/bin/echo --output_postprocess_args=abc123
```
aspotashev
added a commit
to aspotashev/rust
that referenced
this issue
Mar 8, 2024
Add the following two optional flags to `libtest` (rustc's built-in unit-test
framework), in order to support postprocessing of the test results using a
separate executable:
* `--output_postprocess_executable [PATH]`
* `--output_postprocess_args [ARGUMENT]` (can be repeated.)
If you don't pass `--output_postprocess_executable [PATH]`, the behavior stays
the same as before this commit. That is, the test results are sent to stdout.
If you pass `--output_postprocess_executable [PATH]`, `libtest` will
1. Spawn a child process from the executable binary (aka *postprocessor*) at the given path.
2. Pass the arguments from the `--output_postprocess_args [ARGUMENT]` flags (if
any) to the child process. If `--output_postprocess_args` was used multiple
times, all listed arguments will be passed in the original order.
3. Propagate the environment variables to the child process.
The *postprocessor* executable is expected to wait for the end of input (EOF) and then terminate.
Usage example #1: Filter lines of the test results
```shell
$ LD_LIBRARY_PATH=$(pwd) ./test-05daf44cb501aee6 --output_postprocess_executable=/usr/bin/grep --output_postprocess_args="test result"
test result: ok. 59 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.31s
```
Usage example rust-lang#2: Save test results into a file
```shell
$ LD_LIBRARY_PATH=$(pwd) ./test-05daf44cb501aee6 --output_postprocess_executable=/usr/bin/sh --output_postprocess_args=-c --output_postprocess_args="cat > /tmp/output.txt"
```
Usage example rust-lang#3: Save test results into a file while keeping the command line
output
```shell
$ LD_LIBRARY_PATH=$(pwd) ./test-05daf44cb501aee6 --output_postprocess_executable=/usr/bin/tee --output_postprocess_args="/tmp/output.txt"
running 60 tests
...
```
Usage example rust-lang#4: Prepend every line of test results with the value of an environment variable (to demonstrate environment variable propagation)
```shell
$ LOG_PREFIX=">>>" LD_LIBRARY_PATH=$(pwd) ./test-05daf44cb501aee6 --output_postprocess_executable=/usr/bin/sh --output_postprocess_args=-c --output_postprocess_args="sed s/^/\$LOG_PREFIX/"
>>>
>>>running 60 tests
...
```
Usage example rust-lang#5: Change format of JSON output (using
https://jqlang.github.io/jq/)
```shell
$ LD_LIBRARY_PATH=$(pwd) ./test-05daf44cb501aee6 -Zunstable-options --format=json --output_postprocess_executable=/usr/bin/jq
```
Usage example rust-lang#6: Print test execution time in machine-readable format
```shell
$ LD_LIBRARY_PATH=$(pwd) ./test-05daf44cb501aee6 -Zunstable-options --format=json --output_postprocess_executable=/usr/bin/jq --output_postprocess_args=".exec_time | numbers"
0.234317996
```
Rationale for adding this functionality:
* Bazel (build system) doesn't provide a way to process output from a binary
(in this case, Rust test binary's output) other using a wrapper binary.
However, using a wrapper binary potentially breaks debugging, because Bazel
would suggest to debug the wrapper binary rather than the Rust test itself.
* See bazelbuild/rules_rust#1303.
* Cargo is not used in Rust Bazel rules.
* Although we could wait for rust-lang#96290
and then modify Rust Bazel rules to pass `--logfile` on the command line
to provisionally unblock
bazelbuild/rules_rust#1303, that solution
still wouldn't allow advanced test results postprocessing such as
changing JSON/XML schema and injecting extra JUnit properties.
* Due to limitations of Rust libtest formatters, Rust developers often use a
separate tool to postprocess the test results output (see comments to
rust-lang#85563).
* Examples of existing postprocessing tools:
* https://crates.io/crates/cargo2junit
* https://crates.io/crates/gitlab-report
* https://crates.io/crates/cargo-suity
* For these use cases, it might be helpful to use the new flags
`--output_postprocess_executable`, `--output_postprocess_args` instead
of piping the test results explicitly, e.g. to more reliably separate
test results from other output.
Rationale for implementation details:
* Use platform-dependent scripts (.sh, .cmd) because it doesn't seem to be
possible to enable unstable feature `bindeps`
(https://rust-lang.github.io/rfcs/3028-cargo-binary-dependencies.html) in
"x.py" by default.
* Here's a preexisting test that also uses per-platform specialization:
`library/std/src/process/tests.rs`.
aspotashev
added a commit
to aspotashev/rust
that referenced
this issue
Mar 8, 2024
Add the following two optional flags to `libtest` (rustc's built-in unit-test
framework), in order to support postprocessing of the test results using a
separate executable:
* `--output_postprocess_executable [PATH]`
* `--output_postprocess_args [ARGUMENT]` (can be repeated.)
If you don't pass `--output_postprocess_executable [PATH]`, the behavior stays
the same as before this commit. That is, the test results are sent to stdout.
If you pass `--output_postprocess_executable [PATH]`, `libtest` will
1. Spawn a child process from the executable binary (aka *postprocessor*) at
the given path.
2. Pass the arguments from the `--output_postprocess_args [ARGUMENT]` flags (if
any) to the child process. If `--output_postprocess_args` was used multiple
times, all listed arguments will be passed in the original order.
3. Propagate the environment variables to the child process.
The *postprocessor* executable is expected to wait for the end of input (EOF)
and then terminate.
Usage example #1: Filter lines of the test results
```shell
$ LD_LIBRARY_PATH=$(pwd) ./test-05daf44cb501aee6 --output_postprocess_executable=/usr/bin/grep --output_postprocess_args="test result"
test result: ok. 59 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.31s
```
Usage example rust-lang#2: Save test results into a file
```shell
$ LD_LIBRARY_PATH=$(pwd) ./test-05daf44cb501aee6 --output_postprocess_executable=/usr/bin/sh --output_postprocess_args=-c --output_postprocess_args="cat > /tmp/output.txt"
```
Usage example rust-lang#3: Save test results into a file while keeping the command line
output
```shell
$ LD_LIBRARY_PATH=$(pwd) ./test-05daf44cb501aee6 --output_postprocess_executable=/usr/bin/tee --output_postprocess_args="/tmp/output.txt"
running 60 tests
...
```
Usage example rust-lang#4: Prepend every line of test results with the value of an
environment variable (to demonstrate environment variable propagation)
```shell
$ LOG_PREFIX=">>>" LD_LIBRARY_PATH=$(pwd) ./test-05daf44cb501aee6 --output_postprocess_executable=/usr/bin/sh --output_postprocess_args=-c --output_postprocess_args="sed s/^/\$LOG_PREFIX/"
>>>
>>>running 60 tests
...
```
Usage example rust-lang#5: Change format of JSON output (using
https://jqlang.github.io/jq/)
```shell
$ LD_LIBRARY_PATH=$(pwd) ./test-05daf44cb501aee6 -Zunstable-options --format=json --output_postprocess_executable=/usr/bin/jq
```
Usage example rust-lang#6: Print test execution time in machine-readable format
```shell
$ LD_LIBRARY_PATH=$(pwd) ./test-05daf44cb501aee6 -Zunstable-options --format=json --output_postprocess_executable=/usr/bin/jq --output_postprocess_args=".exec_time | numbers"
0.234317996
```
Rationale for adding this functionality:
* Bazel (build system) doesn't provide a way to process output from a binary
(in this case, Rust test binary's output) other using a wrapper binary.
However, using a wrapper binary potentially breaks debugging, because Bazel
would suggest to debug the wrapper binary rather than the Rust test itself.
* See bazelbuild/rules_rust#1303.
* Cargo is not used in Rust Bazel rules.
* Although we could wait for rust-lang#96290
and then modify Rust Bazel rules to pass `--logfile` on the command line
to provisionally unblock
bazelbuild/rules_rust#1303, that solution
still wouldn't allow advanced test results postprocessing such as
changing JSON/XML schema and injecting extra JUnit properties.
* Due to limitations of Rust libtest formatters, Rust developers often use a
separate tool to postprocess the test results output (see comments to
rust-lang#85563).
* Examples of existing postprocessing tools:
* https://crates.io/crates/cargo2junit
* https://crates.io/crates/gitlab-report
* https://crates.io/crates/cargo-suity
* For these use cases, it might be helpful to use the new flags
`--output_postprocess_executable`, `--output_postprocess_args` instead
of piping the test results explicitly, e.g. to more reliably separate
test results from other output.
Rationale for implementation details:
* Use platform-dependent scripts (.sh, .cmd) because it doesn't seem to be
possible to enable unstable feature `bindeps`
(https://rust-lang.github.io/rfcs/3028-cargo-binary-dependencies.html) in
"x.py" by default.
* Here's a preexisting test that also uses per-platform specialization:
`library/std/src/process/tests.rs`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently
rust_testdoes not support generating junit xml report out-of-the-box.To support this for all rust test target, a workaround is with the
--run_underflag and a script that writes the xml report from stdout to $XML_OUTPUT_FILE:The main downside is having to declare the script as a
datadependency for every rust test target.It would be nice to have built-in support for this in rust_test as an option, would it be sensible to add this to the rust test launcher?
The text was updated successfully, but these errors were encountered: