custom test harness #2
Labels
C-tracking-issue
Category: A tracking issue for something unstable.
S-needs-team-input
Status: Needs input from team on whether/how to proceed.
Milestone
Comments
epage
added
C-tracking-issue
10 tasks
Open
3 tasks
epage
added a commit
to epage/rust
that referenced
this issue
Dec 13, 2024
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
epage
added a commit
to epage/rust
that referenced
this issue
Dec 13, 2024
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
epage
added a commit
to epage/rust
that referenced
this issue
Jan 7, 2025
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
epage
added a commit
to epage/rust
that referenced
this issue
Jan 7, 2025
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
epage
added a commit
to epage/rust
that referenced
this issue
Jan 24, 2025
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
epage
added a commit
to epage/rust
that referenced
this issue
Jan 24, 2025
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
epage
added a commit
to epage/rust
that referenced
this issue
Jan 24, 2025
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
matthiaskrgr
added a commit
to matthiaskrgr/rust
that referenced
this issue
Jan 25, 2025
fix(libtest): Deprecate '--logfile' rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api. Closes rust-lang/testing-devex-team#9
matthiaskrgr
added a commit
to matthiaskrgr/rust
that referenced
this issue
Jan 26, 2025
fix(libtest): Deprecate '--logfile' rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api. Closes rust-lang/testing-devex-team#9
jhpratt
added a commit
to jhpratt/rust
that referenced
this issue
Jan 26, 2025
fix(libtest): Deprecate '--logfile' rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api. Closes rust-lang/testing-devex-team#9
rust-timer
added a commit
to rust-lang-ci/rust
that referenced
this issue
Jan 26, 2025
Rollup merge of rust-lang#134283 - epage:logfile, r=Amanieu fix(libtest): Deprecate '--logfile' rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api. Closes rust-lang/testing-devex-team#9
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summary of discussion with T-libs-api about providing a richer API in
test:See https://epage.github.io/blog/2023/06/iterating-on-test/
Tasks
#[distributed_slice]aka a method to enumerate tests #3mainfrom a dependency? Might get this mostly for free by putting it in a prelude thanks for 1.79'simported_mainThe text was updated successfully, but these errors were encountered: