compiletest: investigate if it's possible to do run-time capability detection against target #135928

jieyouxu · 2025-01-23T11:00:01Z

Some tests are conditioned on capabilities, which are most accurately detected at run-time against the target under cross-compilation scenarios (compared to having to maintain allowlists or denylists of targets). However, implementing this is non-trivial:

You have to build and run the capability run-time test as an executable on the target, not the host. This may involve having to work with custom test runners (e.g. wasmtime) or remote-test-server.
The target may not support std!

The text was updated successfully, but these errors were encountered:

…<try> Implement `needs-subprocess` directive, and cleanup a bunch of tests to use `needs-{subprocess,threads}` ### Summary Closes rust-lang#128295. - Implements `//@ needs-subprocess` directive in compiletest as requested in rust-lang#128295. However, compiletest is a host tool, so we can't just try to spawn process because that spawns the process on *host*, not the *target*, under cross-compilation scenarios. - The short-term solution is to add *Yet Another* list of allow-list targets. - The long-term solution is to first check if a `$target` supports std, then try to run a binary to do run-time capability detection *on the target*. But that is tricky because you have to build-and-run a binary *for the target*. - This PR picks the short-term solution, because the long-term solution is highly non-trivial, and it's already an improvement over individual `ignore-*`s all over the place. - Opened an issue about the long-term solution in rust-lang#135928. - Documents `//@ needs-subprocess` in rustc-dev-guide. - Replace `ignore-{wasm,wasm32,emscripten,sgx}` with `needs-{subprocess,threads}` where suitable in tests. - Some drive-by test changes as I was trying to figure out if I could use `needs-{subprocess,threads}` and found some bits needlessly distracting. Count of tests that use `ignore-{wasm,wasm32,emscripten,sgx}` before and after this PR: | State | `ignore-sgx` | `ignore-wasm` | `ignore-emscripten` | | - | - | - | - | | Before this PR | 96 | 88 | 207 | | After this PR | 36 | 38 | 61 | <details> <summary>Commands used to find out locally</summary> ``` --- before [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-sgx" tests | wc -l 96 [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-wasm" tests | wc -l 88 [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-emscripten" tests | wc -l 207 --- after [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-sgx" tests | wc -l 36 [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-wasm" tests | wc -l 38 [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-emscripten" tests | wc -l 61 ``` </details> ### Review advice - Best reviewed commit-by-commit. - Non-trivial test changes (not mechanically simple replacements) are split into individual commits to help with review. - I *could* split some test changes out into another PR, but I found that I needed to change some tests to `needs-threads`, some to `needs-subprocess`, and some needed to use *both*, so they might conflict and become very annoying. --- r? `@ghost` (need to run try jobs) try-job: x86_64-msvc-1 try-job: i686-msvc-1 try-job: i686-mingw try-job: x86_64-mingw-1 try-job: x86_64-apple-1 try-job: aarch64-apple try-job: aarch64-gnu try-job: test-various try-job: armhf-gnu

jieyouxu · 2025-01-23T11:25:39Z

Alternatively, another print flag: rustc --print=target-supports-std-subprocess.

…<try> Implement `needs-subprocess` directive, and cleanup a bunch of tests to use `needs-{subprocess,threads}` ### Summary Closes rust-lang#128295. - Implements `//@ needs-subprocess` directive in compiletest as requested in rust-lang#128295. However, compiletest is a host tool, so we can't just try to spawn process because that spawns the process on *host*, not the *target*, under cross-compilation scenarios. - The short-term solution is to add *Yet Another* list of allow-list targets. - The long-term solution is to first check if a `$target` supports std, then try to run a binary to do run-time capability detection *on the target*. But that is tricky because you have to build-and-run a binary *for the target*. - This PR picks the short-term solution, because the long-term solution is highly non-trivial, and it's already an improvement over individual `ignore-*`s all over the place. - Opened an issue about the long-term solution in rust-lang#135928. - Documents `//@ needs-subprocess` in rustc-dev-guide. - Replace `ignore-{wasm,wasm32,emscripten,sgx}` with `needs-{subprocess,threads}` where suitable in tests. - Some drive-by test changes as I was trying to figure out if I could use `needs-{subprocess,threads}` and found some bits needlessly distracting. Count of tests that use `ignore-{wasm,wasm32,emscripten,sgx}` before and after this PR: | State | `ignore-sgx` | `ignore-wasm` | `ignore-emscripten` | | - | - | - | - | | Before this PR | 96 | 88 | 207 | | After this PR | 36 | 38 | 61 | <details> <summary>Commands used to find out locally</summary> ``` --- before [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-sgx" tests | wc -l 96 [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-wasm" tests | wc -l 88 [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-emscripten" tests | wc -l 207 --- after [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-sgx" tests | wc -l 36 [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-wasm" tests | wc -l 38 [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-emscripten" tests | wc -l 61 ``` </details> ### Review advice - Best reviewed commit-by-commit. - Non-trivial test changes (not mechanically simple replacements) are split into individual commits to help with review. - I *could* split some test changes out into another PR, but I found that I needed to change some tests to `needs-threads`, some to `needs-subprocess`, and some needed to use *both*, so they might conflict and become very annoying. --- r? `@ghost` (need to run try jobs) try-job: x86_64-msvc-1 try-job: i686-msvc-1 try-job: i686-mingw try-job: x86_64-mingw-1 try-job: x86_64-apple-1 try-job: aarch64-apple try-job: aarch64-gnu try-job: test-various try-job: armhf-gnu

…r=oli-obk Implement `needs-subprocess` directive, and cleanup a bunch of tests to use `needs-{subprocess,threads}` ### Summary Closes rust-lang#128295. - Implements `//@ needs-subprocess` directive in compiletest as requested in rust-lang#128295. However, compiletest is a host tool, so we can't just try to spawn process because that spawns the process on *host*, not the *target*, under cross-compilation scenarios. - The short-term solution is to add *Yet Another* list of allow-list targets. - The long-term solution is to first check if a `$target` supports std, then try to run a binary to do run-time capability detection *on the target*. But that is tricky because you have to build-and-run a binary *for the target*. - This PR picks the short-term solution, because the long-term solution is highly non-trivial, and it's already an improvement over individual `ignore-*`s all over the place. - Opened an issue about the long-term solution in rust-lang#135928. - Documents `//@ needs-subprocess` in rustc-dev-guide. - Replace `ignore-{wasm,wasm32,emscripten,sgx}` with `needs-{subprocess,threads}` where suitable in tests. - Some drive-by test changes as I was trying to figure out if I could use `needs-{subprocess,threads}` and found some bits needlessly distracting. Count of tests that use `ignore-{wasm,wasm32,emscripten,sgx}` before and after this PR: | State | `ignore-sgx` | `ignore-wasm` | `ignore-emscripten` | | - | - | - | - | | Before this PR | 96 | 88 | 207 | | After this PR | 36 | 38 | 61 | <details> <summary>Commands used to find out locally</summary> ``` --- before [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-sgx" tests | wc -l 96 [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-wasm" tests | wc -l 88 [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-emscripten" tests | wc -l 207 --- after [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-sgx" tests | wc -l 36 [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-wasm" tests | wc -l 38 [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-emscripten" tests | wc -l 61 ``` </details> ### Review advice - Best reviewed commit-by-commit. - Non-trivial test changes (not mechanically simple replacements) are split into individual commits to help with review. Their individual commit messages give some basic description of the changes. - I *could* split some test changes out into another PR, but I found that I needed to change some tests to `needs-threads`, some to `needs-subprocess`, and some needed to use *both*, so they might conflict and become very annoying. --- r? `@ghost` (need to run try jobs) try-job: x86_64-msvc-1 try-job: i686-msvc-1 try-job: i686-mingw try-job: x86_64-mingw-1 try-job: x86_64-apple-1 try-job: aarch64-apple try-job: aarch64-gnu try-job: test-various try-job: armhf-gnu

…r=oli-obk Implement `needs-subprocess` directive, and cleanup a bunch of tests to use `needs-{subprocess,threads}` ### Summary Closes rust-lang#128295. - Implements `//@ needs-subprocess` directive in compiletest as requested in rust-lang#128295. However, compiletest is a host tool, so we can't just try to spawn process because that spawns the process on *host*, not the *target*, under cross-compilation scenarios. - The short-term solution is to add *Yet Another* list of allow-list targets. - The long-term solution is to first check if a `$target` supports std, then try to run a binary to do run-time capability detection *on the target*. But that is tricky because you have to build-and-run a binary *for the target*. - This PR picks the short-term solution, because the long-term solution is highly non-trivial, and it's already an improvement over individual `ignore-*`s all over the place. - Opened an issue about the long-term solution in rust-lang#135928. - Documents `//@ needs-subprocess` in rustc-dev-guide. - Replace `ignore-{wasm,wasm32,emscripten,sgx}` with `needs-{subprocess,threads}` where suitable in tests. - Some drive-by test changes as I was trying to figure out if I could use `needs-{subprocess,threads}` and found some bits needlessly distracting. Count of tests that use `ignore-{wasm,wasm32,emscripten,sgx}` before and after this PR: | State | `ignore-sgx` | `ignore-wasm` | `ignore-emscripten` | | - | - | - | - | | Before this PR | 96 | 88 | 207 | | After this PR | 36 | 38 | 61 | <details> <summary>Commands used to find out locally</summary> ``` --- before [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-sgx" tests | wc -l 96 [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-wasm" tests | wc -l 88 [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-emscripten" tests | wc -l 207 --- after [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-sgx" tests | wc -l 36 [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-wasm" tests | wc -l 38 [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-emscripten" tests | wc -l 61 ``` </details> ### Review advice - Best reviewed commit-by-commit. - Non-trivial test changes (not mechanically simple replacements) are split into individual commits to help with review. Their individual commit messages give some basic description of the changes. - I *could* split some test changes out into another PR, but I found that I needed to change some tests to `needs-threads`, some to `needs-subprocess`, and some needed to use *both*, so they might conflict and become very annoying. --- r? ``@ghost`` (need to run try jobs) try-job: x86_64-msvc-1 try-job: i686-msvc-1 try-job: i686-mingw try-job: x86_64-mingw-1 try-job: x86_64-apple-1 try-job: aarch64-apple try-job: aarch64-gnu try-job: test-various try-job: armhf-gnu

Rollup merge of rust-lang#135926 - jieyouxu:needs-subprocess-thread, r=oli-obk Implement `needs-subprocess` directive, and cleanup a bunch of tests to use `needs-{subprocess,threads}` ### Summary Closes rust-lang#128295. - Implements `//@ needs-subprocess` directive in compiletest as requested in rust-lang#128295. However, compiletest is a host tool, so we can't just try to spawn process because that spawns the process on *host*, not the *target*, under cross-compilation scenarios. - The short-term solution is to add *Yet Another* list of allow-list targets. - The long-term solution is to first check if a `$target` supports std, then try to run a binary to do run-time capability detection *on the target*. But that is tricky because you have to build-and-run a binary *for the target*. - This PR picks the short-term solution, because the long-term solution is highly non-trivial, and it's already an improvement over individual `ignore-*`s all over the place. - Opened an issue about the long-term solution in rust-lang#135928. - Documents `//@ needs-subprocess` in rustc-dev-guide. - Replace `ignore-{wasm,wasm32,emscripten,sgx}` with `needs-{subprocess,threads}` where suitable in tests. - Some drive-by test changes as I was trying to figure out if I could use `needs-{subprocess,threads}` and found some bits needlessly distracting. Count of tests that use `ignore-{wasm,wasm32,emscripten,sgx}` before and after this PR: | State | `ignore-sgx` | `ignore-wasm` | `ignore-emscripten` | | - | - | - | - | | Before this PR | 96 | 88 | 207 | | After this PR | 36 | 38 | 61 | <details> <summary>Commands used to find out locally</summary> ``` --- before [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-sgx" tests | wc -l 96 [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-wasm" tests | wc -l 88 [17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-emscripten" tests | wc -l 207 --- after [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-sgx" tests | wc -l 36 [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-wasm" tests | wc -l 38 [17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-emscripten" tests | wc -l 61 ``` </details> ### Review advice - Best reviewed commit-by-commit. - Non-trivial test changes (not mechanically simple replacements) are split into individual commits to help with review. Their individual commit messages give some basic description of the changes. - I *could* split some test changes out into another PR, but I found that I needed to change some tests to `needs-threads`, some to `needs-subprocess`, and some needed to use *both*, so they might conflict and become very annoying. --- r? ``@ghost`` (need to run try jobs) try-job: x86_64-msvc-1 try-job: i686-msvc-1 try-job: i686-mingw try-job: x86_64-mingw-1 try-job: x86_64-apple-1 try-job: aarch64-apple try-job: aarch64-gnu try-job: test-various try-job: armhf-gnu

rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Jan 23, 2025

jieyouxu added this to compiletest maintenance and improvements Jan 23, 2025

github-project-automation bot moved this to Backlog in compiletest maintenance and improvements Jan 23, 2025

jieyouxu mentioned this issue Jan 23, 2025

Implement needs-subprocess directive, and cleanup a bunch of tests to use needs-{subprocess,threads} #135926

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

compiletest: investigate if it's possible to do run-time capability detection against target #135928

compiletest: investigate if it's possible to do run-time capability detection against target #135928

jieyouxu commented Jan 23, 2025 •

edited

Loading

jieyouxu commented Jan 23, 2025

compiletest: investigate if it's possible to do run-time capability detection against *target* #135928

compiletest: investigate if it's possible to do run-time capability detection against *target* #135928

Comments

jieyouxu commented Jan 23, 2025 • edited Loading

jieyouxu commented Jan 23, 2025

compiletest: investigate if it's possible to do run-time capability detection against target #135928

compiletest: investigate if it's possible to do run-time capability detection against target #135928

jieyouxu commented Jan 23, 2025 •

edited

Loading