You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We should automatically run SkipIfVcr tests in "Recording" mode, instead of just skipping them. (We could implement this as running all skipped tests in "Recording" mode, since always-skip tests will still be skipped.)
We will likely need to also have a way to actually skip on PRs (but still run on nightlies) for specific tests that truly can't run in that environment (or for handling cases like stampeding herd of PRs accessing a specific API.)
Alternatively: We currently alert users when a test was added but not run (for example because it was skipped in VCR or is GA-only). However, we could try to detect all tests that are run and skipped in VCR and raise them directly where the reviewer will see them, so they can make a more informed decision about whether manual tests are required. The problem with this approach is that it would be easy to miss and require manual intervention by the reviewer to run and then check the tests.
References
GoogleCloudPlatform/magic-modules#11291 introduced a failure in TestAccGKEHubFeature_FleetDefaultMemberConfigConfigManagement because it was not run on the PR.
The text was updated successfully, but these errors were encountered:
melinath
changed the title
Surface tests skipped due to VCR more directly to users
Surface tests skipped due to VCR more directly to reviewers
Aug 9, 2024
What kind of contribution is this issue about?
Pull request UX
Details
We should automatically run SkipIfVcr tests in "Recording" mode, instead of just skipping them. (We could implement this as running all skipped tests in "Recording" mode, since always-skip tests will still be skipped.)
We will likely need to also have a way to actually skip on PRs (but still run on nightlies) for specific tests that truly can't run in that environment (or for handling cases like stampeding herd of PRs accessing a specific API.)
Alternatively: We currently alert users when a test was added but not run (for example because it was skipped in VCR or is GA-only). However, we could try to detect all tests that are run and skipped in VCR and raise them directly where the reviewer will see them, so they can make a more informed decision about whether manual tests are required. The problem with this approach is that it would be easy to miss and require manual intervention by the reviewer to run and then check the tests.
References
GoogleCloudPlatform/magic-modules#11291 introduced a failure in TestAccGKEHubFeature_FleetDefaultMemberConfigConfigManagement because it was not run on the PR.
The text was updated successfully, but these errors were encountered: