Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run SkipIfVCR tests during recording mode #19079

Open
melinath opened this issue Aug 9, 2024 · 1 comment
Open

Run SkipIfVCR tests during recording mode #19079

melinath opened this issue Aug 9, 2024 · 1 comment

Comments

@melinath
Copy link
Collaborator

melinath commented Aug 9, 2024

What kind of contribution is this issue about?

Pull request UX

Details

We should automatically run SkipIfVcr tests in "Recording" mode, instead of just skipping them. (We could implement this as running all skipped tests in "Recording" mode, since always-skip tests will still be skipped.)

We will likely need to also have a way to actually skip on PRs (but still run on nightlies) for specific tests that truly can't run in that environment (or for handling cases like stampeding herd of PRs accessing a specific API.)

Alternatively: We currently alert users when a test was added but not run (for example because it was skipped in VCR or is GA-only). However, we could try to detect all tests that are run and skipped in VCR and raise them directly where the reviewer will see them, so they can make a more informed decision about whether manual tests are required. The problem with this approach is that it would be easy to miss and require manual intervention by the reviewer to run and then check the tests.

References

GoogleCloudPlatform/magic-modules#11291 introduced a failure in TestAccGKEHubFeature_FleetDefaultMemberConfigConfigManagement because it was not run on the PR.

@melinath melinath changed the title Surface tests skipped due to VCR more directly to users Surface tests skipped due to VCR more directly to reviewers Aug 9, 2024
@melinath
Copy link
Collaborator Author

GoogleCloudPlatform/magic-modules#11153 introduced test failures because the tests were skipped in VCR.

@melinath melinath changed the title Surface tests skipped due to VCR more directly to reviewers Run SkipIfVCR tests during recording mode Aug 19, 2024
@BBBmau BBBmau added this to the Goals milestone Aug 19, 2024
@BBBmau BBBmau added the size/l label Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants