Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Are runtime tests actually required in CI?? #1485

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

paul0403
Copy link
Contributor

@paul0403 paul0403 commented Jan 24, 2025

Context:
There are a couple different items that seem to be running runtime tests in CI. However it seems like the only entry that runs the full runtime test suite, Runtime Code Coverage, is not actually a required CI item.

Moreover, it seems like the "lightning", "lightning-kokkos" and "openqasm" entries are not actually doing any testing at all.

We slip in a expected failed test into all runtime test files to see which CI entries can pick it up.

@paul0403 paul0403 requested a review from a team January 24, 2025 20:45
Copy link
Contributor

Hello. You may have forgotten to update the changelog!
Please edit doc/releases/changelog-dev.md on your branch with:

  • A one-to-two sentence description of the change. You may include a small working example for new features.
  • A link back to this PR.
  • Your name (or GitHub username) in the contributors section.

@paul0403 paul0403 changed the title Are runtime tests actually required in CI? Are runtime tests actually required in CI?? Jan 24, 2025
@paul0403
Copy link
Contributor Author

I don't know how long this has been in there, but apparently none of the required runtime test CI entries right now can actually detect failures! Moreover, the three required runtime test CI entries don't actually run anything!

The only entry that runs the entire runtime test suite is the runtime code coverage, which is not a required CI entry for merging.

This PR slipped a CHECK(1 == 0); into all runtime test files, so if the testing is setup correctly then everything should fail. However, none, other than the runtime coverage, actually failed!

@paul0403 paul0403 added author:build-wheels Run the wheel building workflows on this Pull Request reviewer:require-wheels Pull Requests will need wheel building job successful before being merged urgent Mark a pull request as high priority labels Jan 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author:build-wheels Run the wheel building workflows on this Pull Request reviewer:require-wheels Pull Requests will need wheel building job successful before being merged urgent Mark a pull request as high priority
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant