reduce flaky tests

[josephleekl]

Before submitting

Please complete the following checklist when submitting a PR:

• All new features must include a unit test.
  If you've fixed a bug or added code that should be tested, add a test to the
  tests directory!

• All new functions and code must be clearly commented and documented.
  If you do make documentation changes, make sure that the docs build and
  render correctly by running make docs.

• Ensure that the test suite passes, by running make test.

• Add a new entry to the .github/CHANGELOG.md file, summarizing the
  change, and including a link back to the PR.

• Ensure that code is properly formatted by running make format.

When all the above are checked, delete everything above the dashed
line and fill in the pull request template.

---

Context:
The aim is to reduce the number of flaky tests, and deal with some of the stochastic tests failures observed in CI.

Description of the Change:
4 tests are updated:

1. test_shots_single_measure_obs (TSSMO)
   Previously flaky test, flaky now removed and shot increased 10x. Previous failure due to low shots count.
   Previous observed failure rate (without flaky): LQ 4/1000
   Updated failure rate: LQ 0/1000

2. test_controlled_qubit_gates (TCQG)
   Previously flaky test, flaky now removed. There is no non-determinism in test.
   No failure.

3. test_cnot_controlled_qubit_unitary (TCCQU)
   Previously flaky test, flaky now removed. Only non-determinism in initial state preparation. Initial state now fixed with seed (already used in above test).
   No failure.

4. test_sample_variations (TSV)
   Shots increased 10x and now reference compares with analytical probability calculation in default.qubit rather than with shots.
   Previous observed failure rate: 13/1000
   Updated failure rate: LQ 0/1000

Benefits:

• Reduced flaky tests from 6 to 3
• Reduce likelihood of TSV causing CI failure, which is frequently observed (1 2 3 4)

Possible Drawbacks:
Increased test runtime:

1. TSSMO
   LQ x86: 5s -> 16s
   LG x86: 3s -> 6s
   LK x86 CPU: 1s -> 4s
   LK GPU: 1s -> 4s

2. TSV
   LQ x86: 3s -> 5s
   LG x86: 6s -> 10s
   LK x86 CPU: 1s -> 4s
   LK GPU: 1s -> 4s

However, originally, for TSSMO the time increases if flaky is required to be triggered, and TSV failure would require re-running the tests.

Related GitHub Issues:

[sc-78212]

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.90%. Comparing base (705fe89) to head (8e46a9e).
Report is 1 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (705fe89) and HEAD (8e46a9e). Click for more details.

HEAD has 24 uploads less than BASE

Flag	BASE (705fe89)	HEAD (8e46a9e)
	32	8

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1015      +/-   ##
==========================================
- Coverage   97.90%   91.90%   -6.01%     
==========================================
  Files         230      109     -121     
  Lines       39531    16755   -22776     
==========================================
- Hits        38704    15399   -23305     
- Misses        827     1356     +529     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[multiphaseCFD]

LGTM! Thanks @josephleekl ! Could you please clarify why flaky tests can be reduced with this implementation?

[maliasadi]

LGTM! Happy to approve 🙌

[josephleekl]

LGTM! Thanks @josephleekl ! Could you please clarify why flaky tests can be reduced with this implementation?

Thanks Shuli, the shots increase significantly reduces the failure rate, which means that we should no longer need flaky tests for some tests. ALso some tests are not non-deterministic so no flaky is required.