Add no-optionals and full-optionals test runs #8967

jakelishman · 2022-10-20T11:51:13Z

Summary

This restructures the CI slightly to perform a complete "no-optionals" run, and a complete "all optionals" run (for the optionals that are readily accessible as Python packages, without complex additional setup). Previously, we only did a partial test with some of the oldest optional components, which could have allowed for behaviour to accidentally require optional components without us noticing.

This splits the requirements-dev.txt file into two; lines that remain in the file are what is actually required to run the test suite, run the style checks, and do the documentation build. The rest (and everything that was missing) is added to a new
requirements-optional.txt file, which can be used to pull in (almost) all of the packages that Terra can use to provide additional / accelerated functionality.

Several tests needed to gain additional skips to account for this change. There is a good chance that some tests are missing skips for some libraries that are not the first point of failure, but it's hard to test explicitly for these in one go.

Details and comments

qiskit-bot · 2022-10-20T11:51:21Z

Thank you for opening a new pull request.

Before your PR can be merged it will first need to pass continuous integration tests and be reviewed. Sometimes the review process can be slow, so please be patient.

While you're waiting, please feel free to review other open PRs. While only a subset of people are authorized to approve pull requests for merging, everyone is encouraged to review open pull requests. Doing reviews helps reduce the burden on the core team and helps make the project's code better for everyone.

One or more of the the following people are requested to review this:

@Qiskit/terra-core
@enavarro51
@ikkoham

This restructures the CI slightly to perform a complete "no-optionals" run, and a complete "all optionals" run (for the optionals that are readily accessible as Python packages, without complex additional setup). Previously, we only did a partial test with some of the oldest optional components, which could have allowed for behaviour to accidentally require optional components without us noticing. This splits the `requirements-dev.txt` file into two; lines that remain in the file are what is actually _required_ to run the test suite, run the style checks, and do the documentation build. The rest (and everything that was missing) is added to a new `requirements-optional.txt` file, which can be used to pull in (almost) all of the packages that Terra can use to provide additional / accelerated functionality. Several tests needed to gain additional skips to account for this change. There is a good chance that some tests are missing skips for some libraries that are not the first point of failure, but it's hard to test explicitly for these in one go.

coveralls · 2022-10-20T13:28:11Z

Pull Request Test Coverage Report for Build 5600511995

1 of 1 (100.0%) changed or added relevant line in 1 file are covered.
222 unchanged lines in 21 files lost coverage.
Overall coverage increased (+1.4%) to 87.444%

Files with Coverage Reduction	New Missed Lines	%
crates/accelerate/src/sabre_layout.rs	1	99.43%
qiskit/transpiler/passes/layout/dense_layout.py	1	95.1%
qiskit/visualization/circuit/_utils.py	1	94.35%
crates/qasm2/src/lex.rs	2	91.39%
qiskit/circuit/controlflow/switch_case.py	2	94.08%
qiskit/qasm3/ast.py	2	97.79%
qiskit/circuit/classical/expr/visitors.py	3	95.59%
qiskit/circuit/library/standard_gates/x.py	3	99.12%
qiskit/transpiler/preset_passmanagers/level1.py	3	96.67%
qiskit/transpiler/passes/routing/sabre_swap.py	5	96.89%

Totals
Change from base Build 5592160564:	1.4%
Covered Lines:	73869
Relevant Lines:	84476

💛 - Coveralls

jakelishman · 2022-10-20T17:21:11Z

Tests should pass after #8973.

jakelishman · 2022-10-28T13:22:28Z

Blocked by #9021.

jakelishman · 2022-10-29T09:42:09Z

Ready for review.

# Conflicts: # .azure/test-linux.yml # .azure/tutorials-linux.yml # azure-pipelines.yml # requirements-dev.txt # test/python/dagcircuit/test_dagcircuit.py

requirements-optional.txt

jakelishman · 2023-07-11T23:15:48Z

The failure on Windows is due to #10415.

jakelishman · 2023-07-19T00:03:52Z

With #10441 merged, this is unblocked.

Eric-Arellano · 2023-07-19T00:07:00Z

Will re-review tomorrow. Let's make sure to land this before #10443.

Eric-Arellano

Thanks! Great cleanup.

Feel free to merge without applying my changes - they're somewhat nitty

.azure/test-linux.yml

azure-pipelines.yml

mtreinish

This LGTM, I'm a still mildly concerned about the upper version runs not including the optionals as I just know we've had issues with some optional dependencies lagging on python version support. The nightly jobs provide the coverage, but we've been bad historically about catching failures in the nightly runs. That being said this is a good cleanup and restructuring of how we deal with the optionals and testing. We can always revisit the exact testing matrix and strategy in a followup pr if we have issues for 3.12.

* Add no-optionals and full-optionals test runs This restructures the CI slightly to perform a complete "no-optionals" run, and a complete "all optionals" run (for the optionals that are readily accessible as Python packages, without complex additional setup). Previously, we only did a partial test with some of the oldest optional components, which could have allowed for behaviour to accidentally require optional components without us noticing. This splits the `requirements-dev.txt` file into two; lines that remain in the file are what is actually _required_ to run the test suite, run the style checks, and do the documentation build. The rest (and everything that was missing) is added to a new `requirements-optional.txt` file, which can be used to pull in (almost) all of the packages that Terra can use to provide additional / accelerated functionality. Several tests needed to gain additional skips to account for this change. There is a good chance that some tests are missing skips for some libraries that are not the first point of failure, but it's hard to test explicitly for these in one go. * Fix typo in coverage workflow * Try relaxing ipython constraints * Squash newly exposed lint failures * Fix typo in tutorials pipeline * Update the 'also update' comments * Remove unneeded qiskit-toqm dependency * Section requirements-optional.txt * Test all optionals on min not max Python version Optionals are generally more likely to have been made available on the older Pythons, and some may take excessively long to provide wheels for the latest versions of Python. * Add missing test skip * Fix optional call * Use correct boolean syntax * Fix tests relying on Jupyter * Install ipykernel in tutorials * Remove HAS_PDFLATEX skip from quantum_info tests For simple LaTeX tests, IPython/Jupyter can handle the compilation internally using MathJax, and doesn't actually need a `pdflatex` installation. We only need that when we're depending on LaTeX libraries that are beyond what MathJax can handle natively. * Include additional tutorials dependencies * Install all of Terra's optionals in tutorial run * Do not install all optionals in docs build * Use class-level skips where appropriate * Do not install ibmq-provider in tutorials run * Include matplotlib in docs requirements * Remove unnecessary whitespace * Split long pip line * Only install graphviz when installOptionals Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com> * Install visualization extras in docs build * Don't `--upgrade` when installing optionals This is to prevent any optionals that have a dependency on Terra from potentially upgrading it to some PyPI version during their installation. This shouldn't happen with the current development model of having only one supported stable branch and the main branch, but the `-U` is unnecessary, and not having it is safer for the future. * Update secondary installation of Terra in docs build * Install all optionals during the docs build I yoyo-ed on this, not wanting it to be too onerous to build the documentation, but in the end, we need to have the optional features installed in order to build most of the documentation of those features, and some unrelated parts of the documentation may use these features to enhance their own output. * Fix test setup job * Remove duplication of matplotlib in requirements files * Update image test installation command * Restore editable build * Move `pip check` to `pip` section * Remove redundant "post-install" description * Expand comment on first-stage choices * pytohn lol --------- Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com> (cherry picked from commit 0438a4b)

* Add no-optionals and full-optionals test runs This restructures the CI slightly to perform a complete "no-optionals" run, and a complete "all optionals" run (for the optionals that are readily accessible as Python packages, without complex additional setup). Previously, we only did a partial test with some of the oldest optional components, which could have allowed for behaviour to accidentally require optional components without us noticing. This splits the `requirements-dev.txt` file into two; lines that remain in the file are what is actually _required_ to run the test suite, run the style checks, and do the documentation build. The rest (and everything that was missing) is added to a new `requirements-optional.txt` file, which can be used to pull in (almost) all of the packages that Terra can use to provide additional / accelerated functionality. Several tests needed to gain additional skips to account for this change. There is a good chance that some tests are missing skips for some libraries that are not the first point of failure, but it's hard to test explicitly for these in one go. * Fix typo in coverage workflow * Try relaxing ipython constraints * Squash newly exposed lint failures * Fix typo in tutorials pipeline * Update the 'also update' comments * Remove unneeded qiskit-toqm dependency * Section requirements-optional.txt * Test all optionals on min not max Python version Optionals are generally more likely to have been made available on the older Pythons, and some may take excessively long to provide wheels for the latest versions of Python. * Add missing test skip * Fix optional call * Use correct boolean syntax * Fix tests relying on Jupyter * Install ipykernel in tutorials * Remove HAS_PDFLATEX skip from quantum_info tests For simple LaTeX tests, IPython/Jupyter can handle the compilation internally using MathJax, and doesn't actually need a `pdflatex` installation. We only need that when we're depending on LaTeX libraries that are beyond what MathJax can handle natively. * Include additional tutorials dependencies * Install all of Terra's optionals in tutorial run * Do not install all optionals in docs build * Use class-level skips where appropriate * Do not install ibmq-provider in tutorials run * Include matplotlib in docs requirements * Remove unnecessary whitespace * Split long pip line * Only install graphviz when installOptionals Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com> * Install visualization extras in docs build * Don't `--upgrade` when installing optionals This is to prevent any optionals that have a dependency on Terra from potentially upgrading it to some PyPI version during their installation. This shouldn't happen with the current development model of having only one supported stable branch and the main branch, but the `-U` is unnecessary, and not having it is safer for the future. * Update secondary installation of Terra in docs build * Install all optionals during the docs build I yoyo-ed on this, not wanting it to be too onerous to build the documentation, but in the end, we need to have the optional features installed in order to build most of the documentation of those features, and some unrelated parts of the documentation may use these features to enhance their own output. * Fix test setup job * Remove duplication of matplotlib in requirements files * Update image test installation command * Restore editable build * Move `pip check` to `pip` section * Remove redundant "post-install" description * Expand comment on first-stage choices * pytohn lol --------- Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com> (cherry picked from commit 0438a4b) Co-authored-by: Jake Lishman <jake.lishman@ibm.com>

* Add no-optionals and full-optionals test runs This restructures the CI slightly to perform a complete "no-optionals" run, and a complete "all optionals" run (for the optionals that are readily accessible as Python packages, without complex additional setup). Previously, we only did a partial test with some of the oldest optional components, which could have allowed for behaviour to accidentally require optional components without us noticing. This splits the `requirements-dev.txt` file into two; lines that remain in the file are what is actually _required_ to run the test suite, run the style checks, and do the documentation build. The rest (and everything that was missing) is added to a new `requirements-optional.txt` file, which can be used to pull in (almost) all of the packages that Terra can use to provide additional / accelerated functionality. Several tests needed to gain additional skips to account for this change. There is a good chance that some tests are missing skips for some libraries that are not the first point of failure, but it's hard to test explicitly for these in one go. * Fix typo in coverage workflow * Try relaxing ipython constraints * Squash newly exposed lint failures * Fix typo in tutorials pipeline * Update the 'also update' comments * Remove unneeded qiskit-toqm dependency * Section requirements-optional.txt * Test all optionals on min not max Python version Optionals are generally more likely to have been made available on the older Pythons, and some may take excessively long to provide wheels for the latest versions of Python. * Add missing test skip * Fix optional call * Use correct boolean syntax * Fix tests relying on Jupyter * Install ipykernel in tutorials * Remove HAS_PDFLATEX skip from quantum_info tests For simple LaTeX tests, IPython/Jupyter can handle the compilation internally using MathJax, and doesn't actually need a `pdflatex` installation. We only need that when we're depending on LaTeX libraries that are beyond what MathJax can handle natively. * Include additional tutorials dependencies * Install all of Terra's optionals in tutorial run * Do not install all optionals in docs build * Use class-level skips where appropriate * Do not install ibmq-provider in tutorials run * Include matplotlib in docs requirements * Remove unnecessary whitespace * Split long pip line * Only install graphviz when installOptionals Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com> * Install visualization extras in docs build * Don't `--upgrade` when installing optionals This is to prevent any optionals that have a dependency on Terra from potentially upgrading it to some PyPI version during their installation. This shouldn't happen with the current development model of having only one supported stable branch and the main branch, but the `-U` is unnecessary, and not having it is safer for the future. * Update secondary installation of Terra in docs build * Install all optionals during the docs build I yoyo-ed on this, not wanting it to be too onerous to build the documentation, but in the end, we need to have the optional features installed in order to build most of the documentation of those features, and some unrelated parts of the documentation may use these features to enhance their own output. * Fix test setup job * Remove duplication of matplotlib in requirements files * Update image test installation command * Restore editable build * Move `pip check` to `pip` section * Remove redundant "post-install" description * Expand comment on first-stage choices * pytohn lol --------- Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>

jakelishman added type: qa Issues and PRs that relate to testing and code quality Changelog: None Do not include in changelog labels Oct 20, 2022

jakelishman requested review from a team, nonhermitian, ikkoham, manoelmarques and woodsp-ibm as code owners October 20, 2022 11:51

jakelishman force-pushed the test-all-optionals branch from 0a14c66 to a66534b Compare October 20, 2022 12:07

jakelishman added 2 commits October 20, 2022 13:33

Fix typo in coverage workflow

a74b00d

Try relaxing ipython constraints

5cbaa18

jakelishman mentioned this pull request Oct 20, 2022

Disable pylint's import-error #8973

Merged

jakelishman added 5 commits October 21, 2022 21:03

Merge branch 'main' into test-all-optionals

69efa6b

Squash newly exposed lint failures

d6f9f0c

Merge remote-tracking branch 'ibm/main' into test-all-optionals

a5c009a

Merge remote-tracking branch 'ibm/main' into test-all-optionals

69b4169

Fix typo in tutorials pipeline

96bb7c6

jakelishman mentioned this pull request Oct 28, 2022

Fix exact floating-point comparison causing flaky test #9021

Merged

Merge remote-tracking branch 'ibm/main' into test-all-optionals

69a6613

# Conflicts: # .azure/test-linux.yml # .azure/tutorials-linux.yml # azure-pipelines.yml # requirements-dev.txt # test/python/dagcircuit/test_dagcircuit.py

jakelishman force-pushed the test-all-optionals branch from 1791636 to 69a6613 Compare November 21, 2022 15:14

jakelishman mentioned this pull request Nov 21, 2022

Add option in user config file to control reverse_bits in circuit_drawer #9150

Closed

mtreinish reviewed Nov 21, 2022

View reviewed changes

requirements-optional.txt Outdated Show resolved Hide resolved

requirements-optional.txt Show resolved Hide resolved

requirements-optional.txt Outdated Show resolved Hide resolved

requirements-optional.txt Outdated Show resolved Hide resolved

jakelishman added 3 commits November 22, 2022 19:15

Update the 'also update' comments

80c5945

Remove unneeded qiskit-toqm dependency

535f270

Section requirements-optional.txt

8f84190

jakelishman added 4 commits May 25, 2023 14:26

Merge remote-tracking branch 'ibm/main' into test-all-optionals

bd37cb9

Merge remote-tracking branch 'ibm/main' into test-all-optionals

1b77b10

Merge remote-tracking branch 'ibm/main' into test-all-optionals

ab254a7

Update image test installation command

981c6b5

Restore editable build

2469e97

jakelishman mentioned this pull request Jul 17, 2023

Remove cyclic-definition handling from Clifford.from_circuit #10441

Merged

Merge remote-tracking branch 'ibm/main' into test-all-optionals

2679adf

Eric-Arellano previously approved these changes Jul 19, 2023

View reviewed changes

.azure/test-linux.yml Outdated Show resolved Hide resolved

.azure/test-linux.yml Outdated Show resolved Hide resolved

azure-pipelines.yml Outdated Show resolved Hide resolved

jakelishman added 2 commits July 19, 2023 15:17

Move pip check to pip section

67682df

Remove redundant "post-install" description

f9b5cba

jakelishman dismissed Eric-Arellano’s stale review via f9b5cba July 19, 2023 14:18

Eric-Arellano previously approved these changes Jul 19, 2023

View reviewed changes

Expand comment on first-stage choices

4c0a986

jakelishman dismissed Eric-Arellano’s stale review via 4c0a986 July 19, 2023 14:23

Eric-Arellano previously approved these changes Jul 19, 2023

View reviewed changes

pytohn lol

db06df9

jakelishman dismissed Eric-Arellano’s stale review via db06df9 July 19, 2023 14:34

Eric-Arellano approved these changes Jul 19, 2023

View reviewed changes

mtreinish approved these changes Aug 4, 2023

View reviewed changes

mtreinish added this pull request to the merge queue Aug 4, 2023

Merged via the queue into Qiskit:main with commit 0438a4b Aug 4, 2023
13 checks passed

jakelishman deleted the test-all-optionals branch August 4, 2023 19:47

mergify bot mentioned this pull request Aug 4, 2023

Add no-optionals and full-optionals test runs (backport #8967) #10575

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add no-optionals and full-optionals test runs #8967

Add no-optionals and full-optionals test runs #8967

jakelishman commented Oct 20, 2022

qiskit-bot commented Oct 20, 2022

coveralls commented Oct 20, 2022 •

edited

Loading

jakelishman commented Oct 20, 2022

jakelishman commented Oct 28, 2022

jakelishman commented Oct 29, 2022

jakelishman commented Jul 11, 2023

jakelishman commented Jul 19, 2023

Eric-Arellano commented Jul 19, 2023

Eric-Arellano left a comment •

edited

Loading

mtreinish left a comment

Add no-optionals and full-optionals test runs #8967

Add no-optionals and full-optionals test runs #8967

Conversation

jakelishman commented Oct 20, 2022

Summary

Details and comments

qiskit-bot commented Oct 20, 2022

coveralls commented Oct 20, 2022 • edited Loading

Pull Request Test Coverage Report for Build 5600511995

💛 - Coveralls

jakelishman commented Oct 20, 2022

jakelishman commented Oct 28, 2022

jakelishman commented Oct 29, 2022

jakelishman commented Jul 11, 2023

jakelishman commented Jul 19, 2023

Eric-Arellano commented Jul 19, 2023

Eric-Arellano left a comment • edited Loading

Choose a reason for hiding this comment

mtreinish left a comment

Choose a reason for hiding this comment

coveralls commented Oct 20, 2022 •

edited

Loading

Eric-Arellano left a comment •

edited

Loading