Slow-test failures in test_bound_pass_manager

[jakelishman]

Environment

• Qiskit Terra version: main at 94e9480
• Python version: 3.10
• Operating system: Linux

What is happening?

The nightly runs of the full test suite including the slowest components are failing following the changes in #9629. The same tests should be running as in the regular CI runs, and this same suite runs in nightly (non-slow) CI on Python 3.10 as well, and all of those runs are passing.

Sample test failure log:

==============================
Failed 2 tests - output below:
==============================

test.python.primitives.test_backend_estimator.TestBackendEstimator.test_bound_pass_manager
------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/runner/work/qiskit-terra/qiskit-terra/test/python/primitives/test_backend_estimator.py", line 362, in test_bound_pass_manager
    self.assertEqual(cm.output, expected)

      File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/unittest/case.py", line 845, in assertEqual
    assertion_func(first, second, msg=msg)

      File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/unittest/case.py", line 1051, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)

      File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/unittest/case.py", line 1033, in assertSequenceEqual
    self.fail(msg)

      File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/unittest/case.py", line 675, in fail
    raise self.failureException(msg)

    AssertionError: Lists differ: [] != ['INFO:LocalLogger:bound_pass_manager', 'INFO:LocalLogger:bound_pass_manager']

Second list contains 2 additional elements.
First extra element 0:
'INFO:LocalLogger:bound_pass_manager'

- []
+ ['INFO:LocalLogger:bound_pass_manager', 'INFO:LocalLogger:bound_pass_manager']


test.python.primitives.test_backend_sampler.TestBackendSampler.test_bound_pass_manager
--------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/runner/work/qiskit-terra/qiskit-terra/test/python/primitives/test_backend_sampler.py", line 395, in test_bound_pass_manager
    self.assertEqual(cm.output, expected)

      File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/unittest/case.py", line 845, in assertEqual
    assertion_func(first, second, msg=msg)

      File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/unittest/case.py", line 1051, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)

      File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/unittest/case.py", line 1033, in assertSequenceEqual
    self.fail(msg)

      File "/opt/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/unittest/case.py", line 675, in fail
    raise self.failureException(msg)

    AssertionError: Lists differ: [] != ['INFO:LocalLogger:bound_pass_manager', 'INFO:LocalLogger:bound_pass_manager']

Second list contains 2 additional elements.
First extra element 0:
'INFO:LocalLogger:bound_pass_manager'

- []
+ ['INFO:LocalLogger:bound_pass_manager', 'INFO:LocalLogger:bound_pass_manager']

How can we reproduce the issue?

See the comments on the slow-test issue from this comment: #7864 (comment). Currently, that comment and the two following comments are about this problem.

What should happen?

Tests pass.

Any suggestions?

I'm suspicious that the use and testing of a logger with a fixed name (LocalLogger) from multiple different files might be allowing race conditions, exacerbated by longer-running tests with the slow-tests active. It may be more appropriate to use a unique logger for each individual test to alleviate this, or perhaps to change the test more significantly to explicitly only touch local state (e.g. by instead using a TransformationPass that just has a unittest.Mock instance as its run method, which can be directly queried).

I'm just guessing at that being the cause, though - I don't know.

@ElePT and @Cryoris, please could you look at this? It's going to keep pinging the error log for the nightly runs until fixed.