Variances for non-commuting observables are wrong

[austingmhuang]

Describe the bug
Given some observable that has non-commuting elements e.g. "IX" + "IZ", the variances that is used to compute the standard error is incorrect

Steps to reproduce
Define some circuit e.g.

qc = QuantumCircuit(2)
qc.rx(np.pi/3, 0)
qc.rz(np.pi/3, 0)

Then define your observable (SparsePauliOp)
observable = SparsePauliOp(["IX", "IZ"])

Check answer to find that it is not the expected result. Since we cannot get the variance directly from the device, we compute the variance from the standard error. In the source code, the standard error is computed from the variance that is measured, so this is just reversing that step.

res = estimator.run([(qc, observable)])
var = (res.result()[0].data.stds/res.result()[0].metadata["target_precision"]) ** 2 ## 1.2167353630065918

Expected behavior
var should be around 0.43

Suggested solutions
Account for covariance when calculating variances of non-commuting observables.

Additional Information
You can verify that the variance is wrong by computing the variance manually.
Using Var(A+B) = E((A+B)^2) - (E(A+B))^2
var = estimator.run([(qc, observable.dot(observable))]).result()[0].data.evs - estimator.run([(qc, observable)]).result()[0].data.evs ** 2

• qiskit-ibm-runtime version: 0.23
• Python version: 3.11.9
• Operating system: macOS

[Cryoris]

Can this be closed since Qiskit/qiskit#12670 is merged or do you prefer to wait until the fix is release in the upcoming 1.2 release (it's already available in the release candidate)?