Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEST: 1.15.x + blas variants #295

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

h-vetinari
Copy link
Member

Following the same scheme as #280, #273, #267, #242, #224, #223, #199, #172, #153 & #130.

@conda-forge-admin
Copy link
Contributor

conda-forge-admin commented Jan 8, 2025

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe/meta.yaml:

  • ℹ️ The recipe is not parsable by parser conda-souschef (grayskull). This parser is not currently used by conda-forge, but may be in the future. We are collecting information to see which recipes are compatible with grayskull.
  • ℹ️ The recipe is not parsable by parser conda-recipe-manager. The recipe can only be automatically migrated to the new v1 format if it is parseable by conda-recipe-manager.

This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/12674220704. Examine the logs at this URL for more detail.

@h-vetinari
Copy link
Member Author

h-vetinari commented Jan 9, 2025

Scipy 1.15.0

TL;DR: problems with blis & openblas, rest OK

From 6 failures out of 43 runs for 1.14.0, we're now at 23 failures out of 85 CI runs

Notable

Bad News

Restrictions

  • test on aarch64/ppc64le only run tests with label='fast', not the full test suite
  • the following test skips:
# on the slowest agents, these tests take more than 20min in emulation
{% set tests_to_skip = tests_to_skip + " or test_mixed_threads_processes or test_sf_gh18506" %}   # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or test_quad_vec_pool" %}                                # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_lsq_linear and test_large_rank_deficient)" %}   # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_iterative and test_precond_inverse)" %}         # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_multiprocess and fft)" %}                       # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_optimize and TestBrute and test_workers)" %}    # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (TestLstsq and test_random_complex_exact)" %}         # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (TestLevyStable and test_location_scale)" %}          # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (TestShgoArguments and test_19_parallelization)" %}   # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (TestDifferentialEvolutionSolver and (test_immediate_updating or test_parallel))" %}    # [aarch64 or ppc64le]
# tests that run into timeouts (set in scipy test suite) in emulation
{% set tests_to_skip = tests_to_skip + " or (test_propack and test_examples)" %}                  # [aarch64 or ppc64le]

Details

lib for 1.14.0 now updated
version
updated
build
scipy 1.14.0 1.15.0 X
numpy 2.0.0 2.2.1 X
libblas 3.9.0-22 3.9.0-26 X
blis 0.9.0-2 0.9.0-2
openblas 0.3.27-0 0.3.28-1 X
mkl 2024.1.0-692 (linux/win)
2023.2.0-50500 (osx)
2024.2.2-16 (linux)
2023.2.0-50500 (osx)
2024.2.2-15 (win)
X
netlib 3.9.0-5 3.9.0-8 X
pythran 0.16.1-0 0.17.0-1 X
qemu-user-static 8.2.4-1 8.2.8-2 X

variant 1.14.0 now
linux + aarch64 + openblas 60 failures, mostly QR-related;
apparently due to emulation
remain
osx + x64 + accelerate - 1 failure
{osx,win} + x64 + blis - 4 failures
win + x64 + openblas some failures & hangs turned into crashes, apparently due to memory pressure

variant accelerate blis mkl netlib openblas sum**
linux / x64 ✔️ ✔️ ✔️ ✔️ -
linux /
aarch64
✔️ 5F
linux /
ppc64le
✔️ ✔️ -
osx / arm -
osx / x64 ✔️ ✔️ ✔️ 8F
win / x64 ✔️ ✔️ 10F
sum** 3F 10F - - 10F 23F

** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 85 CI combinations being tested

Build logs:
Azure


linux + aarch64 + openblas: 60 failures
=========================== short test summary info ============================
FAILED linalg/tests/test_basic.py::TestLstsq::test_random_overdet - AssertionError: 
Not equal to tolerance rtol=2.98023e-06, atol=2.98023e-06
driver: gelsd
Mismatched elements: 45 / 45 (100%)
Max absolute difference among violations: 23.948553
Max relative difference among violations: 1.0819287
 ACTUAL: array([[-2.175597e-02,  1.430922e-01,  1.740812e-02],
       [ 2.515155e-02,  2.834495e-02, -6.470114e-03],
       [-5.741564e-05,  1.434418e-02,  4.162814e-02],...
 DESIRED: array([[  4.199284,   5.158097,   2.723418],
       [ 11.474334,  11.579959,   5.939476],
       [  0.678559,   0.93205 ,   0.438899],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_tall_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.35266799
Max relative difference among violations: 1.35266799
 ACTUAL: array([[ 0.51534 , -0.066137, -0.282195, -0.078835,  0.126846,  0.063667,
         0.254653,  0.145272,  0.297032, -0.312945,  0.047386, -0.375106],
       [-0.066137,  0.058695,  0.107405, -0.020373, -0.37564 , -0.306869,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_tall_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.3851347
Max relative difference among violations: 1.3851347
 ACTUAL: array([[ 0.194025,  0.271456,  0.193256, -0.103866,  0.639289,  0.325099,
        -0.308913,  0.225063,  0.009908,  0.531324,  0.418844,  0.486349],
       [ 0.271456, -0.024757,  0.531958, -0.127783,  0.087165,  0.124488,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_delete_last_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.39604264
Max relative difference among violations: 1.39604264
 ACTUAL: array([[ 7.312894e-01,  2.936632e-02, -2.486337e-01, -3.261323e-01,
         4.044744e-03, -1.884402e-02,  4.260744e-01,  4.652548e-01,
         4.959362e-02,  2.202505e-04, -5.590689e-01, -1.013072e-04],...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_non_unit_strides_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.35266799
Max relative difference among violations: 1.35266799
 ACTUAL: array([[ 0.51534 , -0.066137, -0.282195, -0.078835,  0.126846,  0.063667,
         0.254653,  0.145272,  0.297032, -0.312945,  0.047386, -0.375106],
       [-0.066137,  0.058695,  0.107405, -0.020373, -0.37564 , -0.306869,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_non_unit_strides_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.37619862
Max relative difference among violations: 1.37619862
 ACTUAL: array([[ 1.469218e-01,  5.787523e-01,  5.345628e-02, -4.102920e-01,
         5.512089e-01, -6.510303e-02, -3.209545e-01, -2.358619e-01,
         2.342796e-01,  2.828279e-01,  3.863395e-01,  1.333238e-01],...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_neg_strides_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.35266799
Max relative difference among violations: 1.35266799
 ACTUAL: array([[ 0.51534 , -0.066137, -0.282195, -0.078835,  0.126846,  0.063667,
         0.254653,  0.145272,  0.297032, -0.312945,  0.047386, -0.375106],
       [-0.066137,  0.058695,  0.107405, -0.020373, -0.37564 , -0.306869,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_neg_strides_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.37619862
Max relative difference among violations: 1.37619862
 ACTUAL: array([[ 1.469218e-01,  5.787523e-01,  5.345628e-02, -4.102920e-01,
         5.512089e-01, -6.510303e-02, -3.209545e-01, -2.358619e-01,
         2.342796e-01,  2.828279e-01,  3.863395e-01,  1.333238e-01],...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_non_itemize_strides_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.35266799
Max relative difference among violations: 1.35266799
 ACTUAL: array([[ 0.51534 , -0.066137, -0.282195, -0.078835,  0.126846,  0.063667,
         0.254653,  0.145272,  0.297032, -0.312945,  0.047386, -0.375106],
       [-0.066137,  0.058695,  0.107405, -0.020373, -0.37564 , -0.306869,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_non_itemize_strides_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.37619862
Max relative difference among violations: 1.37619862
 ACTUAL: array([[ 1.469218e-01,  5.787523e-01,  5.345628e-02, -4.102920e-01,
         5.512089e-01, -6.510303e-02, -3.209545e-01, -2.358619e-01,
         2.342796e-01,  2.828279e-01,  3.863395e-01,  1.333238e-01],...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_non_native_byte_order_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.35266799
Max relative difference among violations: 1.35266799
 ACTUAL: array([[ 0.51534 , -0.066137, -0.282195, -0.078835,  0.126846,  0.063667,
         0.254653,  0.145272,  0.297032, -0.312945,  0.047386, -0.375106],
       [-0.066137,  0.058695,  0.107405, -0.020373, -0.37564 , -0.306869,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_non_native_byte_order_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.37619862
Max relative difference among violations: 1.37619862
 ACTUAL: array([[ 1.469218e-01,  5.787523e-01,  5.345628e-02, -4.102920e-01,
         5.512089e-01, -6.510303e-02, -3.209545e-01, -2.358619e-01,
         2.342796e-01,  2.828279e-01,  3.863395e-01,  1.333238e-01],...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_sqr_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 81 / 81 (100%)
Max absolute difference among violations: 1.4709349
Max relative difference among violations: 1.4709349
 ACTUAL: array([[-0.019964,  0.137794,  0.149978,  0.332288, -0.219051, -0.040983,
         0.081796,  0.176545,  0.126362],
       [ 0.137794,  0.374147,  0.331801,  0.035937, -0.212025,  0.319611,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_sqr_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 121 / 121 (100%)
Max absolute difference among violations: 1.56323719
Max relative difference among violations: 1.56323719
 ACTUAL: array([[ 0.5352  ,  0.172625,  0.575634,  0.555047,  0.251488, -0.130139,
        -0.484157,  0.022222, -0.660154,  0.202653,  0.07694 ],
       [ 0.172625,  0.498765,  0.109751,  0.198007,  0.048938,  0.10279 ,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_tall_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 169 / 169 (100%)
Max absolute difference among violations: 1.37422016
Max relative difference among violations: 1.37422016
 ACTUAL: array([[ 0.306785, -0.078271, -0.09774 ,  0.629711, -0.112268,  0.031813,
         0.131607, -0.096303,  0.016277, -0.31381 ,  0.084271, -0.120308,
        -0.174024],...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_tall_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 225 / 225 (100%)
Max absolute difference among violations: 1.35956195
Max relative difference among violations: 1.35956195
 ACTUAL: array([[-3.609179e-02, -2.703785e-01, -5.440032e-02,  2.898523e-01,
         4.581065e-01, -3.596952e-02, -1.910010e-01,  2.072426e-01,
         3.869596e-02, -3.996934e-02,  8.291176e-02,  1.208157e-01,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_tall_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.31610107
Max relative difference among violations: 1.31610107
 ACTUAL: array([[ 0.395182,  0.061126,  0.091705, -0.038793,  0.292402,  0.588824,
        -0.201593,  0.357648,  0.331584, -0.058131,  0.26981 , -0.237627],
       [ 0.061126,  0.015904, -0.288082, -0.067372, -0.097557, -0.253215,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_tall_p_col_tall - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.52502376
Max relative difference among violations: 1.52502376
 ACTUAL: array([[ 0.379029, -0.304094, -0.144745,  0.232346,  0.242104,  0.303838,
        -0.098809, -0.085326,  0.059822, -0.257439,  0.053225,  0.262986],
       [-0.304094,  0.240901, -0.288143,  0.263892, -0.158313, -0.124594,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_tall_p_col_sqr - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.45840535
Max relative difference among violations: 1.45840535
 ACTUAL: array([[ 0.776305,  0.245598,  0.123602,  0.498223,  0.180076,  0.054498,
        -0.131358,  0.039556,  0.279728, -0.137389,  0.057484, -0.142773],
       [ 0.245598,  0.024506, -0.404031, -0.019139,  0.177571,  0.147957,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_tall_p_col_fat - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.31002095
Max relative difference among violations: 1.31002095
 ACTUAL: array([[-0.027368,  0.404199, -0.194338,  0.261452,  0.421794, -0.155244,
         0.020775, -0.034648,  0.262335, -0.158642,  0.412529,  0.051271],
       [ 0.404199,  0.482341, -0.092184, -0.538004,  0.253629,  0.381795,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_fat_p_row_fat - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 100 / 100 (100%)
Max absolute difference among violations: 1.27009049
Max relative difference among violations: 1.27009049
 ACTUAL: array([[-0.018354,  0.506641, -0.266757, -0.375645,  0.40916 ,  0.44026 ,
         0.289925,  0.036174,  0.228546,  0.336693],
       [ 0.506641, -0.27009 , -0.076236, -0.087398,  0.241975, -0.15332 ,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_fat_p_row_sqr - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.42828497
Max relative difference among violations: 1.42828497
 ACTUAL: array([[-0.261911,  0.45862 , -0.059198,  0.21579 ,  0.463723,  0.289269,
         0.099015, -0.111441,  0.253587,  0.020857, -0.210805,  0.171284],
       [ 0.45862 , -0.170591,  0.746429, -0.371978, -0.015335, -0.203765,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_fat_p_row_tall - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 196 / 196 (100%)
Max absolute difference among violations: 1.38900569
Max relative difference among violations: 1.38900569
 ACTUAL: array([[ 0.309889,  0.520779,  0.256557,  0.611373,  0.234448,  0.186756,
        -0.144043, -0.432629, -0.29988 ,  0.091021,  0.347191, -0.49002 ,
         0.092618,  0.417969],...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_economic_p_col_eco - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 100 / 100 (100%)
Max absolute difference among violations: 1.64193898
Max relative difference among violations: 1.64193898
 ACTUAL: array([[ 0.423202,  0.197114,  0.279268, -0.263687, -0.343731,  0.08245 ,
         0.237208,  0.200332,  0.058084,  0.011772],
       [ 0.197114,  0.382135, -0.288262, -0.060497, -0.156316,  0.304202,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_economic_p_col_sqr - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.38882607
Max relative difference among violations: 1.38882607
 ACTUAL: array([[ 0.776305,  0.430198,  0.123602,  0.498223,  0.180076, -0.230487,
        -0.131358,  0.039556,  0.279727,  0.156331,  0.057484, -0.142773],
       [ 0.430198,  0.608486, -0.404032, -0.019139,  0.177571, -0.126041,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_economic_p_col_fat - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.23673192
Max relative difference among violations: 1.23673192
 ACTUAL: array([[ 0.200447,  0.404199,  0.069552,  0.07444 ,  0.725081, -0.155244,
        -0.242666,  0.2079  , -0.142786, -0.158642,  0.225283, -0.573266],
       [ 0.404199,  0.482341, -0.087215, -0.278327, -0.002819,  0.381795,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_Mx1_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 81 / 81 (100%)
Max absolute difference among violations: 1.35408488
Max relative difference among violations: 1.35408488
 ACTUAL: array([[ 0.202564,  0.080353,  0.546481, -0.499241,  0.44599 , -0.365503,
         0.281745,  0.047513, -0.147732],
       [ 0.080353,  0.104208, -0.049411,  0.717046,  0.188095, -0.23352 ,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_Mx1_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 121 / 121 (100%)
Max absolute difference among violations: 1.18590231
Max relative difference among violations: 1.18590231
 ACTUAL: array([[ 0.513253,  0.469261, -0.348298, -0.281501,  0.105035,  0.292235,
        -0.237656, -0.20907 , -0.253257, -0.175457,  0.859724],
       [ 0.469261, -0.044185, -0.367757, -0.218692, -0.324741, -0.363129,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_unit_strides_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 81 / 81 (100%)
Max absolute difference among violations: 1.4709349
Max relative difference among violations: 1.4709349
 ACTUAL: array([[-0.019964,  0.137794,  0.149978,  0.332288, -0.219051, -0.040983,
         0.081796,  0.176545,  0.126362],
       [ 0.137794,  0.374147,  0.331801,  0.035937, -0.212025,  0.319611,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_unit_strides_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 121 / 121 (100%)
Max absolute difference among violations: 1.56323719
Max relative difference among violations: 1.56323719
 ACTUAL: array([[ 0.5352  ,  0.172625,  0.575634,  0.555047,  0.251488, -0.130139,
        -0.484157,  0.022222, -0.660154,  0.202653,  0.07694 ],
       [ 0.172625,  0.498765,  0.109751,  0.198007,  0.048938,  0.10279 ,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_unit_strides_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.31610107
Max relative difference among violations: 1.31610107
 ACTUAL: array([[ 0.395182,  0.061126,  0.091705, -0.038793,  0.292402,  0.588824,
        -0.201593,  0.357648,  0.331584, -0.058131,  0.26981 , -0.237627],
       [ 0.061126,  0.015904, -0.288082, -0.067372, -0.097557, -0.253215,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_unit_strides_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.52502376
Max relative difference among violations: 1.52502376
 ACTUAL: array([[ 0.379029, -0.304094, -0.144745,  0.232346,  0.242104,  0.303838,
        -0.098809, -0.085326,  0.059822, -0.257439,  0.053225,  0.262986],
       [-0.304094,  0.240901, -0.288143,  0.263892, -0.158313, -0.124594,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_neg_strides_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 81 / 81 (100%)
Max absolute difference among violations: 1.4709349
Max relative difference among violations: 1.4709349
 ACTUAL: array([[-0.019964,  0.137794,  0.149978,  0.332288, -0.219051, -0.040983,
         0.081796,  0.176545,  0.126362],
       [ 0.137794,  0.374147,  0.331801,  0.035937, -0.212025,  0.319611,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_neg_strides_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 121 / 121 (100%)
Max absolute difference among violations: 1.56323719
Max relative difference among violations: 1.56323719
 ACTUAL: array([[ 0.5352  ,  0.172625,  0.575634,  0.555047,  0.251488, -0.130139,
        -0.484157,  0.022222, -0.660154,  0.202653,  0.07694 ],
       [ 0.172625,  0.498765,  0.109751,  0.198007,  0.048938,  0.10279 ,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_neg_strides_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.31610107
Max relative difference among violations: 1.31610107
 ACTUAL: array([[ 0.395182,  0.061126,  0.091705, -0.038793,  0.292402,  0.588824,
        -0.201593,  0.357648,  0.331584, -0.058131,  0.26981 , -0.237627],
       [ 0.061126,  0.015904, -0.288082, -0.067372, -0.097557, -0.253215,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_neg_strides_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.52502376
Max relative difference among violations: 1.52502376
 ACTUAL: array([[ 0.379029, -0.304094, -0.144745,  0.232346,  0.242104,  0.303838,
        -0.098809, -0.085326,  0.059822, -0.257439,  0.053225,  0.262986],
       [-0.304094,  0.240901, -0.288143,  0.263892, -0.158313, -0.124594,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_itemsize_strides_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 81 / 81 (100%)
Max absolute difference among violations: 1.4709349
Max relative difference among violations: 1.4709349
 ACTUAL: array([[-0.019964,  0.137794,  0.149978,  0.332288, -0.219051, -0.040983,
         0.081796,  0.176545,  0.126362],
       [ 0.137794,  0.374147,  0.331801,  0.035937, -0.212025,  0.319611,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_itemsize_strides_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 121 / 121 (100%)
Max absolute difference among violations: 1.56323719
Max relative difference among violations: 1.56323719
 ACTUAL: array([[ 0.5352  ,  0.172625,  0.575634,  0.555047,  0.251488, -0.130139,
        -0.484157,  0.022222, -0.660154,  0.202653,  0.07694 ],
       [ 0.172625,  0.498765,  0.109751,  0.198007,  0.048938,  0.10279 ,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_itemsize_strides_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.3161011
Max relative difference among violations: 1.3161011
 ACTUAL: array([[ 0.395182,  0.061126,  0.091705, -0.038793,  0.292402,  0.588824,
        -0.201593,  0.357648,  0.331584, -0.058131,  0.26981 , -0.237627],
       [ 0.061126,  0.015904, -0.288082, -0.067372, -0.097557, -0.253215,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_itemsize_strides_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.52502376
Max relative difference among violations: 1.52502376
 ACTUAL: array([[ 0.379029, -0.304094, -0.144745,  0.232346,  0.242104,  0.303838,
        -0.098809, -0.085326,  0.059822, -0.257439,  0.053225,  0.262986],
       [-0.304094,  0.240901, -0.288143,  0.263892, -0.158313, -0.124594,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_native_byte_order_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 81 / 81 (100%)
Max absolute difference among violations: 1.4709349
Max relative difference among violations: 1.4709349
 ACTUAL: array([[-0.019964,  0.137794,  0.149978,  0.332288, -0.219051, -0.040983,
         0.081796,  0.176545,  0.126362],
       [ 0.137794,  0.374147,  0.331801,  0.035937, -0.212025,  0.319611,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_native_byte_order_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 121 / 121 (100%)
Max absolute difference among violations: 1.56323719
Max relative difference among violations: 1.56323719
 ACTUAL: array([[ 0.5352  ,  0.172625,  0.575634,  0.555047,  0.251488, -0.130139,
        -0.484157,  0.022222, -0.660154,  0.202653,  0.07694 ],
       [ 0.172625,  0.498765,  0.109751,  0.198007,  0.048938,  0.10279 ,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_native_byte_order_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.31610107
Max relative difference among violations: 1.31610107
 ACTUAL: array([[ 0.395182,  0.061126,  0.091705, -0.038793,  0.292402,  0.588824,
        -0.201593,  0.357648,  0.331584, -0.058131,  0.26981 , -0.237627],
       [ 0.061126,  0.015904, -0.288082, -0.067372, -0.097557, -0.253215,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_native_byte_order_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.52502376
Max relative difference among violations: 1.52502376
 ACTUAL: array([[ 0.379029, -0.304094, -0.144745,  0.232346,  0.242104,  0.303838,
        -0.098809, -0.085326,  0.059822, -0.257439,  0.053225,  0.262986],
       [-0.304094,  0.240901, -0.288143,  0.263892, -0.158313, -0.124594,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_tall_rank_1 - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.5631814
Max relative difference among violations: 1.5631814
 ACTUAL: array([[-0.054046,  0.205639, -0.123517,  0.260355,  0.474265, -0.249005,
         0.177618, -0.301207,  0.618385, -0.037394,  0.023149,  0.252298],
       [ 0.205639,  0.756895, -0.286572,  0.154587,  0.321836,  0.283857,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_tall_rank_p - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.5631814
Max relative difference among violations: 1.5631814
 ACTUAL: array([[-0.054046,  0.205639, -0.123517,  0.260355,  0.474265, -0.249005,
         0.177618, -0.301207,  0.618385, -0.037394,  0.023149,  0.252298],
       [ 0.205639,  0.756895, -0.286572,  0.154587,  0.321836,  0.283857,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_non_unit_strides_rank_1 - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.56318134
Max relative difference among violations: 1.56318134
 ACTUAL: array([[-0.054046,  0.205639, -0.123517,  0.260355,  0.474265, -0.249005,
         0.177618, -0.301207,  0.618385, -0.037394,  0.023149,  0.252298],
       [ 0.205639,  0.756895, -0.286572,  0.154587,  0.321836,  0.283857,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_non_unit_strides_rank_p - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.51972252
Max relative difference among violations: 1.51972252
 ACTUAL: array([[ 0.030667,  0.051494, -0.262698, -0.204111,  0.552454,  0.260673,
        -0.072537,  0.140756,  0.215893, -0.045971,  0.289605, -0.198233],
       [ 0.051494,  0.300885,  0.159218,  0.002972, -0.091647,  0.648181,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_neg_strides_rank_1 - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.5631814
Max relative difference among violations: 1.5631814
 ACTUAL: array([[-0.054046,  0.205639, -0.123517,  0.260355,  0.474265, -0.249005,
         0.177618, -0.301207,  0.618385, -0.037394,  0.023149,  0.252298],
       [ 0.205639,  0.756895, -0.286572,  0.154587,  0.321836,  0.283857,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_neg_strides_rank_p - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.51972252
Max relative difference among violations: 1.51972252
 ACTUAL: array([[ 0.030667,  0.051494, -0.262698, -0.204111,  0.552454,  0.260673,
        -0.072537,  0.140756,  0.215893, -0.045971,  0.289605, -0.198233],
       [ 0.051494,  0.300885,  0.159218,  0.002972, -0.091647,  0.648181,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_non_itemsize_strides_rank_1 - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.56318134
Max relative difference among violations: 1.56318134
 ACTUAL: array([[-0.054046,  0.205639, -0.123517,  0.260355,  0.474265, -0.249005,
         0.177618, -0.301207,  0.618385, -0.037394,  0.023149,  0.252298],
       [ 0.205639,  0.756895, -0.286572,  0.154587,  0.321836,  0.283857,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_non_itemsize_strides_rank_p - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.51972252
Max relative difference among violations: 1.51972252
 ACTUAL: array([[ 0.030667,  0.051494, -0.262698, -0.204111,  0.552454,  0.260673,
        -0.072537,  0.140756,  0.215893, -0.045971,  0.289605, -0.198233],
       [ 0.051494,  0.300885,  0.159218,  0.002972, -0.091647,  0.648181,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_non_native_byte_order_rank_1 - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.5631814
Max relative difference among violations: 1.5631814
 ACTUAL: array([[-0.054046,  0.205639, -0.123517,  0.260355,  0.474265, -0.249005,
         0.177618, -0.301207,  0.618385, -0.037394,  0.023149,  0.252298],
       [ 0.205639,  0.756895, -0.286572,  0.154587,  0.321836,  0.283857,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_non_native_byte_order_rank_p - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference among violations: 1.51972252
Max relative difference among violations: 1.51972252
 ACTUAL: array([[ 0.030667,  0.051494, -0.262698, -0.204111,  0.552454,  0.260673,
        -0.072537,  0.140756,  0.215893, -0.045971,  0.289605, -0.198233],
       [ 0.051494,  0.300885,  0.159218,  0.002972, -0.091647,  0.648181,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_lapack.py::test_pftri - AssertionError: 
Arrays are not almost equal to 4 decimals

Mismatched elements: 21 / 400 (5.25%)
Max absolute difference among violations: 0.00037931
Max relative difference among violations: 0.64988735
 ACTUAL: array([[ 5.0451e-02, -1.1576e-03,  2.7290e-04, -1.5024e-03, -1.7656e-03,
        -1.3410e-03, -1.4870e-03, -7.1398e-04, -1.4734e-03, -2.5769e-03,
        -2.0850e-03, -9.8616e-04, -7.3829e-04, -1.9978e-03,  1.7069e-04,...
 DESIRED: array([[ 5.0565e-02, -1.0042e-03,  2.1366e-04, -1.5335e-03, -1.5981e-03,
        -1.4567e-03, -1.5799e-03, -5.9890e-04, -1.4266e-03, -2.5949e-03,
        -2.0850e-03, -9.8616e-04, -7.3829e-04, -1.9978e-03,  1.7069e-04,...
FAILED linalg/tests/test_lapack.py::test_sfrk_hfrk - AssertionError: 
Arrays are not almost equal to 4 decimals

Mismatched elements: 165 / 400 (41.2%)
Max absolute difference among violations: 1.14951146
Max relative difference among violations: 14.46566384
 ACTUAL: array([[ 3.9575e+01,  1.3012e+00,  5.5721e-02,  1.6919e+00,  1.3906e+00,
         1.7789e+00,  8.5231e-01,  8.1178e-01,  1.2308e+00,  2.1547e+00,
         1.7334e+00,  6.0473e-01,  2.1888e-01,  2.0501e+00, -1.1836e-01,...
 DESIRED: array([[ 3.9575e+01,  1.3012e+00,  2.2163e-01,  1.2076e+00,  1.4051e+00,
         1.8348e+00,  1.0907e+00,  1.0859e+00,  1.3009e+00,  2.1895e+00,
         1.3301e+00,  1.5722e+00,  1.7302e-01,  1.5784e+00,  1.6231e-01,...
FAILED linalg/tests/test_lapack.py::TestBlockedQR::test_geqrt_gemqrt - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-05

Mismatched elements: 320 / 400 (80%)
Max absolute difference among violations: 1.4186668
Max relative difference among violations: 1.4186668
 ACTUAL: array([[ 4.153938e-01,  3.214428e-01, -1.697152e-01, -2.782871e-01,
         2.549523e-01,  3.815205e-01, -1.836173e-01, -8.294297e-03,
         2.499609e-01,  1.465444e-02,  1.211795e-01, -2.663904e-01,...
 DESIRED: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,...
FAILED linalg/tests/test_lapack.py::TestBlockedQR::test_tpqrt_tpmqrt - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-05

Mismatched elements: 1024 / 1600 (64%)
Max absolute difference among violations: 1.2731317
Max relative difference among violations: 1.2731317
 ACTUAL: array([[ 3.782889e-03,  1.856932e-02,  1.523624e-02, ..., -1.549972e-01,
         2.169387e-01, -3.143150e-01],
       [ 1.856932e-02,  4.894964e-02, -5.576465e-02, ...,  2.150273e-02,...
 DESIRED: array([[1., 0., 0., ..., 0., 0., 0.],
       [0., 1., 0., ..., 0., 0., 0.],
       [0., 0., 1., ..., 0., 0., 0.],...
FAILED linalg/tests/test_lapack.py::test_pstrf - AssertionError: 
Not equal to tolerance rtol=0, atol=0.000119209

Mismatched elements: 85 / 100 (85%)
Max absolute difference among violations: 3.529682
Max relative difference among violations: 1.4060138
 ACTUAL: array([[3.487075, 2.2597  , 1.555289, 2.763015, 3.431685, 2.046993,
        2.797032, 2.700427, 3.422926, 2.103169],
       [2.2597  , 2.130606, 1.477844, 1.633565, 2.117459, 1.866201,...
 DESIRED: array([[3.487075, 2.2597  , 1.555289, 2.763015, 3.431685, 2.046993,
        2.797032, 2.700427, 0.      , 0.      ],
       [2.2597  , 2.38366 , 1.007861, 2.456765, 2.693789, 1.169568,...
FAILED linalg/tests/test_lapack.py::test_pstf2 - AssertionError: 
Not equal to tolerance rtol=0, atol=0.000119209

Mismatched elements: 85 / 100 (85%)
Max absolute difference among violations: 3.529682
Max relative difference among violations: 1.4060138
 ACTUAL: array([[3.487075, 2.2597  , 1.555289, 2.763015, 3.431685, 2.046993,
        2.797032, 2.700427, 3.422926, 2.103169],
       [2.2597  , 2.130606, 1.477844, 1.633565, 2.117459, 1.866201,...
 DESIRED: array([[3.487075, 2.2597  , 1.555289, 2.763015, 3.431685, 2.046993,
        2.797032, 2.700427, 0.      , 0.      ],
       [2.2597  , 2.38366 , 1.007861, 2.456765, 2.693789, 1.169568,...
= 60 failed, 58585 passed, 3731 skipped, 190 xfailed, 15 xpassed, 2587 warnings in 4896.49s (1:21:36) =
osx + x64 + accelerate: 1 failure

this occurred 3 times out of 5 runs

=========================== short test summary info ============================
FAILED differentiate/tests/test_differentiate.py::TestDerivative::test_maxiter_callback[numpy] - assert np.float64(8.526512829121202e-14) == np.float64(8.171241461241152e-14)
= 1 failed, 73851 passed, 4672 skipped, 329 xfailed, 13 xpassed, 2587 warnings in 630.95s (0:10:30) =
win + blis: 4 failures
=========================== short test summary info ===========================
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-float64-csr_array] - AssertionError: 
Not equal to tolerance rtol=1e-08, atol=1e-08

Mismatched elements: 1 / 3 (33.3%)
Max absolute difference among violations: 0.00198541
Max relative difference among violations: 0.00015558
 ACTUAL: array([17.875834, 13.45001 , 12.759742])
 DESIRED: array([17.875834, 13.45001 , 12.761728])
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-float64-csc_array] - AssertionError: 
Not equal to tolerance rtol=1e-08, atol=1e-08

Mismatched elements: 1 / 3 (33.3%)
Max absolute difference among violations: 0.00198541
Max relative difference among violations: 0.00015558
 ACTUAL: array([17.875834, 13.45001 , 12.759742])
 DESIRED: array([17.875834, 13.45001 , 12.761728])
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-float64-csr_array] - numpy.linalg.LinAlgError: k=3 singular triplets did not converge within kmax=11 iterations
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-float64-csc_array] - numpy.linalg.LinAlgError: k=3 singular triplets did not converge within kmax=11 iterations
= 4 failed, 73843 passed, 4677 skipped, 325 xfailed, 17 xpassed, 2593 warnings in 1217.42s (0:20:17) =
win + openblas: crashes & hangs

Each CI run seems to behave differently, but it mostly seems to be due to memory pressure; e.g. many logs have a crash like the following

.......................................................sss.sssssssss.sss [  2%]
..............................................s.s....................... [  2%]
##[warning]Free disk space on D:\ is lower than 5%; Currently used: 98.21%
##[warning]Free disk space on D:\ is lower than 5%; Currently used: 98.21%
.............F..............s....ss...sss..ssss.s....s..[gw0] node down: Not properly terminated
F
replacing crashed worker gw0
..ss...sss..ssss.ss...ss...ss...sss..ssss.sss..sss..sss..sss..ssss.ssss. [  2%]
ssss.ssss.ssss.ssss......s....ss...sss..ssss.s....s....ss...sss..ssss.ss [  2%]

with traces as follows

================================== FAILURES ===================================
_______________________ test_public_modules_importable ________________________
[gw0] win32 -- Python 3.13.1 %PREFIX%\python.exe
..\_test_env\Lib\site-packages\scipy\_lib\tests\test_import_cycles.py:18: in test_public_modules_importable
    assert pid.wait() == 0, f'Failed to import {PUBLIC_MODULES[i]}'
E   AssertionError: Failed to import scipy.cluster
E   assert 1 == 0
E    +  where 1 = wait()
E    +    where wait = <Popen: returncode: 1 args: ['C:\\bld\\scipy-split_1736356435827\\_test_env\...>.wait
        i          = 0
        pid        = <Popen: returncode: 1 args: ['C:\\bld\\scipy-split_1736356435827\\_test_env\...>
        pids       = [<Popen: returncode: 1 args: ['C:\\bld\\scipy-split_1736356435827\\_test_env\...>, <Popen: returncode: None args: ['C:...plit_1736356435827\\_test_e...>, <Popen: returncode: None args: ['C:\\bld\\scipy-split_1736356435827\\_test_e...>, ...]
Captured stderr call -----------------------------
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import scipy.cluster
  File "%PREFIX%\Lib\site-packages\scipy\__init__.py", line 47, in <module>
    from numpy import __version__ as __numpy_version__
  File "%PREFIX%\Lib\site-packages\numpy\__init__.py", line 298, in <module>
    _core.getlimits._register_known_types()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "%PREFIX%\Lib\site-packages\numpy\_core\getlimits.py", line 287, in _register_known_types
    smallest_subnormal_dd = ld(nextafter(0., 1.))
MemoryError
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import scipy.io.wavfile
  File "%PREFIX%\Lib\site-packages\scipy\__init__.py", line 47, in <module>
    from numpy import __version__ as __numpy_version__
  File "%PREFIX%\Lib\site-packages\numpy\__init__.py", line 298, in <module>
    _core.getlimits._register_known_types()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "%PREFIX%\Lib\site-packages\numpy\_core\getlimits.py", line 287, in _register_known_types
    smallest_subnormal_dd = ld(nextafter(0., 1.))
MemoryError
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import scipy.fftpack
  File "%PREFIX%\Lib\site-packages\scipy\__init__.py", line 47, in <module>
    from numpy import __version__ as __numpy_version__
  File "%PREFIX%\Lib\site-packages\numpy\__init__.py", line 298, in <module>
    _core.getlimits._register_known_types()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "%PREFIX%\Lib\site-packages\numpy\_core\getlimits.py", line 287, in _register_known_types
    smallest_subnormal_dd = ld(nextafter(0., 1.))
MemoryError
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import scipy.datasets
  File "%PREFIX%\Lib\site-packages\scipy\__init__.py", line 47, in <module>
    from numpy import __version__ as __numpy_version__
  File "%PREFIX%\Lib\site-packages\numpy\__init__.py", line 181, in <module>
    from . import lib
  File "%PREFIX%\Lib\site-packages\numpy\lib\__init__.py", line 36, in <module>
    from . import _arraypad_impl
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1022, in exec_module
  File "<frozen importlib._bootstrap_external>", line 1118, in get_code
  File "<frozen importlib._bootstrap_external>", line 1218, in get_data
MemoryError
________________________ _lib/tests/test_public_api.py ________________________
[gw0] win32 -- Python 3.13.1 %PREFIX%\python.exe
worker 'gw0' crashed while running '_lib/tests/test_public_api.py::test_dir_testing'

and results like

=========================== short test summary info ===========================
FAILED _lib/tests/test_import_cycles.py::test_public_modules_importable - AssertionError: Failed to import scipy.cluster
assert 1 == 0
 +  where 1 = wait()
 +    where wait = <Popen: returncode: 1 args: ['C:\\bld\\scipy-split_1736356435827\\_test_env\...>.wait
FAILED _lib/tests/test_public_api.py::test_dir_testing
= 2 failed, 73843 passed, 4679 skipped, 325 xfailed, 17 xpassed, 2596 warnings in 1118.77s (0:18:38) =

In one case the test suite crashed completely

[  2%]
.......................................x.................sssssssssssssss [  2%]
##[warning]Free disk space on D:\ is lower than 5%; Currently used: 98.25%
##[warning]Free disk space on D:\ is lower than 5%; Currently used: 98.25%
...........................................[gw1] node down: Not properly terminated
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\_pytest\main.py", line 283, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\_pytest\main.py", line 337, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\pluggy\_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\pluggy\_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\pluggy\_callers.py", line 139, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\pluggy\_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\_pytest\logging.py", line 803, in pytest_runtestloop
INTERNALERROR>     return (yield)  # Run all the tests.
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\pluggy\_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\_pytest\terminal.py", line 673, in pytest_runtestloop
INTERNALERROR>     result = yield
INTERNALERROR>              ^^^^^
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\pluggy\_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\xdist\dsession.py", line 138, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\xdist\dsession.py", line 163, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\xdist\dsession.py", line 243, in worker_errordown
INTERNALERROR>     crashitem = self.sched.remove_node(node)
INTERNALERROR>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\xdist\scheduler\load.py", line 235, in remove_node
INTERNALERROR>     self.check_schedule(node)
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\xdist\scheduler\load.py", line 207, in check_schedule
INTERNALERROR>     self._send_tests(node, min(num_send, maxschedchunk))
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\xdist\scheduler\load.py", line 307, in _send_tests
INTERNALERROR>     node.send_runtest_some(tests_per_node)
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\xdist\workermanage.py", line 355, in send_runtest_some
INTERNALERROR>     self.sendcommand("runtests", indices=indices)
INTERNALERROR>   File "%PREFIX%\Lib\site-packages\xdist\workermanage.py", line 373, in sendcommand
INTERNALERROR>     self.log(f"sending command {name}(**{kwargs})")
INTERNALERROR>                                         ^^^^^^^^
INTERNALERROR> MemoryError

=========== 1677 passed, 20 skipped, 2 xfailed in 63.19s (0:01:03) ============

Finally, in one case, a more concrete culprit for the failure in test_public_modules_importable appeared in the logs:

  File "%PREFIX%\Lib\site-packages\scipy\linalg\_misc.py", line 3, in <module>
    from .blas import get_blas_funcs
  File "%PREFIX%\Lib\site-packages\scipy\linalg\blas.py", line 213, in <module>
    from scipy.linalg import _fblas
ImportError: DLL load failed while importing _fblas: The paging file is too small for this operation to complete.

That's despite enabling a paging file (between 8-16GB depending on availability) in f2826f3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants