Trac #33473: numpy incompatibility when using system python and numpy…

… 1.22 is installed in the system

This retroactively affects sage-9.5, meaning sage-9.5 used to build ok
in my system but now that system numpy has been upgraded to 1.22 it
breaks. This is using the standard tarball, unpatched, as long as system
python is used and numpy 1.22 happens to be installed in the system when
sage is build.

NOTE: this is NOT trying to use system numpy. Sage builds and installs
its own numpy (1.21) but headers for the system numpy (1.22) get
incorrectly mixed up when building sagelib.

The symptom is that any sage module that uses `cimport numpy` will fail
to load, making sage almost unusable, e.g.:
{{{
sage: import sage.rings.polynomial.real_roots
------------------------------------------------------------------------
---
ValueError                                Traceback (most recent call
last)
...
ValueError: numpy.ndarray size changed, may indicate binary
incompatibility. Expected 96 from C header, got 88 from PyObject
}}}

Background:
 - numpy 1.21 and numpy 1.22 are ABI incompatible (ndarray changed size)
 - building sagelib needs to know location of include files:
   - for python: since I'm using system python, this is a system include
dir, here `/usr/include/python3.10`
   - for numpy: since numpy is bundled with sage, this is a dir in the
venv, here `/home/tornaria/src/sage/sage-9.5/local/var/lib/sage/venv-
python3.10/lib/python3.10/site-packages/numpy/core/include`
 - my system numpy (1.22) has its own headers installed in
`/usr/lib/python3.10/site-packages/numpy/core/include` but there's also
a symlink `/usr/include/python3.10/numpy -> ../../lib/python3.10/site-
packages/numpy/core/include/numpy`.
 - in the CC command line the python include location (in the system) is
before the numpy include location (in the venv)
 - since the system python contains the system numpy, all numpy headers
will be included from system and not from the venv, triggering the
incompatibility.

NOTE: the symlink `/usr/include/python3.10/numpy` is not shipped by
numpy itself, but it seems some distros add it. I'm using void linux,
which has it, but at least debian also has it (see
https://packages.debian.org/bullseye/amd64/python3-numpy/filelist)

The solution should be changing the order of the include locations so
that the numpy location comes before the python location.

URL: https://trac.sagemath.org/33473
Reported by: tornaria
Ticket author(s): Gonzalo Tornaría
Reviewer(s): Matthias Koeppe

build/pkgs/scipy/patches/numpy-include.patch

@@ -0,0 +1,11 @@
+--- a/scipy/spatial/setup.py	2021-11-23 14:19:51.100109800 -0300
++++ b/scipy/spatial/setup.py	2022-03-06 23:01:17.458072945 -0300
+@@ -40,7 +40,7 @@
+     inc_dirs = [get_python_inc()]
+     if inc_dirs[0] != get_python_inc(plat_specific=1):
+         inc_dirs.append(get_python_inc(plat_specific=1))
+-    inc_dirs.append(get_numpy_include_dirs())
++    inc_dirs.insert(0, get_numpy_include_dirs())
+     inc_dirs.append(join(dirname(dirname(__file__)), '_lib'))
+     inc_dirs.append(join(dirname(dirname(__file__)), '_build_utils', 'src'))
+

src/sage/env.py

@@ -357,8 +357,9 @@ def sage_include_directories(use_sources=False):
 
         sage: import sage.env
         sage: sage.env.sage_include_directories()
-        ['.../include/python...',
-        '.../python.../numpy/core/include']
+        ['.../site-packages',
+         '.../site-packages/numpy/core/include',
+         '.../include/python...']
 
     To check that C/C++ files are correctly found, we verify that we can
     always find the include file ``sage/cpython/cython_metaclass.h``,
@@ -380,7 +381,7 @@ def sage_include_directories(use_sources=False):
             distutils.sysconfig.get_python_inc()]
     try:
         import numpy
-        dirs.append(numpy.get_include())
+        dirs.insert(1, numpy.get_include())
     except ModuleNotFoundError:
         pass
     return dirs