Python 3.10: RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf

[grahamgower]

stdpopsim CI looks like its having problems because of the numpy version kastore is built against. Presumably triggered now that numpy have released 1.23.0?
https://github.com/popsim-consortium/stdpopsim/runs/7081344760?check_suite_focus=true

I guess kastore needs a build dependency on oldest-supported-numpy?

[jeromekelleher]

I think we hit something like this the other day with tsinfer but couldn't reproduce. Any idea what's going on @benjeffery?

[benjeffery]

Yes, I couldn't reproduce that either. I'll try to repro this one (should be easier as it is CI) and dig in.

[grahamgower]

Here's how to reproduce. Make sure you install the wheel package before installing kastore. By default, wheel builds wheels in an isolated environment. If wheel is not installed you'll see Using legacy 'setup.py install' for kastore, since package 'wheel' is not installed., which builds the wheel in the current environment. Because kastore doesn't have a build requirement on any specific numpy version, the latest version (1.23.0) is installed in the isolated build env and kastore is built against that. If the running environment has numpy 1.22.x, then kastore cannot be imported.

t490:tmp $ python --version
Python 3.10.5
t490:tmp $ python -m venv venv
t490:tmp $ source venv/bin/activate
(venv) t490:tmp $ pip install -U pip wheel
Requirement already satisfied: pip in ./venv/lib/python3.10/site-packages (22.0.4)
Collecting pip
  Using cached pip-22.1.2-py3-none-any.whl (2.1 MB)
Collecting wheel
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, pip
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Uninstalling pip-22.0.4:
      Successfully uninstalled pip-22.0.4
Successfully installed pip-22.1.2 wheel-0.37.1
(venv) t490:tmp $ pip install --no-cache kastore
Collecting kastore
  Downloading kastore-0.3.1.tar.gz (28 kB)
  Preparing metadata (setup.py) ... done
Collecting numpy>=1.7
  Downloading numpy-1.23.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.0/17.0 MB 5.9 MB/s eta 0:00:00
Collecting humanize
  Downloading humanize-4.2.2-py3-none-any.whl (102 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 102.5/102.5 kB 8.6 MB/s eta 0:00:00
Building wheels for collected packages: kastore
  Building wheel for kastore (setup.py) ... done
  Created wheel for kastore: filename=kastore-0.3.1-cp310-cp310-linux_x86_64.whl size=65908 sha256=b601778b6d2733e430952b04df4e252534faeeaedfde2daf6e6800e03ee8038b
  Stored in directory: /tmp/pip-ephem-wheel-cache-57x0pjr0/wheels/66/ad/12/738e945bf90a9536540e155cb57e74c723633e171b65c6d142
Successfully built kastore
Installing collected packages: numpy, humanize, kastore
Successfully installed humanize-4.2.2 kastore-0.3.1 numpy-1.23.0
(venv) t490:tmp $ python -c "import kastore"
(venv) t490:tmp $ pip install "numpy<1.23"
Collecting numpy<1.23
  Using cached numpy-1.22.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.23.0
    Uninstalling numpy-1.23.0:
      Successfully uninstalled numpy-1.23.0
Successfully installed numpy-1.22.4
(venv) t490:tmp $ python -c "import kastore"
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/venv/lib/python3.10/site-packages/kastore/__init__.py", line 7, in <module>
    from . import store
  File "/tmp/venv/lib/python3.10/site-packages/kastore/store.py", line 24, in <module>
    import kastore.exceptions as exceptions
  File "/tmp/venv/lib/python3.10/site-packages/kastore/exceptions.py", line 4, in <module>
    from _kastore import FileFormatError
ImportError: numpy.core.multiarray failed to import

[benjeffery]

Ok, so summarising here -

• a normal install works
• installing an old numpy, then kastore works
• installing kastore, then an old numpy, is the broken case.

We can fix this by changing kastore's install_requires to oldest-supported-numpy, installing an old numpy after now works, however the user does get this warning from pip:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
oldest-supported-numpy 2022.5.28 requires numpy==1.21.6; python_version == "3.10" and platform_machine != "loongarch64" and platform_python_implementation != "PyPy", but you have numpy 1.23.0 which is incompatible.

Which isn't ideal.

[grahamgower]

Why not add a pyproject.toml file with the following

[build-system]
requires=[..., "oldest-supported-numpy"]

Is that the recommended approach these days?

[benjeffery]

Yes, we could modernise and use build. I did this recently for msprime.

[jeromekelleher]

Updating to pyproject.toml seems like the way to go. This is probably what should happen in tsinfer as well I guess?

[benjeffery]

Note that the kastore wheels have always used oldest-supported-numpy so this was only an issue for the sdist.

[benjeffery]

@grahamgower I've released 0.3.2 with this PR in. Hopefully that fixes your issue!