pythran fails to build with setuptools 73.0.0

[tommilligan]

Found while building scipy which depends on pythran

#28 59.45 pdm.termui: [651/1475] Generating 'scipy/stats/_stats_pythran.cpython-311-x86_64-linux-gnu.so.p/_stats_pythran.cpp'
#28 59.45 pdm.termui: FAILED: scipy/stats/_stats_pythran.cpython-311-x86_64-linux-gnu.so.p/_stats_pythran.cpp 
#28 59.45 pdm.termui: /tmp/pdm-build-env-4jkfvkat-shared/bin/pythran -E ../scipy/stats/_stats_pythran.py -o scipy/stats/_stats_pythran.cpython-311-x86_64-linux-gnu.so.p/_stats_pythran.cpp
#28 59.45 pdm.termui: Traceback (most recent call last):
#28 59.45 pdm.termui:   File "/tmp/pdm-build-env-4jkfvkat-shared/bin/pythran", line 8, in <module>
#28 59.45 pdm.termui:     sys.exit(run())
#28 59.45 pdm.termui:              ^^^^^
#28 59.45 pdm.termui:   File "/tmp/pdm-build-env-4jkfvkat-shared/lib/python3.11/site-packages/pythran/run.py", line 181, in run
#28 59.45 pdm.termui:     pythran.compile_pythranfile(args.input_file,
#28 59.45 pdm.termui:     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
#28 59.45 pdm.termui:   File "/tmp/pdm-build-env-4jkfvkat-shared/lib/python3.11/site-packages/pythran/__init__.py", line 127, in __getattr__
#28 59.45 pdm.termui:     import pythran.toolchain
#28 59.45 pdm.termui:   File "/tmp/pdm-build-env-4jkfvkat-shared/lib/python3.11/site-packages/pythran/toolchain.py", line 11, in <module>
#28 59.45 pdm.termui:     from pythran.dist import PythranExtension, PythranBuildExt
#28 59.45 pdm.termui:   File "/tmp/pdm-build-env-4jkfvkat-shared/lib/python3.11/site-packages/pythran/dist.py", line 139, in <module>
#28 59.45 pdm.termui:     class PythranBuildExt(PythranBuildExtMixIn, LegacyBuildExt, metaclass=PythranBuildExtMeta):
#28 59.45 pdm.termui: TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

Build environment

#28 27.59 pdm.termui: ======== Resolution Result ========
#28 27.59 pdm.termui: Stable pins:
#28 27.59 pdm.termui:               python None
#28 27.59 pdm.termui:               cython 3.0.11
#28 27.59 pdm.termui:         meson-python 0.16.0
#28 27.59 pdm.termui:                numpy 2.1.0
#28 27.59 pdm.termui:             pybind11 2.12.0
#28 27.59 pdm.termui:              pythran 0.15.0
#28 27.59 pdm.termui:              beniget 0.4.2.post1
#28 27.59 pdm.termui:                 gast 0.5.5
#28 27.59 pdm.termui:                meson 1.5.1
#28 27.59 pdm.termui:            packaging 24.1
#28 27.59 pdm.termui:                  ply 3.11
#28 27.59 pdm.termui:   pyproject-metadata 0.8.0
#28 27.59 pdm.termui:           setuptools 73.0.0

See relevant discussion in scipy repo: scipy/scipy#21416

[rgommers]

(brief comment because I'm not feeling 100% and offline for a few days)

1. The current state is that all recent releases of SciPy are now broken when built from source (with -Duse-pythran=false the only workaround). They can only be fixed by a revert in setuptools, so we should try to get whatever the problem is reverted.
   • I can't stay focused long enough to wrap my head around this and write a good bug report for setuptools at the moment. I hope someone else can.
2. An unpinned setuptools is a very annoying dependency to have - setuptools is always going to be very fragile and continue to do breaking releases. I'd really like pythran to not need setuptools at all for the transpiling functionality it offers, but localize that to the extension module building functionality. If that's a lot of work, adding upper bounds to pythran releases is a reasonable shorter-term workaround.

[agriyakhetarpal]

I can't stay focused long enough to wrap my head around this and write a good bug report for setuptools at the moment. I hope someone else can.

@rgommers, I just bisected, and the first bad commit in setuptools seems to be f9398b1f. I'll have a look and try to write a bug report for this there.

[serge-sans-paille]

What about #2231 ?

[serge-sans-paille]

I think we can consider this fixed: setuptools fixed the issue upstream, and pythran no longer depends on setuptools for the .cpp generation part.