ENH: upstream setuptools patches to upstream setuptools

[mattip]

This would simplify building projects that have migrated to HPy. This task would be something like

• find the places in the current code that need more support from setuptools
• submit patches to upstream to incorporate them into the official setuptools

[antocuni]

This is a summary of what we do after merging #338 :

• The final goal is to be able to write something like this

from setuptools import setup, Extension
setup(
    name="hpy-simple-example",
    hpy_ext_modules=[
        Extension('simple', sources=['simple.c')]),
    ],
    setup_requires=['hpy'],
)

• another requirement is to be able to build "universal modules": these require a different set of compilation options, and produces a file with a different extension: simple.hpy.so. Currently, we do that by doing setup.py --hpy-abi=universal build.

• we are using a setuptools entry point to detect hpy_ext_modules=[...] and hook into setuptools:

hpy/hpy/devel/__init__.py

Lines 128 to 142 in 2211087

	def handle_hpy_ext_modules(dist, attr, hpy_ext_modules):
	""" Distuils hpy_ext_module setup(...) argument and --hpy-abi option.
	See hpy's setup.py where this function is registered as an entry
	point.
	"""
	assert attr == 'hpy_ext_modules'
	# add a global option --hpy-abi to setup.py
	dist.__class__.hpy_abi = DEFAULT_HPY_ABI
	dist.__class__.global_options += [
	('hpy-abi=', None, 'Specify the HPy ABI mode (default: %s)' % DEFAULT_HPY_ABI)
	]
	hpydevel = HPyDevel()
	hpydevel.fix_distribution(dist)

• inside the hook, we automatically patch the distribution to use our own build_hpy_ext command for build_ext:

hpy/hpy/devel/__init__.py

Lines 95 to 99 in 2211087

	# ============= build_ext ==========
	build_ext = dist.cmdclass.get("build_ext", cmd.build_ext.build_ext)
	self.build_ext_sanity_check(build_ext)
	build_ext_hpy = make_mixin(build_ext, build_ext_hpy_mixin)
	dist.cmdclass['build_ext'] = build_ext_hpy

hpy/hpy/devel/__init__.py

Line 261 in 2211087

class build_ext_hpy_mixin:

• this is where we modify the Extension to add the extra c files, include dirs, etc.:

hpy/hpy/devel/__init__.py

Lines 285 to 301 in 2211087

	def _finalize_hpy_ext(self, ext):
	if hasattr(ext, "hpy_abi"):
	return
	ext.name = HPyExtensionName(ext.name)
	ext.hpy_abi = self.distribution.hpy_abi
	ext.include_dirs += self.hpydevel.get_extra_include_dirs()
	ext.sources += self.hpydevel.get_extra_sources()
	ext.define_macros.append(('HPY', None))
	if ext.hpy_abi == 'cpython':
	ext.sources += self.hpydevel.get_ctx_sources()
	ext._hpy_needs_stub = False
	elif ext.hpy_abi == 'universal':
	ext.define_macros.append(('HPY_UNIVERSAL_ABI', None))
	ext._hpy_needs_stub = True
	else:
	raise DistutilsError('Unknown HPy ABI: %s. Valid values are: '
	'cpython, universal' % ext.hpy_abi)

On top of these, there are various hacks and bad monkey-patching to make things works. For example, we have to use this ugly hack to be able to distinguish hpy and normal extensions:

hpy/hpy/devel/__init__.py

Lines 176 to 195 in 2211087

	class HPyExtensionName(str):
	""" Wrapper around str to allow HPy extension modules to be identified.
	The following build_ext command methods are passed only the *name*
	of the extension and not the full extension object. The
	build_ext_hpy_mixin class needs to detect when HPy are extensions
	passed to these methods and override the default behaviour.
	This str sub-class allows HPy extensions to be detected, while
	still allowing the extension name to be used as an ordinary string.
	"""
	def split(self, *args, **kw):
	result = str.split(self, *args, **kw)
	return [self.__class__(s) for s in result]
	def translate(self, *args, **kw):
	result = str.translate(self, *args, **kw)
	return self.__class__(result)

[mattip]

We could think about supporting pyproject.toml build systems as well or instead of setuptools. The scientific python projects have moved to meson/meson-python.

[fangerer]

Agreed. Btw. this relates to #435 .