Pass the Py_LIMITED_API macro to Extensions

[rauletorresc]

Context: Extensions also need to be informed about the usage of the limited API.

Description of the Change: Pass the Py_LIMITED_API macro to Extensions

Benefits: Extensions will conform with the limited API.

[joeycarter]

Looks good @rauletorresc, I think this should do it! 💯

Just had one comment, then I'll be happy to approve.

[dime10]

The last thing I'm concerned about is the third-party modules we (atleast attempt) to ship in our wheels:

• CUDA: python only
• OQD: C++ sources, but no Python module (nanobind/pybind) as far as I can see
• OQC: this one has C++ sources and still builds a Python module with Pybind11

For OQC the situation is rather confusing, the Python module is built separately from the device backend

catalyst/frontend/catalyst/third_party/oqc/src/CMakeLists.txt

Line 41 in 0719a87

add_library(rtd_oqc SHARED OQCDevice.cpp)

catalyst/frontend/catalyst/third_party/oqc/src/CMakeLists.txt

Lines 55 to 59 in 0719a87

	pybind11_add_module(oqc_python_module SHARED oqc_python_module.cpp)
	target_include_directories(oqc_python_module PRIVATE ${runtime_includes})
	add_dependencies(rtd_oqc oqc_python_module)
	target_compile_definitions(rtd_oqc PUBLIC -DOQC_PY=\"$<TARGET_FILE_NAME:oqc_python_module>\")

but we only ship the device backend (without the module):

catalyst/Makefile

Lines 196 to 197 in 0719a87

	cp $(OQC_BUILD_DIR)/librtd_oqc* $(MK_DIR)/frontend/catalyst/lib
	cp $(OQC_BUILD_DIR)/backend/*.toml $(MK_DIR)/frontend/catalyst/lib/backend

Furthermore, the only thing linking the module to the device backend is the compiler macro OQC_PY shown in the cmake snippet above, but since that value is set at build time it must be out of date at install time on a different system. It is used by the device backend here:

catalyst/frontend/catalyst/third_party/oqc/src/OQCRunner.hpp

Line 114 in 0719a87

DynamicLibraryLoader libLoader(OQC_PY);

To recap, the OQC plugin is still being built with Pybind11, which is not stable abi compatible, but the relevant module isn't even shipped in the wheels or configured for distribution 🤔

[joeycarter]

The last thing I'm concerned about is the third-party modules we (atleast attempt) to ship in our wheels:

• CUDA: python only
• OQD: C++ sources, but no Python module (nanobind/pybind) as far as I can see
• OQC: this one has C++ sources and still builds a Python module with Pybind11

The OQD modules shouldn't cause issues with building the wheels with the stable ABI since they don't use the Python C API at all (for now, at least).

For OQC, I think you're right that this is not correct. I just tried building the wheels locally and running a simple test and it fails:

import pennylane as qml
from catalyst.third_party.oqc import OQCDevice

import os
os.environ["OQC_EMAIL"] = "email"
os.environ["OQC_PASSWORD"] = "password"
os.environ["OQC_URL"] = "oqc_url"

device = OQCDevice(backend="lucy", shots=1000, wires=1)
@qml.qjit
@qml.qnode(device)
def counts_1qbit(x: float):
    qml.RX(x, wires=0)
    return qml.counts()

counts_1qbit(1.0)

RuntimeError: [.../catalyst/frontend/catalyst/third_party/oqc/src/../../../../../runtime/include/DynamicLibraryLoader.hpp][Line:75][Function:getSymbol] Error in Catalyst Runtime: .../catalyst/frontend/catalyst/third_party/oqc/src/build/oqc_python_module.cpython-312-x86_64-linux-gnu.so: undefined symbol: counts

I think the reason we missed this is because oqc_python_module only defines a counts() function, but as far as I can tell we have no tests covering it.

I don't think the OQC issue is blocking this PR, but we should address it in a future PR.

[joeycarter]

Furthermore, we skip the OQC frontend tests entirely if the qcaas_client package is not installed:

catalyst/frontend/test/test_oqc/conftest.py

Lines 22 to 27 in 0719a87

	try:
	import qcaas_client
	oqc_available = True
	except ImportError:
	oqc_available = False

catalyst/frontend/test/test_oqc/conftest.py

Lines 44 to 49 in 0719a87

	def pytest_collection_modifyitems(items):
	"""A pytest items modifier method"""
	if not oqc_available:
	# If OQC QCAAS is not installed, mark all collected tests to be skipped.
	for item in items:
	item.add_marker(pytest.mark.skip(reason="OQC qcaas is not installed"))

[joeycarter]

Happy to approve, but we should look into the OQC issue I mentioned above.

[dime10]

I don't think the OQC issue is blocking this PR, but we should address it in a future PR.

So I think it's relevant to this PR in the sense that to properly ship OQC with the stable abi we need to convert it to nanobind, but you're right we can do this in a separate PR 👍

[rauletorresc]

I don't think the OQC issue is blocking this PR, but we should address it in a future PR.

So I think it's relevant to this PR in the sense that to properly ship OQC with the stable abi we need to convert it to nanobind, but you're right we can do this in a separate PR 👍

#1391