stubgen and Cython modules

[dgrunwald]

Hello,

I'm trying to use mypy with a Cython module (via stubgen).

Example Cython module:

#cython: binding=True

import typing

def f(path: str, a: int = 0, b: bool = True) -> typing.List[str]:
    return []

cdef class MyClass(object):
    def __init__(self, name: str = None):
        self.name = name

    def run(self, action: str) -> None:
        pass

In general, there's two ways in which Cython can expose the function signature:

• #cython: embedsignature=True: Cython will embed the signature in the docstring.
• #cython: binding=True: Cython will use a custom function object (not a PyCFunction) that has the relevant attributes set -- this allows using inspect.signature().

The embedsignature approach has some problems that I think are on the Cython side of things, see cython/cython#3150.
For the binding approach, stubgen currently outputs:

f: Any

class MyClass:
    def __init__(*args, **kwargs) -> None: ...
    @classmethod
    def run(self, unicodeaction: str) -> None: ...
    def __reduce__() -> Any: ...
    def __setstate__(state) -> Any: ...

Both f and MyClass.run are of type cyfunction, which does not match this test for a C function:

def is_c_function(obj: object) -> bool:
    return inspect.isbuiltin(obj) or type(obj) is type(ord)

The cyfunction is effectively a custom callable that has most of the attributes expected for a function:

>>> dir(testmodule.f)
['__annotations__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__get__', '__getattribute__', '__globals__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__self__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']
>>> testmodule.f.__annotations__
{'path': 'unicode', 'a': 'int', 'b': 'bool', 'return': typing.List[str]}
>>> inspect.signature(testmodule.MyClass.run)
<Signature (self, action:'unicode') -> None>

The str->unicode replacement is a Cython bug, but that can be worked around by defining a unicode = str type alias.

To allow using mypy for a project that imports Cython libraries, I think stubgen should:

• on top-level, recognize the cyfunction as a function and emit a function declaration based on inspect.signature()
• on class-level, recognize the cyfunction and use inspect.signature() instead of relying on the docstring

[JukkaL]

Stubgen support for Cython modules would be nice. Would you like to create a PR? If you have questions, feel free to ask them here. (And apologies for the slow response.)

[TH3CHARLie]

Both interesting and nice feature to have and I'd like to take a shot

[pbotros]

I decided to take a stab at this issue - see PR #8631 and lmk feedback, thanks!

[matteosantama]

@dgrunwald

For the binding approach, stubgen currently outputs:

Are you able to run stubgen on that code? I get

Critical error during semantic analysis: <filename>.py:8: error: invalid syntax

It appears stubgen does not like the cdef

[dgrunwald]

@matteosantama I was running stubgen on the compiled extension module, not on the .pyx source code.

[JotaRata]

Bump