On macos arm64, objc.objc_msgSend seg-faults when called from ipython

[erykoff]

Related to #561 , ipython/ipykernel/etc will seg-fault when running the following test code (based on appnope):

import ctypes
import ctypes.util

objc = ctypes.cdll.LoadLibrary(ctypes.util.find_library('objc'))

def _utf8(s):
    """ensure utf8 bytes"""
    if not isinstance(s, bytes):
        s = s.encode('utf8')
    return s

def n(name):
    """create a selector name (for methods)"""
    return objc.sel_registerName(_utf8(name))

def C(classname):
    """get an ObjC Class by name"""
    return objc.objc_getClass(_utf8(classname))

NSString = C('NSString')

objc.objc_msgSend(NSString, n("stringWithUTF8String:"), _utf8("reason"))

The stack trace on an arm64 mac (DTK) says: python[97504:4905234] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '*** +[NSString stringWithUTF8String:]: NULL cString'

Note that this code will work with both system python (macos 11.0.1) and conda-forge python 3.9.0 for osx-arm64. However, it will seg-fault with ipython/ipykernel from conda-forge or with ipython installed with the system python via pip.

[geohot]

Nice repro! Breaks from normal python3 (3.9 from homebrew) by adding

import IPython

to the top. Trying to figure out exactly what it is that breaks it. Relevant traceback

* thread #2, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x3854940)
  * frame #0: 0x000000019baf2168 libobjc.A.dylib`objc_msgSend + 8
    frame #1: 0x00000001a86b4050 libffi.dylib`ffi_call_SYSV + 80
    frame #2: 0x00000001a86bc9d8 libffi.dylib`ffi_call_int + 944
    frame #3: 0x00000001034d3ec0 _ctypes.cpython-39-darwin.so`_ctypes_callproc + 1300
    frame #4: 0x00000001034cdea8 _ctypes.cpython-39-darwin.so`PyCFuncPtr_call + 1088
    frame #5: 0x00000001001484b4 Python`_PyObject_MakeTpCall + 308
    frame #6: 0x00000001001f49c4 Python`call_function + 512

[geohot]

Tracked it back through ipython.

from urllib.request import urlopen

at the top breaks it too. Nothing ipython it seems. Actually breaks it is the wrong word. Without the import, NSString == 0. With it, it has a value and crashes.

Adding

objc.objc_getClass.restype = ctypes.c_void_p
objc.sel_registerName.restype = ctypes.c_void_p
objc.objc_msgSend.restype = ctypes.c_void_p
objc.objc_msgSend.argtypes = [ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p]

fixes it, in both python and ipython.

[erykoff]

@geohot I think the key part of your fix is definitely setting the argtypes for each individual call, which was missing before and probably incorrect. (Though why this worked with python and not ipython is a mystery).
It's unclear to me if the None issue is actually an issue, the code runs without error (though maybe the nope is a no-op in that case, I'm not sure how to check).
Finally, I traced down the mystery of the from urllib.request import python to the import _scproxy call in urllib.
And it seems that the key here is that we need to load the Foundation library which _scproxy pulls in (along with a ton of other stuff):
_ = ctypes.cdll.LoadLibrary(ctypes.util.find_library('Foundation'))

[geohot]

The None thing is definitely an issue, it just means it couldn't find the class, and won't be able to call methods on it. The reason python and ipython were different is just because ipython imported urllib before your code, which loaded Foundation.

You found the fix in the other thread though, obviously you can't find NSString if you don't load Foundation! Fixed in my appnope PR.

[minrk]

I believe this also requires CPython 3.9.1 by python/cpython#21249. ctypes dyld in general doesn't work with cpython 3.9.0 on macOS 11. The patches have been backported in the latest homebrew Python 3.9.0 and conda-forge Python 3.9 builds (3.8, too, perhaps?).