PyO3 generates broken ref to PyUnicode_AsUTF8AndSize on PyPy3

[mgorny]

Bug Description

The headers of PyPy3 define a number of aliases such as:

#define PyUnicode_AsUTF8AndSize PyPyUnicode_AsUTF8AndSize

As a result, extensions built against PyPy3 using PyUnicode_AsUTF8AndSize() API have it translated to PyPyUnicode_AsUTF8AndSize symbol, etc.

However, extensions built on PyPy3 using PyO3 use CPython symbol names and fail to load:

E   ImportError: /tmp/setuptools-rust/examples/html-py-ever/.tox/pypy3/lib/pypy3.9/site-packages/html_py_ever/html_py_ever.pypy39-pp73-x86_64-linux-gnu.so: undefined symbol: PyUnicode_AsUTF8AndSize

Steps to Reproduce

Install PyPy3, e.g. from https://www.pypy.org/download.html or using your distribution packages.

git clone https://github.com/PyO3/setuptools-rust
cd setuptools-rust/examples/html-py-ever
git checkout v0.12.1
tox -e pypy3

Backtrace

=============================================================== ERRORS ================================================================
________________________________________________ ERROR collecting test/test_parsing.py ________________________________________________
ImportError while importing test module '/tmp/setuptools-rust/examples/html-py-ever/test/test_parsing.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/pypy3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test/test_parsing.py:5: in <module>
    import html_py_ever
.tox/pypy3/lib/pypy3.9/site-packages/html_py_ever/__init__.py:1: in <module>
    from .html_py_ever import *
E   ImportError: /tmp/setuptools-rust/examples/html-py-ever/.tox/pypy3/lib/pypy3.9/site-packages/html_py_ever/html_py_ever.pypy39-pp73-x86_64-linux-gnu.so: undefined symbol: PyUnicode_AsUTF8AndSize
_______________________________________________ ERROR collecting test/test_selector.py ________________________________________________
ImportError while importing test module '/tmp/setuptools-rust/examples/html-py-ever/test/test_selector.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/pypy3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test/test_selector.py:5: in <module>
    import html_py_ever
.tox/pypy3/lib/pypy3.9/site-packages/html_py_ever/__init__.py:1: in <module>
    from .html_py_ever import *
E   ImportError: /tmp/setuptools-rust/examples/html-py-ever/.tox/pypy3/lib/pypy3.9/site-packages/html_py_ever/html_py_ever.pypy39-pp73-x86_64-linux-gnu.so: undefined symbol: PyUnicode_AsUTF8AndSize

Your operating system and version

Gentoo Linux

Your Python version (python --version)

Python 3.9.10 (307e102d7222131fee14073e8856df773627186c, Jan 31 2022, 19:21:32) [PyPy 7.3.8 with GCC 11.2.1 20220115]

Your Rust version (rustc --version)

rustc 1.58.1 (gentoo)

Your PyO3 version

tested with 0.12.1 and 0.15.1

How did you install python? Did you use a virtualenv?

PyPy3 from distribution package, actual build inside tox/virtualenv as shown above.

Additional Info

No response

[mgorny]

I'm sorry, I have probably failed to test 0.15.1. When I try again, I am unable to build it:

         Running `rustc --crate-name pyo3 --edition=2018 /home/mgorny/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.15.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="default"' --cfg 'feature="extension-module"' --cfg 'feature="indoc"' --cfg 'feature="macros"' --cfg 'feature="paste"' --cfg 'feature="pyo3-macros"' --cfg 'feature="unindent"' -C metadata=cd65d4deb75c6a41 -C extra-filename=-cd65d4deb75c6a41 --out-dir /tmp/pip-req-build-23clmqnu/target/release/deps -L dependency=/tmp/pip-req-build-23clmqnu/target/release/deps --extern cfg_if=/tmp/pip-req-build-23clmqnu/target/release/deps/libcfg_if-5afdd7b0cfa8cd70.rmeta --extern indoc=/tmp/pip-req-build-23clmqnu/target/release/deps/libindoc-7fcdefefd019aa84.rmeta --extern libc=/tmp/pip-req-build-23clmqnu/target/release/deps/liblibc-effaeae12b4944a0.rmeta --extern parking_lot=/tmp/pip-req-build-23clmqnu/target/release/deps/libparking_lot-9ff9f8c6b0d972e0.rmeta --extern paste=/tmp/pip-req-build-23clmqnu/target/release/deps/libpaste-5343916200d42c36.rmeta --extern pyo3_macros=/tmp/pip-req-build-23clmqnu/target/release/deps/libpyo3_macros-6ba82771b3723a71.so --extern unindent=/tmp/pip-req-build-23clmqnu/target/release/deps/libunindent-1e5253350a431aac.rmeta --cap-lints allow --cfg Py_3_6 --cfg Py_3_7 --cfg Py_3_8 --cfg Py_3_9 --cfg PyPy --cfg 'py_sys_config="WITH_THREAD"' --cfg track_caller --cfg min_const_generics --cfg addr_of`
    error[E0425]: cannot find function `_PyObject_CallNoArg` in module `ffi`
       --> /home/mgorny/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.15.1/src/instance.rs:569:62
        |
    569 |                     PyObject::from_owned_ptr_or_err(py, ffi::_PyObject_CallNoArg(self.as_ptr()))
        |                                                              ^^^^^^^^^^^^^^^^^^^ help: a function with a similar name exists: `PyObject_Calloc`
        |
       ::: /home/mgorny/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.15.1/src/ffi/objimpl.rs:9:5
        |
    9   |     pub fn PyObject_Calloc(nelem: size_t, elsize: size_t) -> *mut c_void;
        |     --------------------------------------------------------------------- similarly named function `PyObject_Calloc` defined here
    
    error[E0425]: cannot find function, tuple struct or tuple variant `PyObject_CallMethodNoArgs` in module `ffi`
       --> /home/mgorny/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.15.1/src/instance.rs:623:62
        |
    623 |                       PyObject::from_owned_ptr_or_err(py, ffi::PyObject_CallMethodNoArgs(self.as_ptr(), name.as_ptr()))
        |                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^ help: a function with a similar name exists: `PyObject_CallMethodObjArgs`
        |
       ::: /home/mgorny/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.15.1/src/ffi/abstract_.rs:60:5
        |
    60  | /     pub fn PyObject_CallMethodObjArgs(
    61  | |         o: *mut PyObject,
    62  | |         method: *mut PyObject,
    63  | |         ...
    64  | |     ) -> *mut PyObject;
        | |_______________________- similarly named function `PyObject_CallMethodObjArgs` defined here
    
    error[E0425]: cannot find function `_PyObject_CallNoArg` in module `ffi`
       --> /home/mgorny/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.15.1/src/types/any.rs:341:58
        |
    341 |                     self.py().from_owned_ptr_or_err(ffi::_PyObject_CallNoArg(self.as_ptr()))
        |                                                          ^^^^^^^^^^^^^^^^^^^ help: a function with a similar name exists: `PyObject_Calloc`
        |
       ::: /home/mgorny/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.15.1/src/ffi/objimpl.rs:9:5
        |
    9   |     pub fn PyObject_Calloc(nelem: size_t, elsize: size_t) -> *mut c_void;
        |     --------------------------------------------------------------------- similarly named function `PyObject_Calloc` defined here
    
    error[E0425]: cannot find function, tuple struct or tuple variant `PyObject_CallMethodNoArgs` in module `ffi`
       --> /home/mgorny/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.15.1/src/types/any.rs:470:50
        |
    470 |               self.py().from_owned_ptr_or_err(ffi::PyObject_CallMethodNoArgs(self.as_ptr(), name.as_ptr()))
        |                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^ help: a function with a similar name exists: `PyObject_CallMethodObjArgs`
        |
       ::: /home/mgorny/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.15.1/src/ffi/abstract_.rs:60:5
        |
    60  | /     pub fn PyObject_CallMethodObjArgs(
    61  | |         o: *mut PyObject,
    62  | |         method: *mut PyObject,
    63  | |         ...
    64  | |     ) -> *mut PyObject;
        | |_______________________- similarly named function `PyObject_CallMethodObjArgs` defined here
    

[mattip]

This may need to wait for the next pypy3.9 rc. It turns out we are missing some of the new C-API functions in pypy3.9rc1 like PyObject_CallMethodNoArgs.

[mattip]

I had some help from the PyO3 people to set up a CI which runs PyO3 nightly + PyPy nightly, but have not yet updated it to include PyPy3.9. In the last two weeks it broke. Help to get it going again would be appreciated.

[birkenfeld]

I though we map the names using link_name? E.g. https://github.com/PyO3/pyo3/blob/main/pyo3-ffi/src/unicodeobject.rs#L168

[davidhewitt]

More generally we haven't attempted to support PyPy 3.9 dev builds at all yet. If anyone is interested in adding it to our Github workflow we could begin implementing the required updates. @mgorny can you use PyPy3 3.8 instead? Is supported and should work no problem.

[mattip]

@mgorny is trying out PyPy's 3.9rc1, which I unleashed a week ago. I will try to add PyPy 3.9 to the pypy/binary-builds repo before the rc2 release, which should give some feedback about what is still missing.

[davidhewitt]

Note that as well as adding to the pypy/binary-builds repo I expect we'll need to make updates here in PyO3. Might take a few iterations to get green builds!

[ofek]

Any update on this? New 3.9: https://www.pypy.org/posts/2022/02/pypy-v738-release.html