Segmentation fault before method call with dict and unsendable

[konstin]

A #[pyclass(dict, unsendable)] crashes with a segmentation fault when trying to call a method.

🌍 Environment

• Your operating system and version: Ubuntu 18.04
• Your python version: 3.7
• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: pyenv and virtualenv
• Your Rust version (rustc --version): rustc 1.44.1 (c7087fe00 2020-06-17)
• Your PyO3 version: 0.11.1
• Have you tried using latest PyO3 master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?: Yes

💥 Reproducing

lib.rs

use pyo3::prelude::*;

#[pyclass(dict, unsendable)] // <- NOTE THESE TWO
struct BeautifulSoup {}

#[pymethods]
impl BeautifulSoup {
    #[new]
    fn new() -> PyResult<BeautifulSoup> {
        println!("new called");
        Ok(BeautifulSoup {})
    }

    fn select_one(&self) -> Option<()> {
        println!("select_one called");
        Some(())
    }
}

#[pymodule]
fn html_py_ever_repo(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_class::<BeautifulSoup>()?;

    Ok(())
}

Cargo.toml

[package]
name = "html-py-ever-repo"
version = "0.1.0"
authors = ["konstin <konstin@mailbox.org>"]
edition = "2018"

[dependencies]
#pyo3 = { version = "0.11.1", features = ["extension-module"] }
pyo3 = { git = "https://github.com/PyO3/pyo3", features = ["extension-module"] }

[lib]
name = "html_py_ever_repo"
crate-type = ["cdylib"]

Crashing python code

from html_py_ever_repo import BeautifulSoup
BeautifulSoup().select_one()

backtrace

#0  _PyObject_GetMethod (obj=obj@entry=0x7ffff7e516c0, name=0x7ffff7e65070, method=method@entry=0x7fffffffd770) at Objects/object.c:1165
#1  0x00005555555b1075 in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3040
#2  0x000055555568952d in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff7edd450) at Python/ceval.c:547
#3  _PyEval_EvalCodeWithName (_co=_co@entry=0x7ffff7f148a0, globals=globals@entry=0x7ffff7f148a0, locals=locals@entry=0x7ffff7f5b390, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0, 
    kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:3930
#4  0x00005555556895f3 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=locals@entry=0x7ffff7f5b390, globals=globals@entry=0x7ffff7f148a0, 
    _co=_co@entry=0x7ffff7f148a0) at Python/ceval.c:3959
#5  PyEval_EvalCode (co=co@entry=0x7ffff7f148a0, globals=globals@entry=0x7ffff7f45b90, locals=locals@entry=0x7ffff7f45b90) at Python/ceval.c:524
#6  0x00005555556c4d8b in run_mod (arena=0x7ffff7f5b390, flags=0x7fffffffda50, locals=0x7ffff7f45b90, globals=0x7ffff7f45b90, filename=0x7ffff7e911f0, mod=0x555555b09118) at Python/pythonrun.c:1035
#7  PyRun_FileExFlags (fp=0x555555b22650, filename_str=<optimized out>, start=<optimized out>, globals=0x7ffff7f45b90, locals=0x7ffff7f45b90, closeit=1, flags=0x7fffffffda50) at Python/pythonrun.c:988
#8  0x00005555556c4f8d in PyRun_SimpleFileExFlags (fp=0x555555b22650, filename=<optimized out>, closeit=1, flags=0x7fffffffda50) at Python/pythonrun.c:429
#9  0x00005555555be035 in pymain_run_file (p_cf=0x7fffffffda50, filename=<optimized out>, fp=0x555555b22650) at Modules/main.c:456
#10 pymain_run_filename (cf=0x7fffffffda50, pymain=0x7fffffffdb60) at Modules/main.c:1635
#11 pymain_run_python (pymain=0x7fffffffdb60) at Modules/main.c:2896
#12 pymain_main (pymain=pymain@entry=0x7fffffffdb60) at Modules/main.c:3057
#13 0x00005555555be2a9 in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:3092
#14 0x00007ffff7041b97 in __libc_start_main (main=0x5555555ae140 <main>, argc=2, argv=0x7fffffffdcb8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdca8)
    at ../csu/libc-start.c:310
#15 0x00005555555b815a in _start ()

The crash occurs before the method is called, so select_one called is never printed.

I used the following command inside the virtualenv for reproducing:

maturin build -i python3.7 -m html-py-ever-repo/Cargo.toml && pip install -U html-py-ever-repo/target/wheels/html_py_ever_repo-0.1.0-cp37-cp37m-manylinux1_x86_64.whl && python -c "from html_py_ever_repo import BeautifulSoup; BeautifulSoup().select_one()"

[kngwyu]

Thank you for reporting!
I think it's tdue to the order of members 😓

[sebpuetz]

I think I have found the culprit, the offset for the __dict__ attribute was incorrect when the non-dummy ThreadChecker is used. The last field of PyCell is currently the ThreadChecker, so if both unsendable and dict are specified, the ThreadChecker takes up 8 bytes. Subtracting T::Dict::OFFSET from the size of the struct then returns a pointer to the beginning of the ThreadChecker instead of the dictionary. Moving the ThreadChecker before the WeakRef and Dict pointers in PyCell solves the segfaults. From what I can tell, this didn't cause any failing tests.

[davidhewitt]

Fixed via #1058