PyObject -> &'py PyAny

[1tgr]

I'm implementing a function within an API that's expected to return &'py PyAny. The function will serialize a Rust object to a Python object and return a reference to it, bound to the GIL lifetime of Python<'py>.

It would be easy to implement a function that returns PyObject, and there are various types in PyO3 that will return &'py PySomething, eg PyBool::new, PyDict::new. However I don't see a route to expose Rust objects generically as &'py PyAny.

Conceptually I think I want to: allocate an object on the Python heap; move my instance T into it; add the new Python object to the auto-release pool; return a reference to the Python object.

The closest I could see was: PyObject::as_ref (but that returns a reference bound to the lifetime of &self, not to &'py); or something involving *mut ffi::PyObject and unsafe Python::from_owned_ptr.

In case it's relevant, I'm working against PyO3 v0.8.5.

[davidhewitt]

Are you able to update to pyO3 0.10.1 or higher? Then you can use PyCell::new(py, my_rust_instance). It actually allocates the object on the Python heap and registers a reference to it on the auto-release pool exactly as you suggest :)

This returns &'py PyCell<T>, which derefs to &'py PyAny.

[1tgr]

Thanks, is there a conversion that works for primitive types as well as #[pyclass]?

[davidhewitt]

You could use let any: &PyAny = unsafe { py.from_owned_ptr(x.into_py(py).unwrap().into_ptr()) }, where x is any rust type which can be converted to a Python type.

However I think a safe API should be trivial to add, so I'll add this for 0.12.

[1tgr]

Thanks, this one-liner is working nicely.

[kngwyu]

I don't know what you need precisely, but doesn't x.into_py(py).as_ref(py) work?

[davidhewitt]

@kngwyu I think if you do that the lifetime isn't long enough to return from the function. (The reference is owned by the intermediary PyObject?)

[1tgr]

Indeed, as_ref(&'a self, py: Python<'py>) returns PyRef<'a, Self> not PyRef<'py, Self>. Fundamentally the reference needs to come from an into_ conversion (move ownership over to the pool) not an as_ conversion (caller retains ownership).

[davidhewitt]

Indeed, the API I'm considering is fn register(self, py: Python<'py>) -> &'py PyAny

[kngwyu]

@davidhewitt
As a trait method of PyClass?
Sounds good for me, though I think the naming can be improved(e.g., into_pyany or so).
Now PyClass has no method so we can make it more user-friendly by adding some short-hand methods.

[davidhewitt]

I was thinking that register would be added to PyObject and Py<T>. I'll submit a PR to show what I mean maybe this evening or by the weekend.

[kngwyu]

So that is only for making things one-liner?
Then it's not so appealing for me, though I don't think it shoudn't be exist.

[1tgr]

So that is only for making things one-liner?

It avoids an unsafe one-liner.

Or rather, it moves the unsafety out of user code and into pyo3.

[kngwyu]

As an alternative, how about:

impl<'p> Python<'p> {
    pub fn register<T: crate::IntoPy<PyObject>>(self, value: T) -> &'p PyAny {
        unsafe { self.from_owned_ptr(value.into_py(self).into_ptr()) }
    }
}

?

@1tgr
You misunderstand x.into_py(py).as_ref(py).
As @davidhewitt says it does not work without one more let, but it works like:

    let obj: PyObject = EmptyClassWithNew {}.into_py(py);
    let any = obj.as_ref(py);

And this is the intended path to PyAny.

[1tgr]

@kngwyu the lifetime on the reference returned by as_ref is too short for what I need: it gives PyRef<'a, PyAny> not PyRef<'py, PyAny>, where 'a is the lifetime on the obj local variable. This is fine if I'm using the any local variable immediately, but I can't return it from a function as I'd want.

FYI, the context is a serde serializer that turns a Rust struct into a Python dict. The serializer returns &'py PyAny given py: Python<'py>. I need the conversion discussed here to serialize primitive types, ie to convert T: IntoPy<PyObject> to &'py PyAny given py: Python<'py>.

[davidhewitt]

As an alternative, how about:

impl<'p> Python<'p> {
    pub fn register<T: crate::IntoPy<PyObject>>(self, value: T) -> &'p PyAny {
        unsafe { self.from_owned_ptr(value.into_py(self).into_ptr()) }
    }
}

Yeah I considered that too. I think the nice thing about it being on Py<T> especially is you can get

impl<T> Py<T> {
    pub fn register(self, py: Python) -> &T::AsRefTarget {
        unsafe { py.from_owned_ptr(self.into_ptr()) }
    }
}

Though T: crate::IntoPy<PyObject> on Python is also quite convenient. Maybe we should add both? 😆

[kngwyu]

@1tgr
Ah now I understand, thank you for your explaination!

@davidhewitt
Looks nice 👍 , but maybe into_ref is better?

[gilescope]

@davidhewitt keen on having only one way to do something if possible to keep things simple

[davidhewitt]

FWIW I'm planning on implementing .into_ref() soon, but I want to see the outcome of #1063 (and maybe #1056) beforehand so I don't spend too much time writing and documenting a solution which gets reworked shortly after!