POC for step 1 of #419 - replace Handlemaps with Arcs.

The context for this is Ryan's [proposal for passing around object instances]( #419)

By ipmlementing `ViaFfi` for `Arc<T>` we get a huge amount of functionality
for free - `Arc` wrappers around Objects can suddenly appear in dictionaries
and function params, and be returned from functions once we remove the explicit
code that disallows it.

It's also incomplete (needs docs, kotlin tests for the new test cases,
swift hasn't been touched at all, etc)

docs/manual/src/internals/object_references.md

@@ -1,7 +1,7 @@
 # Managing Object References
 
 UniFFI [interfaces](../udl/interfaces.md) represent instances of objects
-that have methods and contain shared mutable state. One of Rust's core innovations
+that have methods and contain state. One of Rust's core innovations
 is its ability to provide compile-time guarantees about working with such instances,
 including:
 
@@ -16,6 +16,8 @@ system. UniFFI itself tries to take a hands-off approach as much as possible and
 depends on the Rust compiler itself to uphold safety guarantees, without assuming
 that foreign-language callers will be "well behaved".
 
+## Concurrency
+
 UniFFI's hands-off approach means that all object instances exposed by UniFFI must be safe to
 access concurrently. In Rust terminology, they must be `Send+Sync` and must be useable
 without taking any `&mut` references.
@@ -27,18 +29,26 @@ of the component - as much as possible, UniFFI tries to stay out of your way, si
 that the object implementation is `Send+Sync` and letting the Rust compiler ensure that
 this is so.
 
-## Handle Maps
+Typically this will mean your implementation uses some data structures
+explicitly designed for this purpose, such as a `Mutex` or `RwLock` - but this
+detail is completely up to you - as much as possible, uniffi tries to stay out
+of your way, so ultimately it is the Rust compiler itself which is the ultimate
+arbiter.
+
+## Lifetimes
+
+In order to allow for instances to be used as flexibly as possible, uniffi
+works with `Arc`s holding a pointer to your instances and leverages their
+reference-count based lifetimes, allowing uniffi to largely stay out
+of handling lifetimes entirely for these objects.
 
-For additional typechecking safety, UniFFI indirects all object access through a
-"handle map", a mapping from opaque integer handles to object instances. This indirection
-imposes a small runtime cost but helps us guard against errors or oversights
-in the generated bindings.
+However, this does come at a cost - when you want to return instances from
+your dictionaries or methods, you must return an `Arc<>` directly. When
+accepting instances as arguments, you can choose to accept it as an `Arc<>` or
+as the underlying struct - there are different use-cases for each scenario.
 
-For each interface declared in the UDL, the UniFFI-generated Rust scaffolding
-will create a global handlemap that is responsible for owning all instances
-of that interface, and handing out references to them when methods are called.
-The handlemap requires that its contents be `Send+Sync`, helping enforce requirements
-around thread-safety.
+The exception to the above is constructors - these are expected to just provide
+the instance and uniffi will wrap it in the `Arc<>`.
 
 For example, given a interface definition like this:
 
@@ -50,53 +60,82 @@ interface TodoList {
 };
 ```
 
-The Rust scaffolding would define a lazyily-initialized global static like:
+On the Rust side of the generated bindings, the instance constructor will create an instance of the
+corresponding `TodoList` Rust struct, wrap it in an `Arc<>` and return a raw
+pointer to the foreign language code:
 
 ```rust
-lazy_static! {
-    static ref UNIFFI_HANDLE_MAP_TODOLIST: ArcHandleMap<TodoList> = ArcHandleMap::new();
+pub extern "C" fn todolist_12ba_TodoList_new(
+    err: &mut uniffi::deps::ffi_support::ExternError,
+) -> *const std::os::raw::c_void /* *const TodoList */ {
+    uniffi::deps::ffi_support::call_with_output(err, || {
+        let _new = TodoList::new();
+        let _arc = std::sync::Arc::new(_new);
+        uniffi::UniffiVoidPtr(<std::sync::Arc<TodoList> as uniffi::ViaFfi>::lower(_arc))
+    })
 }
 ```
 
-On the Rust side of the generated bindings, the instance constructor will create an instance of the
-corresponding `TodoList` Rust struct, insert it into the handlemap, and return the resulting integer
-handle to the foreign language code:
+and the uniffi runtime defines:
 
 ```rust
-pub extern "C" fn todolist_TodoList_new(err: &mut ExternError) -> u64 {
-    // Give ownership of the new instance to the handlemap.
-    // We will only ever operate on borrowed references to it.
-    UNIFFI_HANDLE_MAP_TODOLIST.insert_with_output(err, || TodoList::new())
+unsafe impl<T: Sync + Send> ViaFfi for std::sync::Arc<T> {
+    type FfiType = *const std::os::raw::c_void;
+    fn lower(self) -> Self::FfiType {
+        std::sync::Arc::into_raw(self) as Self::FfiType
+    }
 }
 ```
 
-When invoking a method on the instance, the foreign-language code passes the integer handle back
-to the Rust code, which borrows a reference to the instance from the handlemap for the duration
-of the method call:
+which does the "arc to pointer" dance for us. Note that this has "leaked" the
+`Arc<>` reference - if we never see that pointer again, our instance will leak.
+
+When invoking a method on the instance, the foreign-language code passes the
+raw pointer back to the Rust code, which turns it back into a cloned `Arc<>` which
+lives for the duration of the method call:
 
 ```rust
-pub extern "C" fn todolist_TodoList_add_item(handle: u64, todo: RustBuffer, err: &mut ExternError) -> () {
-    let todo = <String as uniffi::ViaFfi>::try_lift(todo).unwrap()
-    // Borrow a reference to the instance so that we can call a method on it.
-    UNIFFI_HANDLE_MAP_TODOLIST.call_with_result_mut(err, handle, |obj| -> Result<(), TodoError> {
-        TodoList::add_item(obj, todo)
+pub extern "C" fn todolist_12ba_TodoList_add_item(
+    ptr: *const std::os::raw::c_void,
+    todo: uniffi::RustBuffer,
+    err: &mut uniffi::deps::ffi_support::ExternError,
+) -> () {
+    uniffi::deps::ffi_support::call_with_result(err, || -> Result<_, TodoError> {
+        let _obj = <std::sync::Arc<TodoList> as uniffi::ViaFfi>::try_lift(ptr).unwrap();
+        let _retval =
+            TodoList::add_item(&_obj, <String as uniffi::ViaFfi>::try_lift(todo).unwrap())?;
+        Ok(_retval)
     })
 }
 ```
 
-Finally, when the foreign-language code frees the instance, it passes the integer handle to
-a special destructor function so that the Rust code can delete it from the handlemap:
+where the uniffi runtime defines:
 
 ```rust
-pub extern "C" fn ffi_todolist_TodoList_object_free(handle: u64) {
-    UNIFFI_HANDLE_MAP_TODOLIST.delete_u64(handle);
-}
+unsafe impl<T: Sync + Send> ViaFfi for std::sync::Arc<T> {
+    type FfiType = *const std::os::raw::c_void;
+    fn try_lift(v: Self::FfiType) -> Result<Self> {
+        let v = v as *const T;
+        // We musn't drop the `Arc<T>` that is owned by the foreign-language code.
+        let foreign_arc = std::mem::ManuallyDrop::new(unsafe { Self::from_raw(v) });
+        // Take a clone for our own use.
+        Ok(std::sync::Arc::clone(&*foreign_arc))
+    }
 ```
 
-This indirection gives us some important safety properties:
+Notice that we take care to ensure the reference added by the constructor
+remains alive. Finally, when the foreign-language code frees the instance, it
+passes the raw pointer a special destructor function so that the Rust code can
+drop that initial final reference (and if that happens to be the final reference,
+the rust object will be dropped.)
 
-* If the generated bindings incorrectly pass an invalid handle, or a handle for a different type of object,
-  then the handlemap will throw an error with high probability, providing some amount of run-time typechecking
-  for correctness of the generated bindings.
-* The handlemap can ensure we uphold Rust's requirements around unique mutable references and threadsafey,
-  by specifying that the contained type must be `Send+Sync`, and by refusing to hand out any mutable references.
+```rust
+pub extern "C" fn ffi_todolist_12ba_TodoList_object_free(ptr: *const std::os::raw::c_void) {
+    if let Err(e) = std::panic::catch_unwind(|| {
+        assert!(!ptr.is_null());
+        unsafe { std::sync::Arc::from_raw(ptr as *const TodoList) };
+    }) {
+        uniffi::deps::log::error!("ffi_todolist_12ba_TodoList_object_free panicked: {:?}", e);
+    }
+}
+```

uniffi/src/lib.rs

@@ -45,7 +45,6 @@ pub mod deps {
     pub use anyhow;
     pub use bytes;
     pub use ffi_support;
-    pub use lazy_static;
     pub use log;
     pub use static_assertions;
 }
@@ -476,6 +475,74 @@ unsafe impl<V: ViaFfi> ViaFfi for HashMap<String, V> {
     }
 }
 
+/// Support for passing reference-counted shared objects via the FFI.
+///
+/// To avoid dealing with complex lifetime semantics over the FFI, any data passed
+/// by reference must be encapsulated in an `Arc`, and must be safe to share
+/// across threads.
+unsafe impl<T: Sync + Send> ViaFfi for std::sync::Arc<T> {
+    // Don't use a pointer to <T> as that requires a `pub <T>`
+    type FfiType = *const std::os::raw::c_void;
+
+    /// When lowering, we have an owned `Arc<T>` and we transfer that ownership
+    /// to the foreign-language code, "leaking" it out of Rust's ownership system
+    /// as a raw pointer. The foreign-language code is responsible for freeing
+    /// this via TODO calling a special destructor for `T`.
+    ///
+    /// (This works safely because we have unique ownership of `self`).
+    fn lower(self) -> Self::FfiType {
+        std::sync::Arc::into_raw(self) as Self::FfiType
+    }
+
+    /// When lifting, we receive a "borrow" of the `Arc<T>` that is owned by
+    /// the foreign-language code, and make a clone of it for our own use.
+    ///
+    /// Safety: the provided value must be a pointer previously obtained by calling
+    /// the `lower()` method of this impl.
+    fn try_lift(v: Self::FfiType) -> Result<Self> {
+        let v = v as *const T;
+        // We musn't drop the `Arc<T>` that is owned by the foreign-language code.
+        let foreign_arc = std::mem::ManuallyDrop::new(unsafe { Self::from_raw(v) });
+        // Take a clone for our own use.
+        Ok(std::sync::Arc::clone(&*foreign_arc))
+    }
+
+    /// When writing as a field of a complex structure, make a clone and transfer ownership
+    /// of it to the foreign-language code by writing its pointer into the buffer.
+    fn write<B: BufMut>(&self, buf: &mut B) {
+        let ptr = std::sync::Arc::clone(self).lower();
+        buf.put_u64(ptr as u64); // TODO: assertions about pointer size
+    }
+
+    /// When reading as a field of a complex structure, we receive a "borrow" of the `Arc<T>`
+    /// that is owned by the foreign-language code, and make a clone for our own use.
+    ///
+    /// Safety: the buffer must contain a pointer previously obtained by calling
+    /// the `lower()` method of this impl.
+    fn try_read<B: Buf>(buf: &mut B) -> Result<Self> {
+        check_remaining(buf, 8)?;
+        Self::try_lift(buf.get_u64() as Self::FfiType)
+    }
+}
+
+// This type exists only because `IntoFfi` is not implemented for
+// `*const std::os::raw::c_void`. It seems reasonable that it should be, so
+// if we update ffi_support to do that, this can die entirely.
+pub struct UniffiVoidPtr(pub *const std::os::raw::c_void);
+
+use ffi_support::IntoFfi;
+unsafe impl IntoFfi for UniffiVoidPtr {
+    type Value = *const std::os::raw::c_void;
+    #[inline]
+    fn ffi_default() -> Self::Value {
+        std::ptr::null_mut()
+    }
+    #[inline]
+    fn into_ffi_value(self) -> Self::Value {
+        self.0
+    }
+}
+
 #[cfg(test)]
 mod test {
     #[test]

uniffi_bindgen/src/bindings/gecko_js/gen_gecko_js.rs

@@ -332,6 +332,7 @@ mod filters {
             FFIType::Float32 => "float".into(),
             FFIType::Float64 => "double".into(),
             FFIType::RustCString => "const char*".into(),
+            FFIType::RustArcPtr => unimplemented!("object pointers are not implemented"),
             FFIType::RustBuffer => context.ffi_rustbuffer_type(),
             FFIType::RustError => context.ffi_rusterror_type(),
             FFIType::ForeignBytes => context.ffi_foreignbytes_type(),

uniffi_bindgen/src/bindings/kotlin/gen_kotlin.rs

@@ -112,6 +112,8 @@ mod filters {
             FFIType::Float32 => "Float".to_string(),
             FFIType::Float64 => "Double".to_string(),
             FFIType::RustCString => "Pointer".to_string(),
+            // XXX - make this `Pointer` (but things like Helpers.kt need upgrading in non-obvious ways. )
+            FFIType::RustArcPtr => "Long".to_string(),
             FFIType::RustBuffer => "RustBuffer.ByValue".to_string(),
             FFIType::RustError => "RustError".to_string(),
             FFIType::ForeignBytes => "ForeignBytes.ByValue".to_string(),

uniffi_bindgen/src/bindings/python/gen_python.rs

@@ -73,6 +73,7 @@ mod filters {
             FFIType::Float32 => "ctypes.c_float".to_string(),
             FFIType::Float64 => "ctypes.c_double".to_string(),
             FFIType::RustCString => "ctypes.c_voidp".to_string(),
+            FFIType::RustArcPtr => "ctypes.c_voidp".to_string(),
             FFIType::RustBuffer => "RustBuffer".to_string(),
             FFIType::RustError => "ctypes.POINTER(RustError)".to_string(),
             FFIType::ForeignBytes => "ForeignBytes".to_string(),

uniffi_bindgen/src/bindings/swift/gen_swift.rs

@@ -164,6 +164,7 @@ mod filters {
             FFIType::Float32 => "float".into(),
             FFIType::Float64 => "double".into(),
             FFIType::RustCString => "const char*_Nonnull".into(),
+            FFIType::RustArcPtr => "UnsafeRawPointer".into(), // ??
             FFIType::RustBuffer => "RustBuffer".into(),
             FFIType::RustError => "NativeRustError".into(),
             FFIType::ForeignBytes => "ForeignBytes".into(),

uniffi_bindgen/src/interface/ffi.rs

@@ -35,6 +35,10 @@ pub enum FFIType {
     /// If you've got one of these, you must call the appropriate rust function to free it.
     /// This is currently only used for error messages, and may go away in future.
     RustCString,
+    /// A `*const c_void` pointer to a rust-owned `Arc<T>`.
+    /// If you've got one of these, you must call the appropriate rust function to free it.
+    /// The templates will generate a unique `free` function for each T.
+    RustArcPtr,
     /// A byte buffer allocated by rust, and owned by whoever currently holds it.
     /// If you've got one of these, you must either call the appropriate rust function to free it
     /// or pass it to someone that will.

uniffi_bindgen/src/interface/object.rs

@@ -95,6 +95,10 @@ impl Object {
         &self.name
     }
 
+    pub fn type_(&self) -> Type {
+        Type::Object(self.name.clone())
+    }
+
     pub fn constructors(&self) -> Vec<&Constructor> {
         self.constructors.iter().collect()
     }
@@ -127,8 +131,8 @@ impl Object {
     pub fn derive_ffi_funcs(&mut self, ci_prefix: &str) -> Result<()> {
         self.ffi_func_free.name = format!("ffi_{}_{}_object_free", ci_prefix, self.name);
         self.ffi_func_free.arguments = vec![FFIArgument {
-            name: "handle".to_string(),
-            type_: FFIType::UInt64,
+            name: "ptr".to_string(),
+            type_: FFIType::RustArcPtr,
         }];
         self.ffi_func_free.return_type = None;
         for cons in self.constructors.iter_mut() {
@@ -198,7 +202,7 @@ impl APIConverter<Object> for weedle::InterfaceDefinition<'_> {
 
 // Represents a constructor for an object type.
 //
-// In the FFI, this will be a function that returns a handle for an instance
+// In the FFI, this will be a function that returns a pointer to an instance
 // of the corresponding object type.
 #[derive(Debug, Clone)]
 pub struct Constructor {
@@ -232,7 +236,7 @@ impl Constructor {
         self.ffi_func.name.push('_');
         self.ffi_func.name.push_str(&self.name);
         self.ffi_func.arguments = self.arguments.iter().map(Into::into).collect();
-        self.ffi_func.return_type = Some(FFIType::UInt64);
+        self.ffi_func.return_type = Some(FFIType::RustArcPtr);
     }
 
     fn is_primary_constructor(&self) -> bool {
@@ -282,8 +286,8 @@ impl APIConverter<Constructor> for weedle::interface::ConstructorInterfaceMember
 
 // Represents an instance method for an object type.
 //
-// The in FFI, this will be a function whose first argument is a handle for an
-// instance of the corresponding object type.
+// The FFI will represent this as a function whose first/self argument is a
+// `FFIType::RustArcPtr` to the instance.
 #[derive(Debug, Clone)]
 pub struct Method {
     pub(super) name: String,
@@ -313,7 +317,7 @@ impl Method {
 
     pub fn first_argument(&self) -> Argument {
         Argument {
-            name: "handle".to_string(),
+            name: "ptr".to_string(),
             // TODO: ideally we'd get this via `ci.resolve_type_expression` so that it
             // is contained in the proper `TypeUniverse`, but this works for now.
             type_: Type::Object(self.object_name.clone()),

uniffi_bindgen/src/interface/types/mod.rs

@@ -129,8 +129,8 @@ impl Into<FFIType> for &Type {
             // Strings are always owned rust values.
             // We might add a separate type for borrowed strings in future.
             Type::String => FFIType::RustBuffer,
-            // Objects are passed as opaque integer handles.
-            Type::Object(_) => FFIType::UInt64,
+            // Objects are pointers to an Arc<>
+            Type::Object(_) => FFIType::RustArcPtr,
             // Callback interfaces are passed as opaque integer handles.
             Type::CallbackInterface(_) => FFIType::UInt64,
             // Errors have their own special type.