Rust functions that can be called from C

[brson]

We have a lot of scenarios now where people creating bindings want to be able to provide a callback that a C function can call. The current solution is to write a native C function that uses some unspecified internal APIs to send a message back to Rust code. Ideally it involves writing no C code.

Here is a minimal solution for creating functions in Rust that can be called from C code. The gist is: 1) we have yet another kind of function declaration, 2) this function cannot be called from Rust code, 3) it's value can be taken as an opaque unsafe pointer, 4) it bakes in the stack switching magic and adapts from the C ABI to the Rust ABI.

Declarations of C-to-Rust (crust) functions:

crust fn callback(a: *whatever) {
}

Getting an unsafe pointer to a C ABI function:

let callbackptr: *u8 = callback;

We could also define some type specifically for this purpose.

Compiler implementation:

It's mostly straightforward, but trans gets ugly. In trans we will need to do basically the opposite of what we do for native mod functions:

• Generate a C ABI function using the declared signature
• Generate a shim function that takes the C arguments in a struct
• The C function stuffs the arguments into a struct
• The C function calls upcall_call_shim_on_rust_stack with the struct of arguments and the address of the shim function
• Generate a Rust ABI function using the declared signature
• The shim function pulls the arguments out of the struct and calls the Rust function

Runtime implementation:

The runtime has to change in a few ways to make this happen:

• A new upcall for switching back to the Rust stack
• Tasks need to maintain a stack of Rust contexts and C contexts
• Needs a strategy to deal with failure after reentering the Rust stack
• Needs a strategy to deal with yielding after reentering the Rust stack

Failure:

We can't simply throw an exception after reentering the Rust stack because there's no guarantee the native code can be unwound with C++ exceptions. The Go language apparently will just skip over all the native frames in this scenario, leaking everything along the way. We, instead will abort - if the user want's to avoid catastrophic failure they should use their Rust callback to dispatch a message and immediately return.

Yielding:

Without changes to the way we handle C stacks we cannot allow Rust functions to context-switch to the scheduler after reentering the Rust stack from C code. I see two solutions:

1. Yielding is different after reentering the Rust stack and simply blocks. Tasks that want to do this should make sure they have their own scheduler (Implement scheduler domains #1721).
2. Instead of running native code using the scheduler's stack, tasks will check out C stacks from a pool located in each scheduler. Each time a task reenters the C stack it will check if it already has one and reuse it, otherwise it will request a new one from the scheduler. This would allow Rust code to always yield normally without tying up the scheduler.

I prefer the second option.

See also #1508

[beoran]

Sorry to jump in here, but I'd like to stress that calling C functions should be fast, as in blazing fast. If I want to write a game in rust using a C library like Allegro, SDL, or Opengl, this is essential. Otherwise the game will slow down in the rendering code where there are a lot of C calls, which is unacceptable. The default Go language compiler with cgo has such problems.

So I'd prefer a solution that is fast, even though it may restrict what the function on the Rust side can do.

Also, wouldn't it be an idea to use "native fn" in stead of "crust fn" or does that have another planned meaning?

[nikomatsakis]

@beoran you have this backward. We're talking about C calling Rust. Calling C functions from Rust is already pretty fast (could be made somewhat faster).

[beoran]

Ok, I see. Is there any way I can help with speeding up calling C from rust?

[nikomatsakis]

@beoran as I said, it's kind of orthogonal to this issue... but probably the best thing you could do is make up a benchmark showing how the performance is inadequate. :)

[beoran]

OK, I will do that when I get far enough in wrapping Allegro to compare the overhead of calling it from rust Rust with calling it from C. I'll leave this issue alone for now and I'll open a new one once I have the benchmark.

[pcwalton]

Could we avoid one of the copies of the arguments by having the C function write directly to the Rust stack, as Rust->C calls currently do?

[brson]

I'm hoping that the final copy of the arguments out of the shim struct and into the arguments of the rust function will be eliminated through inlining. I'm not sure if that's what you are referring to though.

I believe that our C calls currently copy the arguments into a struct on the Rust stack then copy that struct to the C stack.

[nikomatsakis]

@pcwalton Rust->C do not currently write directly into the C stack because that is specific to i386. I wanted to avoid having to write code that was specific to a particular calling convention, even at the price of some performance, in order to get 64-bit working. (I think such optimizations might make sense now, though---particularly as #1402 points out that LLVM doesn't really handle the calling conventions completely anyway)

[brson]

After reading the stack switching function, the arg struct isn't copied across stacks at all, the pointer to the previous stack is just passed to the function that runs on the new stack, which makes perfect sense.

[nikomatsakis]

The arg struct is not copied, but the shim function will load values out of it and re-push them onto the new stack. The old code used to literally write the argument values directly into the target stack. This made sense on i386 but on x86_64 it's much more complex to figure out which values will go on the stack etc.

[brson]

After some testing I've found that it is going to be very difficult for crust functions to guarantee that they don't fail. What currently happens is that, when a top-level task (like main) fails every task is told to fail, so as soon as the callback tries to send a message (or when it returns from sending a message) it can end up failing and causing the runtime to abort abnormally.

[brson]

I guess we can change rust_task to ignore kill requests once tasks have reentered the rust stack. For tasks implementing event loops they can peek at some monitor port looking for a message indicating that the runtime is failing and figure out how to terminate gracefully.

[nikomatsakis]

So when a top-level task fails, it propagates errors to its children? I guess I don't understand our error propagation model, I thought it went from the leaves up. It seems like code running on C stacks ought to be able to run in an unsupervised task or something like that.

[brson]

It basically acts as if 'main' is supervised by the kernel, so if main fails everything fails.

[brson]

I'm calling this done. There's a little cleanup, and I've filed separate bugs for remaining issues.

[alexchandel]

Please forgive this resurrection, but this issue is linked from a y-combinator piece and a few other sites, and I recently had a noob ask about it, so I'm noting that, per this issue & later changes, calling Rust from C is simple:

#[no_mangle]
pub extern fn hello_rust() -> *const u8 {
    "Hello, world!\0".as_ptr()
}

#include "stdio.h"
const char *hello_rust(void);
int main(void) {
    printf("%.32s\n", hello_rust());
}