LuaUserDataType shouldn't require a static lifetime

[jonas-schievink]

It's sufficient if the userdata outlives the Lua state. That way it can store references to other Lua objects or the main Lua state, which makes userdata more flexible.

[kyren]

It needs a static lifetime for TypeId, right?

Edit: I guess it no longer does now that RFC #1849 has landed. I know that practically the lifetime only needs to outlive Lua itself, but I know there are lots of dragons around TypeId and lifetimes other than static, and I might be unqualified to say whether doing this would actually be safe. I feel like it wouldn't be? It's morally equivalent to Rc<Any>, and there are LOTS of warnings here saying that basically it is impossible to use this safely with Any. Maybe it's not, because you could make the lifetime of anything you get out of it simply be 'lua?

[jonas-schievink]

Oh, you're right, good catch. Well, since the RFC isn't yet implemented I guess this will stay a wishlist issue at best, at least for the time being.

[kyren]

I think that this would also run straight into the same issues as #33. I'm a little gun-shy now about making any of the lifetimes of the transmuted types anything but 'static, so unless we come up with a general strategy to deal with that, if it even exists, we probably shouldn't do this.

[jonas-schievink]

Agreed, closing for now.

[t-veor]

Hello,
I actually just ran into this problem, where basically I would like to pass some mutable references as part of some user data to the lua script. The context is that I have a system in specs which gives me mutable references to components and I would like to let a function in the lua script modify them, by passing in some user data which exposes a bunch of getters and setters that retrieve data from and modify the components through the references.

I recognise the obvious lifetime issue here which is that the lua script shouldn't be able to hang on to the user data after the references expire, but I thought this was sort of what Lua::scope was for as it would expire values it created when it was dropped. Scope objects still require values to be UserData however, and UserData requires 'static.

It seems to me that UserData could take on some lifetime to become UserData<'a>, and then normally lua would only take UserData<'static> or UserData<'lua>s and scopes would allow UserData<'scope>s.

I've thought about some other ways to fix this which is to still produce a 'static structure that can edit the components I get from specs, but they both seem horrible. One way to do this is to keep weak references to components in the structure but this requires me to wrap literally every component I want the lua script to be able to touch in an Rc or Arc, which seems like it'd impose some overhead for other systems and require me to change a bunch of code. Or, I could have some sort of structure which can be passed to lua which takes a copy of the components and then accumulates changes passed to it, and then performs those changes on the actual components when passed back. This also seems awful.

[kyren]

@t-veor I recently tried again to lift this limitation, and though I no longer think it's impossible, I think it would require an entirely parallel userdata API. Let me explain in more detail...

The problem with non-'static UserData is that TypeId and Any or Any-like types are just sort of fundamentally unsound with non-'static types. TypeId doesn't know anything about lifetimes, and there is no reflection support for lifetime types, so there is just no way once something non-'static is placed into Lua as a userdata to tell that it's the same lifetime when it comes back out of Lua.

But, you might say, why is this restriction able to be lifted for functions via Scope and not for UserData? The reason is that the API for functions in rlua does not have to use TypeId, once you place a Fn like type into Lua, you can't get it back out again like you can with AnyUserData::borrow, so there's no need to do dynamic type checking with TypeId like there is with UserData.

HOWEVER, one could imagine a UserData API that also doesn't require TypeId, but it would have some bad properties:

• You couldn't have a registry of UserData metatables, when creating a new instance, a new metatable would have to be created every time for every new userdata, as you can't store them and look them up via TypeId.
• Once you create an AnyUserData, there would be no way to get the instance back out of the AnyUserData, because again you can't use TypeId.
• Because of the above restriction, the API for UserDataMethods would need to be redesigned to avoid using AnyUserData. You'd need a separate implementation that stored the methods differently without using the internal Callback type, and you couldn't have the "function"-style API for UserDataMethods, only the "method"-style one since you can't pass an AnyUserData to a user since they wouldn't be able to do anything with it.

So, this is sort of possible but it's a pretty involved change and the best version I can imagine still increases the API surface area quite a lot for a very small amount of gain.

But, here's some good news: You can probably already do what you want to do by encoding everything as functions rather than methods on a UserData, it's just much more inconvenient. The reason I know this is that I've actually had to deal with a very similar problem making a scripting API to "query" an ECS-like store, which required locking components and lots of non-'static types, and since I couldn't get non-'static UserData to work, I ended up making the API using callback closures instead. From a purely technical perspective I found the solution more or less fine, but I wasn't very fond of the API you end up with so I wrapped the API with additional Lua to "look" like methods on userdata rather than the magic closures that they actually were. If you're interested I can give you more details.

So, the next question is, is it worth it to make a parallel API for UserData that lifts the 'static restriction while imposing a bunch more restrictions on top of it? I'm not super sure, but I think the answer is probably not, unfortunately :( You would only be able to use that API through "scope" which is already kind of niche, you can pass types in but you can't get them back out which is surprising and limiting, it's slow because you have to create big metatables from scratch for every instance, and you can already encode the same behavior by manually creating non-'static functions (which is also slow, but at least it's more honest about how slow it is).

I'll keep thinking about how to design a reasonable API and if I come up with one I'll revisit this.