Investigate the possibility to interact with data channel in workers

[youennf]

We could try to allow processing of data channel messages in workers, without trying to solve the bigger issue of creating a data channel in a worker.
That would solve some known use cases like Zoom or streaming

[zenhack]

This is possibly related to some things I've been thinking about lately: in particular, I'd like to be able to transfer various webrtc-related objects between iframes in a page using postMessage() -- it seems like making RTCDataChannel Transferrable would solve this for workers as well.

(For my own use case I'm also interested in doing this with media streams and other objects)

[youennf]

@zenhack, could you share more of your use case?
Plan is to present something along the lines of https://docs.google.com/presentation/d/1crumgYj4eHkjo04faLktPTg0QoYJhTFoosEBudfJBuw/present?slide=id.g7957cd038b_9_43 and I could add your usecase there as well.

As of transferring MediaStreamTrack objects, this might best be tracked in a separate issue in https://github.com/w3c/mediacapture-extensions.
This might help other use cases such as use cases discussed in w3c/mediacapture-screen-share#158

[zenhack]

Sure!

This is in the context of Sandstorm. Basically, it's a platform for self-hosting web apps, and we don't want apps to be able to "phone home" to app developers without the consent of the user -- so on the server side apps are containerized and not given network access, and in the browser we use Content-Security-Policy to block most ways an app could contact the outside world (there's currently no way to do this for webrtc, which I'm trying to help solve in w3c/webappsec-csp#457).

We also run the app's page itself from a separate origin than the Sandstorm UI, inside of an iframe, with the Sandstorm UI around it. You can get a feel for it at https://demo.sandstorm.io. The iframe has the unfortunate consequence that apps can't request access to mic & camera directly.

So, I'd like to provide a way through which apps could request this access (and also controlled use of the networking portions of webrtc) from Sandstorm itself via a postMessage() API; we already use this mechanism for requesting access to server-provided resources, so it would be a natural extension: https://docs.sandstorm.io/en/latest/developing/powerbox/

With that in place, Sandstorm can request access to resources from the browser itself, and then mediate apps' access to them. Note that if/when the CSP change is implemented, Sandstorm itself can still create webrtc connections, since it is served from a different origin than the apps, with a different (more relaxed) CSP.

[jesup]

I proposed this quite a few years ago in the WG - making datachannels transferrable so that apps could process and respond to network input without waiting on the mainthread. A good example is a game with a datachannel connection to a server and/or to each other player. Game state maintenance could occur in a worker, while mainthread is frequently busy with user input and rendering.

Similarly, we discussed possibly transferring mediastreams, and possibly already-connected connections

[jimmywarting]

I wish to be able to construct a WebRTC datachannel in (service) workers rather then having to get them from some main page as transfered

[trezy]

I'll provide my use case here in case this presentation hasn't been given yet. I'm working on a browser-based MMORPG. To make the game as efficient as possible, I'm working to move everything except for rendering off of the UI thread and into workers.

My current work is centered around moving network communications into their own worker. My Worker has been processing messages from WebSockets, then using a BroadcastChannel to relay any updates to the UI thread. While WebSockets are great, TCP connections have plenty of well-documented issues in games networking. I'm trying to move to using a much better UDP connection via the Geckos.io library. The problem I'm encountering is that @yandeu's library requires RTCPeerConnection to be available. If we added the ability for an RTCPeerConnection to be transferred to a Worker, then I could initialize the Geckos.io library in the UI thread, then pass it off to my network communications Worker to replace what is currently a WebSocket connection.

[youennf]

I did a prototype implementation in WebKit and have the following implementation feedback:

• If the peer connection that created the original data channel closes, the data channel gets closed.
• Transferring a data channel while the data channel is sending data is probably doable but a bit painful, especially when sending blobs. We could disallow transferring data channels if they are in 'open' state, or if their bufferAmount is not zero. Current prototype restricts transferring data channels to 'connecting' state at the moment.
• Transferring a data channel multiple times should be fine. If an intermediate context goes away, this might not trigger closing the data channel (contrary to streams currently).

[lgrahl]

Cool!

If the peer connection that created the original data channel closes, the data channel gets closed.

That should only really affect service worker use cases, right?

Current prototype restricts transferring data channels to 'connecting' state at the moment.

Since setting the connecting state is queued in a task, sending the data channel should still be possible directly after the createDataChannel call. If so, that would be a pragmatic and simple solution even when the SCTP association is already up.

[youennf]

That should only really affect service worker use cases, right?

That might also affect cases where you transfer a channel to another window, though WebKit prototype does not yet support out-of-process channels.

Since setting the connecting state is queued in a task, sending the data channel should still be possible directly after the createDataChannel call

Right, ditto for datachannel event.

[steely-glint]

Great news!
(Looking forward to a chance to experiment with that!)
Does that mean you can only transfer channels you created (as opposed to ones that were created by your peer)?

[youennf]

Does that mean you can only transfer channels you created (as opposed to ones that were created by your peer)?

You can transfer any data channel, either ones you get from createDataChannel or ondatachannel events, provided they are in connecting state.

[steely-glint]

But how would one catch an inbound channel that was still connecting ? What event could you wait for? By the time you know the label it is already open I think.

[youennf]

Hum, you are right. The idea is that it would be fine to at least transfer the datachannel in the datachannel event handler.

[steely-glint]

In a lot of cases it won't matter, you mostly want to transfer channels where you control the lifecycle, but it would make some P2P usages inconveniently asymmetric.

[lgrahl]

OOB negotiated data channels would work fine, just IB negotiated data channels would not. Perhaps an alternative would be to only allow transfer of data channels where no event handler has been attached and send has not been called.

Hum, you are right. The idea is that it would be fine to at least transfer the datachannel in the datachannel event handler.

Or that. IIRC the channel is marked open at that point but announcement is still pending. If that makes any difference for the implementation. 🙂

[youennf]

Right, we have a few options:

1. Transfer a data channel at creation time: easy non-main-thread blocking implementation, ensure all events can be dispatched to the application.
2. Transfer a data channel if app is not listening and not writing: easy non-main-thread blocking implementation, some events can be lost but app does not seem to care since it is not listening.
3. Transfer a data channel whatever the state while ensuring proper event dispatching and ordering: easy implementation if we rely on main thread to do the synchronisation, but we probably do not want that. More difficult implementation otherwise.

WebKit prototype is doing 1.
It might be good to get feedback on how much 2 would be more appealing.
From the feedback I received so far, 3 does not seem worth the effort.