MSE in Workers

[wolenetz]

Request for Mozilla Position on an Emerging Web Specification: MSE-in-Workers feature addition to MSE in the MSEv2 specification.

• Specification Title: MSE-in-Workers (feature addition to the MediaSource Extensions Specification)
• Specification or proposal URL: MSE-in-Workers draft feature specification w3c/media-source#282
• Caniuse.com URL (optional):
• Bugzilla URL (optional):
• Mozillians who can provide input (optional):

Other information

This feature has previously been discussed at HTML Media WG F2F and FOMS conferences as early as Fall 2018 and March 2019, with Mozilla representatives present. It expands the availability of the MSE API to be useable from a DedicatedWorker context, and retains the existing MediaSource objectURL idiom/mechanism for attaching/extending an HTMLMediaElement for use with the MediaSource API. Chromium has an experimental implementation and there is a demo [1] of the feature and how it can effectively improve media buffering and playback performance especially when the main Window context is under contention.

[1] https://wolenetz.github.io/mse-in-workers-demo/mse-in-workers-demo.html

[wolenetz]

The associated MSE feature-tracking issue in the MSE specification's repository is w3c/media-source#175

[annevk]

It seems rather unfortunate that this further cements URL.createObjectURL() rather than building on media.srcObject in some way. It's rather easy to create memory leaks with the former.

cc @padenot @jan-ivar

[jan-ivar]

@annevk I agree. Oddly, the spec mentions the url hack, but doesn't mention that MediaSource can already be directly assigned to srcObject, which is the superior way that should be encouraged (an oversight?). It looks like only Safari implements srcObject correctly here so far, maybe that's a factor?

Why isn't MediaSource Serializable? (copy semantics seem preferable here over transferable).

I'd consider this a minor issue however, as urls would probably need to work too, and I wouldn't block on removing urls. Overall I'm supportive of MSE in workers and welcome it. Cc @youennf

[annevk]

Unless I'm missing something, the current design for MediaSource in workers depends on URL.createObjectURL(), so unless that is addressed we'll further spread it. And since MediaSource isn't exposed in workers, we could make URL.createObjectURL() in workers throw for it once we expose MediaSource there. It's somewhat minor relative to the entire feature, but I would like to see it addressed.

[johnBartos]

I'd like to put in a big +1 on behalf of Twitch. I believe we'll see a very significant drop in rebuffering, especially during periods of main thread contention like the initial page load. We have already implemented this thanks to @wolenetz 's efforts in Chrome, and I'd love to see this in FF as well (and Safari, but one can dream).

[wolenetz]

The initial approach in the explainer for MSE-in-Workers was to use a transferable handle [0], usable via createObjectURL or srcObject, but feedback from Apple [1] was to simplify to not use transferring of any MSE object and instead continue to use object URL of a MediaSource object constructed in a worker. This was discussed further at the face-to-face (including Mozilla, Apple, MSFT and Google representatives) adjacent to FOMS in March of 2019. The absence of objection to continuing to use object URLs until now motivated using the simplified object URL attachment semantic. It is not exclusive; adding srcObject for both main and worker attachments can be done separately. Furthermore, adding a transferable handle object (to possibly reduce cold start attachments to worker MSE) and new methods like "MediaSource.open()" are not in conflict with the current approach. Regarding transferability/serializability of MediaSource, it was also discussed prior to the original handle-based proposal, and was deemed infeasible (first mentioned in [2]) due to MediaSource being an event target, and lack of formal clarity on how to transfer objects that are event targets. Hence, the original proposal used a transferable handle instead. [0] w3c/media-source#175 (comment) [1] w3c/media-source#175 (comment) [2] w3c/media-source#175 (comment) In short, the originally proposed transferable handle was requested to be simplified to follow existing object URL attachment semantic, then discussed with no objection throughout the design and prototyping stage until now. Eventual deprecation of main-thread and worker-thread MediaSource object URL creation could still be done at a later time; though the semantic is well understood and widely prevalent since it is the only way currently to use the MSE API in multiple user agents.

…

On Wed, Jul 28, 2021 at 8:17 AM John Bartos ***@***.***> wrote: I'd like to put in a big +1 on behalf of Twitch. I believe we'll see a very significant drop in rebuffering, especially during periods of main thread contention like the initial page load. We have already implemented this thanks to @wolenetz <https://github.com/wolenetz> 's efforts in Chrome, and I'd love to see this in FF as well (and Safari, but one can dream). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#547 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACYK3Q4DAYUEWAOVPLDYF4TT2ANPHANCNFSM47TFENBA> .

[karlt]

Use of object URLs across threads/realms would introduce new problems that are not present with use in a single realm.
Some of these issues were raised previously.

There is also a potential race at creation of the URL. I guess the implementation is expected to somehow pass meaning for the URL to Window scope prior to receiving any postMessage() message from the worker? Content can also pass URLs via shared memory.

Is the set of Window scopes that must associate a URL with a Worker MediaSource after or during createObjectURL() defined?

IIUC the original MediaSourceHandle approach involved creating the MediaSource in Window scope and transferring a handle to Worker scope, which handle had the complexity of a number of methods and/or attributes to be replicated to behave as a MediaSource.

Has consideration been given to a transferable object that would solely pass a MediaSource identifier from Worker to Window?
Wouldn't an explicit transfer of a MediaSource reference be simpler than having to specify and implement a new mechanism for when an object URL might have meaning in different realms, including when such meaning would expire?

[wolenetz]

Note that using srcObject for MSE-in-Workers attachments is a change to the feature that the media wg arrived at last September [1], and I'm working on getting that into both the feature spec and Chromium's experimental implementation currently.

[1] https://www.w3.org/2021/09/14-mediawg-minutes.html

[wolenetz]

@#547 (comment) -

Has consideration been given to a transferable object that would solely pass a MediaSource identifier from Worker to Window?

This is the approach I'm currently considering: since the MediaSource itself isn't transferrable as it's an EventTarget, and since the experiment with worker-originated objectURLs has already shown significant improvements, having the attach object be constructable in a dedicated worker for worker-originated-then-transferred-to-main for use in srcObject attachment is the precise mechanism I'm considering now (to more readily replace the existing objectURL worker attachment mechanism).

Other ways of using srcObject (e.g. main-thread-only MSE attachments, or even main-thread origination of the transfer) are out of scope of this work but are not precluded.

-edited for format

[wolenetz]

Transferable MediaSourceHandle settable on main thread media element srcObject, and obtained via [SameObject] readonly MediaSource.handle has landed in the MSE specification following discussions here and especially in MSE spec PRs w3c/media-source#305 and w3c/media-source#306. I believe I've addressed all concerns.

[wolenetz]

With the specification, explainer and demo all updated now, please revisit this official request for position. Thank you.