MediaSource in Worker

[DanielBaulig]

The Videos team at Facebook has a growing need to move calls to Media Source Extensions off of the main thread and into a Worker. While most parts of your off-the-shelve streaming video player can be implemented in a Worker (fetching and parsing of the manifest as well as selecting and fetching of segments) the calls to and interactions with the Media Source Extensions API (e.g. SourceBuffer.prototype.appendBuffer, SourceBuffer.prototype.remove, etc) remain a bottleneck and have to happen on the main browser rendering thread.

On a sizable and complex web site large blocks of JS execution can regularly block the main thread making videos slow to start playing and causing lots of rebuffering, especially in live videos where you can only buffer ahead very little data. To ensure great web video performance in the long run, even as web apps continue to grow in size and complexity, we believe we need to allow video players to move into Workers and off of the main thread.

We would like to propose this change to the working group and involve other API users as well as implementors in the discussion.

[bmaurer]

#14 also seems relevant here -- based on my reading of that bug it would help with part of this problem (once you fetch a segment you'd pass a stream to that audio to MediaSource and would bypass the main thread, or even a worker thread in terms of getting that data to where it needs to go).

But that still leaves the issue of fetching a manifest and initiating a request for a segment

[jyavenard]

the calls to and interactions with the Media Source Extensions API (e.g. SourceBuffer.prototype.appendBuffer, SourceBuffer.prototype.remove, etc) remain a bottleneck and have to happen on the main browser rendering thread.

That it's a bottleneck would be mostly a user agent implementation issue.
Those APIs are designed to be asynchronous, they should take no time at all and not block the main thread.
With current Firefox, running those two operations in a worker would make little (or no) performance difference, they are in effect running internally in a worker.

I wish issue #100 progressed further, it would certainly facilitate handling those operations.

[bmaurer]

The issue here isn't the cost of video on the main thread. The issue is that if the main thread is busy with other work (say the javascript needed to render a page) that it can not be responsive to input from the network. This means that the video will be slower to buffer even though the client has been able to get the video over the network.

The goal is for the video loading code to be isolated on a worker thread where it can operate in real time without the risk of latency induced by work on the main thread.

[zhaoz]

We are seeing the same issue on YouTube and I'd like to add our voice to support a solution here as well. Moving append buffer off the main thread would greatly simplify the juggling we need to do to prioritize video playback.

[bmaurer]

It's great to see that youtube has the same take on this issue. Would it be worth taking the next step here and drafting out what a spec change would look like? I think we can model this on the offscreen canvas api (https://hacks.mozilla.org/2016/01/webgl-off-the-main-thread/)

var canvas = document.getElementById('myCanvas');
var offscreen = canvas.transferControlToOffscreen();
var worker = new Worker('worker.js');
worker.postMessage({ canvas: offscreen }, [offscreen]);

Essentially MediaSource would offer an API which would pass control of the media source APIs to a worker thread. Any APIs that MediaSource offers (eg to add source buffers) would only be callable from the worker.

MediaSource is a fairly complex API with a number of events / methods. It'd be great to get some feedback from @zhaoz, @DanielBaulig as well as folks who work on browsers and this spec to comment on what kind of threading issues might come up with this API.

[DanielBaulig]

The main limitation with transferring MediaSource object that I can currently see is that in general most video player implementations heavily depend on certain video node attribute for determining which video segments to fetch and play. These are as far as I can tell primarily video.currentTime, video.paused and video.buffered.
The main thread will need to regularly update the worker with the current values of these for most video player implementations to be able to operate correctly.

However, the API itself could have support for this: the worker could get copies of these values on every event loop exposed. This exposure could happen as part of the transferred MediaSource object, e.g. on a bag of exposed values or directly on the object:

onmessage = function(message) {
  if (message.mediaSource.mediaElementState.paused) {
    // don't do anything while the video is paused
    return;
  }
};

Another problem may be that the main thread indirectly retains a reference to the MediaSource as an object URL on the videos src attribute. I am not sure how this will complicate the safe usage of object URLs.

[bmaurer]

That makes total sense. It does seem like the MediaSource API should be responsible for relaying that information -- accessing it directly from the video element seems like it violates layering and would make the threading far more difficult here.

One other thing we should think about -- maybe this is a totally separate issue -- I worry about the latency of starting the worker, downloading the necessary JS, etc. On Facebook we've found that the amount of JS necessary to have the logic for streaming the video can be pretty high. I wonder if there's a way to allow the server to list the URLs necessary to start streaming so that they can be fetched and inserted into the buffer while the more advanced control starts up.

[DanielBaulig]

One other thing we should think about -- maybe this is a totally separate issue -- I worry about the latency of starting the worker, downloading the necessary JS, etc. On Facebook we've found that the amount of JS necessary to have the logic for streaming the video can be pretty high. I wonder if there's a way to allow the server to list the URLs necessary to start streaming so that they can be fetched and inserted into the buffer while the more advanced control starts up.

I don't think there's anything preventing video player implementations from doing something like this currently. That said, in the context of a transferable MediaSource, this becomes a little harder. This could potentially mean creating a MediaSource object and several sourceBuffers on the main thread, appending data to the sourceBuffers and only later transferring ownership to the Worker. This transfer won't be simple anymore though since there's a complex mix of objects and possibly tasks that has to be dealt with.

[jyavenard]

One of the issue in dealing with a sourcebuffer in a worker thread, is how, currently, the source buffer notifies that the current operation has completed: through events fired on the attached media element and which is only ever accessed on the main thread.

So if your main thread is busy, no update/updateend event and you're back to the issue of not being able to keep up with your source buffer.

This problem could be easily worked around if rather than relying on events such as update/updateend to know when you can feed new data to the source buffer we used Promises.
I opened an issue a while back to implement this:
#100

The beauty with SourceBuffer::appendBuffer and SourceBuffer::remove with returning Promises is that it would be fully backward compatible. Once the appendBuffer operation has completed, resolve the Promise and queue update/updateend
Up to the player to use one mechanism or the other, never both.

Making a SourceBuffer usable on a worker thread wouldn't be a too complicated task (famous last words) in gecko, the architecture is already asynchronous and thread-safe.
The interaction with the media element would be the issue.

But if we limit the interaction to the source buffer to appendBuffer and remove operations, and checking of the buffered range to be performed on the source buffers alone, and we use Promises then it should be straight forward.

[wolenetz]

@#175 (comment), 'updateend' is fired on the SourceBuffer , not HTMLMediaElement. However, readyState transitions (and consequent events like 'loadedmetadata') would occur on the media element on the main thread.

I'm less clear that Promises are required for MSE-on-SW, though I'm not opposed (I just don't want to tangle the two features).

[wolenetz]

Here's a summary of this issue so far, from my understanding. I have a set of questions, below, for which web author input would be greatly appreciated.

Proposals in [2] and [3] help to limit the scope of this to be specific to just MSE, and not require exposing all of HTMLMediaElement API also to Service Worker (SW):

[2] Proposes that the MediaSource API be made “transferrable” to a SW
[3] Adds that major HTMLMediaElement attributes necessary for MSE API usage be periodically copied from the main thread to SW (perhaps as part of the transferred MediaSource object).

[1] #175
[2] #175 (comment)
[3] #175 (comment)

Questions:

• Should the “transfer” be allowed on a MediaSource object which has already had SourceBuffers added to it? (This might allow for initial buffering to occur on main thread on page load, while the SW context is still spinning up.) (I’m not clear this will help so much in practice, as some web players already experience high time to first frame due to other activities contending on the main thread immediately on page load.)
• Is there potential “safe usage of Object URLs” problem with transferring the MediaSource context to a SW? (I would defer to web platform security experts. I suspect that a set of reasonable restrictions/checks on the transfer may be needed. Any suggestions?)
• What portion of the MediaSource and SourceBuffer API need to be transferred?
• What portion of the HTMLMediaElement state need to be periodically copied into the transferred MediaSource’s SW context?
• Precisely when should such state be copied?
• Are any HTMLMediaElement events needed in the SW context? (This would increase complexity; I’m not certain any are needed in SW context, since the web app could forward these -- though with consequent potentially high latency -- into the SW context itself.)
• Once transferred to SW, would the original MediaSource be restricted from access from any other context?
• I assume the HTMLMediaElement itself must provide continued access; though would queries of it like “buffered”, which MSE extends, need to round-trip into the SW context?

-- edit: substitute "Worker" for "Service Worker" in this post, please. I had conflated them.

[DanielBaulig]

Am I assuming correctly that you are using the term Service Worker and the general Worker API interchangeably?

I'll try to answer most of the questions for which I believe I have enough context:

Should the “transfer” be allowed on a MediaSource object which has already had SourceBuffers added to it? (This might allow for initial buffering to occur on main thread on page load, while the SW context is still spinning up.) (I’m not clear this will help so much in practice, as some web players already experience high time to first frame due to other activities contending on the main thread immediately on page load.)

This is probably not necessary. The sourceopen event takes some time to fire and the application probably needs time to load data to append to the source buffers too. If those steps take significantly less time than spinning up the Worker then there may be some room for optimization, but I believe that this is likely better done by improving Worker startup time, than introducing bloat here. This obviously assumes that we do not have to wait for the "sourceopen" event to fire before we can transfer the MediaSource object.

Is there potential “safe usage of Object URLs” problem with transferring the MediaSource context to a SW? (I would defer to web platform security experts. I suspect that a set of reasonable restrictions/checks on the transfer may be needed. Any suggestions?)

I am not sure I can answer this question.

What portion of the MediaSource and SourceBuffer API need to be transferred?

We've spent some time internally thinking about this and we came to the conclusion that the entirety of the SourceBuffer API is probably the minimally viable subset of the API for the application.

Having the MediaSource object itself available is preferred but not necessarily required. We believe though due to the amount of dependencies between the MediaSource object and it's SourceBuffer objects, that transferring the MediaSource object is likely the easier approach from a spec and implementation perspective.

We'd love to hear some thoughts from implementors on this though.

What portion of the HTMLMediaElement state need to be periodically copied into the transferred MediaSource’s SW context?

Are any HTMLMediaElement events needed in the SW context? (This would increase complexity; I’m not certain any are needed in SW context, since the web app could forward these -- though with consequent potentially high latency -- into the SW context itself.)

The Worker would generally require access to information that is helpful in making decisions on how much, how fast and what kind of data to buffer. Generally this could be the current playhead (currentTime), the current state of the video buffer (buffered attribute) and current payback rate (playbackRate), if audio is audible (muted and volume) and probably more.
The exact information here however is highly dependent on the details and sophistication of the application and probably highly vary from application to application. Note also, that a small delay in updating the Worker with this information should not lead to any significant problems for the application. In fact, there are plenty of applications and scenarios conceivable in which no state synchronization at all would be required. I think generally the same is true for any HTMLMediaElement events.

For those reasons and to make the specification and implementation process easier we would probably suggest to leave this synchronization completely up to the application. The application can use postMessage to update any necessary state in the Worker as required.

Once transferred to SW, would the original MediaSource be restricted from access from any other context?

I think that is the current understanding.

I assume the HTMLMediaElement itself must provide continued access; though would queries of it like “buffered”, which MSE extends, need to round-trip into the SW context?

This seems to be more of an implementation / specification related question that I am not entirely sure I am able to give feedback on. My understanding here however is that the buffered attribute both for sourceBuffer as well as the video node already need information from the (already) asynchronous and in a different thread running process of media decoding.

[wolenetz]

Thanks for your answers - those do help describe the possible feature scope more precisely. w.r.t.:

Am I assuming correctly that you are using the term Service Worker and the general Worker API interchangeably?

You're indeed correct - I had mistakenly constrained the scope too far :). Unless the general Worker API is too broad, let's target it, not just Service Workers.

[wolenetz]

@#175 (comment):

My understanding here however is that the buffered attribute both for sourceBuffer as well as the video node already need information from the (already) asynchronous and in a different thread running process of media decoding.

Decoding itself doesn't impact buffered ranges (unless there's a decode error and transition of MediaSource readyState to closed.) Rather, operations on the MSE API itself can change the buffered ranges. What I was trying to get at is: what are the potentially required synchronization points between the HTMLMediaElement attached by a transferred MSE worker context? For example:

• HTMLMediaElement.buffered and duration: If an app synchronously queries the 'buffered' attribute of such an HTMLMediaElement, yet the MSE in worker context is in the process of, or perhaps has recently completed, a SourceBuffer appendBuffer(), SourceBuffer remove(), SourceBuffer mode or timestampOffset change when previously 'ended', MediaSource endOfStream(), or MediaSource set duration operation, is there some guarantee of synchronization of the result of such operation that needs to be handled? What should such a guarantee look like? Note that actual playback/decode doesn't factor into this (buffered time range updates due to appendBuffer()/remove()/duration setting don't require actual decode). I assume there needs to be some specified synchronization of such results; in non-worker/current case, if the SourceBuffer.updating attribute is false across all SourceBuffers in the attached MediaSource, then the buffered range query on the parent media element should be stable and correct. Is some similar guarantee required in the worker case?

• HTMLMediaElement.seekable and set/clearLiveSeekableRanges (also involves HTMLMediaElement.buffered state): ditto: should there be some guarantee of synchronization with the worker before a query of seekable can be satisfied?

• Worker MediaSource.buffered (potential new API method): Note that the worker context can itself inspect and intersect manually the SourceBuffer buffered ranges to mimic what the parent element's logic would do. Should the transferred MSE API provide such copy of the parent media element's logic that can be processed without synchronizing state with the parent media element?

[bmaurer]

These are great questions. I think my intuition is that generally the worker thread which holds the MSE will be communicating with the main thread, eg via postMessage. I think it would make sense to enforce logical causality. For example, if you call appendBuffer, then send a message to the main thread the result of that appendBuffer should be observable when processing the send message.

I agree that MediaSource.buffered probably makes sense -- there's no need to query the main thread for this information.

[wolenetz]

For example, if you call appendBuffer, then send a message to the main thread the result of that appendBuffer should be observable when processing the send message.

To be clear, I assume you mean more like "if you call appendBuffer, and upon later receiving updateend you send a message to the main thread, the result of the appendBuffer should be visible on the main thread when processing the receipt of that message." Please correct me if I misunderstand :)

[wolenetz]

Also, does it make sense to use such a transferred MSE API from a Service Worker context (I don't think so, because such a context is independent of a web page, and therefore from a DOM containing a media element). ISTM, this makes sense from a web worker context, not from SW context.

[bmaurer]

To be clear, I assume you mean more like "if you call appendBuffer, and upon later receiving updateend you send a message to the main thread, the result of the appendBuffer should be visible on the main thread when processing the receipt of that message." Please correct me if I misunderstand :)

yep, exactly.

Also, does it make sense to use such a transferred MSE API from a Service Worker context (I don't think so, because such a context is independent of a web page, and therefore from a DOM containing a media element). ISTM, this makes sense from a web worker context, not from SW context.

By worker I meant web worker here

[DanielBaulig]

I agree with what @bmaurer said: maintaining logical causality with postMessage probably makes sense.

If we are considering replicating the buffered HTMLMediaElement extension on MediaSource - which I totally think makes sense -, we should probably do the same thing for seekable.

[wolenetz]

Thanks for your responses to my questions. Here's my newbie-to-workers question: can a Worker do a fetch that is intercepted by a Service Worker, thereby enabling a combination of offline MSE-in-Worker being fed by data provided by the Service Worker? Where are known gaps in spec or implementation to accomplish such a scenario (other than, of course, this issue's MSE-in-Worker discussion :) )?

[kentuckyfriedtakahe]

I'm with @jyavenard thinking that adding #100 while workerising SourceBuffer would be worthwhile. It may be timely because the worker codepath may end up being a fork of the non-worker path. I'd like to hear from @bmaurer and @zhaoz about whether this will make the player implementation simpler or better.

[bakulf]

I would like to share my point of view about the transferring of MediaSource and SourceBuffer objects to workers. If I have understood the spec correctly, a normal use of a MediaSource is via HTMLMediaElement.src = URL.createObjectURL(MS). The output of URL.createObjectURL() is a DOMString, fully cloneable via postMessage().

Would be probably cleaner to expose MediaSource and SourceBuffer constructors to workers.
A worker can create a MediaSource and SourceBuffer(s) then, simply send a blob URL to the window: worker.onmessage = e => { mediaElement.src = e.data; }

The change I propose is mark MediaSource interface in this way:

[Constructor, Exposed=(Window,Worker)]
interface MediaSource {...

same for SourceBuffer.

Note that SourceBuffer supports the appending of ArrayBuffer and ArrayBufferView. Both are already transferable to workers.
Another important point, maybe the most important, so far, EventTargets are not transferable to workers because EventHandler cannot be handled correctly when the object is transferred.

[alastor0325]

Agree with comment, it would be easy if we restrict the media source and source buffer could only be used in one thread all the time and disallow them being transferrable.

If they’re transferable, how should the behavior of event handlers be defined? It seems a difficult problem, and the spec doesn't mention anything about it.

eg.
If I transfer an media source before its “onsourceopen” be called, which thread the event should be dispatched?
If the source buffer can also be transferred, whether we can transfer multiple source buffers to multiple workers?
…e.t.c

—

If what I understand it correctly, the usage scenario would like the following.

First, we create both media source and source buffer on the WT, and send the URL back to MT after wrapping media source as URL. After the media source is attached successfully to media element, “sourceopen” would be triggered on WT. Next, MT should send the video URL to WT, then all things could be done independently inside the WT from fetching data to appending data into buffer (and with some new API for getting the status of media element).

Please correct me if I was wrong and welcome to propose other possible scenarios.

[DanielBaulig]

I see the intentions here and agree that this would make several aspects easier. The only concern I have is that this would add a significant amount of additional startup time.

On top of the time it takes to complete sourceopen, now we may also have to wait for a Worker to spin up and a message to be sent form the Worker to the main thread (which may be busy doing other stuff). All these things will have to happen in sequence and it will require two dedicated time slices on the main thread to complete.

In a scenario with a transferable MediaSource, waiting for sourceopen and the bootstrapping of the Worker can happen in parallel and the Worker will not have to round trip back to the main thread to complete the transaction reducing the main thread time slices require to one.

The biggest risks associated with moving MSE into Workers from a product perspective is probably the startup latency. The solution as suggested in #175 (comment) increases this risk significantly.

[alastor0325]

This might be a stupid question, why couldn't we bootstrap worker in advance, before starting playing video?

Can we bootstrap worker in the beginning? and then use postMessage() to notify worker to return the URL of media source when we would start MSE soon?

In a scenario with a transferable MediaSource, waiting for sourceopen and the bootstrapping of the Worker can happen in parallel and the Worker will not have to round trip back to the main thread to complete the transaction reducing the main thread time slices require to one.

If we can start worker in advance, the following might be the possible use case. In this case, MT only need one time slice to handle the message sent from WT.

worker.postMessage(url /* used for fetching on WT */);
worker.onMessage = e => { mediaElement.src = e.data; };

[alastor0325]

The main problem to make media source transferable is "how to handle the event handler of transferable object", because it's undefined behavior.

I think this topic is out of the media scope, and I would like to know how strong intention we want to go with this direction? If do so, we need to discuss and modify with worker's working group and make sure the behavior can be well defined.

However, it might be take a long time, and @bakulf, who is our worker expert, also suggest don't try to transfer an object with event handler.

[DanielBaulig]

@alastor0325 We can only bootstrap the Worker ahead of time, if we have an opportunity to do so. If the first thing we are doing on a blank page load is to create and assign a MediaSource, there's no way for us to bootstrap a Worker ahead of time.

In addition to that, creating the object URL on the Worker thread will require us to touch the main thread twice. Consider the following example:

    /* main thread code */
    // first main thread execution
    const video = document.querySelector('video');
    const worker = new Worker('mse_worker.js');
    worker.addEventListener('message', (event) => {
      // second main thread execution
      video.src = event.data;
    });

But if we can create the MediaSource on the main thread, we only need one time slice on the main thread:

    /* main thread code */
    // first main thread execution
    const video = document.querySelector('video');
    const worker = new Worker('mse_worker.js');
    const ms = new MediaSource();
    const url = window.URL.createObjectURL(ms);
    video.src = url;
    worker.postMessage(ms, [ ms ]);

We don't need a second time slice on the main thread before we can start loading video.

Talking to some folks at Mozilla, they wanted to look into why the sourceopen event even is required/asynchronous in the first place.

Another option could be to require transferring of the MediaSource to the Worker before the URL assignment happens. In that case any events firing on the MediaSource could be deterministically scheduled to execute on the Worker instead of the main thread.

Lastly, I could imagine something TransferableMediaSource, that internally does all the steps MediaSource would do upon creation, but doesn't expose any external API interface except for a getMediaSource() method. After the method was called, the TransferableMediaSource looses it's ability to be transferred.
This could be created on the main thread, assigned to the video node's src through an Object URL and then transferred to the Worker, where the Worker would call getMediaSource to obtain a reference on the "true", underlying MediaSource.

[jyavenard]

Why not only having the source buffers in a worker?
They really are the only ones we care about when it comes to feeding data fast enough.

[alastor0325]

@DanielBaulig I think it's a good idea to have "TransferableMediaSource", it could fix the problem about the event handler and also it would only need one time slice on the main thread.

In addition, you've mention we would need something like "mediaSource.mediaElementState" which can be reflected the status of media element. Maybe we could start to define the specific attributes which we need.

[DanielBaulig]

In addition, you've mention we would need something like "mediaSource.mediaElementState" which can be reflected the status of media element. Maybe we could start to define the specific attributes which we need.

For the most part I think it's sufficient to provide these media element states via postMessage to the worker at a regular rate.

Unless the platform can significantly improve update performance or maybe even provide thread-safe, but otherwise real-time variations of these, I don't think they are of highest priority.

The most important states/properties are probably currentTime and error. Virtually any media related property can be useful though to optimize certain conditions, prevent over- or underfetching and allow for a more consistent player and data fetching state (e.g. buffered, playbackRate, paused, seeking, ended, readyState, duration).

Why not only having the source buffers in a worker?
They really are the only ones we care about when it comes to feeding data fast enough.

Generally this is correct. I remember that this was brought up before as an option, but I'm not entirely sure what the arguments against it were. There's definitely the limitation of having to touch the main thread twice though (you have to wait for 'sourceopen' to fire before you can create SourceBuffer instances), but I believe the major concern was about how to model the APIs around this, since SourceBuffers are exposed on several MediaSource APIs and all of them would become invalid when the SourceBuffer is transferred into a Worker.

[johnBartos]

LGTM from our app's perspective, as long as typeof MediaSourceHandle === function on the main thread. We'd like some way to check if MSEIW is supported before we set up our worker.

[wolenetz]

@#175 (comment) thanks for your input. We're currently considering two options: Change from exception to null value for the 'handle' attribute when accessed on a main thread MediaSource, or change visibility of that handle getter to only be visible on the dedicated worker context (for now at least). I presume we'll disallow exception even in the latter case.

Either way, it looks like, on main thread (or on worker thread), MediaSource.canConstructInDedicatedWorker === true will remain the overall feature detection for MSE-in-Workers, and, when it's true, both dedicated worker and main thread contexts should have typeof MediaSourceHandle === 'function' on conforming implementations.

[wolenetz]

From discussion during Media Working Group meeting at TPAC 2022 today, both @jernoble and @dontcallmedom also supported the visibility restriction option for the handle getter. I presented both options (null with custom spec replacement for [SameObject], or just exposed=dedicatedworker -- neither can throw exception).

I'll update (or just abandon/replace) the PR #316 to instead do visibility restriction of that getter, without exception throwing.

[wolenetz]

In Chrome, the feature has been fixed to behave per #317 and builds of Chrome >= 108.0.5334.0 should include it enabled by default. See also the Chromium feature tracking issue 878133.