MediaSource throws QuotaExceededError errors when playing live DASH stream in Edge on Xbox One eventually stopping playback

[Moussa]

Have you read the FAQ and checked for duplicate open issues?: Yes

What version of Shaka Player are you using?: 2.3.3

Can you reproduce the issue with our latest release version?: Yes

Can you reproduce the issue with the latest code from master?: Have not tried, but commits since 2.3.5 release do not look relevant

Are you using the demo app or your own custom app?: Custom

If custom app, can you reproduce the issue using our demo app?: Yes

What browser and OS are you using?: Edge 42.17133,2007.0, Windows 10 (Xbox One)

What are the manifest and license server URIs?: https://moussa.online/manifest.mpd
(NOTE: you can send the URIs to shaka-player-issues@google.com instead, but please use GitHub and the template for the rest)
(NOTE: a copy of the manifest text or an attached manifest will not be enough to reproduce your issue, and we will ask you to send a URI instead)

What did you do?

Play a live DASH stream in Edge on the Xbox One

What did you expect to happen?

For the stream to play all the way through without issue

What actually happened?

On Xbox One OS 10.0.16299.5101 Edge after a couple of minutes it begins to stop and buffering every few minutes, and then eventually stops to buffer indefinitely with playback never resuming.

On Xbox One OS 10.0.17133.2007 Edge after ~8mins it starts throwing MediaSource threw QuotaExceededError errors, which reduces the buffering goal each time. A few minutes later it eventually throws MediaSource threw QuotaExceededError too many times and playback stops.

We do not observe the same issue in other browsers, or indeed Edge on Windows 10 desktop.

The manifest linked above is what our server manifest looks like by the time we encounter the fatal error count. Segments are added to the manifest over time. The largest segment size is ~4mb, but otherwise they're all ~1mb each.

[joeyparrish]

We can't reproduce your issue with a snapshot of a live manifest. To reproduce, we would need a playable manifest URI. (We would also need an XBox One, which we do not currently have.) If there is any way you can share a live URI with us, even privately, please do. We can accept private URIs at shaka-player-issues@google.com . Please reference the issue number if you email us there.

Let's see what we can do for you abstractly, though.

If Edge is throwing QuotaExceeded errors, it means its MediaSource implementation can't handle the amount of data it has buffered. Embedded devices such as the XBox One typically have smaller quotas than desktop computers, so this is not unusual on its own. Since we are trying to buffer too much data, we try to reduce the buffering goals to compensate.

We have seen situations in the past where a single segment is larger than the quota for a device, so no reduction in the buffering goal can possibly allow us to recover. My suggestion in such cases is that we should try to avoid the particular representation that caused this error, since it is too large or too high-bitrate for the device to handle. That wouldn't work for you, though, because you only appear to have one video stream and one audio stream. Besides all this, 4MB would be a very small segment to break a device's quota. It's still possible, but I find it unlikely.

A browser is supposed to try to make room in its buffers by evicting older data before throwing QuotaExceeded. If this is broken in the browser, that might explain the behavior you're seeing. One thing you can try to test this theory is to reduce the streaming.bufferBehind configuration. By default, we keep 30 seconds in buffer behind the playhead. By reducing that value, you are less likely to be in a situation where the browser should have to evict any of that old data. Since your manifest appears to have 1-second segments, try setting bufferBehind to 2 or 5. If that works, we know it's a browser bug, and we have a workaround.

If that doesn't work, let us know, and we will look for other possibilities.

[Moussa]

We can't reproduce your issue with a snapshot of a live manifest. To reproduce, we would need a playable manifest URI. (We would also need an XBox One, which we do not currently have.) If there is any way you can share a live URI with us, even privately, please do.

I've confirmed the issue is reproducible with the snapshot manifest I grabbed, but as mentioned it's only reproducible in Edge on the Xbox One, and not desktop Edge. Providing a live manifest URI ourselves is very difficult, as we have a server component that generates the manifest. We would need to start a session on the server at the same time for someone using the manifest.

Embedded devices such as the XBox One typically have smaller quotas than desktop computers, so this is not unusual on its own.

Indeed, our measurements show the media source buffer has a size of around ~90mb. I've tried setting a streaming.bufferBehind value of 2 as suggested but that seems to have no impact on the issue. We still see failure at roughly the same timestamp. Let me know if there's any logging I can provide that might help identify whether it's a browser issue or Shaka issue.

[joeyparrish]

I'm not sure what else we should try. Is there a JS debugger on that platform that you can use somehow to get logs?

I will see if any of my colleagues have an XBox One in the office for testing on other projects. We might be able to borrow it for a while to try to reproduce this.

I have confirmed that we can generally play the content at the manifest URI you provided.

[joeyparrish]

Update: I'm buying us an XBox, which should be available in our lab next week.

[Moussa]

That's great! With a retail Xbox One console there isn't a way afaik to access the browser console or attach a debugger (I suppose if you really wanted to you could modify Shaka to emit logging to another local server rather than the console). But if you register your console as a development console (there's a small fee), you can create a simple HTML5 UWP app and run it on the device from Visual Studio, which has a debugger/console. I can provide the UWP app if it helps.

In the meantime I grabbed some Shaka logs myself, though I'm not sure how helpful they'll be.

[TheModMaker]

I cannot reproduce any problem with the manifest you provided in either the nightly or on v2.3.3. I am also able to play the Sintel demo asset at 1080p just fine (4k is getting decoder problems). Can you try from the nightly demo page: https://bit.ly/2HbtCbp.

I've confirmed the issue is reproducible with the snapshot manifest I grabbed

How? I needed to manually alter the start time to get the manifest to play. Can you elaborate more on how you are reproducing this issue?

Also, in the "Configuration" section, there is a checkbox "Log to screen" that lets you easily see the logs. Plus there is a drop-down menu to change the log level, increasing the log level to "Verbose" would probably help. Alternatively, in a custom app you can use shaka.log.setLevel(shaka.log.Level.V1).

[Moussa]

@TheModMaker To clarify, are you running it on an Xbox One? The issue is only present when running inside Edge on the Xbox One.

To reproduce I only need to load the nightly demo/our custom player with the manifest I linked, and let it play for ~12 minutes. But to avoid network issues I'm running a local server which servers the manifest and segments. I've rerun playback with verbose logging, here is the output. Hope that's more helpful.

[TheModMaker]

Yes, I was testing on the Xbox One.

Your logs show that the browser isn't evicting the content when we tell it to. Near the bottom, there are lines like buffer behind too large: [...] overflow=755.6861656 and evicted 755.6861656 seconds, but the next append will still try to evict 700+ seconds. So either your segments are somehow much larger than you think, or the browser is not actually removing the data when we tell it to. But that is not causing all the problems since we get a QuotaExceededError before that point.

I won't be able to look at this today, but I'll look at it more tomorrow.

[Moussa]

@TheModMaker Ah, I think the behaviour is slightly different depending on which OS your console is on. There is currently a big update being beta tested by some users, which comes with a new Edge version. In both cases playback stops at around the same point, but in the new Edge I see the QuotaExceededErrors. If need be I can get your and your colleague's Xbox Ones into the testing ring via my MS contacts, so that you can test against the upcoming update. Let me know.

[TheModMaker]

I got our device on the early access OS and can confirm that I get a QuotaExceededError when playing. This is very likely a platform issue since we only have the same buffered when we get the error as when we start. Plus if I load a new asset, I immediately get another QuotaExceededError, so this looks like a platform memory leak causing an OOM. Would you like to file an issue with Edge or should we? If you file an issue, please post a link here so we can follow.

[Moussa]

@TheModMaker I suspected a platform bug too, so I reported it to a Microsoft contact who passed it on to the Edge team. So there is no publicly filed issue we can track. I'll wait to hear back from them first, but feel free to file it if you wish in the meantime. Thanks again!

[rahulkrish]

@Moussa @TheModMaker Has there been any updates on the above issue ? We are getting the QuotaExceededError while playing back a Live stream on Xbox One OS Version - 10.0.17134.3036. The stream happen to playback fine for an hour or so before throwing the error. Any workarounds to this ? We did try tweaking the streaming.bufferBehind as @joeyparrish had suggested but no visible difference in behaviour. Any help or advice would be much appreciated!

[joeyparrish]

As far as we can tell, this is an external issue, on Microsoft Edge. It appears that data is not being removed from buffer when we use the SourceBuffer.remove method.

This is fairly fundamental to MediaSource playback, and we are not aware of any workaround. This issue needs be to raised with Microsoft. Please feel free to CC me if you need help in that conversation.

[rahulkrish]

Sure @joeyparrish I'll CC you in on the mail to our Microsoft point of contact. Appreciate your help on taking this to closure.

[joeyparrish]

No problem. Happy to help. If MS can recommend a workaround that can be applied in JavaScript while we wait for the OS/browser to be fixed, I would be happy to help you get that working. You could deploy a temporary fork of Shaka Player with that workaround to make sure you can serve your audience on time.

[Moussa]

I've been told by our MS contact that the issue has been identified and fixed. They were unsure which Insider build the fix would go out with but I have the 10.0.17134.4034 Insider Alpha ring build that was released on June 12 and can confirm I can no longer reproduce the issue with my sample video. If anyone else can confirm I can close out this issue,

[joeyparrish]

@TheModMaker, can you please take a look and try to repro again?

I'm not sure what build we have on our XB1. I tried to check, but the string I found looked nothing like yours, @Moussa, so I may have been looking at the wrong thing in the OS.

[Moussa]

@joeyparrish The full string looks like 10.0.17134.4034 (rs4_release_xbox_dev_1806.180608-1245). The two parts might be shown separately in Xbox One settings, I'm using the Xbox Device Portal.

[TheModMaker]

Unfortunately we don't have access to the Alpha ring, we only have access to the Omega ring. So we can't verify until that OS version gets promoted to the Omega ring.

[Moussa]

Hopefully makes it down the Insider rings soon. I believe it's in beta ring now too.

[joeyparrish]

According to a private thread with Microsoft, this platform bug is triggered by the combination of MediaSource and DOMParser. We use DOMParser to parse the XML in a DASH manifest. This is the only web API available for XML parsing as far as we know.

I believe that XML parsing should be done by the browser, so we will not rewrite our DASH parser to use some pure-JS XML parser. However, a viable workaround may be to insert a pure-js DOMParser polyfill into your application to replace the native implementation of the browser. Once the fix for the bug has been released to the stable Xbox One firmware, you could then remove the polyfill from your application.

I will explore this polyfill as a possible fix and report back if it can be applied trivially on top of Shaka Player: https://github.com/jindw/xmldom

[joeyparrish]

I found a way to make that polyfill work. I have attached a compiled bundle that will replace DOMParser with a pure-JS version. You can add this to your application before you load Shaka Player to avoid the browser's DOMParser implementation.

xmldom-domparser-polyfill.zip

Here's how I made it:

1. Installed node modules for webpack, webpack-cli, and xmldom
2. Modified xmldom in the following ways:
   a. Added exports.Element = Element; to the bottom of dom.js.
   b. Added window.DOMParser = DOMParser; to the bottom of dom-parser.js.
   c. Added window.Element = require('./dom').Element;' to the bottom of dom-parser.js`.
3. Created a webpack.config.js file:

const path = require('path');

module.exports = {
  entry: './node_modules/xmldom/dom-parser.js',
  output: {
    filename: 'xmldom-domparser-polyfill.js',
    path: path.resolve(__dirname, 'dist'),
  },
};

4. Ran ./node_modules/.bin/webpack-cli --config webpack.config.js
5. Added <script src="../dist/xmldom-domparser-polyfill.js"></script> to the Shaka Player demo app.

Please test on the XBox and let me know if this is an effective workaround.

[jayku4u]

According to a private thread with Microsoft, this platform bug is triggered by the combination of MediaSource and DOMParser. We use DOMParser to parse the XML in a DASH manifest. This is the only web API available for XML parsing as far as we know.

If this issue is caused by the combination of MediaSource and DOMParser, then having a js-polyfill will not be an effective workaround. Is there anyway to mitigate the MediaSource bug?

[joeyparrish]

If this issue is caused by the combination of MediaSource and DOMParser, then having a js-polyfill will not be an effective workaround.

That's not true. Because the issue was triggered by the combination of the two, a js-polyfill that replaces native DOMParser is an effective workaround. The polyfill I posted above does just that. If the native DOMParser is not used, the bug is not triggered.

Is there anyway to mitigate the MediaSource bug?

Microsoft stated that a firmware update was on the way which would fix the bug. Since that time, version 10.0.17134.4056 was released on July 5. Since the firmware update is out, I will close this issue. If the issue can still be reproduced with that firmware, please let us know, and we will reopen the issue.

Thanks!