(Deprecate sequence muxed support) Sequence AppendMode can produce surprisingly bad results with muxed media

[wolenetz]

I think various co-editors and web authors over time, including myself, have noticed problems with spec-compliant processing of muxed coded frames in sequence AppendMode. The issues I see usually stem from the auto-updated timestampOffset, calculated on the first frame processed after a discontinuity in one track, becomes applied to all of the tracks. This can result in loss of A/V sync. There are other cases that, depending on implementation choices, can result in buffered range gaps and playback stalls, too.

Though I recall we had chatted about possibly deprecating/disallowing 'sequence' mode usage for SourceBuffers that have more than 1 track, I couldn't find a resulting spec bug. Let's please discuss here.

Thanks!

[jyavenard]

I support this...

In fact it could be argued that there are many ways the current spec could be interpreted when there are multiple tracks.

[wolenetz]

So far, preliminary (Chrome M61 - dev channel) shows very miniscule % of occurrences of sequence muxed mode usage. However, this data is very preliminary, I'll update with public numbers once the related telemetry logging is in a more significant portion of Chrome usage.

[guest271314]

Note, not sure about the exacting technical intent or correct interpretation of the specification, here.

As a user of Chromium and Firefox, the implementations of each browser differ significantly enough that adjustments to code to solve an issue at one browser in this case https://bugzilla.mozilla.org/show_bug.cgi?id=1400587 exposed an issue at the browser where code previously rendered expected result.

[wolenetz]

Lack of interop among implementations (and among developer expectations of what should happen following discontinuities in multi-track 'sequence' mode) is precisely why this form of support is strongly being considered to be deprecated.

[andrewmd5]

@wolenetz to chime in, we currently use sequence mode with multiple tracks to achieve real-time < 20MS streaming. The issue we see with segment (and this could be an miss implementation on our end) is that if a single buffer is missed the playback stops. This is problematic when using WebRTC unreliable channels to pipe the video feed. It seems when using segment mode there is an inherit latency as well making remote desktop applications unusable.

I hope at some point the abstraction layer of MSE is lifted and we can access the decoder ourselves, but until then we're working within the confines of the API. Any tips are welcomed.

Of course if this is deprecated the obvious solution is to split the audio/video into their own streams, its just not as clean.

[jyavenard]

@Codeusa that's likely due to a gap in your data. You can verify this by checking the buffered range. If the gaps are too big, playback will stop (as per spec, as you can only play content when readyState is HAVE_FUTURE_DATA).

That's one downside of MSE, it doesn't cope with missing data.
We hope to address this with a "live" MSE mode that would allow for such issue.

in the mean time, you must monitor the buffered range as well as the waiting event, and seek accordingly over the gap (this is what most DASH players are already doing)

[andrewmd5]

@jyavenard Do you have a particular link where I can follow the progress of the "live" mode spec?

[wolenetz]

@Codeusa , we have several related MSE spec issues around:

• an API to let apps request for the engine to play through unbuffered regions of the timeline (Support playback through unbuffered ranges, and allow app to provide buffered gap tolerance #160)

• a (slightly) more explicit low-latency API (not a decoder API) (Expose an explicit set/get low-latency versus "smoothing" MSE API rather than relying on implementation-specific, implicit bytestream hints that the stream might be "live" #21 and Low-latency mode #133)

Along with some other highly asked for items, these are likely candidates for upcoming feature incubations as we begin gaining traction on vNext features.

[wolenetz]

With existing ability to inspect buffered ranges and adjust timeStampOffset (and append window), and with sequence mode usage in muxed SourceBuffers continuing to be very low rate, deprecating this support looks in scope for V2. Further, other V2 work (#160 and #21) can assist API users that have relied on sequence muxed appends to adapt to lack of sequence muxed append support.