Update SourceBuffer.appendStream() and related algorithms to use ReadableByteStream

[wolenetz]

Migrated from w3c bugzilla tracker. For history prior to migration, please see:
https://www.w3.org/Bugs/Public/show_bug.cgi?id=27239.

[wolenetz]

Needs follow-up by editors to confirm whether this is blocked still by churning normative dependency on Streams API. We could also consider marking this as an at-risk feature in the spec.

[wolenetz]

From discussion with Paul Cotton, we need a stable Streams spec at minimum (even if not a w3c normative reference, and just a whatwg reference.)

[wolenetz]

I discussed the concerns about stable Streams API spec reference with @domenic and received the following advice:

1. [Streams API spec editors] will never break incoming anchor links without first coordinating with all consumer specs so they can update.
2. Commit snapshots are available at https://streams.spec.whatwg.org/commit-snapshots/?C=M;O=D if MSE desires the ability to link to a historical reference to what [STREAMS] looked like at a particular point in time, e.g. in something like “NOTE: the algorithms here are based on the state of [STREAMS] as of [its b98007c… commit snapshot.”

See also Paul Cotton's earlier comment in the w3c bug at https://www.w3.org/Bugs/Public/show_bug.cgi?id=27239#c8 that seemed to indicate that ReadableByteStream had a specific reference available for use in MSE spec: https://streams.spec.whatwg.org/#rbs

[wolenetz]

(Pending Jerry's update from TPAC MSE f2f) w.r.t "MattWolenetz: just want jdsmith to respond to github proposal saying "concern would be satisfied w/ stable reference"
[17:56] jdsmith: that's fine"

[jdsmith3000]

If we can define a stable implementation based on the new streams api, then I would support making that change. It aligns with previous commitments made by the task force on this topic.

[yfuna]

@wolenetz, @jdsmith3000 : The current spec for ReadableByteStream still looks immature, lacking most of its algorithm definitions. I'd like to make sure with you if making it okay to allow a normative reference to such a spec is in line with or a part of the TF's decision in the past.

[wolenetz]

Retaining as V1 with understanding from Media Extensions WG meeting Mar 15th, 2016, that re-triage to later milestone may be necessary if this issue remains unsolved or blocked in early April.

[paulbrucecotton]

Media Extensions WG meeting Mar 15th, 2016

The meeting that Matt is referring to was an Editors-Chair-Team meeting only. It was not a WG meeting.

/paulc

[wolenetz]

My apologies - Paul is correct.

[wolenetz]

@tyoshino, please update on the status of whatwg/streams#300 and https://streams.spec.whatwg.org/#rbs algorithms. Is whatwg/streams#430 meant to solve the problem, and if so, when is it expected to make it into the spec? MSE appendStream() needs a spec'ed and interoperable contract for how to have a stream's fetch and append be constrained by a 'maxSize', which is IIUC the main reason why ReadableByteStream is needed beyond just the existing ReadableStream (see step #9 in the current https://w3c.github.io/media-source/#sourcebuffer-stream-append-loop). I also welcome any better text around how to spec this in MSE that you may have.

Context: this MSE spec bug for updating SourceBuffer.appendStream() to use the new STREAMS API relies upon having a stable reference Streams spec.

[tyoshino]

@wolenetz Thank you for waiting, and sorry for long long delay. whatwg/streams#430 is done. The Fetch API would be updated to adopt the new ReadableStream API soon.

The Fetch API gives you a ReadableStream instance, not an instance of a different class (ReadableByteStream) as before. The ReadableStream instance is constructed with an underlying source which generates bytes. The ReadableStream knows that and activates the getBYOBReader() method (the naming can be changed. see whatwg/streams#294. we'll try to finalize it soon) in addition to getReader().

A reader obtained from the getBYOBReader() method is called ReadableStreamBYOBReader. It has read(view) method which takes TypedArrays and DataView and writes generated data into the region specified by the view. So, if you allocate a view with maxSize bytes, the reader puts bytes only up to maxSize bytes (or can be less).

So, the "Stream Append Loop` algorithm of the media source extensions would include algorithm like the followings:

const buffer = new Uint8Array(maxSize);
const byobReader = response.body.getBYOBReader();
byobReader.read(buffer).then(result => {
  if (result.done) {
    // Jump to the loop done step
    return;
  }
  const filledView = result.value;
  addToTheEndOfInputBuffer(filledView);
});

filledView is representing the region filled with generated bytes.

[domenic]

However, if there's a desire to have 2 compatible implementations in order to advance through The Process, it might be better to remove appendStream() entirely for now, since there are 0 implementations of [STREAMS]-compatible appendStream() at this point, and as far as I know integration is not on anyone's roadmap.

[wolenetz]

@tyoshino Thanks for the details. Is there a Fetch spec bug associated with adopting the new ReadableStream API? Does any UA currently provide the updated ReadableStream API?
I'd like to get this into the MSE spec as much as possible ASAP, so that UA implementors see the motivation (MSE appendStream() support) in implementing the updated ReadableStream (and Fetch) APIs. This is rather time-sensitive, and continuing delay of prerequisite specs is severely risking the ability of updating appendStream() in both MSE spec and user agent implementations.

[tyoshino]

@wolenetz Just created whatwg/fetch#267. But the required change to the Fetch API is internal to the API algorithm. You could leave a small note saying that the Fetch spec is going to be updated soon so that the .body attribute returns a ReadableStream with byte source (whatwg/fetch#267), and update the MSE spec to assume that the ReadableStream returned by .body would give you a ReadableStreamBYOBReader.

Regarding UA, the earliest realistic milestone for Chromium is M52.

[annevk]

1. https://www.w3.org/Bugs/Public/show_bug.cgi?id=27239#c5 never got addressed.
2. I thought the problem with MSE was that it wanted opaque data. Are you okay with only CORS-same-origin data now? Because that issue has not been solved and I'm still not really convinced we want to solve it.

[wolenetz]

@annevk:

1. Regarding https://www.w3.org/Bugs/Public/show_bug.cgi?id=27239#c5, do you have a better name than appendStream(). I believe the reason appendStream() was used was to clearly differentiate the mechanism by which the media data is sourced versus appendBuffer(). Both "append" media data into the coded frame processing algorithm of the SourceBuffer.

2. There are two things MSE appendStream() is trying to solve:
   2a) Ease the ability for web authors to use streams to manage the media data being sent to the SourceBuffer for processing. Note that a polyfill could suffice as a workaround for this use case.
   2b) Mixed content HTMLMediaElement src=URL playback is optionally blockable, yet MSE's extension of HTMLMediaElement provides no way for Mixed Content to work. The route planned out for enabling Mixed Content MSE relies on an as-yet undefined-in-spec opaque-to-web-app stream of media data, appended to the MSE SourceBuffer using the same appendStream() API as (2a). This reduces risk of potential leakage of media data sourced from an insecure origin into the context of a web app from a secure origin. Unless I misunderstand, CORS-same-origin data would fail this mixed content scenario, because the origins are different (secure vs insecure). Hence, CORS-same-origin is not enough for mixed content (and if it were, a secure-origin web app could be exposed to risk of leakage of insecure-origin-sourced media).

[wolenetz]

@tyoshino
Thanks for the updates. Regarding https://www.w3.org/Bugs/Public/show_bug.cgi?id=27239#c2
"The MSE spec also needs to make sure it uses whatever Streams mechanisms are available to guarantee that it becomes the exclusive reader of the stream."
Is there such a mechanism in the updated STREAMS spec?

[annevk]

1. It seems something like useStream() would be clearer. A stream isn't really appended. Or is there a list of them? Or even better would be useResponse or some such as I suggested a couple times, meaning we don't have to figure out opaque streams.
2. Mixed content won't work with the current design. There's no opaque streams.

[wolenetz]

1. I'm fine with a name change. I certainly don't want something like an unclear name causing delay.
2. Correct. And that is definitely a problem for MIX MSE.
   Both (1) and (2), please see also my response on the opaque Fetch+Streams issue: Support mixed-content opaque stream fetches yutakahirano/fetch-with-streams#56 (comment)

[paulbrucecotton]

@wolenetz: At our Editor's meeting on Mar 29, we agreed to republish the MSE Candidate Recommendation with appendstream() marked at risk. When we do this is referring to the following parts of MSE sufficient? I know there are other references to appendstream() but these seem like the two most important places which defined appendstream().
a) Section 3.2 Methods - appendstream
https://www.w3.org/TR/media-source/#methods-1
b) Section 3.5.6 Stream Append Loop
https://www.w3.org/TR/media-source/#sourcebuffer-stream-append-loop

[paulbrucecotton]

The HTML Media Extensions WG has decided to publish a revised MSE Candidate Recommendation document with SourceBuffer.appendStream() method marked 'at risk'. See:
https://lists.w3.org/Archives/Public/public-html-media/2016Apr/0021.html

[wolenetz]

Based on timing, I believe there is no time remaining in V1 to achieve a stable implementation of appendStream() since there is still pending STREAMS spec change that hasn't landed yet, and UA implementations of same will necessarily come after such change and before eventually unblocking work on this MSE spec issue and associated implementation changes.

Therefore, I'm moving this issue to VNext. Comments welcome.

[domenic]

Streams spec is updated, but I don't have any comments on implementation timeline.

[wolenetz]

@domenic Thanks for the update. Given MSE V1 needs to show interop by mid-June to achieve PR on our current schedule, it seems too much risk to depend on unknown stabilizations of streams in this time frame across multiple user agents.

[annevk]

That does seem impossible indeed. I'm not sure if you meant opaque streams above by the way with "pending STREAMS spec change", but as far as I can tell those still don't exist (and I don't think they should).

[wolenetz]

Note, #88 is asking to remove 'stalled' and 'progress' events from an MSE-attached-HTMLMediaElement. Is there a way an app could retain a handle on a stream sent to appendStream() such that the app could listen for events that might indicate progress or stalling of the stream fetch? Pardon if this is Streams API FAQ.

[domenic]

Is there a way an app could retain a handle on a stream sent to appendStream() such that the app could listen for events that might indicate progress or stalling of the stream fetch? Pardon if this is Streams API FAQ.

Not an FAQ; this is a solid question.

So in general the idea is that once you give a stream to someone else (like appendStream), that consumer will lock the stream and nobody else will be able to observe what happens to it. This allows optimizations where the UA consumes the data off the main thread using whatever optimizations it sees fit to perform. So on the face of it, no, there are no such events.

However, it's easy enough for appendStream to purposefully expose information to the outside world. For example, SourceBuffer itself could gain such events, so that when you do sb.appendStream(s), the code that is consuming s might say "I seem to have stalled in consuming s; let me dispatch a "stalled" event on sb."

An alternate approach would be for a consumer who cared about progress to create a no-op transform stream to monitor progress. This is somewhat speculative, as transform streams are still under development, but I think the idea would basically be

const rs = getReadableStreamFromSomewhere();

let bytesSoFar = 0;
const monitoringTransform = new TransformStream({
  transform(chunk) {
    bytesSoFar += chunk.byteLength;
    console.log("Bytes so far:", bytesSoFar);

    return chunk;
  }
});

bufferSource.appendStream(rs.pipeThrough(monitoringTransform));

This would, however, avoid any opportunity for the user agent to bypass the main thread, since chunk is a main-thread representation of the data.

[wolenetz]

For MSE v1, I believe we have editors' consensus to remove this appendStream feature (there's a CfC currently in progress). I'll prepare a PR to remove it from MSE v1, and land it shortly (we can revert it if the CfC for the removal fails, or if we wish to refine the spec and implementation in VNext, in the appropriate branch).

[paulbrucecotton]

Streams spec is updated, but I don't have any comments on implementation timeline

@domenic: Can you provide an update on the status of the Streams spec? Is it ready to be referenced from other specs?

/paulc

[domenic]

Yes.

[paulbrucecotton]

I have confirmed that the Streams spec is now ready so this feature could be added back to a future version of MSE.

Is there support to work on incubation of this feature?

/paulc
HME WG Chair

[wolenetz]

Lack of recently requested support encourages this be in Backlog for V2.

[wilaw]

@wolenetz - i'd strongly urge you to consider adding appendStream() to the V2 milestone. WebTransport provides a clear new case for support against what is now a five year old request. The ability to pipe the output of a WebTransport (QuicTransport, Http3Transport) stream to a source buffer provides the lowest possible latency path for live media, addressing multiple target use-cases. The combination of WebTransport, WebCodecs and MSE v2 offer exciting inflection points in both performance and features for web media. It would be a lost opportunity if this triad were deferred another 5 years.

[jan-ivar]

FWIW appendStream seems easy to polyfill with appendBuffer and works in all the browsers.

Except it probably shouldn't fire updateend until the stream is done (suppressing events is harder to polyfill).

@annevk For me, the name seems fine for consistency with appendBuffer, and to not break the model which seems to be that we're adding to whatever is already in the source buffer.

[jan-ivar]

Of course I found #185 (comment) only after posting this.

[jan-ivar]

Ok, so even simpler to polyfill based on #185 (comment) (though note WritableStream doesn't work in Firefox or Safari yet).

Which issue is the right one to track this? Should we open a new one?

[annevk]

I'll defer to @domenic and @ricea when it comes to naming stream-related features.

[ricea]

I don't have much context, but from a Streams perspective the most natural thing to do would be to expose a writable attribute on SourceBuffer which developers could then pipe a readable stream to (or get a writer and write manually if they preferred).

[wolenetz]

See also #185, probably a duplicate if the approach is to add a writable endpoint to a SourceBuffer to which a stream/sequence of chunks could be piped.