From 8073f6918d82e41e3f086deae9dff7d93e3e1815 Mon Sep 17 00:00:00 2001 From: Matt Wolenetz Date: Wed, 29 Sep 2021 17:45:32 -0700 Subject: [PATCH 1/6] MSE-for-WebCodecs initial feature specification * Updates MediaSource and SourceBuffer sections' IDL to reference the new methods and types. * Updates SOTD substantives list format and content to include this feature. See #184 for the spec issue tracking this feature's addition. (Remove this set of lines during eventual squash and merge:) * Adds placeholder notes including references to the WebCodecs spec to let the updated IDLs' references to definitions from that spec succeed. Upcoming commits will remove the placeholders and include exposition on the behavior of the updated IDL, possibly also refactoring reused steps into subalgorithms. --- media-source-respec.html | 50 ++++++++++++++++++++++++++++++---------- 1 file changed, 38 insertions(+), 12 deletions(-) diff --git a/media-source-respec.html b/media-source-respec.html index c04fcb7..76e3ace 100644 --- a/media-source-respec.html +++ b/media-source-respec.html @@ -127,7 +127,14 @@
-

On top of editorial updates, substantives changes since publication as a W3C Recommendation in November 2016 are the addition of a {{SourceBuffer/changeType()}} method to switch codecs, the possibility to create and use {{MediaSource}} objects off the main thread in dedicated workers, and the removal of the createObjectURL() extension to the {{URL}} object following its integration in the File API [[FILEAPI]]. For a full list of changes done since the previous version, see the commits.

+

On top of editorial updates, substantive changes since publication as a W3C Recommendation in November 2016 are +

+ For a full list of changes done since the previous version, see the commits.

The working group maintains a list of all bug reports that the editors have not yet tried to address.

Implementors should be aware that this specification is not stable. Implementors who are not taking part in the discussions are likely to find the specification changing out from under them in incompatible ways. Vendors interested in implementing this specification before it eventually reaches the Candidate Recommendation stage should track the GitHub repository and take part in the discussions.

@@ -323,15 +330,26 @@

MediaSource Object

attribute EventHandler onsourceended; attribute EventHandler onsourceclose; static readonly attribute boolean canConstructInDedicatedWorker; - SourceBuffer addSourceBuffer (DOMString type); - undefined removeSourceBuffer (SourceBuffer sourceBuffer); - undefined endOfStream (optional EndOfStreamError error); - undefined setLiveSeekableRange (double start, double end); - undefined clearLiveSeekableRange (); - static boolean isTypeSupported (DOMString type); -}; + SourceBuffer addSourceBuffer (optional TypeOrConfig typeOrConfig = {}); + undefined removeSourceBuffer (SourceBuffer sourceBuffer); + undefined endOfStream (optional EndOfStreamError error); + undefined setLiveSeekableRange (double start, double end); + undefined clearLiveSeekableRange (); + static boolean isTypeSupported (DOMString type); +}; + +dictionary SourceBufferConfig { + // Precisely one of these must be populated to signal via addSourceBuffer or + // changeType the intent to buffer the corresponding WebCodecs media. + AudioDecoderConfig audioConfig; // For appending EncodedAudioChunks + VideoDecoderConfig videoConfig; // For appending EncodedVideoChunks +}; + +typedef (DOMString or SourceBufferConfig) TypeOrConfig; +

Attributes

+

REMOVE THIS PLACEHOLDER NOTE: TODO describe SourceBufferConfig and the updated addSourceBuffer and the linkage to [[WEBCODECS]]

sourceBuffers of type SourceBufferList, readonly
Contains the list of SourceBuffer objects associated with this MediaSource. When {{MediaSource/readyState}} equals {{ReadyState/""closed""}} this list will be empty. Once {{MediaSource/readyState}} transitions to {{ReadyState/""open""}} SourceBuffer objects can be added to this list by using {{MediaSource/addSourceBuffer()}}.
activeSourceBuffers of type SourceBufferList, readonly
@@ -1138,7 +1156,8 @@

SourceBuffer Object

-
[Exposed=(Window,DedicatedWorker)]
+
+[Exposed=(Window,DedicatedWorker)]
 interface SourceBuffer : EventTarget {
                     attribute AppendMode          mode;
     readonly        attribute boolean             updating;
@@ -1155,13 +1174,20 @@ 
SourceBuffer Object

                     attribute EventHandler        onerror;
                     attribute EventHandler        onabort;
     undefined appendBuffer (BufferSource data);
+    Promise < undefined > appendEncodedChunks(EncodedChunks chunks);
     undefined abort ();
-    undefined changeType (DOMString type);
+    undefined changeType (optional TypeOrConfig typeOrConfig = {});
     undefined remove (double start, unrestricted double end);
-};

+};
+
+typedef (sequence< (EncodedAudioChunk or EncodedVideoChunk) >
+            or EncodedAudioChunk or EncodedVideoChunk) EncodedChunks;
+
[[HTML]] {{AudioTrackList}}, {{VideoTrackList}} and {{TextTrackList}} need Window+DedicatedWorker exposure.
-

Attributes

mode of type AppendMode
+

Attributes

+

REMOVE THIS PLACEHOLDER NOTE: TODO describe encodedchunks, appendEncodedChunks, and the updated changeType and linkage to [[WEBCODECS]].

+
mode of type AppendMode

Controls how a sequence of [=media segments=] are handled. This attribute is initially set by {{MediaSource/addSourceBuffer()}} after the object is created, and can be updated by {{SourceBuffer/changeType()}} or setting this attribute.

On getting, Return the initial value or the last value that was successfully set.

On setting, run the following steps:

From 7811830f182697016db33e0fdc2d4909ef0de57a Mon Sep 17 00:00:00 2001 From: Matt Wolenetz Date: Thu, 30 Sep 2021 16:18:21 -0700 Subject: [PATCH 2/6] MSE-for-WebCodecs: Add example --- media-source-respec.html | 136 +++++++++++++++++++++++++++++++++++---- 1 file changed, 123 insertions(+), 13 deletions(-) diff --git a/media-source-respec.html b/media-source-respec.html index 76e3ace..b9d2883 100644 --- a/media-source-respec.html +++ b/media-source-respec.html @@ -330,7 +330,12 @@

MediaSource Object

attribute EventHandler onsourceended; attribute EventHandler onsourceclose; static readonly attribute boolean canConstructInDedicatedWorker; + + // "optional" with default empty SourceBufferConfig dictionary is used here + // to pass WebIDL verification. Behavior of the method enforces that either + // a valid mime-type string or a valid SourceBufferConfig are provided. SourceBuffer addSourceBuffer (optional TypeOrConfig typeOrConfig = {}); + undefined removeSourceBuffer (SourceBuffer sourceBuffer); undefined endOfStream (optional EndOfStreamError error); undefined setLiveSeekableRange (double start, double end); @@ -345,6 +350,8 @@

MediaSource Object

VideoDecoderConfig videoConfig; // For appending EncodedVideoChunks }; +// This typedef simplifies the syntax for overloading addSourceBuffer and +// changeType to receive either a mime-type string or a WebCodecs config. typedef (DOMString or SourceBufferConfig) TypeOrConfig;
@@ -380,7 +387,7 @@

Attributes

  • If the {{SourceBuffer/updating}} attribute equals true on any SourceBuffer in {{MediaSource/sourceBuffers}}, then throw an {{InvalidStateError}} exception and abort these steps.
  • Run the [=duration change=] algorithm with |new duration:unrestricted double| set to the value being assigned to this attribute.

    The [=duration change=] algorithm will adjust |new duration| higher if there is any currently buffered coded frame with a higher end time.

    -

    {{SourceBuffer/appendBuffer()}} and {{MediaSource/endOfStream()}} can update the duration under certain circumstances.

    +

    {{SourceBuffer/appendBuffer()}}, {{SourceBuffer/appendEncodedChunks()}} and {{MediaSource/endOfStream()}} can update the duration under certain circumstances.

    • onsourceopen of type {{EventHandler}}
    @@ -820,7 +827,7 @@

    Attaching to a media element

  • Continue the by running the remaining steps, with these clarifications:
      -
    1. Text in the or the that refers to "the download", "bytes received", or "whenever new data for the current media resource becomes available" refers to data passed in via {{SourceBuffer/appendBuffer()}}.
      2. +
    2. Text in the or the that refers to "the download", "bytes received", or "whenever new data for the current media resource becomes available" refers to data passed in via {{SourceBuffer/appendBuffer()}} or {{SourceBuffer/appendEncodedChunks()}}.
    3. References to HTTP in the and the do not apply because the HTMLMediaElement does not fetch media data via HTTP when a MediaSource is attached.
    • @@ -880,7 +887,7 @@

    Seeking

    to {{HTMLMediaElement/HAVE_METADATA}}.

    Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

    -
  • The media element waits until an {{SourceBuffer/appendBuffer()}} call causes the [=coded frame processing=] algorithm to set +
  • The media element waits until an {{SourceBuffer/appendBuffer()}} or {{SourceBuffer/appendEncodedChunks()}} call causes the [=coded frame processing=] algorithm to set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to a value greater than {{HTMLMediaElement/HAVE_METADATA}}.

    The web application can use {{SourceBuffer/buffered}} and {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}} to determine what the media element needs to resume playback.

    • @@ -1176,17 +1183,28 @@

    SourceBuffer Object

    undefined appendBuffer (BufferSource data); Promise < undefined > appendEncodedChunks(EncodedChunks chunks); undefined abort (); + + // "optional" with default empty SourceBufferConfig dictionary is used here + // to pass WebIDL verification. Behavior of the method enforces that either + // a valid mime-type string or a valid SourceBufferConfig are provided. undefined changeType (optional TypeOrConfig typeOrConfig = {}); + undefined remove (double start, unrestricted double end); }; +// This typedef simplifies the syntax for describing either a single audio or +// video chunk, or a sequence of chunks. While this typedef does not provide the +// ability to enforce that all members of a sequence of chunks are the same kind +// (audio versus video), the behavior of the appendEncodedChunks method enforces +// that constraint: the implementation MUST reject a sequence containing both +// audio and video chunks. typedef (sequence< (EncodedAudioChunk or EncodedVideoChunk) > or EncodedAudioChunk or EncodedVideoChunk) EncodedChunks;
    [[HTML]] {{AudioTrackList}}, {{VideoTrackList}} and {{TextTrackList}} need Window+DedicatedWorker exposure.

    Attributes

    -

    REMOVE THIS PLACEHOLDER NOTE: TODO describe encodedchunks, appendEncodedChunks, and the updated changeType and linkage to [[WEBCODECS]].

    +

    REMOVE THIS PLACEHOLDER NOTE: TODO describe appendEncodedChunks, the updated changeType and linkage to [[WEBCODECS]].

    mode of type AppendMode

    Controls how a sequence of [=media segments=] are handled. This attribute is initially set by {{MediaSource/addSourceBuffer()}} after the object is created, and can be updated by {{SourceBuffer/changeType()}} or setting this attribute.

    On getting, Return the initial value or the last value that was successfully set.

    @@ -1211,7 +1229,7 @@

    SourceBuffer Object

  • Update the attribute to |new mode|.
    • updating of type {{boolean}}, readonly
    -

    Indicates whether the asynchronous continuation of an {{SourceBuffer/appendBuffer()}} or {{SourceBuffer/remove()}} +

    Indicates whether the asynchronous continuation of an {{SourceBuffer/appendBuffer()}}, {{SourceBuffer/appendEncodedChunks()}} or {{SourceBuffer/remove()}} operation is still being processed. This attribute is initially set to false when the object is created.

    buffered of type {{TimeRanges}}, readonly

    Indicates what {{TimeRanges}} are buffered in the SourceBuffer. This @@ -1329,7 +1347,7 @@

    SourceBuffer Object

    No parameters.
    Return type: {{undefined}}
    changeType
    -

    Changes the MIME type associated with this object. Subsequent {{SourceBuffer/appendBuffer()}} calls will expect the newly appended bytes to conform to the new type. +

    Changes the MIME type associated with this object (TODO: split logic by arg type, reference appendEncodedChunks() too). Subsequent {{SourceBuffer/appendBuffer()}} calls will expect the newly appended bytes to conform to the new type.

    1. If |type:DOMString| is an empty string then throw a {{TypeError}} exception and abort these steps.
    2. If this object has been removed from the {{MediaSource/sourceBuffers}} attribute of the [=parent media source=], then throw an {{InvalidStateError}} exception and abort these steps.
      3. @@ -1542,7 +1560,7 @@

      Segment Parser Loop

      empty when the {{SourceBuffer}} object is created.

      Each {{SourceBuffer}} object has a [[\buffer full flag]] internal - slot that keeps track of whether {{SourceBuffer/appendBuffer()}} is allowed to accept more bytes. It is set + slot that keeps track of whether {{SourceBuffer/appendBuffer()}} is allowed to accept more bytes or {{SourceBuffer/appendEncodedChunks()}} is allowed to accept more encoded chunks. It is set to false when the {{SourceBuffer}} object is created and gets updated as data is appended and removed.

      Each {{SourceBuffer}} object has a [[\group start timestamp]] @@ -1702,7 +1720,7 @@

      Prepare Append

      Buffer Append

      -

      When {{SourceBuffer/appendBuffer()}} is called, the following steps are run to process the appended data.

      +

      When {{SourceBuffer/appendBuffer()}} (TODO split out chunk processing) is called, the following steps are run to process the appended data.

      1. Run the [=segment parser loop=] algorithm.
      2. If the [=segment parser loop=] algorithm in the previous step was aborted, then abort this algorithm.
        3. @@ -2908,9 +2926,12 @@

        Byte Stream Formats

        Examples

        Example use of the Media Source Extensions

        -
        -
        - 
        <script>
+      
        
+        
        Buffering bytes from a bytestream format
        
+        
        
+          
        
+            
        +<script>
   function onSourceOpen(videoTag, e) {
     var mediaSource = e.target;
 
@@ -2999,9 +3020,98 @@ 
        Examples
        
   mediaSource.addEventListener('sourceopen', onSourceOpen.bind(this, video));
   video.src = window.URL.createObjectURL(mediaSource);
 </script>
-          
        
+
        +
        - +
        +
        +

        Buffering WebCodecs encoded chunks

        +
        +
        + 
        +<body>
+<script>
+  // Very simple demuxer bound to a specific media file.
+  // Replace with other source of WebCodecs configs and encoded chunks.
+  let next_chunk_index = 0;
+  let buffer;
+  let chunk_duration = 100 * 1000;  // 100 milliseconds for this example.
+  let metadata = [
+    {
+      offset: /* chunk offset, e.g. 42 */,
+      size: /* chunk size, e.g. 4242 */,
+      type: /* chunk type, e.g. "key" or "delta" */
+    }, // Plus additional chunks' metadata listed here.
+  ];
+
+  function getConfig() {
+    return( /* e.g. { videoConfig: { codec: "vp09.00.10.08" } } */ );
+  }
+
+  async function getNextEncodedChunk() {
+    if (next_chunk_index >= metadata.length)
+      return null;
+    if (next_chunk_index == 0)
+      buffer = await (await fetch(/* e.g. "vp9.chunks" */)).arrayBuffer();
+    let chunk_metadata = metadata[next_chunk_index];
+    let chunk_timestamp = chunk_duration * next_chunk_index;
+    next_chunk_index++;
+    // EncodedAudioChunks could also be buffered into MSE, but only one form of
+    // media (audio versus video) can be buffered into a specific SourceBuffer,
+    // so this simple example is video-only.
+    return new EncodedVideoChunk( {
+      type: chunk_metadata.type,
+      timestamp: chunk_timestamp,
+      duration: chunk_duration,
+      data: new Uint8Array(buffer, chunk_metadata.offset, chunk_metadata.size)
+    } );
+  }
+
+  // MSE player using the WebCodecs content generated, above.
+  async function getOpenMediaSource() {
+    return new Promise(async (resolve, reject) => {
+      const v = document.createElement("video");
+      document.body.appendChild(v);
+      const mediaSource = new MediaSource();
+      const url = URL.createObjectURL(mediaSource);
+      mediaSource.addEventListener("sourceopen", () => {
+        URL.revokeObjectURL(url);
+        if (mediaSource.readyState != "open")
+          reject();
+        else
+          resolve([ v, mediaSource ]);
+      }, { once: true });
+      v.src = url;
+    });
+  }
+
+  async function bufferMedia() {
+    let [ videoElement, mediaSource ] = await getOpenMediaSource();
+    let sourceBuffer = mediaSource.addSourceBuffer(await getConfig());
+
+    // This simple player attempts to buffer everything immediately. Full
+    // players will condition buffering, for example, on playback progress
+    // and availability of chunks.
+    while (null != (chunk = await(getNextEncodedChunk()))) {
+      // Note that appending a sequence instead of a single chunk here
+      // could be more performant.
+      await sourceBuffer.appendEncodedChunks(chunk);
+    }
+
+    mediaSource.endOfStream();
+    videoElement.controls = true;  // Show default controls.
+  }
+
+  if (!SourceBuffer.prototype.hasOwnProperty("appendEncodedChunks"))
+    console.log("MSE-for-WebCodecs support is missing.")
+  else
+    bufferMedia();
+</script>
+</body>
+
        +
        +
        +
      From ea5ad851eeea19872f8b4806aa22dc04b419e5c7 Mon Sep 17 00:00:00 2001 From: Matt Wolenetz Date: Thu, 30 Sep 2021 17:32:53 -0700 Subject: [PATCH 3/6] MSE-for-WebCodecs: Update the MediaSource interface details Needs verification later of correct linkage to WebCodecs dfns for "valid AudioDecoderConfig" and "valid VideoDecoderConfig" once WebRef updates. (See their associated exports added in https://github.com/w3c/webcodecs/pull/372). --- media-source-respec.html | 67 ++++++++++++++++++++++++++++++---------- 1 file changed, 51 insertions(+), 16 deletions(-) diff --git a/media-source-respec.html b/media-source-respec.html index b9d2883..b9d7be1 100644 --- a/media-source-respec.html +++ b/media-source-respec.html @@ -118,7 +118,7 @@ - +
      This specification extends {{HTMLMediaElement}} [[HTML]] to allow JavaScript to generate media streams for playback. @@ -344,8 +344,9 @@

      MediaSource Object

      }; dictionary SourceBufferConfig { - // Precisely one of these must be populated to signal via addSourceBuffer or - // changeType the intent to buffer the corresponding WebCodecs media. + // Precisely one of these WebCodecs config objects must be populated to + // signal via addSourceBuffer or changeType the intent to buffer the + // corresponding WebCodecs media. AudioDecoderConfig audioConfig; // For appending EncodedAudioChunks VideoDecoderConfig videoConfig; // For appending EncodedVideoChunks }; @@ -356,7 +357,6 @@

      MediaSource Object

      Attributes

      -

      REMOVE THIS PLACEHOLDER NOTE: TODO describe SourceBufferConfig and the updated addSourceBuffer and the linkage to [[WEBCODECS]]

      sourceBuffers of type SourceBufferList, readonly
      Contains the list of SourceBuffer objects associated with this MediaSource. When {{MediaSource/readyState}} equals {{ReadyState/""closed""}} this list will be empty. Once {{MediaSource/readyState}} transitions to {{ReadyState/""open""}} SourceBuffer objects can be added to this list by using {{MediaSource/addSourceBuffer()}}.
      activeSourceBuffers of type SourceBufferList, readonly
      @@ -403,25 +403,60 @@

      Attributes

      creating and using a {{MediaSource}} object in a dedicated worker, and mitigates the need for higher latency detection polyfills like attempting creation of a {{MediaSource}} object from a dedicated worker, especially if the feature is not supported.

      -

      Methods

      addSourceBuffer
      +
      +

      Methods

      +
      addSourceBuffer

      Adds a new SourceBuffer to {{MediaSource/sourceBuffers}}.

        -
      1. If |type:DOMString| is an empty string then throw a {{TypeError}} exception and abort these steps.
        2. -
      2. If |type| contains a MIME type that is not supported or contains a MIME type that is not supported with the types specified for the other SourceBuffer objects in {{MediaSource/sourceBuffers}}, then throw a {{NotSupportedError}} exception and abort these steps.
        3. -
      3. If the user agent can't handle any more SourceBuffer objects or if creating a SourceBuffer - based on |type| would result in an unsupported [=SourceBuffer configuration=], - then throw a {{QuotaExceededError}} exception and abort these steps. +
      4. If |typeOrConfig:TypeOrConfig| is a DOMString, then run the following steps: +
          +
        1. If |typeOrConfig| is empty then throw a {{TypeError}} exception and abort these steps and + this method.
          2. +
        2. If |typeOrConfig| contains a MIME type that is not supported or contains a MIME type that is not + supported with the types specified for the other {{SourceBuffer}} objects in + {{MediaSource/sourceBuffers}}, then throw a {{NotSupportedError}} exception and abort these + steps and this method.
          3. +
        +
        5. +
      5. If |typeOrConfig| is not a DOMString, then run the following steps: +
          +
        1. If |typeOrConfig| has both an {{SourceBufferConfig/audioConfig}} and a + {{SourceBufferConfig/videoConfig}} or has neither, then throw a {{TypeError}} exception and abort + these steps and this method.
          2. +
        2. If |typeOrConfig| has neither a neither a [=valid AudioDecoderConfig=] in + {{SourceBufferConfig/audioConfig}} nor a [=valid VideoDecoderConfig=] in + {{SourceBufferConfig/videoConfig}}, then throw a {{TypeError}} exception and abort these steps and + this method.
          3. +
        3. If |typeOrConfig| contains a decoder configuration that is not supported or is not supported with + the types specified for the other {{SourceBuffer}} objects in {{MediaSource/sourceBuffers}}, then + throw a {{NotSupportedError}} exception and abort these steps and this method.
          4. +
        +
        6. +
      6. If the user agent can't handle any more {{SourceBuffer}} objects or if creating a {{SourceBuffer}} based + on |typeOrConfig| would result in an unsupported [=SourceBuffer configuration=], then throw a + {{QuotaExceededError}} exception and abort these steps.

        For example, a user agent MAY throw a {{QuotaExceededError}} exception if the media element has reached the {{HTMLMediaElement/HAVE_METADATA}} readyState. This can occur if the user agent's media engine does not support adding more tracks during playback.

      7. If the {{MediaSource/readyState}} attribute is not in the {{ReadyState/""open""}} state then throw an {{InvalidStateError}} exception and abort these steps.
        8. -
      8. Create a new SourceBuffer object and associated resources.
        9. -
      9. Set the {{SourceBuffer/[[generate timestamps flag]]}} on the new object to the value in the "Generate - Timestamps Flag" column of the byte stream format registry [[MSE-REGISTRY]] entry that is associated with - |type|. -
      10. +
      11. Create a new {{SourceBuffer}} object and associated resources.
        12. +
      12. +
        +
        If |typeOrConfig| is a DOMString:
        +
        Set the {{SourceBuffer/[[generate timestamps flag]]}} on the new object to the value in the + "Generate Timestamps Flag" column of the byte stream format registry [[MSE-REGISTRY]] entry that is + associated with |type|. +
        +
        Otherwise:
        +
        Set the {{SourceBuffer/[[generate timestamps flag]]}} on the new object to false. +

        WebCodecs encoded chunks are required to have timestamps, so there is no need to + generate them.

        +
        +
        +
        13. +
      13. If the {{SourceBuffer/[[generate timestamps flag]]}} equals true:
        @@ -1204,7 +1239,7 @@

        SourceBuffer Object

        [[HTML]] {{AudioTrackList}}, {{VideoTrackList}} and {{TextTrackList}} need Window+DedicatedWorker exposure.

        Attributes

        -

        REMOVE THIS PLACEHOLDER NOTE: TODO describe appendEncodedChunks, the updated changeType and linkage to [[WEBCODECS]].

        +

        REMOVE THIS PLACEHOLDER NOTE: TODO describe appendEncodedChunks, the updated changeType and linkage to WEBCODECS.

        mode of type AppendMode

        Controls how a sequence of [=media segments=] are handled. This attribute is initially set by {{MediaSource/addSourceBuffer()}} after the object is created, and can be updated by {{SourceBuffer/changeType()}} or setting this attribute.

        On getting, Return the initial value or the last value that was successfully set.

        From 73bdac1ad910706c241aee7af967de0d27e21d47 Mon Sep 17 00:00:00 2001 From: Matt Wolenetz Date: Thu, 30 Sep 2021 18:47:19 -0700 Subject: [PATCH 4/6] MSE-for-WebCodecs: More partial updates Adds TODO items to track more work necessary in this draft. Fixes the addSourceBuffer parameter type to be TypeOrConfig. References a not-yet-defined internal slot that tracks the pending append chunks promise, and adds steps to reject that promise on abort() or removeSourceBuffer(), if necessary. Updates various places that describe BSF to also mention the possibility that a SourceBuffer may instead be configured to expect WebCodecs encoded chunks appends. Adds a note to isTypeSupported() for how to use the WebCodecs isConfigSupported() methods to proactively check support for buffering encoded chunks into a SourceBuffer. --- media-source-respec.html | 43 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 39 insertions(+), 4 deletions(-) diff --git a/media-source-respec.html b/media-source-respec.html index b9d7be1..f4617e5 100644 --- a/media-source-respec.html +++ b/media-source-respec.html @@ -1,5 +1,14 @@ + Media Source Extensions™ @@ -251,9 +260,16 @@

        Definitions

        A position in a [=media segment=] where decoding and continuous playback can begin without relying on any previous data in the segment. For video this tends to be the location of I-frames. In the case of audio, most audio frames can be treated as a random access point. Since video tracks tend to have a more sparse distribution of random access points, the location of these points are usually considered the random access points for multiplexed streams.

        SourceBuffer byte stream format specification
        -

        The specific [=byte stream format specification=] that describes the format of the byte stream accepted by a SourceBuffer instance. The - [=byte stream format specification=], for a SourceBuffer object, is initially selected based on the |type:DOMString| passed to the - {{MediaSource/addSourceBuffer()}} call that created the object, and can be updated by {{SourceBuffer/changeType()}} calls on the object.

        +

        The specific [=byte stream format specification=] that describes the format of the byte stream accepted + by a {{SourceBuffer}} instance, when configured via {{MediaSource/addSourceBuffer()}} or updated by + {{SourceBuffer/changeType()}} to be based on a |typeOrConfig:TypeOrConfig| that is a DOMString. The [=byte + stream format specification=], for a {{SourceBuffer}} object, is initially selected based on the + |type:DOMString| passed to the {{MediaSource/addSourceBuffer()}} call that created the object, and can be + updated by {{SourceBuffer/changeType()}} calls on the object. When the {{SourceBuffer}} is instead using a + WebCodecs {{AudioDecoderConfig}} or {{VideoDecoderConfig}} [[WEBCODECS]] in the most recent of either the + {{MediaSource/addSourceBuffer()}} call that created the {{SourceBuffer}} or a {{SourceBuffer/changeType()}} + call, the {{SourceBuffer}} is not expecting byte stream appends via {{SourceBuffer/appendBuffer()}}; instead + it is expecting WebCodecs chunks to be appended via {{SourceBuffer/appendEncodedChunks()}}.

        SourceBuffer configuration

        A specific set of tracks distributed across one or more SourceBuffer @@ -473,7 +489,9 @@

        Attributes

      14. Add the new object to {{MediaSource/sourceBuffers}} and [=queue a task=] to [=fire an event=] named {{addsourcebuffer}} at {{MediaSource/sourceBuffers}}.
      15. Return the new object.
      -
      ParameterTypeNullableOptionalDescription
      |type|{{DOMString}}✘✘
      Return type: SourceBuffer
      + + +
      ParameterTypeNullableOptionalDescription
      |typeOrConfig|{{TypeOrConfig}}✘✘
      Return type: SourceBuffer
    removeSourceBuffer

    Removes a {{SourceBuffer}} from {{MediaSource/sourceBuffers}}.

    @@ -483,6 +501,10 @@

    Attributes

  • If the |sourceBuffer|.{{SourceBuffer/updating}} attribute equals true, then run the following steps:
    1. Abort the [=buffer append=] algorithm if it is running.
      2. +
    2. Abort the [=chunks append=] algorithm if it is running.
      3. +
    3. If this object has a {{Promise}} in {{SourceBuffer/[[\pending append chunks promise]]}} then reject + that {{Promise}} with an {{AbortError}} {{DOMException}} and unset {{SourceBuffer/[[\pending append + chunks promise]]}}.
    4. Set the |sourceBuffer|.{{SourceBuffer/updating}} attribute to false.
    5. [=Queue a task=] to [=fire an event=] named {{abort}} at |sourceBuffer|.
    6. [=Queue a task=] to [=fire an event=] named {{updateend}} at |sourceBuffer|.
      7. @@ -740,6 +762,11 @@

      Attributes

      This method returning true implies that HTMLMediaElement.canPlayType() will return "maybe" or "probably" since it does not make sense for a MediaSource to support a type the HTMLMediaElement knows it cannot play.

      +

      + For proactively checking support for a WebCodecs {{AudioDecoderConfig}} or {{VideoDecoderConfig}}, use the + respective {{AudioDecoder}} {{AudioDecoder/isConfigSupported()}} or {{VideoDecoder}} + {{VideoDecoder/isConfigSupported()}} method. +

      ParameterTypeNullableOptionalDescription
      |type|{{DOMString}}✘✘
      Return type: {{boolean}}
    • @@ -1371,6 +1398,10 @@

    SourceBuffer Object

  • If the {{SourceBuffer/updating}} attribute equals true, then run the following steps:
    1. Abort the [=buffer append=] algorithm if it is running.
      2. +
    2. Abort the [=chunks append=] algorithm if it is running.
      3. +
    3. If this object has a {{Promise}} in {{SourceBuffer/[[\pending append chunks promise]]}} then reject + that {{Promise}} with an {{AbortError}} {{DOMException}} and unset {{SourceBuffer/[[\pending append + chunks promise]]}}.
    4. Set the {{SourceBuffer/updating}} attribute to false.
    5. [=Queue a task=] to [=fire an event=] named {{abort}} at this SourceBuffer object.
    6. [=Queue a task=] to [=fire an event=] named {{updateend}} at this SourceBuffer object.
      7. @@ -2888,6 +2919,10 @@

      Byte Stream Formats

      The byte stream format specifications in the registry are not intended to define new storage formats. They simply outline the subset of existing storage format structures that implementations of this specification will accept.

      Byte stream format parsing and validation is implemented in the [=segment parser loop=] algorithm.

      +

      When currently configured by {{MediaSource/addSourceBuffer()}} or {{SourceBuffer/changeType()}} to + expect appends of WebCodecs encoded chunks via {{SourceBuffer/appendEncodedChunks()}} instead of a bytestream + via {{SourceBuffer/appendBuffer}}, the {{SourceBuffer}} does not have a specific bytestream format + associated.

      This section provides general requirements for all byte stream format specifications:

        From cd45449361e5ad99c2ded9bf5fa00d25cb0b21e6 Mon Sep 17 00:00:00 2001 From: Matt Wolenetz Date: Thu, 30 Sep 2021 19:09:40 -0700 Subject: [PATCH 5/6] MSE-for-WebCodecs: Further updates Fixes reference to the sourceBuffer in removeSourceBuffer()'s promise abort step. Updates the coded frame eviction algorithm's definition of |new data| to possibly be EncodedChunks being appended. Adds top-level steps for appendEncodedChunks(). Updates abort() and removeSourceBuffer() handling to conditionally reject promise vs abort / updateend firing. Updates reset parser state algorithm to clear any EncodedChunks from the input configs and chunks slot, but leave any WebCodecs config there. References #301 for discussion from each of appendBuffer() and appendEncodedChunks(). --- media-source-respec.html | 126 ++++++++++++++++++++++++++++++++------- 1 file changed, 103 insertions(+), 23 deletions(-) diff --git a/media-source-respec.html b/media-source-respec.html index f4617e5..a853efa 100644 --- a/media-source-respec.html +++ b/media-source-respec.html @@ -1,13 +1,22 @@ @@ -502,12 +511,19 @@

        Attributes

        1. Abort the [=buffer append=] algorithm if it is running.
        2. Abort the [=chunks append=] algorithm if it is running.
          3. -
        3. If this object has a {{Promise}} in {{SourceBuffer/[[\pending append chunks promise]]}} then reject - that {{Promise}} with an {{AbortError}} {{DOMException}} and unset {{SourceBuffer/[[\pending append - chunks promise]]}}.
        4. Set the |sourceBuffer|.{{SourceBuffer/updating}} attribute to false.
          5. -
        5. [=Queue a task=] to [=fire an event=] named {{abort}} at |sourceBuffer|.
          6. -
        6. [=Queue a task=] to [=fire an event=] named {{updateend}} at |sourceBuffer|.
          7. +
        7. +
          If the |sourceBuffer| has a {{Promise}} in {{SourceBuffer/[[\pending append chunks + promise]]}}:
          +
          Reject that {{Promise}} with an {{AbortError}} {{DOMException}} and unset {{SourceBuffer/[[\pending append + chunks promise]]}}.
          +
          Otherwise:
          +
            +
          1. [=Queue a task=] to [=fire an event=] named {{abort}} at |sourceBuffer|.
            2. +
          2. [=Queue a task=] to [=fire an event=] named {{updateend}} at |sourceBuffer|.
            3. +
          +
          +
        @@ -1266,7 +1282,6 @@

        SourceBuffer Object

        [[HTML]] {{AudioTrackList}}, {{VideoTrackList}} and {{TextTrackList}} need Window+DedicatedWorker exposure.

        Attributes

        -

        REMOVE THIS PLACEHOLDER NOTE: TODO describe appendEncodedChunks, the updated changeType and linkage to WEBCODECS.

        mode of type AppendMode

        Controls how a sequence of [=media segments=] are handled. This attribute is initially set by {{MediaSource/addSourceBuffer()}} after the object is created, and can be updated by {{SourceBuffer/changeType()}} or setting this attribute.

        On getting, Return the initial value or the last value that was successfully set.

        @@ -1317,7 +1332,7 @@

        SourceBuffer Object

      • Return the current value of this attribute.
    • timestampOffset of type {{double}}
    -

    Controls the offset applied to timestamps inside subsequent [=media segments=] that are appended to this SourceBuffer. The {{SourceBuffer/timestampOffset}} is initially set to 0 which indicates that no offset is being applied.

    +

    Controls the offset applied to timestamps inside subsequent [=media segments=] TODO that are appended to this SourceBuffer. The {{SourceBuffer/timestampOffset}} is initially set to 0 which indicates that no offset is being applied.

    On getting, Return the initial value or the last value that was successfully set.

    On setting, run the following steps:

      @@ -1378,17 +1393,62 @@

      SourceBuffer Object

      The event handler for the {{error}} event.

    onabort of type {{EventHandler}}

    The event handler for the {{abort}} event.

    -

    Methods

    appendBuffer
    +
    +

    Methods

    +
    appendBuffer
    +

    Appends the segment data in an BufferSource[[!WEBIDL]] to the {{SourceBuffer}}.

      -
    1. Run the [=prepare append=] algorithm.
      2. +
    2. Run the [=prepare append=] algorithm. It may throw exception and abort, and if it does, abort these + steps, letting the caller receive the exception.
    3. Add |data:BufferSource| to the end of the {{SourceBuffer/[[input buffer]]}}.
    4. Set the {{SourceBuffer/updating}} attribute to true.
    5. [=Queue a task=] to [=fire an event=] named {{updatestart}} at this SourceBuffer object.
      6. -
    6. Asynchronously run the [=buffer append=] algorithm.
      7. +
    7. Asynchronously run the [=buffer append=] algorithm. +
      If this {{SourceBuffer}} is currently configured by the most recent + of the {{MediaSource/addSourceBuffer()}} call that created this object or a more recent + {{SourceBuffer/changeType()}} call to expect to buffer WebCodecs {{EncodedChunks}} via + {{SourceBuffer/appendEncodedChunks()}}, the asynchronous [=buffer append=] algorithm will detect this and + trigger the [=append error=] algorithm.
      +
    -
    ParameterTypeNullableOptionalDescription
    |data|{{BufferSource}}✘✘
    Return type: {{undefined}}
    abort
    +
    ParameterTypeNullableOptionalDescription
    |data|{{BufferSource}}✘✘
    Return type: {{undefined}}
    + +
    appendEncodedChunks
    +
    +

    Appends WebCodecs [[WEBCODECS]] {{EncodedChunks}} to the {{SourceBuffer}}.

    + +
      +
    1. Run the [=prepare append=] algorithm. If it throws exception and aborts, return a promise rejected + with that exception and abort these steps.
      2. +
    2. Add |chunks:EncodedChunks| to the end of {{SourceBuffer/[[\input webcodecs configs and + chunks]]}}.
      3. +
    3. Set the {{SourceBuffer/updating}} attribute to true. +
      This promise-based method differs from the asynchronous event-signalling + {{SourceBuffer/appendBuffer()}} method: promise resolution or rejection is the way to detect the + analogues of "abort", "error", "updatestart", "update", and "updateend" events that are not enqueued + by {{SourceBuffer/appendEncodedChunks()}}.
      +
      4. +
    4. Let |promise:Promise| be a new {{Promise}}.
      5. +
    5. Set {{SourceBuffer/[[\pending chunks append promise]]}} to be |promise|.
      6. +
    6. Return |promise| to the caller and run the rest of these steps asynchronously
      7. +
    7. Run the [=chunks append=] algorithm. +
      If this {{SourceBuffer}} is currently configured by the most recent + of the {{MediaSource/addSourceBuffer()}} call that created this object or a more recent + {{SourceBuffer/changeType()}} call to expect to buffer bytes parsed from a {{BufferSource}} via + {{SourceBuffer/appendBuffer()}}, the asynchronous [=chunks append=] algorithm will detect this and + trigger the [=append error=] algorithm.
      +
      8. +
    + + +
    ParameterTypeNullableOptionalDescription
    |chunks|{{EncodedChunks}}✘✘
    +
    Return type: {{Promise}} < {{undefined}} >
    +
    + + +
    abort

    Aborts the current segment and resets the segment parser.

      @@ -1399,12 +1459,20 @@

      SourceBuffer Object

      1. Abort the [=buffer append=] algorithm if it is running.
      2. Abort the [=chunks append=] algorithm if it is running.
        3. -
      3. If this object has a {{Promise}} in {{SourceBuffer/[[\pending append chunks promise]]}} then reject - that {{Promise}} with an {{AbortError}} {{DOMException}} and unset {{SourceBuffer/[[\pending append - chunks promise]]}}.
      4. Set the {{SourceBuffer/updating}} attribute to false.
        5. -
      5. [=Queue a task=] to [=fire an event=] named {{abort}} at this SourceBuffer object.
        6. -
      6. [=Queue a task=] to [=fire an event=] named {{updateend}} at this SourceBuffer object.
        7. +
      7. +
        If this object has a {{Promise}} in {{SourceBuffer/[[\pending append chunks + promise]]}}:
        +
        Reject that {{Promise}} with an {{AbortError}} {{DOMException}} and unset {{SourceBuffer/[[\pending append + chunks promise]]}}.
        +
        Otherwise:
        +
          +
        1. [=Queue a task=] to [=fire an event=] named {{abort}} at this {{SourceBuffer}} object.
          2. +
        2. [=Queue a task=] to [=fire an event=] named {{updateend}} at this {{SourceBuffer}} + object.
          3. +
        +
        +
    1. Run the [=reset parser state=] algorithm.
      2. @@ -1626,8 +1694,9 @@

      Segment Parser Loop

      empty when the {{SourceBuffer}} object is created.

      Each {{SourceBuffer}} object has a [[\buffer full flag]] internal - slot that keeps track of whether {{SourceBuffer/appendBuffer()}} is allowed to accept more bytes or {{SourceBuffer/appendEncodedChunks()}} is allowed to accept more encoded chunks. It is set - to false when the {{SourceBuffer}} object is created and gets updated as data is appended and removed.

      + slot that keeps track of whether {{SourceBuffer/appendBuffer()}} is allowed to accept more bytes or + {{SourceBuffer/appendEncodedChunks()}} is allowed to accept more encoded chunks. It is set to false when the + {{SourceBuffer}} object is created and gets updated as data is appended and removed.

      Each {{SourceBuffer}} object has a [[\group start timestamp]] internal slot that keeps track of the starting timestamp for a new [=coded frame group=] in the @@ -1729,6 +1798,15 @@

      Reset Parser State

    2. If the {{SourceBuffer/mode}} attribute equals {{AppendMode/""sequence""}}, then set the {{SourceBuffer/[[group start timestamp]]}} to the {{SourceBuffer/[[group end timestamp]]}}
    3. Remove all bytes from the {{SourceBuffer/[[input buffer]]}}.
      4. +
    4. If {{SourceBuffer/[[\input webcodecs configs and chunks]]}} contains any {{EncodedChunks}}, remove them + from {{SourceBuffer/[[\input webcodecs configs and chunks]]}}, but retain any {{AudioDecoderConfig}} or + {{VideoDecoderConfig}} that may be in that queue. +
      Keeping an unprocessed WebCodecs config in this queue lets future + {{SourceBuffer/appendEncodedChunks()}} reuse the config in the [=chunks append=] algorithm because + there is no other way to retain such an unprocessed config, except perhaps by the app calling + {{SourceBuffer/changeType()}} with the config again. +
      +
    5. Set {{SourceBuffer/[[append state]]}} to [=WAITING_FOR_SEGMENT=].
    @@ -2429,9 +2507,11 @@

    Coded Frame Removal

    Coded Frame Eviction

    -

    This algorithm is run to free up space in this {{SourceBuffer}} when new data is appended.

    +

    This algorithm is run to free up space in this {{SourceBuffer}} when new data is appended. New data is + either the bytes being appended via {{SourceBuffer/appendBuffer()}} or the WebCodecs {{EncodedChunks}} being + appended via {{SourceBuffer/appendEncodedChunks()}}.

      -
    1. Let |new data:BufferSource| equal the data that is about to be appended to this SourceBuffer. +
    2. Let |new data:BufferSource or EncodedChunks| equal the data that is about to be appended to this SourceBuffer.

      Need to recognize step here that implementations MAY decide to set {{SourceBuffer/[[buffer full flag]]}} true here if it predicts that processing |new data| in addition to any existing bytes in From 57e512d1a06576dd4cdbf4a54ef4b8fb54400950 Mon Sep 17 00:00:00 2001 From: Matt Wolenetz Date: Fri, 1 Oct 2021 03:34:34 -0700 Subject: [PATCH 6/6] MSE-for-WebCodecs: Complete the bulk remainder of the draft Defined [[pending append chunks promise]]. Defined [[input webcodecs configs and chunks]] Enforced all EncodedChunks have non-null duration during synchronous appendEncodedChunks(), after the step that runs the Prepare Append algorithm. Updated init-segment-received to mention [=chunks append=] algorithm can also call it. Updated coded-frame-processing to mention [=chunks append=] algorithm can also call it. Defined [=chunks append=] algorithm, Defined generation of DTS for frames from EncodedChunks (needed by coded frame processing algorithm) to be simply 0. Noted this assumes no discontinuity within or across appendEncodedChunks() calls, except as may be explicitly signalled by the app calling abort() or changeType(). Disambiguated (slightly) "SourceBuffer track configuration" versus "SourceBufferConfig". Updated changeType. Some notes on the current (M95) Chrome prototype implementation of this feature; all are expected to be fixed very soon: * crbug.com/1255048: it doesn't support changeType(SourceBufferConfig)) * crbug.com/1255050: it doesn't support h.264 EncodedVideoChunks' append support, and it assumes encoded chunks' DTS==PTS instead of using just 0 for all DTS of EncodedChunks' frames sent to the coded frame processing algorithm: this may require further refinement as noted in the spec as well if it is not working as expected once the prototype is updated. * crbug.com/1255052: it still hardcodes EncodedAudioChunk durations to be 22ms coded frames due to duration field originally not in EncodedAudioChunk specification, and it checks for EncodedVideoChunk duration in the middle of the Prepare Append steps instead of after that subalgorithm. --- media-source-respec.html | 317 ++++++++++++++++++++++++++++----------- 1 file changed, 232 insertions(+), 85 deletions(-) diff --git a/media-source-respec.html b/media-source-respec.html index a853efa..f792442 100644 --- a/media-source-respec.html +++ b/media-source-respec.html @@ -1,23 +1,5 @@ - Media Source Extensions™ @@ -268,19 +250,22 @@

      Definitions

      Random Access Point

      A position in a [=media segment=] where decoding and continuous playback can begin without relying on any previous data in the segment. For video this tends to be the location of I-frames. In the case of audio, most audio frames can be treated as a random access point. Since video tracks tend to have a more sparse distribution of random access points, the location of these points are usually considered the random access points for multiplexed streams.

      -
      SourceBuffer byte stream format specification
      -

      The specific [=byte stream format specification=] that describes the format of the byte stream accepted - by a {{SourceBuffer}} instance, when configured via {{MediaSource/addSourceBuffer()}} or updated by - {{SourceBuffer/changeType()}} to be based on a |typeOrConfig:TypeOrConfig| that is a DOMString. The [=byte - stream format specification=], for a {{SourceBuffer}} object, is initially selected based on the - |type:DOMString| passed to the {{MediaSource/addSourceBuffer()}} call that created the object, and can be - updated by {{SourceBuffer/changeType()}} calls on the object. When the {{SourceBuffer}} is instead using a - WebCodecs {{AudioDecoderConfig}} or {{VideoDecoderConfig}} [[WEBCODECS]] in the most recent of either the - {{MediaSource/addSourceBuffer()}} call that created the {{SourceBuffer}} or a {{SourceBuffer/changeType()}} - call, the {{SourceBuffer}} is not expecting byte stream appends via {{SourceBuffer/appendBuffer()}}; instead - it is expecting WebCodecs chunks to be appended via {{SourceBuffer/appendEncodedChunks()}}.

      - -
      SourceBuffer configuration
      +
      SourceBuffer byte stream format specification or WebCodecs chunks buffering expectations
      +

      + A {{SourceBuffer}} object can dynamically be configured to either expect to be sent a byte stream via + {{SourceBuffer/appendBuffer()}} that conforms to a specific [=byte stream format specification=], or to be + sent sequences of WebCodecs [[WEBCODECS]] chunks via {{SourceBuffer/appendEncodedChunks()}}. + When configured to expect a byte stream, the SourceBuffer byte + stream format specification is the specific [=byte stream format specification=] that describes the + format of the byte stream expected and accepted by a {{SourceBuffer}} object is initially selected based on + the |type:DOMString| passed to the {{MediaSource/addSourceBuffer()}} call that created the object, and can be + updated by {{SourceBuffer/changeType()}} calls on the object. + When the {{SourceBuffer}} is instead using a WebCodecs {{AudioDecoderConfig}} or {{VideoDecoderConfig}} + [[WEBCODECS]] in the most recent of either the {{MediaSource/addSourceBuffer()}} call that created the + {{SourceBuffer}} or a {{SourceBuffer/changeType()}} call, the {{SourceBuffer}} is instead expecting WebCodecs + chunks to be appended via {{SourceBuffer/appendEncodedChunks()}}.

      + +
      SourceBuffer track configuration

      A specific set of tracks distributed across one or more SourceBuffer objects owned by a single MediaSource instance.

      Implementations MUST support at least 1 MediaSource object with the following @@ -451,14 +436,14 @@

      Attributes

    3. If |typeOrConfig| has neither a neither a [=valid AudioDecoderConfig=] in {{SourceBufferConfig/audioConfig}} nor a [=valid VideoDecoderConfig=] in {{SourceBufferConfig/videoConfig}}, then throw a {{TypeError}} exception and abort these steps and - this method.
      4. + this method.
    4. If |typeOrConfig| contains a decoder configuration that is not supported or is not supported with the types specified for the other {{SourceBuffer}} objects in {{MediaSource/sourceBuffers}}, then throw a {{NotSupportedError}} exception and abort these steps and this method.
  • If the user agent can't handle any more {{SourceBuffer}} objects or if creating a {{SourceBuffer}} based - on |typeOrConfig| would result in an unsupported [=SourceBuffer configuration=], then throw a + on |typeOrConfig| would result in an unsupported [=SourceBuffer track configuration=], then throw a {{QuotaExceededError}} exception and abort these steps.

    For example, a user agent MAY throw a {{QuotaExceededError}} exception if the media element has reached the {{HTMLMediaElement/HAVE_METADATA}} readyState. This can occur if the user agent's media engine does not support adding more tracks during @@ -472,12 +457,18 @@

    Attributes

    If |typeOrConfig| is a DOMString:
    Set the {{SourceBuffer/[[generate timestamps flag]]}} on the new object to the value in the "Generate Timestamps Flag" column of the byte stream format registry [[MSE-REGISTRY]] entry that is - associated with |type|. + associated with |typeOrConfig|.
    Otherwise:
    -
    Set the {{SourceBuffer/[[generate timestamps flag]]}} on the new object to false. -

    WebCodecs encoded chunks are required to have timestamps, so there is no need to - generate them.

    +
      +
    1. Set the {{SourceBuffer/[[generate timestamps flag]]}} on the new object to false. +
      WebCodecs encoded chunks are required to have timestamps, so there is no need to + generate them.
      +
      2. +
    2. Copy the {{SourceBufferConfig}} from |typeOrConfig| into the new object's {{SourceBuffer/[[input + webcodecs configs and chunks]]}} for potential handling later during that object's [=chunks append=] + algorithm.
      3. +
    • @@ -513,10 +504,10 @@

    Attributes

  • Abort the [=chunks append=] algorithm if it is running.
  • Set the |sourceBuffer|.{{SourceBuffer/updating}} attribute to false.
  • -
    If the |sourceBuffer| has a {{Promise}} in {{SourceBuffer/[[\pending append chunks +
    If the |sourceBuffer| has a {{Promise}} in {{SourceBuffer/[[pending append chunks promise]]}}:
    -
    Reject that {{Promise}} with an {{AbortError}} {{DOMException}} and unset {{SourceBuffer/[[\pending append - chunks promise]]}}.
    +
    Reject that {{Promise}} with an {{AbortError}} {{DOMException}} and unset + {{SourceBuffer/[[pending append chunks promise]]}}.
    Otherwise:
    1. [=Queue a task=] to [=fire an event=] named {{abort}} at |sourceBuffer|.
      2. @@ -1231,14 +1222,24 @@

      SourceBuffer Object

      enum AppendMode {
     "segments",
     "sequence"
-};
      - - + + @@ -1586,9 +1637,9 @@

      SourceBuffer Object

      Track Buffers

      -

      A track buffer stores the [=track descriptions=] and [=coded frames=] for an individual - track. The track buffer is updated as [=initialization segments=] and [=media segments=] are appended to the - SourceBuffer.

      +

      A track buffer stores the [=track descriptions=] and [=coded frames=] for an + individual track. The track buffer is updated as [=initialization segments=], [=media segments=], decoder + configurations and {{EncodedChunks}} are given to the {{SourceBuffer}}.

      Each [=track buffer=] has a last decode timestamp variable that stores the decode timestamp of the last [=coded frame=] appended in the current [=coded frame group=]. The variable is initially @@ -1723,6 +1774,7 @@

      Segment Parser Loop

      This flag is set by {{MediaSource/addSourceBuffer()}} when the {{SourceBuffer}} object is created and is updated by {{SourceBuffer/changeType()}}.

      +

      When the segment parser loop algorithm is invoked, run the following steps:

        @@ -1798,10 +1850,10 @@

        Reset Parser State

      1. If the {{SourceBuffer/mode}} attribute equals {{AppendMode/""sequence""}}, then set the {{SourceBuffer/[[group start timestamp]]}} to the {{SourceBuffer/[[group end timestamp]]}}
      2. Remove all bytes from the {{SourceBuffer/[[input buffer]]}}.
        3. -
      3. If {{SourceBuffer/[[\input webcodecs configs and chunks]]}} contains any {{EncodedChunks}}, remove them - from {{SourceBuffer/[[\input webcodecs configs and chunks]]}}, but retain any {{AudioDecoderConfig}} or - {{VideoDecoderConfig}} that may be in that queue. -
        Keeping an unprocessed WebCodecs config in this queue lets future +
      4. If {{SourceBuffer/[[input webcodecs configs and chunks]]}} contains any {{EncodedChunks}}, remove them + from {{SourceBuffer/[[input webcodecs configs and chunks]]}}, but retain any {{AudioDecoderConfig}} or + {{VideoDecoderConfig}} that may be in that internal slot. +
        Keeping an unprocessed WebCodecs config in this internal slot lets future {{SourceBuffer/appendEncodedChunks()}} reuse the config in the [=chunks append=] algorithm because there is no other way to retain such an unprocessed config, except perhaps by the app calling {{SourceBuffer/changeType()}} with the config again. @@ -1817,8 +1869,18 @@

        Append Error

        1. Run the [=reset parser state=] algorithm.
        2. Set the {{SourceBuffer/updating}} attribute to false.
          3. -
        3. [=Queue a task=] to [=fire an event=] named {{error}} at this SourceBuffer object.
          4. -
        4. [=Queue a task=] to [=fire an event=] named {{updateend}} at this SourceBuffer object.
          5. +
        5. +
          If this {{SourceBuffer}} has a {{Promise}} in {{SourceBuffer/[[pending append chunks + promise]]}}:
          +
          Reject that {{Promise}} with an {{AbortError}} {{DOMException}} and unset {{SourceBuffer/[[pending + append chunks promise]]}}.
          +
          Otherwise:
          +
            +
          1. [=Queue a task=] to [=fire an event=] named {{error}} at this {{SourceBuffer}} object.
            2. +
          2. [=Queue a task=] to [=fire an event=] named {{updateend}} at this {{SourceBuffer}} object.
            3. +
          +
          +
        6. Run the [=end of stream=] algorithm with the |error:EndOfStreamError| parameter set to {{EndOfStreamError/""decode""}}.
        @@ -1864,8 +1926,15 @@

        Prepare Append

        Buffer Append

        -

        When {{SourceBuffer/appendBuffer()}} (TODO split out chunk processing) is called, the following steps are run to process the appended data.

        +

        When {{SourceBuffer/appendBuffer()}} is called, the following steps are run to process the appended data.

          +
        1. If this {{SourceBuffer}} is currently configured to expect processing of WebCodecs {{EncodedChunks}}, + then run the [=append error=] algorithm and abort this algorithm. +
          The expectation of processing an appended bytestream versus appended {{EncodedChunks}} + is based on the most recently successful execution of the initial {{MediaSource/addSourceBuffer()}} that + created this {{SourceBuffer}} or potentially a more recent {{SourceBuffer/changeType()}} call that may + have changed the expectation.
          +
        2. Run the [=segment parser loop=] algorithm.
        3. If the [=segment parser loop=] algorithm in the previous step was aborted, then abort this algorithm.
        4. Set the {{SourceBuffer/updating}} attribute to false.
          5. @@ -1874,6 +1943,82 @@

          Buffer Append

        +
        +

        Chunks Append

        +

        Each {{SourceBuffer}} object has a [[\pending append chunks + promise]] internal slot that stores the promise necessary to communicate the completion of + asynchronous steps begun by a call to {{SourceBuffer/appendEncodedChunks()}}. If there is no asynchronous + {{SourceBuffer/appendEncodedChunks()}} operation in progress, then this slot is unset.

        + +

        Each {{SourceBuffer}} object has an [[\input webcodecs configs and + chunks]] internal slot that stores a queue of unprocessed {{EncodedChunks}} and at most one + {{SourceBufferConfig}}. The {{SourceBufferConfig}} is added if the initial call to + {{MediaSource/addSourceBuffer()}} that created this object was provided a {{SourceBufferConfig}}, or if + there is a more recent successful call to {{SourceBuffer/changeType()}} on this {{SourceBuffer}} that + provided a {{SourceBufferConfig}}. The contents of this slot are processed by the [=chunks append=] + algorithm to buffer the necessary decoder configuration and coded frames into the underlying [=track + buffer=].

        + +

        When the chunks append algorithm is invoked, run the following steps to process the appended + {{EncodedChunks}} relative to the current {{SourceBufferConfig}}:

        +
          +
        1. If this {{SourceBuffer}} is currently configured to expect processing of an appended bytestream, + then run the [=append error=] algorithm and abort this algorithm. +
          The expectation of processing an appended bytestream versus appended {{EncodedChunks}} + is based on the most recently successful execution of the initial {{MediaSource/addSourceBuffer()}} that + created this {{SourceBuffer}} or potentially a more recent {{SourceBuffer/changeType()}} call that may + have changed the expectation.
          +
          2. +
        2. If there is a {{SourceBufferConfig}} in this {{SourceBuffer}}'s {{SourceBuffer/[[input webcodecs configs + and chunks]]}}, then run the following steps: +
            +
          1. Let |config:SourceBufferConfig| be the {{SourceBufferConfig}} from {{SourceBuffer/[[input webcodecs + configs and chunks]]}}. +
            There must be precisely one {{AudioDecoderConfig}} or {{VideoDecoderConfig}} in + |config|; this condition was already enforced by the {{MediaSource/addSourceBuffer()}} or + {{SourceBuffer/changeType()}} call that placed |config| into {{SourceBuffer/[[input webcodecs + configs and chunks]]}}.
            +
            2. +
          2. Remove the {{SourceBufferConfig}} from {{SourceBuffer/[[input webcodecs configs and chunks]]}}.
            3. +
          3. Run the [=initialization segment received=] algorithm with |config| denoting a single track + configuration. If that algorithm aborts, then abort this algorithm.
            4. +
          +
          3. +
        3. In the order in which they were added to {{SourceBuffer/[[input webcodecs configs and chunks]]}}, + prepare the {{EncodedChunks}} for coded frame processing by running the following steps for each chunk: +
            +
          1. Let |chunk:EncodedAudioChunk or EncodedVideoChunk| be the next encoded chunk being prepared, + dequeued (removed) from the front of {{SourceBuffer/[[input webcodecs configs and chunks]]}}.
            2. +
          2. If |chunk| is an {{EncodedAudioChunk}} but the most recently processed |config| by this object was a + {{VideoDecoderConfig}}, or if |chunk| is an {{EncodedVideoChunk}} but the most recently processed + |config| by this object was an {{AudioDecoderConfig}}, then run the [=append error=] algorithm and + abort this algorithm. +
            3. +
          3. Create a coded frame from |chunk|. Treat the frame as a random access point iff |chunk|'s type + is "key". Set the frame's decode timestamp to be 0. +
            Using a constant decode timestamp disables the [=coded frame processing=] + algorithm's discontinuity detection. Note that other mechanisms for signalling discontinuous + sequences of frames survive this simplification, with the most reliable yet lightweight being calling + {{SourceBuffer/abort()}} when {{SourceBuffer/updating}} is false. A more heavyweight option would be + to call {{SourceBuffer/changeType()}}. Also note that, though WebCodecs does not define a decode + timestamp attribute for encoded chunks, if reliable buffering of chunks into MSE needs real decode + timestamps, this spec may be refined to improve that support. +
            +
            4. +
          +
          4. +
        4. Run the [=coded frame processing=] algorithm with the frames in order from the previous step. +
          This algorithm, unlike the [=segment parser loop=] algorithm, deterministically + consumes and clears all of its input state and processes all corresponding coded frames. Even the steps + which might abort this algorithm early do so after the [=append error=] algorithm clears any remaining + input state and rejects the promise. +
          +
          5. +
        5. Set the {{SourceBuffer/updating}} attribute to false. +
        6. Resolve the {{SourceBuffer/[[pending append chunks promise]]}} and unset that internal slot.
          7. +
        +
        +

        Range Removal

        Follow these steps when a caller needs to initiate a JavaScript visible range removal @@ -1893,7 +2038,8 @@

        Range Removal

        Initialization Segment Received

        -

        The following steps are run when the [=segment parser loop=] successfully parses a complete [=initialization segment=]:

        +

        The following steps are run when the [=segment parser loop=] successfully parses a complete + [=initialization segment=] or the [=chunks append=] algorithm handles a {{SourceBufferConfig}}:

        Each SourceBuffer object has a [[\first initialization segment received flag]] internal slot that tracks whether the first [=initialization segment=] has been appended and received by this algorithm. This @@ -2254,10 +2400,11 @@

        Initialization Segment Received

        Coded Frame Processing

        -

        When complete [=coded frames=] have been parsed by the [=segment parser loop=] then the following steps are run:

        +

        When complete [=coded frames=] have been parsed by the [=segment parser loop=] or emitted by the [=chunks + append=] algorithm, then the following steps are run:

        1. -

          For each [=coded frame=] in the [=media segment=] run the following steps:

          +

          For each [=coded frame=] run the following steps:

          1. Loop Top:
            If {{SourceBuffer/[[generate timestamps flag]]}} equals true:
            @@ -2444,7 +2591,7 @@

            Coded Frame Processing

            If the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute is {{HTMLMediaElement/HAVE_FUTURE_DATA}} and the new [=coded frames=] cause {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}} to have a {{TimeRanges}} that includes the current playback position and [=enough data to ensure uninterrupted playback=], then set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to {{HTMLMediaElement/HAVE_ENOUGH_DATA}}.

            Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

            2. -
          2. If the [=media segment=] contains data beyond the current {{MediaSource/duration}}, then run the +
          3. If the result of coded frame processing buffers data beyond the current {{MediaSource/duration}}, then run the [=duration change=] algorithm with |new duration:unrestricted double| set to the maximum of the current duration and the {{SourceBuffer/[[group end timestamp]]}}.
      Enumeration description
      segments -

      The timestamps in the media segment determine where the [=coded frames=] are placed in the presentation. Media segments can be appended in any order.

      -
      sequence -

      Media segments will be treated as adjacent in time independent of the timestamps in the media segment. Coded frames in a new media segment will be placed immediately after the coded - frames in the previous media segment. The {{SourceBuffer/timestampOffset}} attribute will be updated if a new offset is needed to make the new media segments adjacent to the previous media segment. - Setting the {{SourceBuffer/timestampOffset}} attribute in {{AppendMode/""sequence""}} mode allows a media segment to be placed at a specific position in the timeline without any knowledge - of the timestamps in the media segment. -

      +}; + +
      Enumeration description
      segments +

      The timestamps in the media segment or {{EncodedChunks}} determine where the [=coded frames=] are placed in + the presentation. Media segments can be appended in any order. Discontinuous {{EncodedChunks}} should not be + appended in the same {{SourceBuffer/appendEncodedChunks()}} call: the application should instead wait for the + {{SourceBuffer}} to no longer be {{SourceBuffer/updating}}, and then call {{SourceBuffer/abort()}} or + {{SourceBuffer/changeType()}} before appending {{EncodedChunks}} that are discontinuous with those in the + previous {{SourceBuffer/appendEncodedChunks()}} call.

      +
      sequence +

      Media segments will be treated as adjacent in time independent of the timestamps in the media segment. Coded + frames in a new media segment will be placed immediately after the coded frames in the previous media segment. + As with {{AppendMode/""segments""}} mode, discontinuous {{EncodedChunks}} should not be appended in the same + {{SourceBuffer/appendEncodedChunks()}} call. + The {{SourceBuffer/timestampOffset}} attribute will be updated if a new offset is needed to make the new media + segments adjacent to the previous media segment. Setting the {{SourceBuffer/timestampOffset}} attribute in + {{AppendMode/""sequence""}} mode allows a media segment to be placed at a specific position in the timeline + without any knowledge of the timestamps in the media segment.

      @@ -1332,7 +1333,7 @@ 
      SourceBuffer Object
      
             
    2. Return the current value of this attribute.
      3. 
           
         
      timestampOffset of type {{double}}
      
-          
      Controls the offset applied to timestamps inside subsequent [=media segments=] TODO that are appended to this SourceBuffer. The {{SourceBuffer/timestampOffset}} is initially set to 0 which indicates that no offset is being applied.
      
+        
      Controls the offset applied to timestamps inside subsequent [=media segments=] or {{EncodedChunks}} that are appended to this SourceBuffer. The {{SourceBuffer/timestampOffset}} is initially set to 0 which indicates that no offset is being applied.
      
           
      On getting, Return the initial value or the last value that was successfully set.
      
           
      On setting, run the following steps:
      
           
        
@@ -1400,8 +1401,8 @@ 
        SourceBuffer Object
        
             
        Appends the segment data in an BufferSource[[!WEBIDL]] to the {{SourceBuffer}}.
        
 
           
          
-            
        1. Run the [=prepare append=] algorithm. It may throw exception and abort, and if it does, abort these
-              steps, letting the caller receive the exception.
          2. 
+            
        2. Run the [=prepare append=] algorithm. If it throws exception and aborts, then these steps are also
+              aborted.
          3. 
             
        3. Add |data:BufferSource| to the end of the {{SourceBuffer/[[input buffer]]}}.
          4. 
             
        4. Set the {{SourceBuffer/updating}} attribute to true.
          5. 
             
        5. [=Queue a task=] to [=fire an event=] named {{updatestart}} at this SourceBuffer object.
          6. 
@@ -1420,10 +1421,16 @@ 
          SourceBuffer Object
          
             
          Appends WebCodecs [[WEBCODECS]] {{EncodedChunks}} to the {{SourceBuffer}}.
          
 
             
            
-              
          1. Run the [=prepare append=] algorithm. If it throws exception and aborts, return a promise rejected
-                with that exception and abort these steps.
            2. 
-              
          2. Add |chunks:EncodedChunks| to the end of {{SourceBuffer/[[\input webcodecs configs and
-                chunks]]}}.
            3. 
+              
          3. Run the [=prepare append=] algorithm. If it throws exception and aborts, then these steps are also
+                aborted.
            4. 
+              
          4. If any of the chunks in |chunks:EncodedChunks| have a null value for their duration attribute, throw a
+                {{TypeError}} exception and abort these steps.
+                
            This requirement is necessary to understand all chunks' presentation intervals in
+                  the coded frame processing algorithm, enabling correct buffering when frames overlap other frames, are later
+                  overlapped by other frames, or are in a disjoint portion of the presentation timeline.
+                
            
+              
            5. 
+              
          5. Add |chunks| to the end of {{SourceBuffer/[[input webcodecs configs and chunks]]}}.
            6. 
               
          6. Set the {{SourceBuffer/updating}} attribute to true.
                 
            This promise-based method differs from the asynchronous event-signalling
                   {{SourceBuffer/appendBuffer()}} method: promise resolution or rejection is the way to detect the
@@ -1431,8 +1438,8 @@ 
            SourceBuffer Object
            
                   by {{SourceBuffer/appendEncodedChunks()}}.
            
               
            7. 
               
          7. Let |promise:Promise| be a new {{Promise}}.
            8. 
-              
          8. Set {{SourceBuffer/[[\pending chunks append promise]]}} to be |promise|.
            9. 
-              
          9. Return |promise| to the caller and run the rest of these steps asynchronously
            10. 
+              
          10. Set {{SourceBuffer/[[pending append chunks promise]]}} to be |promise|.
            11. 
+              
          11. Return |promise| to the caller and run the rest of these steps asynchronously.
            12. 
               
          12. Run the [=chunks append=] algorithm.
                 
            If this {{SourceBuffer}} is currently configured by the most recent
                   of the {{MediaSource/addSourceBuffer()}} call that created this object or a more recent
@@ -1461,10 +1468,9 @@ 
            SourceBuffer Object
            
                 
          13. Abort the [=chunks append=] algorithm if it is running.
            14. 
                 
          14. Set the {{SourceBuffer/updating}} attribute to false.
            15. 
                 
          15. 
-                    
            If this object has a {{Promise}} in {{SourceBuffer/[[\pending append chunks
-                    promise]]}}:
            
-                    
            Reject that {{Promise}} with an {{AbortError}} {{DOMException}} and unset {{SourceBuffer/[[\pending append
-                    chunks promise]]}}.
            
+                    
            If this object has a {{Promise}} in {{SourceBuffer/[[pending append chunks promise]]}}:
            
+                    
            Reject that {{Promise}} with an {{AbortError}} {{DOMException}} and unset
+                    {{SourceBuffer/[[pending append chunks promise]]}}.
            
                     
            Otherwise:
            
                     
              
                       
            1. [=Queue a task=] to [=fire an event=] named {{abort}} at this {{SourceBuffer}} object.
              2. 
@@ -1481,12 +1487,38 @@ 
              SourceBuffer Object
              
           
            
         
            No parameters.
            Return type: {{undefined}}
            
         
            changeType
            
-        
            Changes the MIME type associated with this object (TODO: split logic by arg type, reference appendEncodedChunks() too). Subsequent {{SourceBuffer/appendBuffer()}} calls will expect the newly appended bytes to conform to the new type.
+          
            Changes the [=SourceBuffer byte stream format specification or WebCodecs chunks buffering expectations=] for
+            this object. Enables switching among bytestreams, codecs, or even between bytestream parsing versus
+            buffering of WebCodecs encoded chunks in the same {{SourceBuffer}}.
            
           
              
-            
            1. If |type:DOMString| is an empty string then throw a {{TypeError}} exception and abort these steps.
              2. 
-            
            2. If this object has been removed from the {{MediaSource/sourceBuffers}} attribute of the [=parent media source=], then throw an {{InvalidStateError}} exception and abort these steps.
              3. 
-            
            3. If the {{SourceBuffer/updating}} attribute equals true, then throw an {{InvalidStateError}} exception and abort these steps.
              4. 
-            
            4. If |type| contains a MIME type that is not supported or contains a MIME type that is not supported with the types specified (currently or previously) of {{SourceBuffer}} objects in the {{MediaSource/sourceBuffers}} attribute of the [=parent media source=], then throw a {{NotSupportedError}} exception and abort these steps.
              5. 
+            
            5. If |typeOrConfig:TypeOrConfig| is an empty DOMString, then throw a {{TypeError}} exception and abort
+              these steps.
              6. 
+            
            6. If |typeOrConfig| is not a DOMString, then run the following steps:
+              
                
+                
              1. If |typeOrConfig| has both an {{SourceBufferConfig/audioConfig}} and a
+                  {{SourceBufferConfig/videoConfig}} or has neither, then throw a {{TypeError}} exception and abort
+                  these steps and this method.
                2. 
+                
              2. If |typeOrConfig| has neither a neither a [=valid AudioDecoderConfig=] in
+                  {{SourceBufferConfig/audioConfig}} nor a [=valid VideoDecoderConfig=] in
+                  {{SourceBufferConfig/videoConfig}}, then throw a {{TypeError}} exception and abort these steps and
+                  this method.
                3. 
+              
              
+            
              7. 
+
+            
            7. If this object has been removed from the {{MediaSource/sourceBuffers}} attribute of the [=parent media
+              source=], then throw an {{InvalidStateError}} exception and abort these steps.
              8. 
+
+            
            8. If the {{SourceBuffer/updating}} attribute equals true, then throw an {{InvalidStateError}} exception
+              and abort these steps.
              9. 
+
+            
            9. If |typeOrConfig| is a DOMString that contains a MIME type that is not supported or that contains a MIME
+              type that is not supported with the types specified (currently or previously) of {{SourceBuffer}} objects
+              in the {{MediaSource/sourceBuffers}} attribute of the [=parent media source=], then throw a
+              {{NotSupportedError}} exception and abort these steps.
              10. 
+            
            10. If |typeOrConfig| is not a DOMString, and if it contains a decoder configuration that is not supported
+              or is not supported with the types specified (currently or previously) of {{SourceBuffer}} objects in the
+              {{MediaSource/sourceBuffers}} attribute of the [=parent media source=], then throw a {{NotSupportedError}}
+              exception and abort these steps.
              11. 
             
            11. 
               
              If the {{MediaSource/readyState}} attribute of the [=parent media source=] is in the {{ReadyState/""ended""}} state then run the following steps:
              
               
                
@@ -1495,9 +1527,28 @@ 
                SourceBuffer Object
                
               
              
             
              12. 
             
            12. Run the [=reset parser state=] algorithm.
              13. 
-            
            13. Update the {{SourceBuffer/[[generate timestamps flag]]}} on this {{SourceBuffer}} object to the value in
-              the "Generate Timestamps Flag" column of the byte stream format registry [[MSE-REGISTRY]] entry that is
-              associated with |type|.
              14. 
+
+            
            14. 
+              
              
+                
              If |typeOrConfig| is a DOMString:
              
+                
              Update the {{SourceBuffer/[[generate timestamps flag]]}} on this {{SourceBuffer}} object to the value in the
+                "Generate Timestamps Flag" column of the byte stream format registry [[MSE-REGISTRY]] entry that is
+                associated with |typeOrConfig|.
+                
              
+                
              Otherwise:
              
+                
                
+                  
              1. Update the {{SourceBuffer/[[generate timestamps flag]]}} on this {{SourceBuffer}} object to false.
+                    
                WebCodecs encoded chunks are required to have timestamps, so there is no need to
+                generate them.
                
+                  
                2. 
+                  
              2. Remove any {{SourceBufferConfig}} that may be in this {{SourceBuffer}} object's
+                    {{SourceBuffer/[[input webcodecs configs and chunks]]}}.
                3. 
+                  
              3. Copy the {{SourceBufferConfig}} from |typeOrConfig| into this {{SourceBuffer}} object's {{SourceBuffer/[[input
+                    webcodecs configs and chunks]]}} for potential handling later during [=chunks append=] algorithm.
                4. 
+                
              
+                
              
+              
              
+            
              15. 
             
            15. 
                 
              
                   
              If the {{SourceBuffer/[[generate timestamps flag]]}} equals true:
              
@@ -1523,8 +1574,8 @@ 
              SourceBuffer Object
      		Description
      |type|{{DOMString}}|typeOrConfig|{{TypeOrConfig}} ✘ ✘