emsg-processing-model.md

DASH inband event processing using MSE data model

Iraj Sodagar, irajs@live.com

Tencent America

2021-05-17

Introduction

This document provides an extended W3C Media Source Extensions (MSE) model for the processing of DASH inband events.

Note: The current MSE specification does not support the processing of inband events and this document is just one possible illustrative design on how MSE can be extended to support such functionality.

Dispatch modes

See https://www.w3.org/TR/media-timed-events/#event-triggering

For those events that the application has subscribed to receive, the API should:

• Generate a DOM event when an in-band media timed event cue is parsed from the media container or media stream (DASH-IF on-receive mode).
• Generate DOM events when the current media playback position reaches the start time and the end time of a media timed event cue during playback (DASH-IF on-start mode). This applies equally to cues generated by the user agent when parsed from the media container and cues added by the web application.

In general, it is not possible for the UA to know which dispatch mode to use for any given event type. So we introduce the following API so that the web application can tell the UA:

enum DispatchMode {
  "onstart",
  "onreceive"
};

interface InbandEventTrack extends TextTrack {
    undefined subscribe(DOMString eventType, DispatchMode dispatchMode);
    undefined unsubscribe(DOMString eventType);
};

TODO: How to construct eventType from emsg scheme_id_uri and value?

TODO: How to construct an InbandEventTrack? Some UAs already process inband events using TextTrack and create the TextTrack automatically.

Process@append rule

The process@append rule defines how the inband events of a segment are processed, i.e. parsed, and dispatched or scheduled to dispatch at the time of appending the segment to the MSE SourceBuffer using appendBuffer().

In the case of an inband event whose eventType has not been subscribed by the web application:

1. Discard the event

In the case of an inband event whose eventType has been subscribed by the web application with the onreceive dispatch mode:

1. If the event end time is not smaller than the current playback position, and
2. If this event or an equivalent has not been dispatched before,

Then the dispatcher dispatches the event immediately.

In the case of an inband event whose eventType has been subscribed by the web application with the onstart dispatch mode:

1. If the current playback position is not smaller than the event start time, and
2. If the current playback position is not equal or larger than the event end time, and
3. If this event or an equivalent has not been dispatched before,

Then the event is stored in a dispatch buffer for dispatching at the event start time.

Dispatch buffer timing model

Figure 1 demonstrates an inband event with the onstart dispatch mode relative to the MSE timing model.

Figure 1. Media source and in-band event dispatch buffers

Implementation

Figure 2 demonstrates an example of the overlapping events with onstart dispatch mode.

Figure 2. Event buffer model example for onstart events

As is shown above, emsgs E0, E1, and E2 are mapped to the dispatch buffer. With the initial appending of the S1 media segment to the media buffer, the ranges between the event's start and the event's end are marked in the dispatch buffer for E0 and E1.

When S2 is appended to the media buffer, in this case since E2 overlaps with E1 in the dispatch buffer, the corresponding range in the dispatched first is divided into 3 subranges, as shown in the figure.

Figure 3 demonstrates an example of an overwrite, in which the segment S2 is overwritten by a new segment S2' (that does not contain any emsgs) and has a duration that only covers a portion of S2 in the media buffer.

Figure 3. Overwrite of a part of a segment with events having onstart dispatch mode

As shown, since the event E2 has the onstart dispatch mode, its range in the dispatch buffer is unchanged.

Algorithms

Initialization

1. Applications inputs to DASH client
   1. Subscribe SchemeIdURI/value
   2. Provide dispatch mode
2. Event buffer initialization:
   1. Event dispatch (range of event purge may go beyond media buffer)
3. Set Presentation Time Offset (PTO)

Append

Add the following steps to the MSE Segment Parser Loop, which is called from the Buffer Append Algorithm. After step 6.2., which describes handling of coded frames, add:

1. If the input buffer contains one or more inband event messages, then run the inband event processing algorithm.

Inband Event Processing

The inband event processing algorithm is a new algorithm which we propose to add to MSE.

When inband event messages have been parsed by the segment parser loop then the following steps are run:

1. For each inband event message in the media segment run the following steps:
   1. Parse the emsg
   2. Generate the eventType from the emsg.scheme_id_uri and emsg.value
   3. Look up the eventType in the InbandEventTrack's list of subscribed event types
      1. If not present, discard the emsg and abort these steps
   4. Calculate the startTime and endTime values for the DataCue:
      1. For emsg v0 (esmg.version = 0): startTime = segment_start + emsg.presentation_time_delta / emsg.time_scale
      2. For emsg v1 (emsg.version = 1): startTime = emsg.presentation_time / emsg.timescale
      3. If emsg.duration is 0xFFFFFFFF then endTime = +Infinity, else endTime = startTime + emsg.duration / emsg.timescale
   5. If there is an equivalent event message in the InbandEventTrack's list of text track cues, discard the event message and abort these steps. An event message is equivalent if its id, scheme_id_uri, and value values are the same those of any existing
   6. Construct a DataCue instance with the following attributes:
      1. startTime (as calculated above)
      2. endTime (as calculated above)
      3. id = emsg.id
      4. pauseOnExit = false
      5. type = emsg.scheme_id_uri
      6. value = { data: emsg.message_data, emsgValue: emsg.value }
   7. If the subscription's dispatch mode is onreceive, queue a task to fire an event named addcue at the InbandEventTrack object with the cue attribute initialized to the new DataCue object
   8. If the subscription's dispatch mode is onstart, run the HTML addCue() steps with the new DataCue object

Dispatch

TODO: Firing of cue enter and exit for event messages in onstart dispatch mode is handled by the HTML time marches on steps

1. Find the events occurring in the dispatch buffer at the playback position.
2. For each event
   1. If its emsg.id is not in the "already-dispatched" table,
      1. Dispatch the event
      2. Add its emsg.id to the "already-dispatched" table
      3. Remove the event from the dispatch buffer
   2. Otherwise, remove the event from the dispatch buffer

Purge

TODO: Purging is controlled by the web application, by calling SourceBuffer.remove(startTime, endTime). The MSE Range Removal algorithm applies. Should this algorithm also remove cues that lie in the removed time range?

In a purge operation, either a range from the start or a range from the end of the media buffer is purged. This range is referred to as the "purged-range" in this subclause.

1. If any event in the dispatch buffer overlaps with the purged-range
   1. Split the event into two events around the purge-range boundary
2. Remove the purged-range from the dispatch buffer