EMSG box support

[joeyparrish]

We could support EMSG boxes through a network response filter on segments. The filter would parse the MP4 segments looking for EMSG boxes, then emit JavaScript events through Player if found.

Please comment and give specifics if this is of interest to you.

[lwchan2k1]

I am very interested in this feature which can enable advertising support with multi period support together.

[dougdoe]

I have now also subscribed to this issue. Lewis (above) indicates why we need this, but thought I could add a little context. Our third-party advertising vendor requires that the player to support the following features:

1. DASH multi-period manifest (w/ support for switching between DRM and non-DRM streams across periods) – I think this requirement is fulfilled given Various issues with multi-period live streams #310 and Update end time of last segment in a period when new periods are added #315.
2. EMSG box support as per DASH specification, which is this enhancement issue.

Since this enhancement is not planned for Shaka v2.0, we may need to look at implementing this ourselves. If we move ahead with this, we will contact you to discuss a potential solution.

Let us know if you have any feedback or guidance that may help us here. Thanks (as always)!

[joeyparrish]

We would be happy to discuss design and ultimately to review a pull request for this. Thanks for expressing your interest!

What we had in mind is a network filter that would listen for the responses to segment requests and parse those segments before they are processed by StreamingEngine and MediaSourceEngine.

The only tutorial we have right now that discusses network response filters is docs/tutorials/license-wrapping.md, so you can look at that for examples of how response filters work. For EMSG, we'd be filtering on RequestType.SEGMENT instead of RequestType.LICENSE.

I imagine the parsing would require a generic MP4 box parser to locate the EMSG box, if present. There may be some other way to approach it as well, but a generic MP4 box parser would be useful in other places, too, such as VTT (#277) and TTML (#278) text embedded in MP4.

The data extracted from EMSG would be dispatched as an event from Player. All events are constructed with shaka.util.FakeEvent and their types and structures documented in jsdoc annotations in player.js.

The whole feature would ideally be modular and easily omitted from the build. See docs/tutorials/plugins.md for an overview of plugins and how they exist in the build system.

[joeyparrish]

We're changing direction on this slightly. Instead of dispatching emsg events from Player, we will use the presence of an emsg box to trigger an MPD update. For efficiency, we will only filter segments for emsg boxes if the manifest indicates that they are present. This appears to be the only use of the emsg box from what I can tell.

[dougdoe]

Hi Joey, reading over your latest update, I'm not sure this is sufficient. We are planning on using EMSG box for the following:

1. ID3 tags – ID3 tags will be inserted into EMSG box and when detected by the player, we will need to take action
2. Dynamic Ad Insertion

In both cases, I believe we will need EMSG events to be dispatched from the player. Does this make sense?

[joeyparrish]

@dougdoe Can you provide some details on what those EMSG boxes will look like? For example, what scheme URIs will be used for those two cases?

[dougdoe]

Hi @joeyparrish I am about to send you an email with two documents that contain information on EMSG box and ID3 tags.

[baconz]

@dougdoe be careful of #409 -- I think one of your assumptions might be broken.

@joeyparrish it seems like any implementation of EMSG support should be inline with support for Event and EventStream, since the IOP allows you to use either one for the type of data that @dougdoe is interested in. It would be great if Shaka provided an API that allowed you to add listeners for events in the manifest that fire when the player hits the relevant position.

[joeyparrish]

@baconz, can you point me to a reference for Event and EventStream? I admit, I'm not familiar with those yet.

[baconz]

Sure! Check out 5.3.3.7 and 5.3.2.4 of the latest IOP (http://dashif.org/wp-content/uploads/2016/06/DASH-IF-IOP-v3.3.pdf). Here's the example that they give:

<EventStream schemeIdUri="urn:scte:scte35:2014:xml+bin">
 <Event timescale="90000" presentationTime="54054000" duration="5400000" id="1">
 <scte35:Signal>
 <scte35:Binary>
 /DAIAAAAAAAAAAAQAAZ/I0VniQAQAgBDVUVJQAAAAH+cAAAAAA==
 </scte35:Binary>
 </scte35:Signal>
 </Event>
</EventStream>

There's also section 12.1.5 of the SCTE-67 spec, which talks about how SCTE-35 data should be integrated into DASH (http://www.scte.org/documents/pdf/standards/SCTE%2067%202014.pdf).

There isn't a ton of info fleshed out in the IOP, but the features that they hint at would be extremely useful to us.

[joeyparrish]

@baconz, we would consider EventStream and VAST support as a separate enhancement from the basic EMSG support we're working on here.

[baconz]

That sounds reasonable @joeyparrish, should I open a separate issue for EventStream?

[joeyparrish]

Sure, sounds good.

[vto5623]

Hi Joey,

In this last commit (f1a4e25), there are what appear to be example test messages on lines 756 - 784 of file "test/dash/dash_parser_manifest_unit.js", including an example custom message.

var emsgCustom = new Uint8Array([
    0, 0, 0, 55, 101, 109, 115, 103,
    102, 111, 111, 58, 98, 97, 114,
    58, 99, 117, 115, 116, 111, 109,
    100, 97, 116, 97, 115, 99, 104,
    101, 109, 101, 0, 49, 0, 0, 0, 0,
    1, 0, 0, 0, 8, 0, 0, 255, 255, 0,
    0, 0, 1, 116, 101, 115, 116
]);

Decoding it looks as though the box goes straight from the type "emsg" in bytes 4 - 7 (0 indexed) to the EMSG scheme_id_uri, in this case "emsgfoo:bar:customdatascheme".

This might not be correct.
Indeed, according to section 5.10.3.3.3 the spec "ISO/IEC 23009-1 Information technology — Dynamic adaptive streaming over HTTP (DASH) —Part 1: Media presentation description and segment formats", the EMSG box should inherit from FullBox:

aligned(8) class DASHEventMessageBox extends FullBox(‘emsg’, version = 0, flags = 0){
    string scheme_id_uri;
    string value;
    unsigned int(32) timescale;
    unsigned int(32) presentation_time_delta;
    unsigned int(32) event_duration;
    unsigned int(32) id;
    unsigned int(8) message_data[];
}

We believe that this implies that there should be an additional 4 bytes of version and flags (set to 0) between these two fields as defined by Section 4.2 of "ISO/IEC 14496-12 Information technology — Coding of audio-visual objects — Part 12: ISO base media file format":

aligned(8) class FullBox(unsigned int(32) boxtype, unsigned int(8) v, bit(24) f)
    extends Box(boxtype) {
    unsigned int(8) version = v;
    bit(24) flags = f;
}

[joeyparrish]

@ismena, can you please take a look?

[ismena]

@joeyparrish Absolutely!

@vto5623 Thank you for drawing our attention to this, I will work on the fix at once.

[ismena]

Hi @vto5623
I just checked in a change to account for flags in both the parsing code and tests.
Please let me know if you have any questions.