Closed captions in HLS manifest for iOS devices in ME.js

[Dananji]

Description

For closed captions to work with MediaElement.js in iOS devices, the captions needs to be transcoded in the video according to ME.js documentation.
To make this work with streaming media in Avalon, we could add the captions information in our HLS manifests.

Done Criteria

• Investigate on possible ways to add captions to m3u8 manifest

According to Chris, we proxy the wowza manifests through Avalon.
Streaming example with captions from Apple: https://developer.apple.com/streaming/examples/basic-stream-osx-ios5.html with the following manifest:

#EXTM3U

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="bipbop_audio",LANGUAGE="eng",NAME="BipBop Audio 1",AUTOSELECT=YES,DEFAULT=YES
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="bipbop_audio",LANGUAGE="eng",NAME="BipBop Audio 2",AUTOSELECT=NO,DEFAULT=NO,URI="alternate_audio_aac/prog_index.m3u8"


#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English",DEFAULT=YES,AUTOSELECT=YES,FORCED=NO,LANGUAGE="en",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog, public.accessibility.describes-music-and-sound",URI="subtitles/eng/prog_index.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English (Forced)",DEFAULT=NO,AUTOSELECT=NO,FORCED=YES,LANGUAGE="en",URI="subtitles/eng_forced/prog_index.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="Français",DEFAULT=NO,AUTOSELECT=YES,FORCED=NO,LANGUAGE="fr",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog, public.accessibility.describes-music-and-sound",URI="subtitles/fra/prog_index.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="Français (Forced)",DEFAULT=NO,AUTOSELECT=NO,FORCED=YES,LANGUAGE="fr",URI="subtitles/fra_forced/prog_index.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="Español",DEFAULT=NO,AUTOSELECT=YES,FORCED=NO,LANGUAGE="es",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog, public.accessibility.describes-music-and-sound",URI="subtitles/spa/prog_index.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="Español (Forced)",DEFAULT=NO,AUTOSELECT=NO,FORCED=YES,LANGUAGE="es",URI="subtitles/spa_forced/prog_index.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="日本語",DEFAULT=NO,AUTOSELECT=YES,FORCED=NO,LANGUAGE="ja",CHARACTERISTICS="public.accessibility.transcribes-spoken-dialog, public.accessibility.describes-music-and-sound",URI="subtitles/jpn/prog_index.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="日本語 (Forced)",DEFAULT=NO,AUTOSELECT=NO,FORCED=YES,LANGUAGE="ja",URI="subtitles/jpn_forced/prog_index.m3u8"


#EXT-X-STREAM-INF:BANDWIDTH=263851,CODECS="mp4a.40.2, avc1.4d400d",RESOLUTION=416x234,AUDIO="bipbop_audio",SUBTITLES="subs"
gear1/prog_index.m3u8
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=28451,CODECS="avc1.4d400d",URI="gear1/iframe_index.m3u8"

#EXT-X-STREAM-INF:BANDWIDTH=577610,CODECS="mp4a.40.2, avc1.4d401e",RESOLUTION=640x360,AUDIO="bipbop_audio",SUBTITLES="subs"
gear2/prog_index.m3u8
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=181534,CODECS="avc1.4d401e",URI="gear2/iframe_index.m3u8"

#EXT-X-STREAM-INF:BANDWIDTH=915905,CODECS="mp4a.40.2, avc1.4d401f",RESOLUTION=960x540,AUDIO="bipbop_audio",SUBTITLES="subs"
gear3/prog_index.m3u8
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=297056,CODECS="avc1.4d401f",URI="gear3/iframe_index.m3u8"

#EXT-X-STREAM-INF:BANDWIDTH=1030138,CODECS="mp4a.40.2, avc1.4d401f",RESOLUTION=1280x720,AUDIO="bipbop_audio",SUBTITLES="subs"
gear4/prog_index.m3u8
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=339492,CODECS="avc1.4d401f",URI="gear4/iframe_index.m3u8"

#EXT-X-STREAM-INF:BANDWIDTH=1924009,CODECS="mp4a.40.2, avc1.4d401f",RESOLUTION=1920x1080,AUDIO="bipbop_audio",SUBTITLES="subs"
gear5/prog_index.m3u8
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=669554,CODECS="avc1.4d401f",URI="gear5/iframe_index.m3u8"

#EXT-X-STREAM-INF:BANDWIDTH=41457,CODECS="mp4a.40.2",AUDIO="bipbop_audio",SUBTITLES="subs"
gear0/prog_index.m3u8

[cjcolvar]

This looks like a useful blog post with some more details about putting the subtitles/captions into the m3u8: https://www.mux.com/blog/subtitles-captions-webvtt-hls-and-those-magic-flags

[joncameron]

Since we're wrapping and unwrapping m3u8s we should be able to insert this information. Our manifests are manifests of manifests sourced from Wowza, so it might be possible that the player will only respect the child manifest from Wowza. This should be investigated to see what the behavior is.

[masaball]

Adding the subtitle/closed caption information to the HLS manifest is pretty easily accomplished. Adding #EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",LANGUAGE="2 or 3 letter language code",NAME="caption name",AUTOSELECT=YES,URI="<%= caption uri %>" to the main playlist manifest is sufficient. Additionally, the #EXT-X-STREAM-INF line will need to be appended with SUBTITLES="subs".

We will probably need to add a way to designate a caption file's language so that we don't just have "English" hard coded. Also, is Avalon capable of having multiple caption files associated with a work or can only one be uploaded at a time? If there can be multiple, we will need to be sure to code in an each loop to populate the caption group with all of the options.

For the URI field, we can add an endpoint and create a second manifest specifically for the subtitles which the URI would reference. We cannot just pull in the straight caption file. Information on this approach. The most relevant parts are the "HOW TO EDIT THE PLAYLISTS" sections because they give a layout for how the manifests should look but there is good info throughout that answer. Ideally, we will not have to implement any of the request intercepting that answer details since I think the stack overflow conversation is trying to edit the manifests on the fly client-side whereas we can generate everything server side.

The wowza documentation has details for enabling WebVTT caption for iOS. There is a specific config property that needs to be set. However, it sounds like Wowza expects the caption file to be in the same directory as the media file which I do not know if that is the case with our setup currently. I think this is a separate approach from adding the information to the HLS file manually, where we would be letting Wowza handle the formatting. However, I am unsure how the HLS file manipulations discussed will interact with the streaming server. If we manually add the information to the HLS files, it may end up necessitating some special handling. If we use the builtin Wowza settings, then we limit the customizability and have to maintain copies of all caption files on the Wowza infrastructure (Avalon does not currently have guaranteed write access to Wowza filesystem and may require a data migration if we were able to move all the caption files to Wowza). Those are the main approaches as far as I can tell.

[masaball]

Proof of concept PR for adding caption to HLS manifest. Need to test with BrowserStack to verify it does generate the captions properly for iOS. Can then assess the approach and open a new issue for actual implementation.

[masaball]

I have tested the code on Avalon-dev with BrowserStack and was able to get the proof of concept to provide captions on iOS devices. Had to make a minor tweak to how the caption m3u8 file was generated, but it seems to work well now.

[elynema]

Note that using Wowza to add captioning for iOS seems like a more challenging process b/c it requires us to sync caption files over to Wowza server separately; so writing a brand new cross-server integration.