iOS Sidecar VTT TextTrack not working for longer videos (~1hr)

[shlokamin]

Current behavior

When loading in sidecar text track for long videos (~1hr) in iOS, the video will not start. When pressing play, the player will simply just stay idle.

I have had no issues like this with shorter videos (~10-20 mins)

Reproduction steps

Using this video: https://publish.dvlabs.com/democracynow/360/dn2018-0720.mp4
Using this VTT: https://integration.democracynow.org/resources/captions/shows/5874/English.vtt

i.e.

source={{uri:https://publish.dvlabs.com/democracynow/360/dn2018-0720.mp4}}
selectedTextTrack={{
type: "title",
value: "English"
}}
textTracks={[{
title: "English",
language: "en",
type: TextTrackType.VTT,
uri: https://integration.democracynow.org/resources/captions/shows/5874/English.vtt
}]}

Platform

Which player are you experiencing the problem on:

• iOS

[cobarx]

@ashnfb Can you take a look into this? I will also see if I can figure anything out.

[ashnfb]

@cobarx I'll take a look and see if I can figure it out. I have a film not playing too, also when captions are enabled via the system.

[cobarx]

Ok I did some tests and it appears that AVPlayer doesn't like the server that's hosting the subtitles. I downloaded them to my server and they work fine with no other modifications:
http://hamptonmaxwell.com:8080/DemNow.vtt

The video works fine whether I don't have subtitles, use my server, or use other subtitles. If I use integration.democracynow.org, onError shows:

{ code: -11800, domain: "AVFoundationErrorDomain" }

and the error in the native code is:

Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12783), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x600000249a80 {Error Domain=NSOSStatusErrorDomain Code=-12783 "(null)"}}

Looking at AVError.h, -11800 AVErrorUnknown. I've also seen that error when trying to load unsupported subtitle formats on iOS like SRT, so it's not very helpful.

I'd suggest reaching out to the Democracy Now team and seeing if they have had anyone else encounter this issue.

[ashnfb]

@cobarx thanks, I looked into my error as well; it turned out the vtt file wasn't reachable too, and that stopped the video from playing. It's a simple fix to do a check for that, and allow the video to play. Let me know if you want me to PR it, or you can add it?

RCTVideo.m:
AVAssetTrack *textTrackAsset = [textURLAsset tracksWithMediaType:AVMediaTypeText].firstObject;
if (!textTrackAsset) continue; // fix if there's no textTrackAsset

[cobarx]

I think we ought to expand it into a PR that returns only the valid tracks in the onLoad event. The way I'd want to use this is load all the tracks, grab them from onLoad and populate a menu with that data.

…

On Tue, Jul 24, 2018, 3:36 PM Ash Mishra ***@***.***> wrote: @cobarx <https://github.com/cobarx> thanks, I looked into my error as well; it turned out the vtt file wasn't reachable too, and that stopped the video from playing. It's a simple fix to do a check for that, and allow the video to play. Let me know if you want me to PR it, or you can add it? RCTVideo.m: AVAssetTrack *textTrackAsset = [textURLAsset tracksWithMediaType:AVMediaTypeText].firstObject; if (!textTrackAsset) continue; // fix if there's no textTrackAsset — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1143 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD4AF5wtaqXJ9ksDpi0kSCfs3cp7ZOnCks5uJ6FkgaJpZM4VbpFJ> .

[ashnfb]

@cobarx what are you using for an interface for the player? I'm using react-native-video-player; and was intending to add CC controls there. About your suggestion on returning valid tracks from the onLoad event; would we fulfill a url there and pass the string contents of all the subtitles to the video player, or make a second network call to load the subtitles?

[cobarx]

@ashnfb I've built my own custom video controls. Waiting on approval from my company to open source them.

onLoad already returns the text & audio tracks that are available including the ones specified in the textTracks prop. It looks like:

{ 
  ...various other data,
  audioTracks: [
    { language: 'es', title: 'Spanish', type: 'audio/mpeg', index: 0 },
    { language: 'en', title: 'English', type: 'audio/mpeg', index: 1 } ],
  ],
  textTracks: [
    { title: '#1 French', language: 'fr', index: 0, type: 'text/vtt' },
    { title: '#2 English CC', language: 'en', index: 1, type: 'text/vtt' },
    { title: '#3 English Director Commentary', language: 'en', index: 2, type: 'text/vtt' }
  ]
}

So what I'd imagine is you specify textTracks, get the list of ones that are embedded / successfully sideloaded from onLoad, then pass the data to the CC control.

[shlokamin]

Thanks to both of you for taking the time to look into this!

This is very odd. The VTTs load just fine on Android, and we also consume the same VTT files in a separate frontend web application.

I've been debugging and trying to see if others have had similar experiences with this (very vague and unhelpful) error with no luck.

I'm not familiar with Objective C, but can either of you perhaps point me in the right direction as to why this behavior is different in iOS? I'd like to figure this out and hopefully fix it. I can't seem to find where AVPlayer is actually making the request to fetch the VTT.

[ashnfb]

@shlokamin my first guess is that iOS App Transport Security Settings is blocking your https call to nowdemocracy because it's not fulfilling some of the requirements. You can test this by enabling "Allow Arbitrary Loads" in the info.plist for the iOS App, and see if it resolves it.

[cobarx]

@shlokamin I'm not sure what to suggest. I would reach out to Democracy Now and see if they have had any other reports of problems from people building apps with iOS. The relevant code is at:
https://github.com/react-native-community/react-native-video/blob/master/ios/RCTVideo.m#L420
Not sure exactly which line actually fires off loading, but it's in that vicinity.

@ashnfb It's not the Transport Security Settings since the site is https. I have that set and am running into the same issue. My only thought is it could be an SSL cert issue, however I viewed the URL in a web browser and it looks good.

I've dropped the same exact VTT file onto my own server:
http://hamptonmaxwell.com:8080/DemNow.vtt
and it loads fine, so it seems like it has to be the server. It'd be a hack, but you could setup a server that passes the requests on to the DN server.

[ashnfb]

hi @cobarx I'll take a look at your suggestion for fixes probably next week

[shlokamin]

@cobarx doesn't sound like this has ever been an issue before. We also modified the HTTP headers to mimic yours as closely as possible (from Democracy Now's side) and still no luck.

As a potential work around, we tried fetching the VTT and base64 encoded it into a data URI:

RNFetchBlob
.fetch('GET', "http://hamptonmaxwell.com:8080/DemNow.vtt", {
})
.then((res) => {
let text = res.text()
let encoding = btoa(unescape(encodeURIComponent(text)))
this.setState({dataURI:"data:text/vtt;base64," + encoding})
})

And in the video component:

textTracks={[
{
title: "English",
language: "en",
type: TextTrackType.VTT,
uri:this.state.dataURI
}
]}
selectedTextTrack={{
type: "title",
value: "English"
}}

While sort of hacky, this approach works fine on Android Exo Player.

In iOS, however, this also resulted in the same exact error:
error: {domain: "AVFoundationErrorDomain", code: -11800}

It is peculiar that I still get the same error in iOS even though the VTT has been encoded and exists in memory locally. This seems to be a very generic (and poor) error message.

I've googled around to see if others have experienced this issue with AV Player but no luck.

More fun AV Player quirks...

[cobarx]

Do the subtitles work when you load them from my server? Because they worked perfectly for me without needing to Base64 or anything like that. Sounds like there is some underlying issue we're not seeing. Also I'm using the iOS 11.4 simulator, are you using a different version, maybe there's some AVPlayer bug that got fixed.

I'll come back to this later and see if I can get more info about the error.

[shlokamin]

@cobarx yeah when pointing to your server it works just fine! We thought it might have been the HTTP headers from Democracy Now's server but that didn't seem to be the case.

I'm using iOS 11.3 simulator, but I also tried it on my actual phone which runs iOS 11.4.1. Neither worked and resulted in the same error as above.

AV Player not liking data uris as a source seems to be a separate issue, just happens to give the same error.

[ashnfb]

PR for textTracks improvements
#1162

[reisraff]

I am having an issue that seems to be like this.

IDK why, but when I set my texttracks my video does not return data.duration on onLoaded (it is 0 zero), the video never starts.

When I do not set the texttracks the video work perfectly.

I am using HLS for video, and entire vtt files for captions, both use the same s3 bucket and path.

The error only appers for iOS, for android it work correctly.

Could someone help me?

UPDATINIG: I just seen in the README.md that it is not possible

[ajaysaini-sgvu]

@cobarx Do you have any suggestion on this ?

[sannjayy]

@ashnfb I've built my own custom video controls. Waiting on approval from my company to open source them.

onLoad already returns the text & audio tracks that are available including the ones specified in the textTracks prop. It looks like:

{ 
  ...various other data,
  audioTracks: [
    { language: 'es', title: 'Spanish', type: 'audio/mpeg', index: 0 },
    { language: 'en', title: 'English', type: 'audio/mpeg', index: 1 } ],
  ],
  textTracks: [
    { title: '#1 French', language: 'fr', index: 0, type: 'text/vtt' },
    { title: '#2 English CC', language: 'en', index: 1, type: 'text/vtt' },
    { title: '#3 English Director Commentary', language: 'en', index: 2, type: 'text/vtt' }
  ]
}

So what I'd imagine is you specify textTracks, get the list of ones that are embedded / successfully sideloaded from onLoad, then pass the data to the CC control.

Are you getting videoTracks?

[ashnfb]

@sannjayy, for the app I'm working on, we provide the ability to do offline films (mp4), so we then have to pass locally stored captions files alongside those to permit captions to work. If using HLS with encoded references to captions, everything should just work.

Curious about your own controls. I've done the same on my fork of the repo (added custom controls for Android, iOS doesn't need them).