Subtitle parsing broken in 1.4.0-alpha02+

[JonWatson]

Version

Media3 main branch

More version details

Subtitle parsing for a given DASH asset (DRM protected) works fine in 1.4.0-alpha01 but in 1.4.0-alpha02 and 1.4.0-beta01 the playback fails with the following exception. Without provding the asset, can you help figure this out? Any info from the DASH manifest that I could provide to help? Again, same asset plays fine and displays subtitles in 1.4.0-alpha01

androidx.media3.exoplayer.ExoPlaybackException: Source error
      at androidx.media3.exoplayer.ExoPlayerImplInternal.handleIoException(ExoPlayerImplInternal.java:741)
      at androidx.media3.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:711)
      at android.os.Handler.dispatchMessage(Handler.java:102)
      at android.os.Looper.loop(Looper.java:223)
      at android.os.HandlerThread.run(HandlerThread.java:67)
  Caused by: androidx.media3.common.ParserException: SubtitleParser failed.{contentIsMalformed=true, dataType=1}
      at androidx.media3.extractor.text.SubtitleExtractor.parseAndWriteToOutput(SubtitleExtractor.java:258)
      at androidx.media3.extractor.text.SubtitleExtractor.read(SubtitleExtractor.java:161)
      at androidx.media3.exoplayer.source.chunk.BundledChunkExtractor.read(BundledChunkExtractor.java:241)
      at androidx.media3.exoplayer.source.chunk.ContainerMediaChunk.load(ContainerMediaChunk.java:132)
      at androidx.media3.exoplayer.upstream.Loader$LoadTask.run(Loader.java:421)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
      at java.lang.Thread.run(Thread.java:923)
  Caused by: java.lang.IllegalArgumentException
      at androidx.media3.common.util.Assertions.checkArgument(Assertions.java:40)
      at androidx.media3.extractor.text.webvtt.WebvttSubtitle.getEventTime(WebvttSubtitle.java:63)
      at androidx.media3.extractor.text.LegacySubtitleUtil.toCuesWithTiming(LegacySubtitleUtil.java:44)
      at androidx.media3.extractor.text.webvtt.WebvttParser.parse(WebvttParser.java:108)
      at androidx.media3.extractor.text.SubtitleParser.parse(SubtitleParser.java:152)
      at androidx.media3.extractor.text.SubtitleExtractor.parseAndWriteToOutput(SubtitleExtractor.java:238)
      at androidx.media3.extractor.text.SubtitleExtractor.read(SubtitleExtractor.java:161) 
      at androidx.media3.exoplayer.source.chunk.BundledChunkExtractor.read(BundledChunkExtractor.java:241) 
      at androidx.media3.exoplayer.source.chunk.ContainerMediaChunk.load(ContainerMediaChunk.java:132) 
      at androidx.media3.exoplayer.upstream.Loader$LoadTask.run(Loader.java:421) 
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) 
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) 
      at java.lang.Thread.run(Thread.java:923) 

Devices that reproduce the issue

Nvidia Shield TV
Pixel 7 Pro

Devices that do not reproduce the issue

None known

Reproducible in the demo app?

Yes

Reproduction steps

player.trackSelectionParameters =
  player.trackSelectionParameters.buildUpon()
    .setTrackTypeDisabled(TRACK_TYPE_TEXT, !showSubtitles)
    .setPreferredTextLanguage("eng")
    .build()

Expected result

Playback succeeds even when displaying subtitles

Actual result

Receive ERROR_CODE_PARSING_CONTAINER_MALFORMED 3001 during playback when attempting to display subtitles

Media

Private URL. Please let me know if there is some info I can provide from the DASH manifest

Bug Report

• You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.

[icbaker]

1.4.0-alpha02 moved subtitle parsing to happen during extraction by default (instead of during rendering). I wonder if your problem is resolved by moving it back to during rendering? You can do this by calling both MediaSource.Factory.experimentalParseSubtitlesDuringExtraction(false) and TextRenderer.experimentalSetLegacyDecodingEnabled(true) (see also the release notes: https://github.com/androidx/media/releases/tag/1.4.0-alpha02).

Looking at the code, it seems the exception you're seeing is basically an index out of bounds issue:

media/libraries/extractor/src/main/java/androidx/media3/extractor/text/webvtt/WebvttSubtitle.java

Lines 61 to 65 in 023e9d1

	public long getEventTime(int index) {
	Assertions.checkArgument(index >= 0);
	Assertions.checkArgument(index < sortedCueTimesUs.length);
	return sortedCueTimesUs[index];
	}

I think I can see the issue with the logic here - we need to either not return subtitle.getEventTimeCount() from getStartIndex(...) or check that startIndex < subtitle.getEventTimeCount() before calling long firstEventTimeUs = subtitle.getEventTime(startIndex). I will have a go at writing a test case to reproduce the stack trace you're seeing, and see if this change resolves it.

media/libraries/extractor/src/main/java/androidx/media3/extractor/text/LegacySubtitleUtil.java

Lines 40 to 52 in 023e9d1

	int startIndex = getStartIndex(subtitle, outputOptions);
	boolean startedInMiddleOfCue = false;
	if (outputOptions.startTimeUs != C.TIME_UNSET) {
	List<Cue> cuesAtStartTime = subtitle.getCues(outputOptions.startTimeUs);
	long firstEventTimeUs = subtitle.getEventTime(startIndex);
	if (!cuesAtStartTime.isEmpty()
	&& startIndex < subtitle.getEventTimeCount()
	&& outputOptions.startTimeUs < firstEventTimeUs) {
	output.accept(
	new CuesWithTiming(
	cuesAtStartTime,
	outputOptions.startTimeUs,
	firstEventTimeUs - outputOptions.startTimeUs));

[icbaker]

I've added a test that creates the same stack trace, and sent a change the fixes the code to not crash in this case - I hope to include this in 1.4.0-rc01. Thanks for raising this, and providing stack trace details so we could work out the issue.

[icbaker]

This should be fixed by the commit above.

[JonWatson]

Hello @icbaker

I'm testing this out in 1.4.0 and still see the same problem. You are correct in that this is the place the IllegalArgumentException is getting thrown:

 public long getEventTime(int index) { 
   Assertions.checkArgument(index >= 0); 
   Assertions.checkArgument(index < sortedCueTimesUs.length);  <-------------- HERE
   return sortedCueTimesUs[index]; 
 } 

and it is indeed because the nextEventTimeIndex is C.INDEX_UNSET, which causes the startIndex to be set to the length of the array

   private static int getStartIndex(Subtitle subtitle, OutputOptions outputOptions) {
    if (outputOptions.startTimeUs == C.TIME_UNSET) {
      return 0;
    }
    int nextEventTimeIndex = subtitle.getNextEventTimeIndex(outputOptions.startTimeUs);
    if (nextEventTimeIndex == C.INDEX_UNSET) {
      return subtitle.getEventTimeCount();        <---------- HERE
    }
    if (nextEventTimeIndex > 0
        && subtitle.getEventTime(nextEventTimeIndex - 1) == outputOptions.startTimeUs) {
      nextEventTimeIndex--;
    }
    return nextEventTimeIndex;
  }

I'm working on changing the setup to try the legacy way and will report on the results there

[JonWatson]

The array of sortedCueTimesUs is like this from our VTT:

[0, 241000, 241000, 1409000, 1409000, 1442000, 1442000, 1475000, 1475000, 1509000, 1509000, 1576000, 1576000, 1642000, 1642000, 1676000, 1676000, 1742000, 1742000, 1776000, 1776000, 1809000, 1809000, 1843000, 1843000, 2009000, 2009000, 2043000, 2043000, 2076000, 2076000, 2109000, 2109000, 2210000, 2210000, 2243000, 2243000, 2276000, 2276000, 2310000, 2310000, 2610000, 2610000, 2677000, 2677000, 2810000, 2810000, 2844000, 2844000, 2877000, 2877000, 3211000, 3211000, 3244000, 3244000, 3277000, 3277000, 3511000, 3511000, 3544000, 3544000, 3578000, 3578000, 3644000, 3644000, 3678000, 3678000, 3711000, 3711000, 4004000]

But the value we are searching for in Util.binarySearchCeil() is 458500275832 so it seems like our third-party VTTs are not in the expected format somehow...

[JonWatson]

I'd like to point out that ExoPlayer doesn't crash, everything is caught.

It's just that our playback is broken (we get a 3001) when trying to play with subtitles on

[icbaker]

Please can you give us a way to play a stream that reproduces this issue? Without that I'm afraid we're just guessing at the possible cause (and it seems that last time we guessed wrong).

[icbaker]

Please either upload it here or send it to android-media-github@google.com with the subject Issue #1516. Please also update this issue to indicate you’ve done this.

[JonWatson]

Our videos are locked with DRM and our DRM tokens are only good for 10 minutes. I cannot share our source so I think we may be stuck here. It may be that I can provide you with a dynamic URL that will return DRM tokens you can use(along with our custom DRM implementation) but I need to consider this..

[JonWatson]

Confirmed our subtitles show properly and no playback errors if we use:

MediaSource.Factory.experimentalParseSubtitlesDuringExtraction(false) TextRenderer.experimentalSetLegacyDecodingEnabled(true)

[icbaker]

@JonWatson Are you able to share the DASH manifest URL via email, without the DRM license/token info? That may be enough for us to reproduce the issue (since it only concerns subtitles, which presumably aren't encrypted).

[JonWatson]

Yes I can do that! An example manifest URL has been sent to android-media-github@google.com

[icbaker]

Thanks for the test stream, it's been really helpful to look into this issue. I've fixed the IndexOutOfBoundsException with the commit linked above, which solves the playback error, but subtitles are still not shown on your stream.

Looking in your DASH manifest it seems that your subtitles are raw WebVTT (not encapsulated in ISOBMFF / MP4) but are segmented. I believe this combination is not supported by the DASH-IF IOP 5.2.10 (emphasis mine):

Some services store text adaptation sets in stand-alone IMSC1 or WebVTT files, without segmentation or [ISOBMFF] encapsulation.
This document requires:

• Timecodes in stand-alone text files SHALL be relative to the period start point.
• @presentationTimeOffset SHALL NOT be present and SHALL be ignored by clients if present.

I appreciate that this stream works on 1.3.1, and not on 1.4.0, so even if this content isn't fully spec compliant it still seems like a regression in the library.

I'm still working on plumbing the right timestamps around to get these subtitles showing up again - I haven't quite got it working yet.

[icbaker]

@JonWatson Ah I think the stream stopped working today - would you be able to email a new link so I can continue debugging? Thanks!

[JonWatson]

Done! Thank you

[icbaker]

@JonWatson I've made some hacky local changes which get subtitles showing for your stream, but I need to clean them up before being able to submit (because at the moment they will break other use-cases). Unfortunately I think the stream stopped working again today - would you be able to send a new one?

[JonWatson]

Thanks @icbaker , new manifest has been emailed