'data' and 'done' callbacks don't get invoked

[ismena]

Repro:
Navigate to https://v2-4-0-dot-shaka-player-demo.appspot.com/demo/#asset=https://s3-ap-southeast-1.amazonaws.com/learnyst/testfolder/sample/expresshls/hls.m3u8;play and start playback.

For the first 2 segments (https://s3-ap-southeast-1.amazonaws.com/learnyst/testfolder/sample/expresshls/zh3il5009f-yxhbaADHqLeCAo3k.ts and https://s3-ap-southeast-1.amazonaws.com/learnyst/testfolder/sample/expresshls/ZtV37gCCgR5cnwUrQliZEPG9zIU.ts) the callbacks get invoked, but not for the third one (https://s3-ap-southeast-1.amazonaws.com/learnyst/testfolder/sample/expresshls/X3cf3WEJqIG6EfX0Yjq8dvGthAc.ts) although they are present in the transmuxer's internal listeners collection.

The code that registers the callbacks:

  this.muxTransmuxer_ = new muxjs.mp4.Transmuxer({
    'keepOriginalTimestamps': true
  });
 
  this.muxTransmuxer_.on('data', this.onTransmuxed_.bind(this));
  this.muxTransmuxer_.on('done', this.onTransmuxDone_.bind(this));

[mjneil]

It looks like the 3rd segment does not contain any audio data, but the PMT says there is an audio track. Because of this code here https://github.com/videojs/mux.js/blob/master/lib/mp4/transmuxer.js#L1110 the transmuxer is waiting for audio data that is never coming.

You should be able to fix this by setting the remux option to false for the transmuxer.

  this.muxTransmuxer_ = new muxjs.mp4.Transmuxer({
    'keepOriginalTimestamps': true,
    'remux': false
  });

Note that this will result in separate data events for audio and video, which may require other code changes within Shaka.

[ismena]

Thanks a lot for replying, @mjneil! Could you give me more context on a couple of things:

• What is PMT?
• From what you said I got an impression that setting 'remux' to false is going to effectively create 2 separate streams - a video and an audio one, is that correct?

Thanks!

[mjneil]

What is PMT?

PMT stands for Program Map Table. It is part of MPEG-TS and essentially says which elementary streams are available. In the source you provided, the PMT packet says there are two elementary streams available, type 15 ADTS AAC audio and type 27 H.264 video.

From what you said I got an impression that setting 'remux' to false is going to effectively create 2 separate streams - a video and an audio one, is that correct?

Essentially yes. The transmuxer separates audio and video when processing the MPEG-TS packets.

When remux === true, the transmuxer's CoalesceStream waits until both the audio and video have been transmuxed, combines the resulting mp4 boxes into a single bye array, and emits a data event where event.type === 'combined'.

When remux === false, the CoalesceStream does not wait for both streams, and instead emits a data event for both with event.type === 'audio' || event.type === 'video'. Note that these events, regardless of the type, will be emitted by the transmuxer, so from Shaka's point of view should only need to setup listeners on the transmuxer like is currently being done, handling events depending on their type

[joeyparrish]

I tried setting 'remux': false today, and sadly, that doesn't fix it. Instead, I get a MediaSource error from Chrome that says "Initialization segment misses expected aac track".

I will try some other workarounds in Shaka Player, but I would appreciate any suggestions you might have.

[joeyparrish]

If I ignore the lack of callbacks and just move on to the next segment, I wind up with a MediaSource error from Chrome that says "CHUNK_DEMUXER_ERROR_APPEND_FAILED: Video track with track_id=258 already present."

I think I'm out of ideas.

[TheModMaker]

With 'remux': false, I think it will produce two different streams, one for audio and one for video. We initialize the SourceBuffer for the combined stream video/mp4; codecs="avc1,mp4a", so when we append the video init segment we get back, the browser complains. We would need to create two SourceBuffers and use event.type to determine which SourceBuffer to append to.

[joeyparrish]

Ah, I see. We would have one input stream, and two SourceBuffers for output. That would be a completely different architecture, though, from what we have today. If that's what it takes to fix it, I think we would have to defer that for quite some time.

@mjneil, is there any way that this could be fixed in mux.js?

[mjneil]

@TheModMaker has the right idea. With remux: false you will need to create seperate source buffers for audio and video instead of a combined source buffer

[mjneil]

Its probably possible to fix within mux. Would need to add logic probably here https://github.com/videojs/mux.js/blob/master/lib/mp4/transmuxer.js#L1108 to determine if it should return or not. Easiest solution (though probably a messy one) would probably be if the flush source is AudioSegmentStream and you are in that block, its probably safe to assume that the segment just had no audio data since the transmuxer operates under the assumption that it processes video before audio, so you can just add this.pendingTracks.push(this.audioTrack) so it gets included when creating the init segment, increase emitted tracks this.emittedTracks++; and then not return so the rest of the function can emit the event

[mjneil]

an alternative could be to re-create the init segment outside of mux within shaka after you've gotten your done event. You would need to create a video and audio track object that contain the properties touched by the generator https://github.com/videojs/mux.js/blob/master/lib/mp4/mp4-generator.js#L762 most of the properties get attached to event.info (though that will need some fixing cause it looks like we don't send the audio info if a video track exists https://github.com/videojs/mux.js/blob/master/lib/mp4/transmuxer.js#L1128)

[sridhard]

@mjneil any updated on this issue?

[psk200]

@mjneil i'm facing media decode error in videojs 7 and shaka. Is there any way to find errors in the audio / video tracks using mux.js. Please guide me with debugging the audio / video tracks with mux.js.

[mjneil]

@ismena @joeyparrish @TheModMaker @sridhard I just created #224 which should address this issue. Please if you have a chance, test it out with your use case and let me know if it solves the issue or if you encounter any issues.

[joeyparrish]

I'm going to be out of the office for a while, so I will not be able to test the PR until November. If someone else can test before then, please do!

[neerajgulati-airtel]

Any update on this