playback issues - gaps and cannot find segment

[jakubvojacek]

Have you read the FAQ and checked for duplicate open issues?:
Yes. I went through a lot of issues similar to mine but they were having different problems, I think.
What version of Shaka Player are you using?:
2.3.5
Can you reproduce the issue with our latest release version?:
Yes, It happens on https://shaka-player-demo.appspot.com/demo/ as well
Are you using the demo app or your own custom app?:
both
If custom app, can you reproduce the issue using our demo app?:
yes
What browser and OS are you using?:
Tested of macOS & chrome / safari
What are the manifest and license server URIs?:
mpd url - goo.gl/L11oh5

What did you do?
We are using ffmpeg and shaka packager to create dash. Unfortunately, shaka packager does not yet support utctimestamp (but I have already filed a issue for that - shaka-project/shaka-packager#311). But I dont think its related to clock issues.
What did you expect to happen?
When we restart packager, everything is working very smoothly and without errors and the erros usually starts to appear after a 20+ hours of running packager.

After 20 or more hours has passed, sometimes the playback works, a minute later it does not and gets stuck on error (video:4) cannot find segment: currentPeriod.startTime=0 lookupTime=174217.48699998856. But if I just refresh the player, it sometimes starts working (but loading time is still longer then other channels). Lots of people who had this error had time synchronization issues in between server and client. But I checked and me and server has pretty much same time. We also have other 10 channels runinng on the same server that are being processed the exact same way and there are zero issues with those. The difference in between the other streams and this one is that the troublesome has 4 audios. But on several other servers, we are having this issues with single audio streams as well.

The generated content also happens to have few gaps here and then. Could those be causing the issue? Is this issue related more the shaka packager, should I ask over there about the gaps? I have tried everything I could but still no results. Can you please check out the MPD and advise what are we doing wrong?

Thank you

[jakubvojacek]

Forgot to mention, that HLS seems to be working better and more reliable then DASH. Both HLS and DASH are being generated with the same packager command. The HLS is on https://goo.gl/LnprQL

[joeyparrish]

I haven't reproduced the error yet, but from the symptoms, it sounds like encoder drift.

This happens when the encoder falls behind the ideal timeline. For example, if after 20 hours, only 19.98 hours of content were produced, then the encoder would have fallen behind by 1.2 minutes. This might be small, or it might be large, depending on some other details. If your timeShiftBufferDepth were only 1 minute, then you'd be unable to play anything at all, because all content would be outside the availability window.

HLS doesn't have this problem because it doesn't have an ideal timeline at all. You are supposed to begin playback 3 segments from the end, regardless of when the stream started. In fact, the player may not know when the stream started at all.

In the future, we will refactor our DASH support to derive the availability window from the list of segments (#999). Then, the player will implicitly tolerate DASH encoder drift by behaving more like HLS.

We have this problem in our own content. The "Shaka Player History" stream eventually falls behind in the DASH version, but is generally working well in the HLS version.

[jakubvojacek]

It is happening to me right now, when it first loads the manifest, its giving me the unable to find segment error. Once its loads the manifest again (in 5 seconds time), it starts to play. It seems like it could be the encoder drift that you described.

Our timeshiftbufferdepth is 250 mins by default. Any idea what can we do to prevent this error from happening? Any setting either in shaka-packager or shaka-player comes to mind until the planned refactoring is completed? Thank you for your support.

[joeyparrish]

I've been playing for 30 minutes now without any issue. Are you seeing this issue on our demo app? Or in your own app?

[joeyparrish]

Actually, if I reload the stream, I do sometimes see those "cannot find segment" logs, but the stream does seem to recover within a few seconds:

https://nightly-dot-shaka-player-demo.appspot.com/demo/#asset=https://edge1.motv.eu///channels/eurosport-1-hd-42/live/eurosport-1-hd-42.mpd;play

[sandersaares]

Can you provide the FFmpeg and Shaka Packager commands you use to generate this problematic stream? Does it also reproduce with FFmpeg test input sources? There is ongoing work to upgrade the live timing validation in the DASH-IF DASH conformance validator and having more test scenarios included would be helpful in ensuring the upgrades can point out all the issues.

[jakubvojacek]

the demo app. Yes, it recovers within few seconds (for me its after 5 seconds when it reloads the manifest) and then its playing without problems. The problem is only when starting the stream. Other streams, that are being processed exactly the same way, are playing instantly. When channel zapping customer will not like this behavior I am afraid.

But it's happening also on our other servers, for example (dash https://goo.gl/URtAU4, https://goo.gl/55Zr2j hls). In this case we are not able to play the HLS after few days of running the packager. Guess there is some other issue because encoder drift does not apply to HLS, right? In this case, restarting packager solves the issue for 20+/- hours too.

[joeyparrish]

As far as I know, encoder drift does not apply to HLS because there are no calculations involving the current time when you start an HLS stream.

I'll see if I can understand the source of that startup delay.

[jakubvojacek]

yes, ffmpeg and packager commands are here ffmpeg: https://goo.gl/j5HGih, packager: https://goo.gl/XJ8TDm

for the other example I shared a minute ago its https://goo.gl/9QdsVe and https://goo.gl/DVqWqE

[jakubvojacek]

another example, It started to happen on another of our live channels - https://goo.gl/dxc7GV. It does not start even after a while, it keeps on saying that it's unable to find segment.

• HLS plays no issues
• plays in dashif player
• dash plays in android exoplayer
• dash plays in ios expressplay player (its using dash and transforms it into hls internally)
• dash plays in settopbox ekt solidui player

do you think its possible that there might be an issue with shaka player since all other players are able to play the content?

Thank you
Jakub

[sandersaares]

Please excuse slight rounding in my below calculations.

Loading the above link at 03.04.2018 09:20:52 UTC gives me MPD@availabilityStartTime of 29.03.2018 10:17:29 UTC, which means that 428 603 seconds have elapsed since the start of the period.

Given MPD@timeShiftBufferDepth of 15000S, the availability window is approximately from 413 603 to 428 603 seconds.

Shaka tries to load 428 483 on startup. This is within the availability window.

I do see the "cannot find segment" error. Why is this? Let's explore, taking audio track as example.

Given timescale of 90 000, the audio track <SegmentTimeline> defines 4689 segments starting at t/timescale=412 508 seconds and lasting for a total of d*(r+1)/timescale=15 004 seconds. This seems to match the time shift buffer depth more or less. The MPD segment timeline ends at 412 508+15 004=427 512 seconds.

You can see the problem, I hope.

• The MPD first says that the available segments range from 413 603 to 428 603 seconds.
• Then it actually gives you segments for 412 508 to 427 512 seconds.

Let me be quite clear: this is an error in the MPD. This content is malformed. Possibly due to encoder drift, possibly due to something else.

While it is laudable that Shaka player developers plan to work around such content defects in a future version, you should not have an expectation that malformed content will play. Yes, there are players that play malformed content due to quirks in their implementation - not every error in the content "matters" because some values get ignored by some players, depending on their algorithm.

At the end of the day, however, if you wish to achieve a good playback experience across the DASH ecosystem, I suggest you fix your content so that the MPD does not contain conflicting information about segment availability. I am not familiar enough with FFmpeg and Shaka Packager to advise what the appropriate corrective action might be, unfortunately.

[jakubvojacek]

Thank you for the explanation, I can see the problem now. Unfortunately, I too have no idea as to why this could be happening. When we restart the packager, audio & video start times are aligned perfectly. But as the time goes, they seem to be drifting apart. I will do some more testing and then probably ask in Shaka packager github to see if they'll have some idea.

Thank you again. I am closing this issue since it's not related to shaka player.

[joeyparrish]

@jakubvojacek, when times are aligned after the packager is started, but not after many hours of playback, this is encoder drift. The live streams that I publish are currently keeping up with real time (ffmpeg reports 1x on the command line), but sometimes they fall behind, with ffmpeg reporting more like 0.999x.

That 0.1% difference can, over 20 hours, accumulate to more than 1 minute of drift. So even what looks like a tiny amount of drift can add up quickly. When you are using free, off-the-shelf tools to encode content, you can't guarantee perfection. (I doubt you can guarantee perfection with any software-only solution.)

Because of this, and because of our recent first-hand experience publishing live streams, we have changed our position on encoder drift. We used to label drifting streams as "bad content", but we now feel that drift is inevitable (at least in DASH), and that there's no reason we should continue to reject or fail such content. (We can't even produce 100% non-drifting content ourselves, so who are we to judge the whole industry?)

In HLS, the lack of a strict timeline means that there's nothing to drift away from in the first place, making HLS more robust for strict players. So the future of our DASH implementation will behave more like HLS, to avoid rejecting things we could otherwise play.

I recommend that you subscribe to #999 to keep tabs on our progress in tolerating drift. This is scheduled for the v2.5 release, and is part of a much larger refactoring effort in that release.

Thanks!

[jakubvojacek]

@joeyparrish thank you for the effort, will be waiting for 2.5 :)