Provide ways to avoid projection for live streams in a playlist

[apptrash]

I'm using HLS event-playlists. There are always two MediaItem's in a playlist. I wish to have seamless transition, but sometimes ExoPlayer doesn't play next item from beginning (it skips around 6-30 seconds and start from this position).

What I do is:

override fun onMediaItemTransition(mediaItem: MediaItem?, reason: Int) {
    if (reason == MEDIA_ITEM_TRANSITION_REASON_AUTO) {
        Log.d(TAG, "position: ${player.getCurrentPosition()}") // <-- NOT ZERO !!!
        player.moveMediaItem(1, 0)
        player.removeMediaItem(1)
        player.addMediaItem(nextMediaItem)
    }
}

Can you help me? Am i doing something wrong?

All of HLS playlists looks the same:

#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
#EXT-X-START:TIME-OFFSET=0
#EXT-X-MEDIA-SEQUENCE:26992
#EXT-X-PLAYLIST-TYPE:EVENT
#EXTINF:10.08000
#EXT-X-PROGRAM-DATE-TIME:2021-06-09T15:34:53.706Z
2021-06-09/15/segment_153453706_010080.ts
#EXTINF:10.08000
#EXT-X-PROGRAM-DATE-TIME:2021-06-09T15:35:03.960Z
2021-06-09/15/segment_153503960_010080.ts
#EXTINF:10.08000
#EXT-X-PROGRAM-DATE-TIME:2021-06-09T15:35:13.953Z
2021-06-09/15/segment_153513953_010080.ts
#EXTINF:10.08000
#EXT-X-PROGRAM-DATE-TIME:2021-06-09T15:35:23.654Z
2021-06-09/15/segment_153523654_010080.ts
#EXTINF:10.08000
#EXT-X-PROGRAM-DATE-TIME:2021-06-09T15:35:34.034Z
2021-06-09/15/segment_153534034_010080.ts
#EXTINF:10.08000
#EXT-X-PROGRAM-DATE-TIME:2021-06-09T15:35:44.006Z
2021-06-09/15/segment_153544006_010080.ts
...
#EXTINF:10.08000
#EXT-X-PROGRAM-DATE-TIME:2021-06-09T16:26:17.988Z
2021-06-09/16/segment_162617988_010080.ts
#EXTINF:10.08000
#EXT-X-PROGRAM-DATE-TIME:2021-06-09T16:26:28.364Z
2021-06-09/16/segment_162628364_010080.ts
#EXTINF:10.08000
#EXT-X-PROGRAM-DATE-TIME:2021-06-09T16:26:38.226Z
2021-06-09/16/segment_162638226_010080.ts

ExoPlayer version r2.14.0

[claincly]

Not sure if I understand, if your playlist contains two media items, ExoPlayer will handle the transition for you, you don't have to manually do transition.

[apptrash]

override fun onMediaItemTransition(mediaItem: MediaItem?, reason: Int) {
    if (reason == MEDIA_ITEM_TRANSITION_REASON_AUTO) {
        Log.d(TAG, "position: ${player.getCurrentPosition()}") // <-- NOT ZERO !!!
        player.addMediaItem(nextMediaItem)
    }
}

I deleted unnecessary lines player.moveMediaItem(1, 0) / player.removeMediaItem(1)
And it doesn't change anything. After transition new item position still != 0

[apptrash]

I'm adding new item when current item is ended (this is just a logic of an app). Maybe I should do it in some other place/callback?

[apptrash]

As you can see Log.d() print position before player.addMediaItem(nextMediaItem). So I think that it's not a reason.

[apptrash]

mediaItem [eventTime=218.51, mediaPos=28.52, window=1, period=1, reason=AUTO]
positionDiscontinuity [eventTime=218.51, mediaPos=28.52, window=1, period=1, reason=AUTO_TRANSITION, PositionInfo:old [window=0, period=0, pos=907200], PositionInfo:new [window=1, period=1, pos=28399]]

There are a couple of logs from 'EventLogger': as you can see mediaPos=28.52 and reason=AUTO_TRANSITION

[apptrash]

I've also notice next log on transition:

droppedFrames [eventTime=64.83, mediaPos=30.24, window=1, period=1, 2]

[apptrash]

I've created sample application to reproduce this issue: https://github.com/apptrash/ExoBugSandbox

[apptrash]

I do think that it's a bug rather than question

[apptrash]

@claincly can you help me with this issue?

[claincly]

I'll involve @christosts since it's related to HLS here. But one thing I wanted to make sure, are you able to reproduce the issue in our demo app? You can verify by adding an entry to
https://github.com/google/ExoPlayer/blob/b2333c86c1eac9a1f95992960a8495f1e5b79200/demos/main/src/main/assets/media.exolist.json

The entry could look like

      {
        "name": "Playlist with HLS streams",
        "playlist": [
          {
            "uri": "HLS URI 1"
          },
          {
            "uri": "HLS URI 2"
          },
          {
            "uri": "HLS URI 3"
          }
        ]
      }

[apptrash]

How can I add an entry to this repository? By using pull request?

[apptrash]

Of course, I will try to reproduce issue on demo app, but I don't think that it is possible to reproduce on it because there are some specific steps (like changing playWhenReady on PLAYER_READY event)

[apptrash]

@christosts, Hello! Any updates here?

[christosts]

I can reproduce on the demo app on 2.14.1.

I picked the two playlist URLs from https://github.com/apptrash/ExoBugSandbox and added them as playlist to the ExoPlayer demo app. When the player transitions to the 2nd item, it starts playing from around 10". But if we play the second playlist as a standalone item then it plays from 0.

I need to investigate further.

[apptrash]

Hello! Any progress on this task? Maybe you can suggest a workaround?

[marcbaechinger]

The start position for playback is projected, because your playlist has no end tag which indicates that further segments may be added and the stream is treated as a live stream. Because of this, when ExoPlayer starts loading data for buffering like for instance 20 seconds before the actual transition happens, it assumes that it should start 20 seconds later to have the correct latency from the live edge.

So if this is not a live stream the best fix would be to add an #EXT-X-ENDLIST tag at the end of the playlist.

(https://developer.apple.com/documentation/http_live_streaming/example_playlists_for_http_live_streaming/event_playlist_construction)

Is this a viable solution on your end?

[apptrash]

Unfortunately no, and there are situations (in my case) where transitions occurs between two playlists with type=event and the second one without endlist tag.

[apptrash]

There is EXT-X-START tag with 0 value in playlists. Shouldn't that solve the problem?

[marcbaechinger]

EXT-X-START does not help in this case of playing in a playlist I'm afraid. When played in a playlist the default start position is projected, so that the target latency to the live edge is not changed. The start time is 0 (zero), but the projection makes it start later. You are playing a live stream that actually isn't a live stream as it seems.

I think you can try to wrap your media source with a ClippingMediaSource and tell the clipping media source to start at position 0 and end at C.TIME_END_OF_SOURCE:

new ClippingMediaSource(
    secondMediaSource,  /* startPositionUs= */ 0, /* endPositionUs= */ C.TIME_END_OF_SOURCE);

[apptrash]

Unfortunately, it doesn't help, I tried to do the following:

player.setMediaItems(
    listOf(
        MediaItem.Builder().setUri(FIRST_MEDIA_ITEM_URL).setClipStartPositionMs(0)
            .setClipEndPositionMs(C.TIME_END_OF_SOURCE).build(),
        MediaItem.Builder().setUri(SECOND_MEDIA_ITEM_URL).setClipStartPositionMs(0)
            .setClipEndPositionMs(C.TIME_END_OF_SOURCE).build()
    )
)

Player still does not start playing from the beginning

[marcbaechinger]

Yes, that's a limitation of the DefaultMediaSourceFactory that is used internally by the player by default. If the clipStartPositionMs is 0 and the clipEndPositionMs is C.TIME_END_OF_SOURCE the media source is not automatically wrapped with a ClippingMediaSource, because it does not clip at all. Your case is special because of the playlist is actually a live playlist that isn't one and we only use the ClippingMediaSource to avoid projection. So you need to manually do that and use player.setMediaSources(listOfMediaSources), the media item API does not help out-of-the box.

To try that out in a hacky way you can start at the first millisecond. With this it clips the first millisecond of the stream. I would do that only to see that it works as expected:

MediaItem.Builder().setUri(FIRST_MEDIA_ITEM_URL)
    .setClipStartPositionMs(1)      <===
    .setClipEndPositionMs(C.TIME_END_OF_SOURCE)
    .build(),

A proper way to do this is using the player.setMediaSources(listOfMediaSources) method instead and create the source programmatically:

List<MediaSource> mediaSources = new ArrayList<>();
DefaultMediaSourceFactory mediaSourceFactory =
    new DefaultMediaSourceFactory(/* context= */ this);
for (int i = 0; i < mediaItems.size(); i++) {
  MediaSource mediaSource = mediaSourceFactory.createMediaSource(mediaItems.get(i));
  mediaSources.add(
      new ClippingMediaSource(
          mediaSource, /* startPositionUs= */ 0, /* endPositionUs= */ C.TIME_END_OF_SOURCE));
}
player.setMediaSources(mediaSources);

As an alternative you could do your custom MediaSourceFactory that does the wrapping and inject it when you build your player. That's a bit more work, but you could keep the media item approach. You also need to make sure that the factory only wraps these media sources with a ClippingMediaSource that really need it.

player =
    new SimpleExoPlayer.Builder(/* context= */ this, renderersFactory)
        .setMediaSourceFactory(customMediaSourceFactory)
        .setTrackSelector(trackSelector)
        .build();

[alex3d]

Hi, @marcbaechinger.
Could you explain the rationale for projecting the EVENT start position? It looks like Exoplayer bug for us.
AFAIK hls standard does not specify event playlist playback semantics, but the majority of player api's start EVENT playlist playback from the first segment. Exoplayer also starts playback from the first segment regardless of X-ENDLIST the presence in all cases except for the case described in this issue.

Our use case:
Live TV programs are mapped to EVENT hls playlists with X-ENDLIST added to the past programs.
This issue prevents us from playing two consecutive tv programs seamlessly in case if the second program is not finished yet i.e. program's EVENT playlist doesn't have X-ENDLIST in it.

[apptrash]

@marcbaechinger Thanks for explaining how the workaround works! I tried it - works fine.

[marcbaechinger]

@alex3d

A playlist without an X-ENDLIST tag is handled as a live stream. When such a stream is added to a playlist on a position different to 0, ExoPlayer starts buffering data of that live stream before it actually transitions to that stream for playback.

Lets assume the player starts buffering from the defaultPosition of the live stream 20 seconds before actually playing it. If the defaultPosition of the live stream tells us to start playback at 1:00:00am wall clock time and we prebuffer 20 seconds but we keep playing the last 20 seconds of the previous VOD stream, we actually start playback of the live stream 20 secs later at 1:00:20am. So the player projects 20 seconds ahead (meaning skipping the 20 seconds of overlapping media). Without doing that we would start playback at 1:00:00am media time at 1:00:20am wall clock time which would increase the target latency to the live edge by the duration of the overlapping buffered media data.

I don't think this is a bug, but rather a feature that tries to not to increase the live latency of a live stream in a playlist which looks like a sensible thing for some use cases. I certainly also see how this is problematic for your use case.

It appears to me that both are valid use case, so I think we should mark this issue as an enhancement to provide ways that users could configure whether projection should be omitted. We already have ways to configure behaviour for live streams by providing a MediaItem.LiveConfiguration object. Without thinking about whether this is the best solution, we could for example add a flag like LiveConfiguration.suppressProjectionInPlaylist or similar (disregard naming :). This could then be taken into account when transitioning so that projection can be suppressed.

[ojw28]

I don't think this is a bug, but rather a feature that tries to not to increase the live latency of a live stream in a playlist which looks like a sensible thing for some use cases. I certainly also see how this is problematic for your use case.

Hm. If #EXT-X-START:TIME-OFFSET=0 is being used then this feels like a bug to me. That's basically saying, "please start playback at the beginning", and that's not what the player is doing. I'm not sure the user should be expected to configure the intended behavior using LiveConfiguration in this case.

I would suggest reclassifying this issue as a bug again.

[alex3d]

I think the most reasonable solution is to not apply start time projection for EVENT (and VOD if it is not already so) playlists with positive #EXT-X-START:TIME-OFFSET (including implicit TIME-OFFSET=0).
A positive TIME-OFFSET indicates an offset from the beginning of the playlist. The beginning of the playlist is not "live". EVENT playlists could only be appended to. So no time projection is needed in this case.
A negative TIME-OFFSET indicates a negative time offset from the end of the last media segment in the playlist. The end of the EVENT playlist is a running target. So time projection seems pretty reasonable in this case.

[marcbaechinger]

A negative TIME-OFFSET indicates a negative time offset from the end of the last media segment in the playlist. The end of the EVENT playlist is a running target.

Agreed this is a valid point to distinguish these cases for HLS EVENT streams.

[alex3d]

@marcbaechinger is it intended you still project start position when EXT-X-START is unset (playback starts from the second segment from playlist beginning in this case)
In our product, we could explicitly set EXT-X-START=0, but it seems a little illogical.

[marcbaechinger]

Technically streams with an EXT-X-START that is positive or zero have the projection suppressed. I think this is intended in the sense that an EVENT stream is handled as a live stream when the end tag is missing and the start offset is undefined. So in this case projection is still applied to answer your question.

The change above does not change how the start position is calculated for streams with a start offset that is unset. I would expect that in absence of any other configuration by the playlist and the media source/item, the actual start position is calculated like (duration - (3 * targetDuration)). When you say this isn't the case when you see it starting from at the second segment?

Can you provide me with a test stream so I can look into the behavior for your stream? It depends on various properties of the media playlist how this the start position is calculated. If you're unable to share publicly, please send them to dev.exoplayer@gmail.com using a subject in the format "Issue #9037".

[apptrash]

@marcbaechinger Hello! I found a case where neither the proposed workaround nor today's fix solve the problem: if segments are added to the second media playlist. I can provide you with an example if required, as well as any clarifying information to reproduce.

[apptrash]

In my previous example, segments were not added to the second media playlist (it was static actually) and your workaround worked correctly.

[apptrash]

Should I create another issue about it?

[marcbaechinger]

@apptrash Please provide me with an example. If the start offset is set to 0 or a positive value by the playlist it does not projection anymore. I'm interested to see why it's not working. Please send me a stream url for your case.

[apptrash]

@marcbaechinger Everything works on version 2.14.2, thanks a lot!