Manifest Names / Paths with Reused Jobs

[j0sh]

Is your feature request related to a problem? Please describe.

In the OS release proposal, one of the requirements is for "full playlist" support which would hold a collection of all the segments in the stream. Assuming we keep this feature for the release, the question becomes how to handle full playlists for live streams that have been attached to a reused job. For example,

Connection 1, Job 1
Segments 0, 1, 2, 3
Full Manifest - 0, 1, 2, 3

Broadcaster Disconnects, Reconnects

Connection 2, Job 1
Segments 4, 5, 6, 7
Full Manifest: 0, 1, 2, 3, 4, 5, 6, 7 ?
      Or perhaps a new manifest for 4, 5, 6, 7?

Because we auto-start jobs, and automatically try to reuse jobs if possible, it is difficult to infer the intent of the broadcaster: do they want to resume an existing stream, or start a new one altogether? This has implications for how the full manifest is constructed.

Possible Solutions

1. Append to the existing manifest corresponding to the JobID

2. Split manifests on a per RTMP connection basis, eg playlist_<nonce>.m3u8

3. A hybrid: combine manifests if the stream has been offline for less than a certain period of time, eg 1 hour. If the stream has been offline for longer than 1 hour, create a new manifest with a new nonce. Otherwise, reuse the old manifest.

4. Use a more concrete method of disambiguating a RTMP stream vs a job vs a named manifest, perhaps using the RTMP path. For example, if the RTMP session goes to "rtmp://localhost/stream/foo" then a master playlist called foo.m3u8 would be created -- regardless of the actual JobId or RTMP connection. Append to any existing foo.m3u8 manifest that the broadcaster is aware of.

5. This could be an opportunity to integrate job creation step along with stream keys. The creation step could specify an output location, RTMP stream key / aliases, and so forth.

The last one is my favorite approach, but probably requires the most work. We will likely have to do this to support PM / Streamflow, so it might be good to spec this out sooner rather than later. This is one reason I think it'd be better to defer this particular feature until later; the path forward should be clearer after VOD is implemented.

Additional context
This will likely be a non-issue in VOD, where the API endpoint that starts a transcode job is parameterized by the output name, without a direct mapping to the job itself. The jobID/streamID then becomes an accounting artifact of the underlying blockchain protocol.

Also related: how to determine output paths for these files? #559

[eladmallel]

👍 for solution #5, since it's indeed on the path to Streamflow and usability at scale. Are there strong reasons to prefer a different solution for now, rather than waiting for the right time to execute on #5?

thanks!

[j0sh]

Are there strong reasons to prefer a different solution for now

Not anymore since we'll hold this feature until after VOD, so we can take the time to spec out #5.

[darkdarkdragon]

@j0sh I still want to make option 2 in this release (but with small change, playlist name should be streamid_seqno.m3u8, where seqno is sequence number, from which current session started.
Couple of reasons why I still want this:

1. This feature has nothing to do whit our future VOD release, it is just convenience for user, so after stream user will have playlist with all the stream inside it.
2. It is very easy to implement, just dozen lines of code, almost nothing to test.
3. It is immediately useful for users.

[eladmallel]

How is #2 useful before we support more than one RMTP per B?

[darkdarkdragon]

@eladmallel for #2 it doesn't matter how much RTMP streams B supports.

[j0sh]

We can do #2 but the node itself wouldn't have a record of the full manifest after a restart or when a new job comes in, so it'd be kind of floating around in S3 or wherever. Granted, this is also true for the segments right now.

One of the tasks for VOD will be to make this persistent, so the whole thing becomes much more useful on a long-term basis, including for live streams.

[darkdarkdragon]

@j0sh yes, it will be floating around in s3 or local storage. whole point of this is to have them manifest float around segments, so user that needed this segments will have manifest that stitches segments together, not just bunch of video segments with no idea how to make video out of them.

[j0sh]

This issue is a bit dated since a lot of things have changed since then. However, full playlist support is important and we have just as many (if not more) questions about how to handle those. Captured these questions in a more up-to-date issue: #869