AppWorker Queue Overflow - feature request to expand alert module

[bchah]

We already know that transcoders will consume all system memory if their queues cannot be kept up with. Now as customers are streaming more in 4K and higher bitrates, we are also seeing memory leak issues with AppWorker. Specifically, even if you assign a higher number of threads to AppWorker it will max out at one thread per stream and in cases where that one thread cannot handle the workload, it will eat system memory at the same rate new data is incoming.

Some solutions could be:

• Expand the alert module to allow these leaks to be handled and stopped, even if it means stopping the incoming stream.
• Improve the workers to spread their workload across available threads dynamically.
• Improve any processes which utilize a transcode/packetization queue to either 'flush' the queue or shut down the process in cases where memory overrun reaches a certain threshold, without stopping that same process for everybody else who is using it. This might also fix the problem where high-bitrate streams slowly get more and more behind real-time.

None of these solutions are easy, but this is becoming more of a real-world problem over time and it is ultimately translating into "OvenMediaEngine can't handle high-bitrate streams" which is not really true.

Example: If we have an OME server with 10 HD streams that use a Thumbnail publisher, then somebody starts streaming HEVC, the only way to prevent OME from crashing for everybody is to go in and disable the Thumbnail publisher for everybody, then restart OME.

Another example: If we have a well-balanced server where many users are streaming in, it only takes one user sending a high bitrate to end the party for everybody. We could use the alert module to cancel that one high-bitrate stream, but it doesn't make sense to do this when more than 80% of the CPU is sitting idle.

Sorry if I seem a little frustrated here, but both myself and my customers have been seeing these problems a lot lately. And I don't have a solution for them besides "OME doesn't handle high-bitrate streams and you need to disable thumbnails/transcodes entirely if you are sending even one HEVC or high resolution stream". Hopefully together we can find a better solution.

[getroot]

TranscodeWebhook can help you.
https://airensoft.gitbook.io/ovenmediaengine/transcoding/transcodewebhook

Unlike AdmissionWebhook, TranscodeWebhook passes the profile information of the input stream exactly. If you know the maximum performance of your system, this can help you decide if you should block the stream.

[getroot]

And Memory Leak means that the memory is increasing because it is not freed due to a bug. In this case, how about we use Queue overflow or other more clear terms to express the problem more clearly?

[bchah]

Thanks @getroot and sorry for using the wrong terminology. You're right about TranscodeWebhook but that only applies before a stream begins and as you stated, is effective if you know the maximum performance of each system - which is difficult to calculate consistently from system to system, stream to stream.

My feature request here is to expand the alert module - how about sending a webhook to alert for queue overflow incidents and allow us to cancel just the process that is causing the queue overflow? For example, if there are 10 streams with 10 thumbnail publishers, and one of the thumbnail transcodes begins to overflow the queue - we would be able to stop or pause that transcode until the queue is under the threshold again.

[getroot]

Defining terms is always a topic of healthy discussion, so we're doing it right.

It's possible to monitor the queue, but it's almost impossible to dynamically stop/restart the transcoder. However, if you could set a max size for the queue, you could achieve the same result you want.

So should we list all the internal queues and allow the user to set the max size for each queue? This is really expert work, but I think it makes sense.

[bchah]

That's a great idea! Just restarting the transcoder if the queue threshold is exceeded. It would also help prevent latency issues when the transcoded outputs start falling behind real-time.

Listing and maintaining sizes of all queues is indeed a lot of work. What if we could set a % value for maximum queue overflow based on the built-in thresholds? So more than for example 20% over any threshold would cause the process holding that queue to restart. Does that seem like an easier implementation?

[getroot]

As mentioned before, it is impossible to stop/restart the transcoder dynamically while it is running. It is very difficult. Instead, it is possible to fix the queue so that it does not exceed the specified max size. With that, we can achieve the same result as stopping the transcoder.

However, this requires several considerations. When the system slows down, all the queues start to overflow, so if you limit the max size of all the queues, you will simultaneously experience problems such as no thumbnails and poor playback. Therefore, you can limit the max size of queues that you consider less important, such as thumbnail encoding. However, as mentioned before, it is difficult to expect that the whole system will run smoothly with this alone. In fact, the best thing to do is to block the stream that is causing the problem.