Info: More RTSP insight

[philipp-schmidt]

Hello @rarzumanyan

It's a petty I couldn't be more active and responsive in this repo the last few months, been very busy at work. I'm not sure what the current state of RTSP is in this repo and I can see there have been some commits. Is this using ffmpeg bitstreams via process pipes now?

Anyway:
I have a few insights to share what I think might have gone wrong a while ago when we were discussing RTSP. Maybe it gives a few helpful pointers. For me it looks like that audio in RTSP might have been the big issue after all. I've been fixing this in our video pipeline recently where some RTSP streams would freeze randomly, similar to my original issue here on github. It's using gstreamer to extract the h264 and the pipeline just wouldn't play video sometimes. I debugged the issue and it turns out that there's just a slim chance that the audio pad in the gstreamer pipeline gets pad ID 0 (probably just by chance whether audio or video gets ID 0) and therefore gets selected by the automatic decoder blocks coming after that. The gstreamer graph between rtsp and a cpu decoder then looks something like this:

The pipeline is now indeed running, because it's running on the audio channel, but the video/h264 is stalled infinitely. The video pad is right there but it was ignored. From here it was an easy fix.

I know the implementation with AV here is different, but maybe it's worth checking how rtsp audio channels were handled. Maybe also not handling them explicitly was creating issues as well. If I have time I might go back and check the IP cameras whether only the ones with audio channels weren't working.

Hope this gives some insight. You can close this issue right away if you want.