I don't know why using pyav for rtsp stream object detection results in errors, but not when using ffmpeg and opencv

[hanruiyao163]

I use the following code to get the rtsp stream, perform object detection and then push the stream

import cv2
from ultralytics import YOLO
import av

input_container = av.open("rtsp://192.168.1.107/live/zoo2")
input_stream = input_container.streams.video[0]
rate = input_stream.base_rate

width = input_stream.width
height = input_stream.height
fps = input_stream.base_rate

output_container = av.open("rtsp://192.168.1.107/live/zoo-detect", mode="w", format="rtsp")
stream = output_container.add_stream("h264", rate=30)
stream.width = width
stream.height = height
stream.pix_fmt = "yuv420p" 

for frame in input_container.decode(input_stream):
    frame = frame.to_ndarray(format="rgb24")
    results = model(frame, verbose=False)
    annotated_frame = results[0].plot()

    av_frame = av.VideoFrame.from_ndarray(annotated_frame, format="rgb24")

    for packet in stream.encode(av_frame):
        output_container.mux(packet)

input_container.close()
output_container.close()

It throws many errors and causes screen stuttering and tearing

mmco: unref short failure
reference picture missing during reorder
reference picture missing during reorder
Missing reference picture, default is 65596
Missing reference picture, default is 65596
co located POCs unavailable
reference picture missing during reorder
reference picture missing during reorder
...

But the following code works fine:

import cv2
from ultralytics import YOLO
import av
import subprocess as sp
import numpy as np

model = YOLO("yolo11n.pt", verbose=False)

cap = cv2.VideoCapture("rtsp://192.168.1.102/live/zoo1")
fps = int(cap.get(cv2.CAP_PROP_FPS))
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

command = [
    'ffmpeg',
    '-y',
    '-f', 'rawvideo',
    '-vcodec', 'rawvideo',
    '-pix_fmt', 'bgr24',
    '-s', "{}x{}".format(width, height),
    '-r', str(fps),
    '-i', '-',
    '-c:v', 'libx264',
    '-pix_fmt', 'yuv420p',
    '-preset', 'veryfast',
    '-f', 'rtsp',
    '-rtsp_transport', 'tcp',
    '-g', '5',
    "rtsp://192.168.1.107/live/zoo-detect"
]
pipe = sp.Popen(command, stdin=sp.PIPE)
while cap.isOpened():
    ret, frame = cap.read()
    results = model(frame, verbose=False)
    annotated_frame = results[0].plot()
    pipe.stdin.write(annotated_frame.tobytes())

[UltimateCodeWarrior]

I wonder if the video buffers are getting overwritten /overflowing because yolo is taking too long to identify things. Maybe you could clone the frame, and queue them in a separate buffer and process them in another thread so that it won't block the frame retrieval. From there you can see how many backlog frames you are getting. If it starts growing too large, you could skip frames to keep up or shrink them down or get a CUDA NVIDIA GPU with optimized opencv/ffmpeg code, or maybe a Mac M4 Max, or wait for the NVIDIA Digits Computer.