Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stream not recovering when interrupted (reading from itself) #1611

Open
motiejus opened this issue Feb 19, 2025 · 1 comment
Open

stream not recovering when interrupted (reading from itself) #1611

motiejus opened this issue Feb 19, 2025 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@motiejus
Copy link

motiejus commented Feb 19, 2025

Once a stream disconnects for some reason, it doesn't recover. It can take hours or days until it gets interrupted; but it does not recover itself -- go2rtc needs to be restarted.

I have a specific example of vno4-dome-panorama-low, but sometimes other streams fail (I just captured the full suite of logs with this one).

Here is the moment when it gets disconnected and a bit around (the last recorded frame is 21:31:38):

Feb 19 21:31:24 fwminex go2rtc[847109]: 21:31:24.785 TRC [rtsp] server request:
Feb 19 21:31:24 fwminex go2rtc[847109]: OPTIONS rtsp://localhost:8554/vno4-dome-panorama-low RTSP/1.0
Feb 19 21:31:24 fwminex go2rtc[847109]: User-Agent: FFmpeg Frigate/0.14.1
Feb 19 21:31:24 fwminex go2rtc[847109]: Session: 32236871
Feb 19 21:31:24 fwminex go2rtc[847109]: Cseq: 6
Feb 19 21:31:24 fwminex go2rtc[847109]:
Feb 19 21:31:24 fwminex go2rtc[847109]: 21:31:24.785 TRC [rtsp] server response:
Feb 19 21:31:24 fwminex go2rtc[847109]: RTSP/1.0 200 OK
Feb 19 21:31:24 fwminex go2rtc[847109]: Cseq: 6
Feb 19 21:31:24 fwminex go2rtc[847109]: Session: 32236871
Feb 19 21:31:24 fwminex go2rtc[847109]:
Feb 19 21:31:38 fwminex go2rtc[847109]: 21:31:38.024 TRC [rtsp] server request:
Feb 19 21:31:38 fwminex go2rtc[847109]: TEARDOWN rtsp://localhost:8554/vno4-dome-panorama-low RTSP/1.0
Feb 19 21:31:38 fwminex go2rtc[847109]: Cseq: 7
Feb 19 21:31:38 fwminex go2rtc[847109]: User-Agent: FFmpeg Frigate/0.14.1
Feb 19 21:31:38 fwminex go2rtc[847109]: Session: 32236871
Feb 19 21:31:38 fwminex go2rtc[847109]:
Feb 19 21:31:38 fwminex go2rtc[847109]: 21:31:38.024 DBG [rtsp] handle error="read tcp [::1]:8554->[::1]:38742: read: connection reset by peer"
Feb 19 21:31:38 fwminex go2rtc[847109]: 21:31:38.024 DBG [streams] stop producer url=ffmpeg:rtsp://localhost:8554/vno4-dome-panorama-orig#hardware=vaapi#video=h264#width=1280
Feb 19 21:31:38 fwminex go2rtc[847109]: 21:31:38.024 TRC [exec] kill with signal=9
Feb 19 21:31:38 fwminex go2rtc[847109]: 21:31:38.029 DBG [rtsp] handle error=EOF
Feb 19 21:31:38 fwminex go2rtc[847109]: 21:31:38.029 DBG [rtsp] disconnect stream=vno4-dome-panorama-orig
Feb 19 21:31:41 fwminex go2rtc[847109]: 21:31:41.017 TRC [rtsp] server request:
Feb 19 21:31:41 fwminex go2rtc[847109]: OPTIONS rtsp://localhost:8554/vno4-dome-panorama-orig RTSP/1.0
Feb 19 21:31:41 fwminex go2rtc[847109]: Session: 12820664
Feb 19 21:31:41 fwminex go2rtc[847109]: Cseq: 40
Feb 19 21:31:41 fwminex go2rtc[847109]: User-Agent: go2rtc/ffmpeg
Feb 19 21:31:41 fwminex go2rtc[847109]:
Feb 19 21:31:41 fwminex go2rtc[847109]: 21:31:41.017 TRC [rtsp] server response:
Feb 19 21:31:41 fwminex go2rtc[847109]: RTSP/1.0 200 OK
Feb 19 21:31:41 fwminex go2rtc[847109]: Cseq: 40
Feb 19 21:31:41 fwminex go2rtc[847109]: Session: 12820664
Feb 19 21:31:41 fwminex go2rtc[847109]:
Feb 19 21:31:43 fwminex go2rtc[847109]: 21:31:43.691 TRC [rtsp] server request:
Feb 19 21:31:43 fwminex go2rtc[847109]: OPTIONS rtsp://localhost:8554/vno4-dome-panorama-low RTSP/1.0
Feb 19 21:31:43 fwminex go2rtc[847109]: User-Agent: FFmpeg Frigate/0.14.1
Feb 19 21:31:43 fwminex go2rtc[847109]: Cseq: 1

When ffmpeg routinely tries to connect to it:

Feb 19 22:46:55 fwminex go2rtc[847109]: 22:46:55.412 TRC [rtsp] server request:
Feb 19 22:46:55 fwminex go2rtc[847109]: OPTIONS rtsp://localhost:8554/vno4-dome-panorama-low RTSP/1.0
Feb 19 22:46:55 fwminex go2rtc[847109]: User-Agent: FFmpeg Frigate/0.14.1
Feb 19 22:46:55 fwminex go2rtc[847109]: Cseq: 1
Feb 19 22:46:55 fwminex go2rtc[847109]:
Feb 19 22:46:55 fwminex go2rtc[847109]: 22:46:55.412 TRC [rtsp] server response:
Feb 19 22:46:55 fwminex go2rtc[847109]: RTSP/1.0 200 OK
Feb 19 22:46:55 fwminex go2rtc[847109]: Cseq: 1
Feb 19 22:46:55 fwminex go2rtc[847109]: Public: OPTIONS, SETUP, TEARDOWN, DESCRIBE, PLAY, PAUSE, ANNOUNCE, RECORD
Feb 19 22:46:55 fwminex go2rtc[847109]:
Feb 19 22:46:55 fwminex go2rtc[847109]: 22:46:55.413 TRC [rtsp] server request:
Feb 19 22:46:55 fwminex go2rtc[847109]: DESCRIBE rtsp://localhost:8554/vno4-dome-panorama-low RTSP/1.0
Feb 19 22:46:55 fwminex go2rtc[847109]: Accept: application/sdp
Feb 19 22:46:55 fwminex go2rtc[847109]: Cseq: 2
Feb 19 22:46:55 fwminex go2rtc[847109]: User-Agent: FFmpeg Frigate/0.14.1
Feb 19 22:46:55 fwminex go2rtc[847109]:
Feb 19 22:46:55 fwminex go2rtc[847109]: 22:46:55.413 DBG [rtsp] new consumer stream=vno4-dome-panorama-low
Feb 19 22:46:55 fwminex go2rtc[847109]: 22:46:55.413 TRC [streams] check cons=455 media=video, sendonly, ANY

go2rtc config:

api:
  listen: :1984
ffmpeg:
  bin: /nix/store/wcf97bi1lbl299iamk55jqsr5m85sbmq-ffmpeg-headless-7.1-bin/bin/ffmpeg
log:
  format: text
  level: trace
rtsp: {}
streams:
  vno4-dome-panorama-high:
  - ffmpeg:rtsp://localhost:8554/vno4-dome-panorama-orig#hardware=vaapi#video=h264
  vno4-dome-panorama-low:
  - ffmpeg:rtsp://localhost:8554/vno4-dome-panorama-orig#hardware=vaapi#video=h264#width=1280
  vno4-dome-panorama-orig:
  - ffmpeg:rtsp://frigate:${FRIGATE_RTSP_PASSWORD}@192.168.188.10/cam/realmonitor?channel=1&subtype=2
  vno4-dome-ptz-high:
  - ffmpeg:rtsp://localhost:8554/vno4-dome-ptz-orig#hardware=vaapi#video=h264
  vno4-dome-ptz-low:
  - ffmpeg:rtsp://localhost:8554/vno4-dome-ptz-orig#hardware=vaapi#video=h264#width=1280
  vno4-dome-ptz-orig:
  - ffmpeg:rtsp://frigate:${FRIGATE_RTSP_PASSWORD}@192.168.188.10/cam/realmonitor?channel=2&subtype=0
webrtc: {}

The -orig streams are h.265 (for bandwidth), which get transcoded to two h.264 streams for local processing.

Camera Dahua DH-SDT3E410-8P-MB-A-PV1

I put whatever I could find here, not sure what's useful.

Image

vno4-dome-panorama-low config:

{
  "producers": [
    {
      "id": 114,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "127.0.0.1:46042 forwarded localhost:8554",
      "source": "exec:/nix/store/wcf97bi1lbl299iamk55jqsr5m85sbmq-ffmpeg-headless-7.1-bin/bin/ffmpeg -hide_banner -v error -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_flags allow_profile_mismatch -allowed_media_types video -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://localhost:8554/vno4-dome-panorama-orig -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -an -vf \"format=vaapi|nv12,hwupload,scale_vaapi=1280:-1:out_color_matrix=bt709:out_range=tv:format=nv12\" -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/3932ae285ec55ea2f2cc7c4fb6d1de3c",
      "url": "rtsp://localhost:8554/vno4-dome-panorama-orig",
      "sdp": "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=go2rtc/1.9.7\r\nc=IN IP4 127.0.0.1\r\nt=0 0\r\na=tool:libavformat 61.7.100\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H264/90000\r\na=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z2QMKawrQCgF/zTUFBgHwAAAAwBAAAAMo8IBCoA=,aO44sA==; profile-level-id=640C29\r\na=control:streamid=0\r\n",
      "user_agent": "ffmpeg/go2rtc",
      "medias": [
        "video, recvonly, H264"
      ],
      "receivers": [
        {
          "id": 115,
          "codec": {
            "codec_name": "h264",
            "codec_type": "video",
            "level": 41,
            "profile": "High"
          },
          "bytes": 7302821,
          "packets": 6035
        }
      ],
      "bytes_recv": 7375661
    }
  ],
  "consumers": []
}

vno4-dome-panorama-orig config:

{
  "producers": [
    {
      "id": 67,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "127.0.0.1:40604 forwarded 192.168.188.10",
      "source": "exec:/nix/store/wcf97bi1lbl299iamk55jqsr5m85sbmq-ffmpeg-headless-7.1-bin/bin/ffmpeg -hide_banner -v error -allowed_media_types video+audio -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://frigate:*@192.168.188.10/cam/realmonitor?channel=1\u0026subtype=2 -c copy -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://127.0.0.1:8554/abed6807a35cfbfb900ef6ce8bad7790",
      "url": "rtsp://frigate:*@192.168.188.10/cam/realmonitor?channel=1\u0026subtype=2",
      "sdp": "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=Media Server\r\nc=IN IP4 127.0.0.1\r\nt=0 0\r\na=tool:libavformat 61.7.100\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H265/90000\r\na=fmtp:96 sprop-vps=QAEMAf//AWAAAAMAoAAAAwAAAwB7rAwAAAMABAAAAwBmqA==; sprop-sps=QgEBAWAAAAMAoAAAAwAAAwB7oAFQIAvFjS7kUvzcFBgUEAAAAwAQAAADAZoQ; sprop-pps=RAHA8o54OzQ=\r\na=control:streamid=0\r\n",
      "user_agent": "ffmpeg/go2rtc",
      "medias": [
        "video, recvonly, H265"
      ],
      "receivers": [
        {
          "id": 68,
          "codec": {
            "codec_name": "hevc",
            "codec_type": "video"
          },
          "childs": [
            69
          ],
          "bytes": 1613523802,
          "packets": 1186657
        }
      ],
      "bytes_recv": 1627798574
    }
  ],
  "consumers": [
    {
      "id": 65,
      "format_name": "rtsp",
      "protocol": "rtsp+tcp",
      "remote_addr": "[::1]:41450",
      "sdp": "v=0\r\no=- 1 1 IN IP4 0.0.0.0\r\ns=go2rtc/1.9.7\r\nc=IN IP4 0.0.0.0\r\nt=0 0\r\nm=video 0 RTP/AVP 96\r\na=rtpmap:96 H265/90000\r\na=fmtp:96 sprop-vps=QAEMAf//AWAAAAMAoAAAAwAAAwB7rAwAAAMABAAAAwBmqA==; sprop-sps=QgEBAWAAAAMAoAAAAwAAAwB7oAFQIAvFjS7kUvzcFBgUEAAAAwAQAAADAZoQ; sprop-pps=RAHA8o54OzQ=\r\na=control:trackID=0\r\n",
      "user_agent": "go2rtc/ffmpeg",
      "medias": [
        "video, sendonly, ANY",
        "audio, sendonly, ANY"
      ],
      "senders": [
        {
          "id": 69,
          "codec": {
            "codec_name": "hevc",
            "codec_type": "video"
          },
          "parent": 68,
          "bytes": 1613523802,
          "packets": 1186657
        }
      ],
      "bytes_send": 1632510314
    }
  ]
}

I would appreciate some guidance what to do to make it recover, or how to troubleshoot it further.

Thank you!

motiejus added a commit to motiejus/config that referenced this issue Feb 19, 2025
@AlexxIT AlexxIT added the bug Something isn't working label Feb 20, 2025
@AlexxIT AlexxIT self-assigned this Feb 20, 2025
@motiejus
Copy link
Author

Happened again. Here is an example go2rtc log line with non-debug logging:

Feb 21 04:36:22 fwminex go2rtc[296913]: 04:36:22.848 WRN github.com/AlexxIT/go2rtc/internal/streams/producer.go:170 > error=EOF url=ffmpeg:rtsp://localhost:8554/vno4-dome-panorama-orig#hardware=vaapi#video=h264

... which my poor man's watchdog then promptly detected & mitigated.

If there is any more info I can provide, let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants