Gstreamer RTSP Client causes UVC errors in LibRealsense

[mcelhennyi]

Required Info
Camera Model	D435 / D455 / T265
Firmware Version	5.13.00.50
Operating System & Version	Ubuntu 18.04
Kernel Version (Linux Only)	4.9.201-tegra
Platform	NVIDIA Jetson
SDK Version	2.50
Language	C/C++
Segment	Robot

Issue Description

I have a system where I have three shared objects loaded dynamically at run time. One is handling all of our realsense interaction, one is an RTSP gstreamer server and the other is an RTSP gstreamer client (on the same device).

The reason for the keep alive client is that when the gstreamer server goes from 1 or more clients to no clients there was a 10 second reset period that would cause re-connection errors. Our solution to this problem was a on-device keep-alive client that would connect via loop back to the server such that there would always be 1 or more clients. This keep-alive client would just sink to a "fakesink" and not make use of the incoming video stream data.

For the longest time, using different combinations of SDK versions and D435/D455 firmware versions, we have been getting these UVC errors off and on when setting up the RGB stream OR depth stream on the D4XX cameras.

We have recently come to find that when we disable this keep alive gstreamer module we do not have any UVC errors from librealsense.

So my question is: Does anyone know how a gstreamer RTSP "fakesink" client could cause UVC errors in the realsense side of things?

Also posted on Nvidia forum and Gstreamer GitLab Issues

[MartyG-RealSense]

Hi @mcelhennyi What kind of UVC errors are you receiving, please? Are they uvcvideo: Failed to query (GET_CUR) UVC control errors like those described in #5302 (comment)

[mcelhennyi]

Hi @mcelhennyi What kind of UVC errors are you receiving, please? Are they uvcvideo: Failed to query (GET_CUR) UVC control errors like those described in #5302 (comment)

I get "UVC header is not available" from the rs2::error &err, err.what().

[MartyG-RealSense]

There was a case at #9043 (comment) where Jetson, 400 Series and T265 was being used and the UVC header is not available error was being experienced. The solution in that particular case was to patch the Jetson kernel with the dedicated librealsense patch script for Jetson called patch-realsense-ubuntu-L4T.sh

[mcelhennyi]

There was a case at #9043 (comment) where Jetson, 400 Series and T265 was being used and the UVC header is not available error was being experienced. The solution in that particular case was to patch the Jetson kernel with the dedicated librealsense patch script for Jetson called patch-realsense-ubuntu-L4T.sh

Ironically, that is my coworker.

This UVC error is not seen when we remove the gstreamer module. When that module is not present the UVC error does not happen.

Are you implying that this patch would help with the gstreamer in the loop? We have the newest L4T which I thought came pre-baked with the librealsense patches (?).

[MartyG-RealSense]

At the time of writing this, the most recent JetPack version that the librealsense SDK currently supports from SDK 2.50.0 onwards is L4T 32.6.1, thanks to code contributed by a RealSense user. More information about this can be found at #9855

In the release notes for librealsense 2.43.0, it was announced that kernel patches were no longer required for the D415 and D435 / D435i models. This statement does not cover the D455 or T265.

https://github.com/IntelRealSense/librealsense/wiki/Release-Notes#nvidia-jetson---user-notification

I am not familiar enough with gstreamer to advise about how it would react to the patch. In situations that may be related to a kernel conflict, I would typically recommend building librealsense with the RSUSB backend, which is not dependent on Linux versions or kernel versions and does not require patching. It bypasses the Video4Linux kernel APIs and communicates with the device using a generic USB driver. However, it is not as well suited to multiple camera applications as a non-RSUSB build.

[mcelhennyi]

At the time of writing this, the most recent JetPack version that the librealsense SDK currently supports from SDK 2.50.0 onwards is L4T 32.6.1, thanks to code contributed by a RealSense user. More information about this can be found at #9855

In the release notes for librealsense 2.43.0, it was announced that kernel patches were no longer required for the D415 and D435 / D435i models. This statement does not cover the D455 or T265.

https://github.com/IntelRealSense/librealsense/wiki/Release-Notes#nvidia-jetson---user-notification

I am not familiar enough with gstreamer to advise about how it would react to the patch. In situations that may be related to a kernel conflict, I would typically recommend building librealsense with the RSUSB backend, which is not dependent on Linux versions or kernel versions and does not require patching. It bypasses the Video4Linux kernel APIs and communicates with the device using a generic USB driver. However, it is not as well suited to multiple camera applications as a non-RSUSB build.

Can you clarify your last sentence However, it is not as well suited to multiple camera applications as a non-RSUSB build.? Our setup is 1 D455, 2 D435s and 1 T265 all connected to one application. We do not have RSUSB enabled. Are you saying if we are not using RSUSB then the performance of multiple cameras maybe degraded/faulty (to include time sync)?

Thanks!

[MartyG-RealSense]

A build based on the RSUSB backend installation method can be helpful in situations where there is a conflict with the Linux kernel that cannot be otherwise resolved, or if an exotic hardware / software configuration is being used (such as an industrial PC / single-board computer) that does not work with a conventional patch-based librealsense build.

As metadata support is included automatically in an RSUSB build, this build method can also help to resolve problems with accessing metadata that are caused by a missing or incorrectly applied patch.

RSUSB does have disadvantages though. For a comparison of RSUSB backend (previously also known as libuvc backend) and its advantages and disadvantages compared to a patch-based librealsense build, please visit #5212 (comment) and scroll down through the linked-to comment to the section headed What are the advantages and disadvantages of using libuvc vs patched kernel modules?

RSUSB received improvements to its multicam handling in SDK 2.35.2 and also had further updates to it applied in 2.50.0, as described in #9821

Despite these enhancements, it is my understanding that a patched kernel build is still preferable to RSUSB when using a multiple camera setup though. An RSUSB build will also check for device changes less often than a patched build (RSUSB checks only every 5 seconds by default) and so some events such as requests to change an option setting may be missed by the SDK and have to be re-tried. It is possible to edit the SDK's code to reduce this 5 second time at the cost of slightly higher CPU utilization though.

I would also add that if you have written a librealsense script for a multicam application that accesses multiple cameras from the same application then it is recommended to use poll_for_frames() instead of wait_for_frames(), as discussed in #2422 (comment)

[mcelhennyi]

@MartyG-RealSense We will give RSUSB a shot and I'll post back our results. Thanks for the help!

[MartyG-RealSense]

You are very welcome, @mcelhennyi - I look forward to your next report. Good luck!

[mcelhennyi]

This seems to have fixed the issue for us thus far (Using RSUSB). If anything else comes up I will re-open.

Thanks!

[MartyG-RealSense]

It's great to hear that you achieved an apparent solution, @mcelhennyi - thanks very much for the update!