[Question] Inquiry about Realsense Depth Camera Series

[soribido]

Required Info
Camera Model
Firmware Version
Operating System & Version	Ubuntu22.04 LTS or Windows11
Kernel Version (Linux Only)
Platform	PC
SDK Version	{ legacy / 2.. }
Language	opencv/python }
Segment	{Robot/Smartphone/VR/AR/others }

I have some questions regarding the Realsense Depth Camera Series. I am planning to install a depth camera on the ceiling to detect depth information of objects and people. My intention is to treat the camera and PC as separate entities, receiving real-time video streams from the camera via remote communication.

I am specifically interested in obtaining only the depth image (map) and RGB image. This process will be carried out using Python and OpenCV, avoiding the use of other languages, GUIs, or additional software. More specifically, I aim to use the depth information corresponding to the same location as the RGB image input for a detection model. The distance between the ceiling and the ground is expected to be approximately 8 to 10 meters.

Given this context, I have a few questions. While I am not an expert in cameras, I understand that the answers might differ depending on whether the camera uses stereo or ToF technology. I appreciate your understanding in this regard.

1. Is it possible to install the camera on the ceiling and receive video streams on a remote PC?
2. Does the camera support the RTSP protocol?
3. What is the expected margin of error when detecting a 1.8-meter-tall person from a height of 10 meters?
4. Can the camera provide both RGB images and depth maps simultaneously?
5. Does the depth map represent relative distances, or can it be converted into absolute distances through specific processing? If so, could you explain how this is achieved?
6. In a scenario where the camera is mounted on a 10-meter ceiling, how is depth calculated for objects at different angles? For example, for a 2-meter-tall object directly below the camera and another 2-meter-tall object at a 45-degree angle from the camera, will the depth values differ? If so, will the difference be approximately sqrt(2), or will it be calculated differently?
7. I’ve observed in some cases that depth information might not be captured (e.g., appearing black) in dark areas or other specific conditions. What causes this issue, and are there any solutions?
8. According to this issue, it seems that RGB and depth images might not align properly. Does this issue still persist?

Lastly, I’d like to know if there are any ToF-based Realsense cameras available. If so, could you recommend a suitable camera for this setup?

[MartyG-RealSense]

Hi @soribido Thanks very much for your questions.

1. It is possible to stream RealSense data wirelessly over wi-fi, though a wired connection would be preferable. The camera mounted on the ceiling would need to have a small computer board attached to it (such as Raspberry Pi or Jetson Nano), as the camera does not contain the hardware necessary to transmit data on its own.

2. The RealSense SDK software contains an RTSP-protocol networking tool up until version 2.53.1 of the software (the current version of the time of writing this is 2.56.3). It was removed from version 2.54.1 onwards, so you would need to use that older version of the SDK.

3. The amount of error in an image can be influenced by a range of factors. The camera model that is chosen will be especially important. The amount of error increases as distance from the camera increases. So for a 10 meter ceiling, a model in the D45x class would be most suitable (D455, D455f, D456, D457). These models remain accurate even at 6 meters away from an observed object / surface. Accuracy will start to drift beyond that distance.

The error factor on those models is rated at less than 2% at 4 meters. I would expect it to still have similar accuracy at 6 meters. There would naturally be a reduction in accuracy beyond 6 meters, though for observing a large surface like a human I would not expect it to be too bad.

4. Yes, RealSense cameras can provide depth and RGB simultaneously.

5. RealSense 400 Series output depth values in meters. It is not an absolute range, but is measured parallel to the imaging sensor like in the diagram below. The distance measured with this parallel method will be almost the same as an absolute measurement (like running a tape measure from the camera to the object), with some small drift in accuracy compared to the tape measure depending on the camera model used and lighting / environmental conditions in the camera's location.

6. The parallel-line-from-sensor method of depth measuring that the 400 Series cameras use mean that the angle should not matter when measuring distance.

7. Areas of the depth image can appear black if a surface is colored black or dark grey, because it is a general physics principle (not specific to RealSense cameras) that those colors absorb light. A surface may also be difficult for the camera to read - resulting in a black area - if it is highly reflective or there is a strong concentration of light such as a lamp on a particular spot on the surface. It is possible to use physical filter products over the camera lenses to help to negate these negative effects, or use a RealSense camera with built-in light blocking filters such as the D455f model.

8. It is possible for misalignment to occur on an aligned image, though it does not occur often.

9. The RealSense L515 camera model uses Time of Flight (ToF) principles, though it is now a retired camera model and no longer supported in the RealSense SDK, though still available from the official RealSense Store whilst stocks last. The last SDK version that supports it is 2.53.1. It would likely not be suitable for your particular project though as its maximum depth sensing range is 9 meters.

[soribido]

@MartyG-RealSense
Thank you for your kind and detailed response.

I have a follow-up question based on your response to my first inquiry. As I understood, it seems that devices like a PoE Hub or NVR cannot be used to connect the RealSense camera to the network directly, and instead, a Raspberry Pi or Jetson Nano is required. Is that correct? Could you please clarify why these devices are necessary, and if a PoE Hub or NVR is not suitable in this case?

[MartyG-RealSense]

RealSense 400 Series cameras do not have a power-over-ethernet connector (PoE) and instead draw their power and communicate via a USB cable.

Although the majority of users of the rs-server RealSense ethernet tool at the link below use a Raspberry Pi with it, as far as I am aware it is possible to use another board or a PC computer with it. The important thing is that the computer / computing board that the camera is plugged into runs the Linux OS, as the rs-server networking program can only be compiled on Linux.

https://dev.intelrealsense.com/docs/open-source-ethernet-networking-for-intel-realsense-depth-cameras

An alternative method of networking RealSense cameras is to use the Python-based EtherSense project, whose documentation shows the use of PoE with an Up Board instead of a Pi.

https://github.com/krejov100/EtherSense

https://dev.intelrealsense.com/docs/depth-camera-over-ethernet-whitepaper

[soribido]

@MartyG-RealSense
Thank you for your response.
I will conclude my questions here and open a new issue if I have further inquiries in the future.

[MartyG-RealSense]

@soribido You are very welcome. Please do feel free to open a new issue later if you have additional questions.