Streaming RealSense frame data to allow processing on a remote computer

[marcingold22]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D400
Firmware Version	Currently I don't know, but it's the latest
Operating System & Version	Linux (Ubuntu 14/16/17) - Raspberry Pi OS
Kernel Version (Linux Only)	5.15.61
Platform	Raspberry Pi
SDK Version	2.51.1
Language	C++
Segment	AR

Issue Description

Hello! I'm playing a bit with the RealSense D400 camera on a Raspberry Pi with Raspberry Pi OS. I need to extract the depth data corresponding to various points of the RGB image.

The image below shows a demo of what i need to get:

I've tried getting both the frames but by looking at the examples I've noticed that the two frames are not aligned by default and that a call to the process(rs2::frame frame) function is needed to get an aligned frameset.
Unfortunately my Raspberry Pi is not powerful enough to compute the alignment for each frame in a reasonable amount of time and despite all the performance boost I could attempt with my limited knowledge, I could not get it.

Now I'm trying to send the frameset to my main computer which can compute it in few milliseconds. I've tried with classical TCP but I was unable to send data which the process function could compute, getting mainly segmentation faults.
After many attempts I decided to ask for help since I don't know how I could do it and found nothing on the various resources (community, here on GitHub etc...)

My question is: is there a way to process frame data real-time on a remote machine with the SDK? Thank you for your attention.

[MartyG-RealSense]

Hi @marcingold22 As you are using Raspberry Pi, the ethernet networking system described in the Intel white-paper document at the link below for transmitting data in real-time from a remote camera on a Pi to a host computer may be worth exploring if you have not done so already.

https://dev.intelrealsense.com/docs/open-source-ethernet-networking-for-intel-realsense-depth-cameras

#8423 provides a good demonstration of accessing a camera remotely as a network camera using C++ scripting.

Please note though that this open-source ethernet networking system described in the above link is being removed from the next RealSense SDK version as described at #11318 as it was an experimental proof of concept rather than an official Intel-validated feature. So you will be limited to a maximum SDK version of 2.53.1 (the current one at the time of writing this) if you use this networking tool.

[marcingold22]

Thank you very much @MartyG-RealSense ! I did not know there was such a feature, I'm still learning about the SDK. I'm editing my code to work with this. Unfortunately after re-building the SDK I get some errors in the realsense-viewer app like "Device unavailable error -9" I'm trying to solve this to see if it works.

[MartyG-RealSense]

Hi @marcingold22 If you are using firmware 5.14.0.0 (the latest at the time of writing this) and kernel 5.15 then I recommend using SDK version 2.53.1 instead of 2.51.1.

Firmware 5.14.0.0 is the recommended firmware for use with 2.53.1, and official support for kernel 5.15 was only introduced in 2.53.1.

[marcingold22]

That was it thank you again @MartyG-RealSense for your help! I modified the code and started rs-server on the Raspberry Pi. Unfortunately the main program on my PC gives me this error:

RealSense error calling rs2_pipeline_wait_for_frames(pipe:0xaaaad77f68e0):
Frame didn't arrive within 15000
terminate called after throwing an instance of 'cv::Exception'

Is it something related to firmware/SDK or did I mess up with something?

This is a snipped of my code which is almost a carbon copy of the ones you find on the Intel website and on the community:

rs2::net_device dev("192.168.xx.xx");
    rs2::context context; 
    dev.add_to(context);
    rs2::pipeline pipe(context);
    rs2::config config;
    config.enable_stream(rs2_stream::RS2_STREAM_DEPTH, resX, resY, rs2_format::RS2_FORMAT_Z16, 30);
    config.enable_stream(rs2_stream::RS2_STREAM_COLOR, resX, resY, rs2_format::RS2_FORMAT_RGB8, 30);
    rs2::pipeline_profile pipprof = pipe.start(config);
    rs2::align align_to(RS2_STREAM_COLOR);
    rs2::depth_sensor sensor = pipprof.get_device().first<rs2::depth_sensor>();
    * scale = sensor.get_depth_scale(); // THIS IS A GLOBAL VARIABLE

    // autoexp.
    if (first_boot) 
         for (int i = 0; i < 30; i++) 
                    pipe.wait_for_frames(); // THE PROGRAM CRASHES HERE WITH AN EXCEPTION

[MartyG-RealSense]

The resolution / FPS combinations supported for a networked camera by rs-server is more limited than with a non-networked camera. At 1280x720 resolution, only 6 FPS instead of 30 is supported for depth and color according to the Software Limitations section of the networking guide.

https://dev.intelrealsense.com/docs/open-source-ethernet-networking-for-intel-realsense-depth-cameras#36-software-limitations

[marcingold22]

Mh... But my resX and resY variables are set such as my frames are 640x480 in resolution 🤔

[MartyG-RealSense]

Can you confirm which RealSense camera model you are using please (D435, D455, etc)

[marcingold22]

Oh, so if I understood I'm limited to 6 FPS? Because I see also 15 and 30 FPS (excuse my ignorance)

My camera model is the D345.

[MartyG-RealSense]

1280x720 resolution is limited to 6 FPS but you can use 15 or 30 if you configure your config.enable_stream instructions to use 640x480 resolution.

[marcingold22]

Oh, but I did. My global variables are:

int resX = 640, resY = 480

That's why I don't know why it gives me this error, I'm re-building the SDK again to see if something changes.

[marcingold22]

Just saying for people who will take a look at this in the future: putting the numbers directly as arguments and removing the framerate (30 in my case) does not solve the problem (unfortunately)

[MartyG-RealSense]

Thank you for the clarification about resX and resY. I see now how you are using them in your config lines to set the resolution. You are correct, setting int resX = 640, resY = 480 should have permitted 30 FPS to be used.

Which method are you using to build the SDK on your Pi, please? Did you build from source code with CMake and include -DBUILD_NETWORK_DEVICE=ON in the CMake build instruction to install the SDK with networking support included?

-DBUILD_NETWORK_DEVICE=ON must be used on the remote Pi to install rs-server on it. rs-server should not be built on the main computer that has realsense-viewer on it.

[marcingold22]

First of all thank you again for taking time answering to me ❤️

I have built the SDK from source on the Raspberry Pi with CMake using the following command:

cmake -DBUILD_NETWORK_DEVICE=ON -DFORCE_RSUSB_BACKEND=ON BUILD_WITH_OPENMP=true ..

I used -DFORCE_RSUSB_BACKEND=ON as recommended on the network page and used BUILD_WITH_OPENMP=true because I might do some experiments in the near future on the Raspberry Pi using C++ so I'd like to see if I get some performance improvements (I'm re-building it right now using also CMAKE_BUILD_TYPE=Release for the same reason).

I think I have built it on the main PC using -DBUILD_NETWORK_DEVICE=ON because I needed the rs_net library which was available only when building with this flag, but I did not build the tools (realsense-viewer included) so I don't have rs-server on it.

[MartyG-RealSense]

When using a Pi as the remote computer and using a main computer with realsense-viewer on it to remotely access the Pi via ssh, -DBUILD_NETWORK_DEVICE=ON should be used on the Pi to build rs-server only on the Pi. rs-server is then accessed from the main PC with the ssh connection.

Once rs-server is running on the Pi, you should be able to access the camera that is attached to the Pi from realsense-viewer on the main machine by adding it as a network camera in realsense-viewer's Add Source option.

Edit: I recall that you are using your own script to access the camera as a network device instead of using realsense-viewer. So your script is launched from the main computer?

[marcingold22]

So your script is launched from the main computer?

Yes, exactly! The camera gets recognised, but no frame arrives, even if I increase the timeout

[MartyG-RealSense]

What happens if you comment out this line:

if (first_boot)

and then make the following change:

   rs2::frameset frames;
    for(int i = 0; i < 30; i++)
    {
        // Wait for all configured streams to produce a frame
        frames = pipe.wait_for_frames();
    }

[marcingold22]

Unfortunately the same error happens 😞

[marcingold22]

I'm building the SDK again on the main PC, with realsense-viewer enabled this time

[marcingold22]

Forgot to mention... Might the problem be caused by Docker? I'm on a Docker container on my main PC, do I need to enable any port? On the whitepaper I see no mention of this but I just remembered about this 🤔

[MartyG-RealSense]

As far as I am aware, there have been no previous reports of RealSense users attempting to access rs-server from within a Docker container on the main PC. My knowledge of Docker is limited so I cannot advise on this, unfortunately.

[marcingold22]

Oh, unfortunately I'm forced because I have a Mac with M1 and can't compile with rs_net on it 😞
Thank you anyway for your kind help and useful tips!

[MartyG-RealSense]

This networking system should be usable on a Mac without using Docker, as indicated in the documentation, as the networking components are not installed on the Mac (they only need to be installed on the Pi's SDK build by using -DBUILD_NETWORK_DEVICE=ON). Though the guide was written before the M1 models were introduced.

[marcingold22]

Unfortunately I was unable to install the rs_net.hpp library (even by copy/pasting it in the lib directory it was not recognised) so I used this appoach...
I think you need -DBUILD_NETWORK_DEVICE=ON to get the rs_net library it but I could try again, unfortunately I have to use XCode because the CMake doesn't work on M1 😞 ...

[MartyG-RealSense]

As far as I am aware you should only need to use -DBUILD_NETWORK_DEVICE=ON CMake flag when installing the SDK from source code on the Raspberry Pi and do not need to build the SDK with networking support on MacOS. Instead, an ssh network connection is opened from the Mac to the network components installed on the Pi.

[marcingold22]

I can't show it right now but I tried using the librealsense formulae to install the SDK, but the realsense-viewer kept showing the error "No network, please build with network device" when I tried using the realsense-viewer to connect to a remote camera.

The real problem is that I need the realsense2-net library to compile my C++ code on the main PC and in no way I managed to install it on macOS M1. I don't know if there's a way to get it without using the -DBUILD_NETWORK_DEVICE=ON

I found this on the whitepaper:

[MartyG-RealSense]

Maybe we should step back to a simpler approach for the moment and see whether the camera on the Pi can be accessed from the Mac using realsense-viewerin order to establish whether the network connection can work at all. If the SDK has been installed on the Mac with brew then instructions at #2811 (comment) describe how to launch realsense-viewer.

[marcingold22]

Luckly the realsense-viewer works on the Mac and the main SDK is installed (with brew).
I've already tried following the instructions on the whitepaper to see if the camera connects, unfortunately it asks for the SDK to be built with the network features, which I'm unable to do with brew, certainly due to my incompetence with the macOS system (unfortunately I have only this at the moment since that's the computer I must work on), but I can try on monday, when I'm back working on this project.

Anyway I'd really take a moment to thank you for helping me this whole week, if it weren't for you I was almost desperate, since I had tried my best to make it work on the Raspberry with little to no success, at least now I'm touching the sky with a finger, if it works on the Mac I have finally finished 😅

By the way until I'm able to use the Mac again I'd like to wish you a happy week-end 🙂

[MartyG-RealSense]

You are very welcome. I'm pleased that I have been able to be of help and look forward to continuing this case with you. Good luck!

[marcingold22]

Hello and good monday! This is the error which appears when I try to connect to a remote device from the realsense-viewer app on the M1:

[MartyG-RealSense]

I experienced the same message when attempting to add a camera as a Network Device fro the Viewer on a computer without a network link. I sincerely apologize. I do not have networking equipment at my location and so have not attempted this project myself, and the guide document does not make clear that network components are also required on the main computer.

A Mac installation guide for librealsense at the link below that is M1-compatible demonstrates a method for compiling librealsense from source code with CMake to create an XCode project. This may provide a way to define the network build flag whilst still building librealsense for XCode.

https://lightbuzz.com/realsense-macos/

[marcingold22]

No need to apologize, I'm already glad for your kind help. Before creating the Docker container, I had already tried doing this but had no success since I'm used to the (in my opinion) simpler and beautiful Make of Linux... After the whole process I get a dylib but still no rs-net library, which I need in order to compile the program... I'm going to try again and will let you know if I manage to get it, macOS is just a hassle to me...

[marcingold22]

Hi Marty, using a virtual machine with Ubuntu I finally got to use the realsense-viewer app. Unfortunately I found out that no frame arrives. Here's a screenshot of the situation. I triend executing the commands in the warning but the error keeps happening. I tried also executing the script in the scripts folder for udev but nothing changed.

[MartyG-RealSense]

It's great to hear that you made progress!

The brown debug log at the bottom of the realsense-viewer window indicates that there are missing or outdated udev rules. The udev rules handle devices in Ubuntu, such as RealSense cameras.

If the error message is Missing/outdated UDEV-Rules will cause 'Permissions Denied' errors then #4951 may be helpful.

[marcingold22]

Mh I installed it the manual way but this still happens 😞 I'll take a look at the guide again and see if I forgot anything

[marcingold22]

By the way by looking at it again, this appears before I even add the network camera, so maybe that's not on the Raspberry Pi 🤔

[MartyG-RealSense]

The udev message in the realsense-viewer log would be related to the main PC, yes, as realsense-viewer is launched on the main PC.

The error message suggests the following instruction to input into the Ubuntu terminal.

---

To manually install UDEV-Rules in terminal run:

sudo cp ~/.99-realsense-libusb.rules /etc/udev/rules.d/99-realsense-libusb.rules && sudo udevadm control --reload-rules && udevadm trigger

[marcingold22]

Hi @MartyG-RealSense I'm very happy to say that it was just a macOS issue, since I think it limits the network access for apps (for security reasons I suppose). I've tried running a docker container on my personal PC (with Linux) and it works!

Again thank you very much for your support, you're great!

[MartyG-RealSense]

You are very welcome, @marcingold22 - it's great news that you achieved a solution. Thanks very much for the update!

[MartyG-RealSense]

Case closed due to solution achieved and no further comments received.