Is poll_for_frames with thread suitable for non-realtime application?

[chunhai]

---

Required Info
Camera Model	{ D435 }
Firmware Version	(5.10.3)
Operating System & Version	{Ubuntu 16 (Mate)
Kernel Version (Linux Only)	(4.14.69)
Platform	odroid
SDK Version	{ 2.16.0}
Language	{C++ }
Segment	{others }

Issue Description

Hey,

I am trying to develop a non-realtime application, where it may take long time to process each frame. I used wait_for_frames without thread, but it causes frames dropping during processing. Thus I am thinking to use a background thread to store the stream frames and use another thread to process each frame one by one, something like the following:

rs2::pipeline pipe;
pipe.start();

rs2::frame_queue queue();

std::thread frames_store_thread([&]() {
    auto frames = pipe.wait_for_frames();
    queue.enqueue(frames.get_depth_frame());
});

std::thread frames_processing_thread([&]() {
    while (true)
    {
        rs2::depth_frame frame;
        if (queue.poll_for_frame(&frame))
        {
	        frame.get_data();
	        // Do processing on the frame
        }
    }
});

while (true)
{
// main thread
}

My understanding is that poll_for_frames will wait the user to fetch next frame, unlike wait_for_frame. Is it correct to use poll_for_frames in this way?

Thank you in advance for the help.

[dorodnic]

Hi @chunhai
This approach can help if processing occassionaly takes long time. However, if processing consistently takes longer than what the frame-rate allows, you will either have to accept frame drops or switch to purely off-line processing, using the ros-bag recorder. Otherwise, the application will just run out of memory.
Note: out of the box, librealsense allows you to keep up to 16 frames of each type, for performance reasons. If you wish to keep more in the queue, you need to initialize the queue with larger capacity and call frame.keep() for every frame you enqueue.

[chunhai]

Hi @dorodnic

Thanks for quick replying!
According to your suggestion, I tried to use ros-bag with poll_for_frames like following.

rs2::config cfg;
cfg.enable_device_from_file("test.bag");
rs2::pipeline pipe;
pipe.start(cfg); //File will be opened in read mode at this point
while(1)
{
    rs2::frameset frames;
    if (!pipe.poll_for_frames(&frames))
    {
        continue;
    }
    // process frames here
}
pipe.stop(); //File will be closed at this point

However, it seems there are still some frame loss. How can I make pipe wait until the completion of processing current frame? Am I using poll_for_frames in a correct way?

[dorodnic]

It depends if the drops occurred during recording or just during playback.
To make sure you get all frames in the bag file, we have a special method called set_real_time, please see rs-convert.cpp:94-96 for code sample.
To avoid drops during recording, it is best to record on-to an SSD drive and use the pre-compiled version of the SDK (or build with -DCMAKE_BUILD_TYPE=Release.

[chunhai]

Hi @dorodnic ,
Thank you for the example.
The drops occurred during playback. I tried set_real_time as falase, but it seems that it still drop frames if the processing is long. I made an experiment by sleep 0.5s after read each frame (to simulate the processing time) as following:

rs2::config cfg;
cfg.enable_stream(RS2_STREAM_COLOR);
cfg.enable_stream(RS2_STREAM_DEPTH);
cfg.enable_device_from_file("test.bag");
rs2::pipeline pipe;
pipe.start(cfg);

rs2::device device = pipe.get_active_profile().get_device();
rs2::playback play_back = device.as<rs2::playback>();
play_back.set_real_time(false);

frameNum = 0
while (true)
{
    rs2:frameset frames = pipe.wait_for_frames();
    auto color_frame = frames.get_color_frame();
    cout << "frame number" << color_frame.get_frame_number() << endl;
    if (frameNum > color_frame.get_frame_number())
        break;
   usleep(500000);
}

If there is no sleep, the output frame number is continuous. If it sleeps a while, it will drop frames.

thanks

[chunhai]

After I upgraded to 2.16.1, set_real_time works now! Thank you very much!

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hi chunhai,

Glad to see that set_real_time off 2.16.1 works well for you.
I am going to close this one for now. Please feel free to report back any issues or ask questions.

Thank you!