How to align RGB with stereo at lower resolution

[logidelic]

I'm experimenting with resolutions and performance with my D415. I am running my RGB stream at 1920x1080 and would like to run my stereo stream at a lower resolution (ex 640x360). My understanding is that when the depth and RGB are aligned, depth is upscaled to the RGB resolution, is that right?

Is it possible to do it the other way around (for the sake of performance)? I.e. align based on the downscaled RGB frames?

Thank you!

[MartyG-RealSense]

Hi @logidelic When aligning depth to color, the depth field of view (FOV) size is aligned to the size of the color FOV. On camera models where the depth FOV is larger than the color FOV (D435 / D435i), this may result in the outer areas of the depth coordinates being cropped off. The D415 and D455 camera models have the same FOV size for depth and color.

The rs_align SDK example program provides a way to test the effects of depth to color alignment on different camera models.

The Overview documentation for rs-align provides an explanation for the effects that can occur during aligning, including upscaling and downscaling. If you are using the align processing block, then the processing block should take care of adjustments depending on what settings you are using,

https://github.com/IntelRealSense/librealsense/tree/master/examples/align#overview

In the images below that are taken from depth to color alignment tests that I performed in rs_align the upper image is on a D435i with a larger depth FOV than color FOV, and the lower image is on a D415 with the same depth and color FOV size.

D435i

D415

[logidelic]

Thank you so much for the response Marty. I've read through your answer (as well as the example you linked to). These are very informative, but unless I'm misunderstanding something (possible) I don't think it answers my question. To make it more concrete:

• RGB stream open at 1920x1080
• Depth stream open at 640x360
• (Same FOV)

Currently to align we do something like:

rs2::align align(RS2_STREAM_COLOR);

...
...

rs2::frameset frame;
if (pipeline.try_wait_for_frames(&frame, std::chrono::duration_cast<std::chrono::milliseconds>(timeout_).count())) {
    const auto aligned_frame = align.process(frame);
    auto color_frame = aligned_frame.get_color_frame();
    auto depth_frame = aligned_frame.get_depth_frame();
    ...
}

My understanding is that, to achieve this alignment, the depth frame first gets upsampled, and then gets aligned to the RGB viewport.

What I'm asking: Is it possible to, instead, downsample the RGB frame to the depth resolution, and have the depth frame get aligned to that lower-resolution frame. Again, this is simply with the hope of saving some cycles during the alignment process... (I don't want to open the RGB stream at a lower resolution because I still want access to the high-res RGB...)

Thanks again!

[MartyG-RealSense]

If you would like to align RGB to the depth, you could use color to depth alignment instead of depth to color. The subject of color to depth alignment with C++ is discussed in the link below:

#2567

rs-align also provides an option to test this, via tick-boxes for Align To Depth and for Align To Color at opposite ends of the base of the program window

[MartyG-RealSense]

Hi @logidelic Do you require further assistance with this case, please? Thanks!