project pixel from RGB to IR or depth issue

[haoshuwen3]

Required Info
Camera Model	D415
Firmware Version
Operating System & Version
Kernel Version (Linux Only)
SDK Version

Project the pixel from color to depth with a tolerance of about 10 pixels， How to call the right API for this case？

static void MapColor2Depth(const Point &color, Point &depth, rs2_intrinsics *video_inst, rs2_intrinsics *depth_inst, rs2_extrinsics *extrincs) {
    float pixes[2];
    pixes[0] = color.x;
    pixes[1] = color.y;

    float colorPoint[3];
    rs2_deproject_pixel_to_point(colorPoint, video_inst, pixes, 1);
    float depthPoint[3];
    rs2_transform_point_to_point(depthPoint, extrincs, colorPoint);
    rs2_project_point_to_pixel(pixes, depth_inst, depthPoint);

    depth.x = pixes[0];
    depth.y = pixes[1];
}

rs2::frame color_frame = frames.get_color_frame();
rs2::frame ir_frame = frames.first(RS2_STREAM_INFRARED);
rs2::depth_frame depth_frame = frames.get_depth_frame();

//If one of them is unavailable, continue iteration
if (!color_frame || !ir_frame || !depth_frame)
{
    continue;
}


s_color_inst = color_frame.get_profile().as<rs2::video_stream_profile>().get_intrinsics();
s_depth_inst = depth_frame.get_profile().as<rs2::video_stream_profile>().get_intrinsics();
s_ir_inst = ir_frame.get_profile().as<rs2::video_stream_profile>().get_intrinsics();
s_color2depth_extrincs = color_frame.get_profile().get_extrinsics_to(depth_frame.get_profile());
s_color2ir_extrincs = color_frame.get_profile().get_extrinsics_to(ir_frame.get_profile());

[zivsha]

Hi @haoshuwen3 , can you please elaborate what exactly are you trying to achieve?
You ask about color to depth alignment but your code uses infrared, color and depth frames..
We provide a rs2::align object that can do this for you, and you can take a look at its code as reference for your starting point

[haoshuwen3]

Hi @zivsha , we had some points like eyes that were detected on RGB Image. Then we try to locate the corresponding points on IR image. I took a look at the rs2::align code, and find the useful rsutil tool. But the projected points on IR is far from the real eye position.
Thanks

[zivsha]

Can you please post a snippet that reproduces this?

[haoshuwen3]

Hi @zivsha ,
static void MapColor2Depth(const Point &color, Point &depth, rs2_intrinsics *video_inst, rs2_intrinsics *depth_inst, rs2_extrinsics *extrincs) {
float pixes[2];
pixes[0] = color.x;
pixes[1] = color.y;

float colorPoint[3];
rs2_deproject_pixel_to_point(colorPoint, video_inst, pixes, 1);
float depthPoint[3];
rs2_transform_point_to_point(depthPoint, extrincs, colorPoint);
rs2_project_point_to_pixel(pixes, depth_inst, depthPoint);

depth.x = pixes[0];
depth.y = pixes[1];

}
I call this function to convert points from RGB to depth, but with about 10 pixels offset regularly.

[zivsha]

Hi @haoshuwen3 ,

We've just pushed Pull Request #1068 which addresses the alignment of color to depth.
You can always deproject depth pixels (i,j), transform and project the result to the color image and copy that color pixel to the pixel (i,j) of the output image. See the changes in the PR to see how we do it.

[travisWangXiaowei]

Hi @zivsha ,
what should i do If I want to get depth of some pixels in RGB frame? Since we cannot project pixel in color frame to depth frame.

[zivsha]

The way to map a color pixel to depth pixel is by doing the same process as mapping depth to color, but returning the color pixels instead. The PR I mentioned uses this technique to calculate the entire frame,rather than a single pixel, as you have to through all depth pixels to find the one(s) matching the color pixel. If you use the rs2::align object, you can get the frames mapped to one another and then get the specific pixel.

[travisWangXiaowei]

Hi @zivsha,
I aligned Color frame to Depth frame, it seems that there is still offset between two frames. Since depth frame and color frame are not spatially aligned, I should deproject all the pixels in depth frame , transform them and project them on color frame. Is that correct?

[zivsha]

After using rs2::align with depth and color (either align depth to color or color to depth) the 2 imgaes you processed should spatially align, you don't have to de/project anymore, since this was performed on each pixel during the align process. If you see any offset then that might be a bug and please share how you identify this offset. If you run rs-align example and make it align to depth do you see this issue?

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@haoshuwen3 **Would like to know if you still need help with this issue? **

[mjfoo21]

Hello,

I know this has been closed but I can't use the rs2::align as suggested.

My code is as followed:

rs2_stream align_to = RS2_STREAM_DEPTH;
rs2::align align(align_to);
rs2::frameset frames = pipe.wait_for_frames();
rs2::frameset processedFrames = align.process(frames);
rsFrame = frames.get_color_frame(); //this is the aligned depth frame

It can't get past the align.process(frames); line with an exception occurring at the invoke() under class processing_block:

Exception thrown at 0x00007FFF61AA1F42 (realsense2.dll) in abc.exe: 0xC0000005: Access violation writing location 0x000001E9D6312128.

Actually I posted the same question a year ago and it was fixed. Somehow it is not working now when I set up a new program and downloaded the repository few days ago.

I have looked at this but it is for Matlab and also this which suggesting calling the projecting functions myself.

I am using Windows 10 if it matters.