wrong depth->color extrinsics

[svensht2]

I'm using using RS435 with FW 5.8.9.0. librealsense 2.7.9, Windows 10 1703.
I'm trying to get align depth to color using the calibration information from the camera, but the alignment looks off.
I know my computation for reprojection from depth and projection to color is correct ( I also took into account the fact that rotation is column major now). I suspect the extrinsic parameters that librealsesne is reporting are not correct.
I'm enabling the pipeline successfully using the following settings:

    mpPipeline->enable_stream(rs2_stream::RS2_STREAM_COLOR, 0, mColorWidth, mColorHeight, rs2_format::RS2_FORMAT_BGR8, 30);
    mpPipeline->enable_stream(rs2_stream::RS2_STREAM_DEPTH, 0, DS435CAM_ACTUAL_DEPTH_WIDTH, DS435CAM_ACTUAL_DEPTH_HEIGHT, rs2_format::RS2_FORMAT_Z16, 30);
    mpPipeline->start()

Then I to get depth->color extrinsics:

    // Get depth stream intrinsics
    rs2::video_stream_profile depth_stream_profile = rs2::video_stream_profile(mpPipeline->get_active_streams(RS2_STREAM_DEPTH));

    // Get "Color" stream intrinsics
    rs2::video_stream_profile color_stream_profile = rs2::video_stream_profile(mpPipeline->get_active_streams(RS2_STREAM_COLOR));

    mColorExtrinsics = depth_stream_profile.get_extrinsics_to(color_stream_profile);

The alignment is noticeably worse than IPDEV "Color UV Map" mode on.
When comparing the extrinsics given by librealsense to the the calibration tables in IPDEV the numbers don't really match up either.

[dorodnic]

Hi @svensht2
Since IPDev is a proprietary tool for NDA customers I can't really discuss it on GitHub. Please ping me at sergey.dorodnicov@intel.com
Regarding the calibration in librealsense, I believe there was an issue in 2.7.9 that affected resolutions with aspect ratio other then 16:9 (HD), and it was fixed in 2.8.0 (@ev-mp to verify).
Could this be the issue you'r seeing?
You can attach images to GitHub posts by dragging, I believe it would help us a lot to resolve the issue quickly.

[jlgarcia75]

Hello @svensht2,

Do you still need help with this issue?

Regards,
Jesus G.
Intel Customer Support

[svensht2]

@jlgarcia75 yes, we still see misalignment in rs_align and our application that are not seen by internal tools you provided to us.

[jlgarcia75]

@svensht2,

Can you post here your images as @dorodnic suggested?

Jesus

[dorodnic]

Some improvements to color to depth texture mapping are introduced in current release.
We are looking into additional ways to improve alignment (like adding an occlusion filter)

[svensht2]

we don't really see the issue anymore with 415s, closing

[svensht2]

@dorodnic I noticed this was merged:
#1031
you referenced the #774 as the issue, but seems like it would apply to this issue

[dorodnic]

Yes, my mistake. We are still working on improving both texture mapping and synchronization. A fix related to sync is also pending, to be released in the next firmware update.
Regarding this issue, we found the place where the two tools were reading from different tables.
In addition, we are planning to publish additional preset that would reduce margins around an object and looking into adding an occlusion filter to get rid of "shadow" mapping on to the background.

[svensht2]

@dorodnic when do you expect the next FW release with sync to be out?
We noticed windows does FW upgrade by default now...

[jlgarcia75]

Hello @svensht2,

The release date for the next FW version has not been finalized. The UWP driver for Windows 10 has been removed for functionality changes. However, you can still get the FW update tool and latest FW from [https://downloadcenter.intel.com/en/download/27408/Windows-Device-Firmware-Update-tool-for-Intel-RealSense-D400-Series-Cameras]. The FW update tool for Linux is coming soon.

Best regards,
Jesus G.
Intel Customer Support

[dorodnic]

I am closing this issue since all follow-up activities (aligning librealsense code with IPDev, updating table-reading code, performing IQ validation with PRQ units and adding basic occlusion removal filter) has been completed.
Thank you for your feedback and hope the results are satisfactory.