Misalignment Between Depth and RGB Streams on Intel RealSense D435

[tangyipeng100]

I'm encountering a misalignment between the depth and RGB streams when using the Intel RealSense D435 camera. The two streams do not align accurately in both absolute and relative positions, as shown in the attached image.

Has anyone experienced a similar issue or knows how to fix this? Any advice would be appreciated.**

[MartyG-RealSense]

Hi @tangyipeng100 The RealSense Viewer does not have alignment in its 2D mode. Depth and RGB can be mapped together if you click the '3D' option in the top corner of the Viewer window to enter 3D pointcloud mode and then enable depth first and RGB second.

Without alignment, there will be differences between the depth and RGB images because the position of the RGB sensor on the front of the camera is horizontally offset from the left infrared sensor (which is the origin of depth).

Also on the D435 model, depth has a larger field of view size than the RGB field of view size. So when depth is aligned to color, the outer edges of the aligned image will be cut off.

Furthermore, by default the Viewer has a post-processing filter called the Decimation filter enabled that 'downsamples' the depth resolution by a factor of 2. For example, a 640x480 depth image will be reduced to 320x240 resolution. To turn this filter off and display depth at the resolution that you selected, go to Stereo Module > Post Processing, expand open the list of filters and left-click on the blue icon beside Decimation Filter to turn the icon black (Off).

[tangyipeng100]

thx. I am using the Intel RealSense SDK to stream and align depth and color frames. After aligning the frames using align.process(frames), I noticed that the depth frame (depth_frame.get_data()) contains only zeros, rendering the depth stream unusable. How to solve it?

import pyrealsense2 as rs
import numpy as np
import cv2

# Configure depth and color streams
pipeline = rs.pipeline()
config = rs.config()

# Get device product line for setting a supporting resolution
pipeline_wrapper = rs.pipeline_wrapper(pipeline)
pipeline_profile = config.resolve(pipeline_wrapper)
device = pipeline_profile.get_device()
device_product_line = str(device.get_info(rs.camera_info.product_line))

found_rgb = False
for s in device.sensors:
    if s.get_info(rs.camera_info.name) == 'RGB Camera':
        found_rgb = True
        break
if not found_rgb:
    print("The demo requires Depth camera with Color sensor")
    exit(0)

config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)

if device_product_line == 'L500':
    config.enable_stream(rs.stream.color, 960, 540, rs.format.bgr8, 30)
else:
    config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

# Create alignment object
align_to = rs.stream.color
align = rs.align(align_to)

# Start streaming
pipeline.start(config)

try:
    while True:
        # Wait for a coherent pair of frames: depth and color
        frames = pipeline.wait_for_frames()
        frames = align.process(frames) # if do not comment this code, then depth_image_aligned only contains zero
        depth_frame = frames.get_depth_frame()
        color_frame = frames.get_color_frame()

        depth_image_aligned = np.asanyarray(depth_frame.get_data())
        if np.count_nonzero(depth_image_aligned) == 0:
            print("Aligned depth frame contains only zeros")

[MartyG-RealSense]

Hi @tangyipeng100 If you are getting zeroes like the case at #3631 then a suggestion for correcting this is to convert the numpy array to a list because of how numpy summarizes when printing large arrays. The link below descibes how such a conversion might be performed.

https://numpy.org/doc/stable/reference/generated/numpy.ndarray.tolist.html

[tangyipeng100]

Hi @tangyipeng100 If you are getting zeroes like the case at #3631 then a suggestion for correcting this is to convert the numpy array to a list because of how numpy summarizes when printing large arrays. The link below descibes how such a conversion might be performed.

https://numpy.org/doc/stable/reference/generated/numpy.ndarray.tolist.html

thx, I have used np.unique() to observe the value, and it equals to 0. Besides, if I comment 'frames = align.process(frames) ', then I can get the depth stream correctly.

[MartyG-RealSense]

The common way to program alignment, as demonstrated by the librealsense SDK's official Python depth-color alignment program align_depth2color.py, is to give the aligned frames their own individual variable name such as 'aligned_frames' instead of using 'frames'. That way, you are not overwriting the contents of the original 'frames' variable at frames = pipeline.wait_for_frames()

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/align-depth2color.py

frames = pipeline.wait_for_frames()

aligned_frames = align.process(frames)
aligned_depth_frame = aligned_frames.get_depth_frame()
color_frame = aligned_frames.get_color_frame()

 depth_image = np.asanyarray(aligned_depth_frame.get_data())
 color_image = np.asanyarray(color_frame.get_data())

[tangyipeng100]

thx, In fact, I am using opencv_viewer_example, and I try to follow your strategy, and it works as before, like the image shows:

If I do not use `align.process(frames), then I can get:

`

when I run align-depth2color.py, I get:

[MartyG-RealSense]

Does the alignment work if you use a modified version of the align_depth2color.py script at #11329

[tangyipeng100]

I have changed to another Windows10 PC, it still does not work.
If I set align_to = rs.stream.depth, then get:

if I set align_to = rs.stream.color, then get:

And my Depth Scale is: 0.0010000000474974513

[tangyipeng100]

I've tried running the alignment script on Ubuntu 18.04, but it's still not working. I searched for similar issues, but couldn't find anything that matches my problem. Could this be related to a hardware issue? If I manage to obtain the alignment parameters, would it be possible to manually align the depth and color images?

[MartyG-RealSense]

This is an unusual problem, though is unlikely to be a hardware issue if align_to = rs.stream.depth works. With this instruction, color to depth alignment is being performed instead of the more commonly used depth to color alignment. The aligned images created by the two methods are similar, so if color to depth alignment works then it's fine to use that method.

[tangyipeng100]

Although setting align_to = rs.stream.depth works, the color image does not display correctly. Instead, it looks like this:

The RGB image appears to contain only zeros, and I am still unable to obtain the correct RGB-depth pair alignment.How can I resolve this issue to get the correct alignment between the RGB and depth images?

[tangyipeng100]

If resolving this issue proves too challenging, could I instead use the intrinsic and extrinsic parameters to achieve the alignment?

[MartyG-RealSense]

You could try resetting your camera to its factory-new default calibration in the RealSense Viewer using the instructions at #10182 (comment) to see whether that makes a difference.

In regard to using intrinsic and extrinsic parameters, #1274 (comment) has some suggestions for methods of alignment other than using align.process.

[ozgur-kurt]

@tangyipeng100 I have faced same issue for Realsense D435i camera.
I am trying to improve this code

import pyrealsense2 as rs
import numpy as np
import cv2

# Configure depth and color streams
pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

# Start streaming
profile = pipeline.start(config)
try:
    while True:
        # Wait for a coherent pair of frames: depth and color
        frames = pipeline.wait_for_frames()
        depth_frame = frames.get_depth_frame()
        color_frame = frames.get_color_frame()

        if not depth_frame or not color_frame:
            continue

        # Convert images to numpy arrays
        depth_image = np.asanyarray(depth_frame.get_data())
        color_image = np.asanyarray(color_frame.get_data())

        # Align depth frame to color frame
        align = rs.align(rs.stream.color)
        aligned_frames = align.process(frames)
        aligned_depth_frame = aligned_frames.get_depth_frame()
        aligned_depth_image = np.asanyarray(aligned_depth_frame.get_data())

        depth_colormap = cv2.applyColorMap(cv2.convertScaleAbs(aligned_depth_image, alpha=0.03), cv2.COLORMAP_JET)
        images = np.hstack((color_image, depth_colormap))

        cv2.namedWindow('Aligned Depth and Color', cv2.WINDOW_AUTOSIZE)
        cv2.imshow('Aligned Depth and Color', images)


        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

finally:
    # Stop streaming
    pipeline.stop()

[tangyipeng100]

@ozgur-kurt
Thanks for your reply, I have tried to use your code, but I still get the same result as the fig shows:

Have you solved this problem?

[tangyipeng100]

@ozgur-kurt
I am trying to modify the code
align = rs.align(rs.stream.color)
to
align = rs.align(rs.stream.depth)
so that I can synchronize the depth and RGB streams.

However, I noticed in your code, the RGB frame isn't using the aligned frame, as shown below:

 # Convert images to numpy arrays
depth_image = np.asanyarray(depth_frame.get_data())
color_image = np.asanyarray(color_frame.get_data())

# Align depth frame to color frame
align = rs.align(rs.stream.depth)
aligned_frames = align.process(frames)
aligned_depth_frame = aligned_frames.get_depth_frame()
aligned_depth_image = np.asanyarray(aligned_depth_frame.get_data())

In this snippet, the depth frame is aligned, but the RGB frame isn't, which leads to misalignment between the depth and RGB streams, as shown in the image I’ve provided. Could you provide guidance on how to ensure both frames are properly synchronized?

[MartyG-RealSense]

Hi @tangyipeng100 Do you have an update about this case that you can provide, please? Thanks!

[tangyipeng100]

@MartyG-RealSense Thank you for your response. I've been a bit busy recently, but I plan to use the intrinsic parameters to align the depth and RGB streams next. I’ll share any updates as soon as I have them.

[MartyG-RealSense]

Thanks very much, @tangyipeng100 - I look forward to your next report. Good luck!

[tangyipeng100]

@MartyG-RealSense Hi, I am trying to manually acquire the depth-to-RGB extrinsic parameters to align the depth and RGB streams using the following code:

import pyrealsense2 as rs

pipeline = rs.pipeline()
config = rs.config()

config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

profile = pipeline.start(config)

depth_to_color_extrinsics = profile.get_stream(rs.stream.depth).as_video_stream_profile().get_extrinsics_to(
    profile.get_stream(rs.stream.color))

print("Rotation:", depth_to_color_extrinsics.rotation)
print("Translation:", depth_to_color_extrinsics.translation)

pipeline.stop()

However, I am getting the following results:

Rotation: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
Translation: [0.0, 0.0, 0.0]
Extrinsics from depth to color are all zeros.

It seems like there might be an issue with the extrinsic parameters for my device. Could this be causing the rs.align function to not align the depth and RGB streams correctly? Is there any way to resolve this issue or correctly acquire the extrinsic parameters?

Thanks!

[MartyG-RealSense]

This particular question is outside of my programming knowledge, unfortunately. I do apologize.

However, #10180 (comment) has a Python demonstration of getting depth to color extrinsics.

depth_frame = frames.get_depth_frame()
color_frame = frames.get_color_frame()

depth_to_color_extrin = depth_frame.profile.get_extrinsics_to(color_frame.profile)

[tangyipeng100]

@MartyG-RealSense
Hello,

I've been following the provided code but I continue to get extrinsics parameters as all zeros, as shown in the image below:

I'm wondering if it's possible to obtain the correct extrinsics parameters using the official software "Intel RealSense Dynamic Calibrator"? Unfortunately, when I tried using the Dynamic Calibrator, I encountered the following error:

Any guidance on how to resolve this issue or alternative methods to correctly retrieve extrinsics parameters would be greatly appreciated. Thanks in advance!

[MartyG-RealSense]

The error indicates that the Dynamic Calibration tool is having problems with the 'calibration table' inside the camera which stores the camera's calibration parameters.

When the Dynamic Calibrator tool is installed, it also installs a text-based program called CustomRW that can be used to perform a 'gold reset' of the camera to create a new calibration table inside the camera to replace one that is possibly corrupted.

On Windows you can activate the gold reset to reset the camera to its factory-new default calibration by inputting the text command below into the Windows Command Prompt text interface to launch CustomRW in gold reset mode.

Intel.Realsense.CustomRW.exe -g

[tangyipeng100]

I attempted to run the following commands, but encountered an issue:

C:\Users\TYP>Intel.Realsense.CustomRW.exe -g
CustomRW for Intel RealSense D400, Version: 2.13.1.0

  Device PID: 0B07
  Device name: Intel RealSense D435
  Serial number: 216122070043
  Firmware version: 05.15.01.00
  Type: 3.2
  Physical port: \\?\usb#vid_8086&pid_0b07&mi_00#7&33298fb&0&0000#{e5323777-f976-4f5b-9b55-b94699c46e44}\global

Calibration on device successfully reset to default gold factory settings.

C:\Users\TYP>Intel.Realsense.CustomRW.exe -r
CustomRW for Intel RealSense D400, Version: 2.13.1.0

  Device PID: 0B07
  Device name: Intel RealSense D435
  Serial number: 216122070043
  Firmware version: 05.15.01.00
  Type: 3.2
  Physical port: \\?\usb#vid_8086&pid_0b07&mi_00#7&33298fb&0&0000#{e5323777-f976-4f5b-9b55-b94699c46e44}\global

Error (9901): calibration tables on device are not supported. Please try again with the latest Dynamic Calibrator software. 
If you are already using the latest software, please re-calibrate the device with the latest OEM Calibration tools and try again.
Check CustomRW tool.

I also tried installing a previous firmware version, but the error persists. It seems like I'm encountering a similar issue to this one.

Is this a known issue with the D435? If so, are there any recommended solutions or workarounds for this error?

[MartyG-RealSense]

As you observed, this issue about calibration tables not being supported has occurred a small number of times in the past. There is not a known fix for it though, unfortunately.

The error messages suggests "If you are already using the latest software, please re-calibrate the device with the latest OEM Calibration tools and try again". That OEM version of the Dynamic Calibration software is only available though to purchasers of Intel's $1500 USD calibration target board product, so this may not be an option for you.

https://store.intelrealsense.com/buy-intel-realsense-d400-cameras-calibration-target.html

An alternative that may be worth exploring is to try installing the Dynamic Calibration software on Ubuntu using the instructions on page 14 onwards of the tool's user guide. Those instructions install the earlier 2.11.0.0 version of the tool.

https://www.intel.com/content/www/us/en/support/articles/000026723/emerging-technologies/intel-realsense-technology.html

[tangyipeng100]

@MartyG-RealSense I am now trying to calibrate the depth and RGB manually and have obtained initial results. Recently, I have been quite busy, but I will update you with my findings after this period. Thanks!

[MartyG-RealSense]

You are very welcome. Thanks very much for the update!

[MartyG-RealSense]

Hi @tangyipeng100 Do you have an update about this case that you can provide, please? Thanks!

[MartyG-RealSense]

HI @tangyipeng100 Do you require further assistance with this case, please? Thanks!

[tangyipeng100]

@MartyG-RealSense

I apologize for my delayed response. I've been busy with my paper and interview preparations recently. I have tried manually calibrating the RGB and depth streams but haven’t yet evaluated additional cases to confirm if my manual extrinsic parameters are accurate. Once I’ve completed my current tasks, I’ll continue with this project and update you on the results. Thank you for your patience.

[MartyG-RealSense]

You are very welcome. It's no problem at all. I look forward to your next report. Good luck!

[MartyG-RealSense]

Case closed due to no further comments received.