upload color and depth images and then align them

[ghost]

Required Info
Camera Model	SR300
Operating System & Version	Win 10
Platform	PC
SDK Version	2.10.1

I've been struggling with the next problem for a few days now:
I have a color image and a corresponding depth image that are not aligned, i took them with RS300 (I've attached an example)
I am wondering if there is a way to upload those images and map the depth image to the color one using librealsense with python.
I know align.process get live stream frames, so i wonder if i can use it with uploaded frames

I would be happy to solve this problem because I have a database of such images that would be a problem for me to take again

Thanks in advance

[zivsha]

Hi @adarvit ,
So just to see if I understand right - you took some photos using SR300 and saved them to a file, but they are not aligned and you want to align them?
How were these frames saved to file? are they raw binary data or were they encoded somehow?
BTW - your attached pictures are broken

[ghost]

Thank for the reply
Exactly, you understand it right

[zivsha]

You must have the calibration data of the camera that was used to take these images in order to successfully align the images - i.e the intrinsics and extrinsics data.

If you have it, please continue to read.
If not- I'm not sure I have an idea how this can be done, possibly opencv has a solution for that.

I'll just write down what can be done and let me know what you are missing or don't understand...

There are a few ways to make what you want happen:

1. Create a rs2::software_device (not sure if we have this in python already), add the correct calibration data to it. Then, use it to load images from your data base and use align.process to align each frameset.
   This will provide you with a robust way of using align.process and also allow you to use these frames as if they are live streams, and also use any other functionality of the SDK with this devices.
2. Use the "free functions" of projection: rs2_deproject_pixel_to_point, rs2_transform_point_to_point, rs2_project_point_to_pixel which are available on python. Using this way you should simply iterate each pixel in the depth frame and transform it to get the matching color pixel. This is basically what align.process is doing but you will have to implement it yourself.
3. Use .bag files to create a playback file which can be loaded using playback_device - this is a bit complex since we don't provide any mechanism or example on how to do it but, using rosbag python API, you can write a .bag file and fill it with frames and their appropriate topics and with a few additional messages you can have a complete file that can be playback using the SDK.

[zivsha]

Is there a reason you are saving frames to file and not using the recorder of the SDK to record a sequence to file?

[ghost]

Thank you for your effort,
I saved a lot of frames without noticing that the frames are not aligned and i can't retake those frames again, so i try everything i can to exploit those images.

I have the calibration, i used this code to get it:
depth_intrin = depth_frame.profile.as_video_stream_profile().intrinsics
color_intrin = color_frame.profile.as_video_stream_profile().intrinsics
depth_to_color_extrin = depth_frame.profile.get_extrinsics_to(color_frame.profile)

I had tried the second solution you suggested, using this code:
depth_sensor = pipe_profile.get_device().first_depth_sensor() depth_scale = depth_sensor.get_depth_scale()
depth_pixel = [330, 330]
depth_point = rs.rs2_deproject_pixel_to_point(depth_intrin, depth_pixel, depth_scale)
color_point = rs.rs2_transform_point_to_point(depth_to_color_extrin, depth_point)
color_pixel = rs.rs2_project_point_to_pixel(color_intrin, color_point)

but all of the pixels were out of bound. for example:
The depth and color frames are 480X640
I tried depth_pixel [330,330] and the color pixel that returned was [4341.313,130.892]

[zivsha]

Can you try the following to get the depth scale: (instead of depth_scale = depth_sensor.get_depth_scale()):
depth_scale = depth_sensor.get_option(rs.option.depth_units)
I suspect this might an issue.

Also - you did not answer my question- was the depth and color images saved as raw binary or using some encoder? if it is the latter - the depth data might be lost or modified

[ghost]

both get_option(rs.option.depth_units) and get_depth_scale() return the same value so it didn't solve the problem.
The depth and color images were not saved as raw binary format

[zivsha]

This is more likely the reason then. The depth data probably contains modified values. which format did you use to save the files?

[ghost]

tiff

[zivsha]

did you save the frames as uint16? did you use any compression level?

[zivsha]

I'm not very familiar with tiff format, but please try the following:

1. Start the pipeline (pipe.start())
2. Take a depth frame (depth_frame = pipe.wait_for_frames().get_depth_frame())
3. Convert it to numpy array (depth_arr = np.asanyarray(depth_frame.get_data()))
4. [Optional] Save the array to csv file (numpy.savetxt("depth.csv", depth_arr , delimiter=","))
5. Save the depth_frame to tiff in the same way you did before
6. Compare the content of each cell (in the csv) to each pixel (in the tiff image)

Let me know if the data is the same, so we can deduce depth data is valid inside the tiff image

[ghost]

I dont use any compression level, but the depth frames were saved as 8 bit. the data in the csv is in 16 bit pixels and the tiff images were saved as 8 bit so the data is not the same.
I also notice that if i use config.enable_stream for depth there is no way to set the format with 8 bit.

[zivsha]

This is an issue, you only saved half the data thus causing the depth values to change, from the image it looks to me like you took the lower bits, so each depth pixel could map to a radius of color pixels. Do you have a way to share how exactly the data was saved/altered so maybe we can come up with some way to help?
Regarding config.enable, you are right the depth format is Z16, which is 16 bits per pixel, and there is no 8 bit format for depth.

[zivsha]

@adarvit , can you upload such 2 files here so we can see what we get?
One depth, and one corresponding color image (you can zip them and drag the zip in the comment to upload)

[ghost]

color&depth.zip

[kargarisaac]

Hi,
I used the following code to get point cloud but what I get is not smooth. I save depth frames as numpy array. every position in the matrix has a value and I use it as depth and distance to camera. but the result is a lot noisy and not smooth. what do you think about that:

frames = pipeline.wait_for_frames()
depth_frame = frames.get_depth_frame()
depth_image = np.asanyarray(depth_frame.get_data())
depth_image = cv2.resize(depth_image, (640, 480))
frames_np.append(depth_image)
np.save('frames.npy', np.array(frames_np))

It's part of cattle's body. It should be smoother:

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
@kargarisaac
How do you view the 3D point cloud with format of npy? Also why don't you just export to ply?

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Any update?

[alt01]

You must have the calibration data of the camera that was used to take these images in order to successfully align the images - i.e the intrinsics and extrinsics data.

If you have it, please continue to read.
If not- I'm not sure I have an idea how this can be done, possibly opencv has a solution for that.

I'll just write down what can be done and let me know what you are missing or don't understand...

There are a few ways to make what you want happen:

1. Create a rs2::software_device (not sure if we have this in python already), add the correct calibration data to it. Then, use it to load images from your data base and use align.process to align each frameset.
   This will provide you with a robust way of using align.process and also allow you to use these frames as if they are live streams, and also use any other functionality of the SDK with this devices.
2. Use the "free functions" of projection: rs2_deproject_pixel_to_point, rs2_transform_point_to_point, rs2_project_point_to_pixel which are available on python. Using this way you should simply iterate each pixel in the depth frame and transform it to get the matching color pixel. This is basically what align.process is doing but you will have to implement it yourself.
3. Use .bag files to create a playback file which can be loaded using playback_device - this is a bit complex since we don't provide any mechanism or example on how to do it but, using rosbag python API, you can write a .bag file and fill it with frames and their appropriate topics and with a few additional messages you can have a complete file that can be playback using the SDK.

I followed this steps you suggested @zivsha , but as @adarvit at the end I get an image bigger than expected.
I implemented this for each depth pixel:

depth_pixel = [330, 330]
depth_point = rs.rs2_deproject_pixel_to_point(depth_intrin, depth_pixel, depth_scale)
color_point = rs.rs2_transform_point_to_point(depth_to_color_extrin, depth_point)
color_pixel = rs.rs2_project_point_to_pixel(color_intrin, color_point)

but when at the end I expect an image of 848x480 I get one of 1039x581.
The intrinsic and extrinsic values Im using are:

Depth intrinsics: width: 424, height: 240, ppx: 214.063, ppy: 120.214, fx: 213.177, fy: 213.177, model: Brown Conrady, coeffs: [0, 0, 0, 0, 0]

Color intrinsics: width: 848, height: 480, ppx: 428.062, ppy: 234.166, fx: 614.532, fy: 614.781, model: Brown Conrady, coeffs: [0, 0, 0, 0, 0]

depth to color extrinsics: rotation: [0.999966, 0.00585115, -0.00579111, -0.00586071, 0.999982, -0.00163449, 0.00578144, 0.00166838, 0.999982]
translation: [0.0146822, 0.0002266, -3.92719e-05]

color to depth extrinsics: rotation: [0.999966, -0.00586071, 0.00578144, 0.00585115, 0.999982, 0.00166838, -0.00579111, -0.00163449, 0.999982]
translation: [-0.0146833, -0.000140612, -4.59912e-05]

 depth_scale: 0.0010000000475

---

Hope you can help me.

@RealSense-Customer-Engineering