Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python Queries for D435I #7418

Closed
anandhakrishnantecholution opened this issue Sep 25, 2020 · 22 comments
Closed

Python Queries for D435I #7418

anandhakrishnantecholution opened this issue Sep 25, 2020 · 22 comments

Comments

@anandhakrishnantecholution
  • Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)


Required Info
Camera Model Intel RealSense D435I
Firmware Version 05.11.15.00
Operating System & Version Ubuntu 18.04
Kernel Version (Linux Only) 5.4.0-47-generic
Platform PC
SDK Version 2.38.1
Language python

Issue Description

  1. How do you save the depth frame as a z16 binary file instead of a colorized png file ?
  2. How do you store the intrinsic and extrinsic parameters so that I can map the color and depth frame later when required ?
  3. How to know whether the IR Emitter is turned on or off for a frame ( this is when the IR Emitter On Off is selected which alternates it on and off ) ?
@MartyG-RealSense
Copy link
Collaborator

  1. Do you have a preference please for a binary format to save to?

For example, the link below contains a Python example for exporting to a ply file:

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/export_ply_example.py

  1. My recollection from past cases involving this subject is that storage is more problematic to achieve than retrieval. The links below represent the best references that I know of about this subject:

#4061

https://community.intel.com/t5/Items-with-no-label/How-to-save-load-intrinsics-to-from-file-python/m-p/435340/highlight/true#M2980

The situation regarding storage may have changed since those past cases. @dorodnic do you have any advice about this, please?

  1. It sounds as though you are using the IR emitter per-frame alternating mode (warning: flashing image in link below):

#3066

If that is correct and the alternating is governed by the metadata variable RS2_FRAME_METADATA_FRAME_LASER_POWER_MODE then you may be able to query the frame metadata details for this setting with Python code using frame_metadata_value. For example, something like the line below for storing the result in a variable called emitter:

emitter= rs.frame.get_frame_metadata(frame, rs.frame_metadata_value.frame_laser_power_mode)

@anandhakrishnantecholution
Copy link
Author

anandhakrishnantecholution commented Sep 25, 2020

Hey @MartyG-RealSense , first of all, thanks for your quick response,

  1. I do not have any preference, but would like to save the depth values without any loss in pixel information.

  2. The link you suggested is working for me. Thanks!

  3. I have run the same command
    emitter = rs.depth_frame.get_frame_metadata(depth_frame, rs.frame_metadata_value.frame_laser_power_mode)
    But I get the below error

RuntimeError: Frame does not support this type of metadata
I can fetch the metadata for things like frame counter, frame size, time of arrival, and actual fps but not for laser power mode.

@MartyG-RealSense
Copy link
Collaborator

  1. If you save the depth values as a ply file then you should be able to query the file for XYZ coordinates. For example:

#3835 (comment)

Saving as a PNG image would result in a loss of depth information.

  1. Great - glad that it helped you!

  2. I have not seen the frame_laser_power_mode metadata queried for its current status before, so applied the metadata instruction to the known format for querying metadata in Python in the hope that it would return useful information in the same way that it would for querying metadata variables such as frame count.

@dorodnic Is the emitter on-off status exposed in metadata for access in Python please?

@anandhakrishnantecholution
Copy link
Author

@MartyG-RealSense , I found that similar technique was used here and it looks like it was working, I wonder what is the difference in my case.

@MartyG-RealSense
Copy link
Collaborator

Could you post your script here please so that it can be compared with the one at #6202

@anandhakrishnantecholution
Copy link
Author

anandhakrishnantecholution commented Sep 28, 2020

@MartyG-RealSense

Below is the code for Python

import pyrealsense2 as rs
import numpy as np
import pandas as pd
import cv2

pipeline=rs.pipeline()
config=rs.config()
rs.config.enable_device_from_file(config,'test.bag',False)

config.enable_stream(rs.stream.color, 1280, 720, rs.format.rgb8, 30)
config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)
config.enable_stream(rs.stream.infrared, 1, 1280, 720, rs.format.y8, 30)

profile=pipeline.start(config)
playback = profile.get_device().as_playback()
playback.set_real_time(False)

align_to = rs.stream.color
align = rs.align(align_to)
colorizer = rs.colorizer()
profile = pipeline.get_active_profile()
count = 0

while True:

    frames = pipeline.wait_for_frames(5000)
    playback.pause()

    depth_frame = frames.get_depth_frame()
    color_frame = frames.get_color_frame()
    ir_frame = frames.get_infrared_frame(1)
    ir_colormap = np.asanyarray(colorizer.colorize(ir_frame).get_data())

    emitter = rs.depth_frame.get_frame_metadata(depth_frame, rs.frame_metadata_value.frame_laser_power_mode)  # <- Error is coming from this line
    print("Emitter : ", emitter)
    #
    # cv2.imshow("test",ir_colormap)
    # cv2.waitKey(0)
    # cv2.destroyAllWindows()
    playback.resume()
    count = count + 1

The equivalent C++ code was also tried with no success

rs2_metadata_type val = 0;
rs2_frame_metadata_value t=RS2_FRAME_METADATA_FRAME_LED_POWER;
if (rsDepthFrame.supports_frame_metadata(t))
    val = rsDepthFrame.get_frame_metadata(t);```

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Sep 28, 2020

Comparing your script with the other one carefully, it looks as though the main difference in the metadata statements is simply that you put rs on the front of the instruction. Your script does:

emitter = rs.depth_frame.get_frame_metadata(depth_frame, rs.frame_metadata_value.frame_laser_power_mode)

And if your instruction was written in the same way as the other script (using depth_frame instead of df), it would drop the rs. and be:

emitter = depth_frame.get_frame_metadata(depth_frame, rs.frame_metadata_value.frame_laser_power_mode)

@anandhakrishnantecholution
Copy link
Author

@MartyG-RealSense, I'm afraid that does not work as I get another error

emitter = depth_frame.get_frame_metadata(depth_frame, rs.frame_metadata_value.frame_laser_power_mode)
TypeError: get_frame_metadata(): incompatible function arguments. The following argument types are supported:
    1. (self: pyrealsense2.pyrealsense2.frame, frame_metadata: pyrealsense2.pyrealsense2.frame_metadata_value) -> int

Invoked with: <pyrealsense2.frame Z16 #16958>, <pyrealsense2.frame Z16 #16958>, frame_metadata_value.frame_laser_power_mode

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Sep 29, 2020

This error message is also reported in the case in the link below:

#3179 (comment)

They are advised to drop the first function that is called in the bracket so that the bracket starts with rs.frame. On your line, that would make it look like this:

emitter = depth_frame.get_frame_metadata(rs.frame_metadata_value.frame_laser_power_mode)

@anandhakrishnantecholution
Copy link
Author

@MartyG-RealSense

I don't think that should make much difference and as expected, I get the same error as before

   emitter = depth_frame.get_frame_metadata(rs.frame_metadata_value.frame_laser_power_mode)
RuntimeError: Frame does not support this type of metadata

I am trying to replicate the scenario mentioned in #6202

@MartyG-RealSense
Copy link
Collaborator

An easy way to test whether it is a code error or whether retrieval of the laser metadata in this way is not supported would be to simply retrieve frame_counter instead:

emitter = depth_frame.get_frame_metadata(rs.frame_metadata_value.frame_counter)

@anandhakrishnantecholution
Copy link
Author

@MartyG-RealSense

As I mentioned in my above comment everything such as frame_counter, actual_fps, raw_frame_size, time_of_arrival etc, works but not frame_laser_power_mode or anything similar such as frame_emitter_mode, frame_laser_mode, frame_led_power.

@MartyG-RealSense
Copy link
Collaborator

Please try removing the '5000' out of wait_for_frames(5000) and just have wait_for_frames() instead to check whether or not the wait period is disrupting retrieval of emitter metadata.

@anandhakrishnantecholution
Copy link
Author

@MartyG-RealSense I'm afraid that does not resolve the issue as well. It still says the frame does not support this type of metadata.

I wonder if something is wrong with the way in which the .bag file was generated. Can you confirm whether anything else needs to be done or enabled in the RealSense viewer or pyrealsense to enable collecting such information?

@MartyG-RealSense
Copy link
Collaborator

Anything that was active at the time of recording should get included in the bag data. For example, if you only had the Depth stream active during recording then RGB and IMU data would not be included.

Do you know whether the IR Emitter was enabled during the bag recording?

@anandhakrishnantecholution
Copy link
Author

@MartyG-RealSense The emitter was turned on as well as the feature mentioned in #3066 with the Emitter On Off setting in the Intel RealSense Viewer.

@anandhakrishnantecholution
Copy link
Author

@MartyG-RealSense , I think it might be that metadata is not supported in Kernal 5.4 as mentioned in #6368.

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Sep 30, 2020

I have not seen reports of metadata problems with kernel 5.4. Support for the kernel is referenced here:

#6866 (comment)

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Sep 30, 2020

I note that in your specification at the top of this case you list that you are using firmware 5.11.15.0.
#3066 lists a requirement of firmware 5.12.0 or newer for the emitter alternating function. 5.12.0 is the firmware release that comes directly after 5.11.15.0.

https://dev.intelrealsense.com/docs/firmware-releases

image

@anandhakrishnantecholution
Copy link
Author

@MartyG-RealSense I will check the same and update you if I find something.

@anandhakrishnantecholution
Copy link
Author

@MartyG-RealSense I could not test anything for a week hence the delay, but your suggestion works after updating to 5.12 I could access the metadata.

This issue can be closed.

@MartyG-RealSense
Copy link
Collaborator

Great news - thanks for the update!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants