Pyrealsense2 - Dropped frames

[TheCrafft]

Required Info
Camera Model	D415
Firmware Version	2.33.1
Operating System & Version	{Win (8.1/10) / Linux (Ubuntu 14/16/17) / MacOS
Kernel Version (Linux Only)	-
Platform	PC
SDK Version	2
Language	python
Segment	others

Issue Description

Hi!

I was wondering if anyone could give me some insight regarding the extraction of frames from a .bag file and the loss of frames using the python package.

I have large .bag files which contain both depth (640x360; 30 FPS) and RGB streams (1280 x 720; ~16FPS). It has been a while since I collected them, but I am quite sure that I initially set both at 1280 x 720; 30 FPS. The data were recorded on an external HDD with USB 3.0.

Q1 What causes the change in resolution for the stereo module and why was the RGB camera not capable of achieving 30 FPS?
My current thought is that I actually set depth at 640x360; 30 FPS, assuming the viewer can't change resolution on its own, and that given the amount of data some frames from the RGB stream were dropped. I am currently not interested in the depth stream, but just find it odd.

Q1.1 Could replacing the HDD with and SSD resolve this issue?

The institution that I work for does not like the use of external programs, hence I am using the python package.
I followed the example shown in issue #1919 and made a few changes.

Script

pipe = rs.pipeline()
cfg = rs.config()
cfg.enable_device_from_file(file, repeat_playback=False)
if exportcolor is True:
    cfg.enable_stream(rs.stream.color, 1280, 720,  rs.format.rgb8, 30)
profile = pipe.start(cfg)

# Needed so frames don't get dropped during processing. Will make it slower
profile.get_device().as_playback().set_real_time(False)

# Initiate count
icolor = 0
try:
    while True:
        frames = pipe.wait_for_frames()
        if exportcolor is True:
            color_frame = frames.get_color_frame()

            # Extract system time at capture
            color_frame_time = color_frame.get_timestamp()/1000  # Windows multiplies by a factor 1000
            color_frame_time = datetime.fromtimestamp(color_frame_time).strftime("%H:%M:%S")
            color_frame_time = str(color_frame_time).replace(":", "")

            # Convert frame to array, BGR2RGB, define path and save
            color_image = np.asanyarray(color_frame.get_data())
            color_image = cv2.cvtColor(color_image, cv2.COLOR_BGR2RGB)
            colorfilepath = path + "/" + filename + "/" + filename + "_" + color_frame_time + "_Color_%06d.tiff" % icolor
            cv2.imwrite(colorfilepath, color_image)

            # Count frame number
            icolor += 1

except RuntimeError:
    pipe.stop()
    print("\nNo more frames, end of .BAG file")

This script works for extracting frames, when ran on a laptop it drops more frames than when ran on a HPC. Though, after closer inspection it appears that it only extracts 6 - 9 FPS, still not the 16 FPS it shows in the viewer. When reconstructing the frames in a video (.mp4), it appears that 12 FPS gives the most natural flowing image.

Q2 What can cause this loss of frames?
Q3 Why does 12 FPS result in the best all-round naturally flowing image?

[MartyG-RealSense]

Hi @TheCrafft Was the bag file recorded in the RealSense Viewer program please? There has been past cases of resolution being halved due to a decimation post-processing filter being enabled when saving / recording files. If you are recording in the Viewer, I recommend checking under the post-processing sections of both the Stereo Module and RGB module to ensure that their respective decimation filters are disabled (indicated by a red icon beside them).

Regarding loss of frames: using a fast-access storage device such as an SSD drive can be beneficial to recording in regard to reducing the likelihood of a bottleneck occurring during saving.

It has been observed in the past on some low-end systems that recording with frame compression enabled may also cause performance problems.

#2102 (comment)

Extracting all frames from a bag can be difficult to achieve, and some scripts - of which there are many of them - are better at it than others. You could test the Python image extraction script in the link below to see whether it provides better performance.

#8137 (comment)

I note that in your own script though you seem to be extracting tiff images. You may have faster performance if you write the images as PNG images.

[TheCrafft]

Hi @MartyG-RealSense!

Yes, the bag file was recorded in the RealSense Viewer. Decimation was on for the Stereo Module, but not for the RGB module. Thank you for pointing that one out! I do not remember having enabled frame compression.

A quick scan shows me that my script is comparable to the one you linked. I guess I am already getting the most out of it.

Aah yes! The reason I am using tiff instead of png is that the png frames had quite some loss. I do not mind the speed of performance as I run the script on a cluster. Still not sure what caused the loss.

For future reference, would one stream (RGB) set at 1280 x 720 @30fps recorded to an SSD give the desired FPS, given that the machine is capable enough, or would lowering the resolution have a more consistent effect?

[MartyG-RealSense]

I would agree that 1280x720 RGB at 30 FPS is a reasonable configuration, yes. If you are recording a bag at that configuration, the bag file will be larger in size than a lower resolution. I recently did a test where I recorded a 20 second bag at 30 FPS. 1280x720 produced a 274 mb bag, whilst 848x480 produced a bag of 138 mb size. If you are recording multiple bags for long durations, having a high capacity drive in terabytes instead of gigabytes may be necessary.

[MartyG-RealSense]

Hi @TheCrafft Do you require further assistance with this case, please? Thanks!

[TheCrafft]

Hi @MartyG-RealSense Consider this solved!

[MartyG-RealSense]

Great to hear, @TheCrafft - thanks very much for the update!