RGB and depth sync

[svensht2]

Hi,

I'm using using RS435 with FW 5.8.9.0. librealsense 2.7.9, Windows 10 1703.
How do I enable HW sync for depth and color? seems like you don't have a sample anymore that demonstrates how to pull 2 synced streams. With using pipeline API and enabling color and depth the streams go out of sync.
You can easily reproduce this by using your rs-align tool and waving your hand for a few seconds.

Thanks,

Simon

[dorodnic]

Hi @svensht2

There is the subject of per-frame metadata that is required for proper synchronization. Without hardware timestamps what you get are software estimates and there is nothing preventing the streams from drifting. Unfortunately both Windows and Linux does not offer this feature out of the box yet. You can read more about it here, including how to enable it.

In addition, there is a trade-off between latency and frame-drops, especially when the system is under heavy stress. In such conditions, we can either increase queue sizes (introducing larger potential latency) or suffer significant frame drops making it almost impossible to pick up well-synchronized pairs (freezes).
In the align example, we relax the synchronization requirement and let the sample use not perfectly matched pairs in absence of better options. We are planning to make that configurable.

In practice, changing false to true in image.cpp:785 is likely to reduce latency. Compiling as Release can also improve the situation. We are going to publish a more general patch very soon to address many of these issues.

As for hardware synchronization, if I recall correctly hardware sync between depth and color is automatically enabled when sensors are configured to the same FPS.

[rafaelspring]

I have the same issue with the D435. I assume this one has hardware sync? Attached are two logs, one with a 15 fps setting for both depth & color, another with a 30 fps setting for both depth & color. The logs show color TS (in us), depth TS (in us) and color minus depth TS.

The average lag of one sensor vs. the other is around 0, however the variance is quite high. In other words, the driver tries to give me color and depth frames of approximately matching time stamps but the mismatch can go as high as half the frame rate (see attached files).

30fps.txt
15fps.txt

It would be really helpful if the documentation was explicit about which sensor supports hardware sync and which one doesn't and how it's possible to enable HW sync in librealsense.

[rafaelspring]

Quick update: Using the rs2::syncer class instead of rs2::pipeline to grab the frames has not improved the situation.

[dorodnic]

On software level, both syncer and pipeline are matching the frames using the same logic.
We do plan to perform more comprehensive testing of device synchronization prior to launch and will publish the results here.

[rafaelspring]

Thanks for the clarification.

[rafaelspring]

Ok, that would of course explain it. Does the 415 have HW sync?

[rafaelspring]

I have just tried this on a D415 running firmware 05.08.09 and there is also no hardware synchronization between color and depth, same as on the D435.

Is hardware synchronization between color and depth a feature that is available at all, in at least some configurations? Or is hardware sync just something the D4x5 series doesn't support?
If the D4x5 does support HW RGB-D sync what are the exact steps to make it work? Today we are trying things on a Windows 10 machine with no luck.

It would be good to get a definite answer on this soon.

[vladsterz]

Are there any news on the subject?

[arybalkin]

Here's a forum link discussing the very same topic: https://communities.intel.com/thread/120208

[tedtman]

@svensht2 ,@vladsterz as @arybalkin noted, an answer to this issue is provided in the communities pages, https://communities.intel.com/thread/120208. If you still have questions please reply. This issue will be closed in five days but you can always open a new issue after that.

[jkosmach]

I'd like to see this ticket re-opened and a better clarity explained.

In your release notes you still have this issues (#774) as an issue and then above you reference the intel communities where the last response from Intel is in January. To my knowledge, This is the correct answer:

For HW Color/Depth Sync

1. This will never be possible with the D435 since the RGB and Depth Sensor are on different PCB
2. This is possible with D415 since the RGB and Depth sensors are on same PCB with the following:
   a. FW 5.9.1 or later supports HW sync on D415
   b. RGB manual exposure must be used
   c. There is a known bug that prevents the RGB gain from being adjust to achieve acceptable quality video
   d. Until the gain is fixed (so that it can be set to levels that auto-exposure is achieved), HW sync for Color/Depth on D415 is not usable.

[sjdrc]

MartyG commented on the 26th of November the following on the Intel forum thread referenced above:

"Hardware sync for D415 and D435 is now available. D415 can do color and depth sync. D435 only has depth sync."

I have been able to use the realsense viewer to sync the RGB and depth feeds by turning RGB Camera -> Auto Exposure Priority off, and have verified this by noting that both feeds now have the same framerate (~29.76) which matches the RGB framerate.

[Desperad0]

MartyG commented on the 26th of November the following on the Intel forum thread referenced above:

"Hardware sync for D415 and D435 is now available. D415 can do color and depth sync. D435 only has depth sync."

I have been able to use the realsense viewer to sync the RGB and depth feeds by turning RGB Camera -> Auto Exposure Priority off, and have verified this by noting that both feeds now have the same framerate (~29.76) which matches the RGB framerate.

Could you give more details about D415 color sync. I have 3 D415 cameras, but cannot sync the color. Thanks

[rafaelspring]

AE Priority has to be turned off.

Also, from my own experience, not all USB ports and configurations can handle streaming a high resolution depth and color image, resulting in frame skips and de-sync. I recommend lowering the color resolution in these cases.

[soarwing52]

hi, I also face the same issue, which the frames are not synced when I tried to use wait_for_frames.

I also saw your replies on Githun and wondering if you found the answer to get the two frames synced.

I tried rs.syncer in python and cant get it work, it would be great if you can provide your example for using the syncer.

my frame data is below: data

every 10 second/press space take one frame. and the result frame is actually matched if i accept +-20 ms. But when I use wait_for_frames I can't get the right match when I use wait_for_frames in python and visualize with opencv/matplotlib it gives out unmatching frames.