[Feature-Request] Depth image compression by colorization

[borongyuan]

Start with the why:

VideoEncoder only supports NV12 or GRAY8 format. It cannot be used to encode depth images, only disparity images with subpixel disabled. This creates a big headache for us with PoE models. We hope to be able to transmit depth images with high resolution, high precision and high frame rate. The approach presented in this RealSense white paper seems promising. Hope it works for OAK cameras.

Move to the what:

Add depth image compression by colorization feature.

Move to the how:

The basic idea is to colorize the 16-bit depth image or disparity image according to the method mentioned in the white paper. Then use VideoEncoder for compressed transport. The recovery part on the Host side should be relatively easy to implement. The main feature that needs to be added is colorization on the camera. Currently ImageManip does not support color conversion of 16-bit images. Perhaps this particular color conversion method should be added to ImageManip, or implemented elsewhere.

[themarpe]

This is a great idea!

We already added Colormap feature a while ago and could be easily extended to cover 16bit to RGB/NV12 as well.

[borongyuan]

Any plan for this? It should be possible to create a custom blob and use the new Cast node. But if you can add this to ImageManip node, I'll now waste time on this.

[themarpe]

CC: @moratom on whats the applicability on RVC2

[borongyuan]

If we use disparityWidth of 96 and subpixelFractionalBits of 4, the max disparity should be 1520. This colorization method provides 1529 discrete levels. So there should be no quantization error, and normalization step might be skipped.

[borongyuan]

Another solution is to use the RVL depth image compression algorithm. RVL codec has been added to ROS since Noetic/Jazzy. We are also considering adding it to RTAB-Map.