VPF 🤝 OpenCV

[sandhawalia]

Hi there,

Was wondering if there is any interest in having VPF and OpenCV work together. This revolves around a handshake between CudaSurface and Opencv GpuMat.

We have a fork of OpenCV which does this would be happy to make it public, along with its pre-built Docker Image (Ubuntu only). This would open up whole host of CV application with VPF

     Video -> VPF -> CudaSuface -> 🤝 -> GpuMat -> CV application -> CudaSurface -> Video

CUDA-accelerated video encoding/decoding in OpenCV is not upto date and therefor needs expensive Upload and Download from device to host and vice versa.

[rarzumanyan]

Hi @sandhawalia

Give me couple days to collect more info and find if I could allocate cycles on this.
Will come back a bit later!

[rarzumanyan]

Bringing more people into discussion
@zjdneos, @sandhawalia

WRT better OpenCV integration topic.
Recently CV-CUDA library was announced which brings in GPU-accelerated CV procedures.
Is this something you may be interested in?

Link to CV-CUDA:
https://blogs.nvidia.com/blog/2022/09/20/computer-vision-cloud/

[Gnarlemagne]

This revolves around a handshake between CudaSurface and Opencv GpuMat
We have a fork of OpenCV which does this would be happy to make it public, along with its pre-built Docker Image (Ubuntu only). This would open up whole host of CV application with VPF

@sandhawalia Hello! Could you expound on how this handshake is accomplished? My current work involves a pretty big stack of video processing, and I've just converted video frame reading to use this library. However, to get it over to cuda_GpuMat involves converting CudaSurface -> download to numpy -> upload to GpuMat, like you said.

Figuring out how to go straight from CudaSurface to GpuMat would be a great help. Thank you!

[sandhawalia]

Hey there. Thanks for interest in the proposal. The handshake goes something like this. On VPF/Python world we have a function

def get_gpu_mat(rgb_frame):
    plane = rgb_frame.PlanePtr()

    H, W, pitch = (plane.Height(), plane.Width(), plane.Pitch())

    gpu_mat = cv2.cuda_GpuMat()

    # zero copy of pixels simply using pre-allocated GPU memory
    # W // 3 needded since channels are horizontally aligned
    #
    # r r r r g g g g b b b b
    # r r r r g g g g b b b b
    # r r r r g g g g b b b b
    #
    gpu_mat = cv2.cuda.ptr2mat(plane.GpuMem(), gpu_mat, H, W // 3, pitch, cv2.CV_8UC3)

On OpenCV side we have following C++ code which implements cv2.cuda.ptr2mat

#include "precomp.hpp"

using namespace cv;
using namespace cv::cuda;

#if !defined HAVE_CUDA || defined(CUDA_DISABLER)

void cv::cuda::ptr2mat(size_t, InputOutputArray, int, int, int, Stream&){ throw_no_cuda(); }

#else // HAVE_CUDA

namespace {

  typedef void (*ptr_func_t)(const size_t ptr, GpuMat& src2, int height, int width, int type, Stream& stream);

  void ptr2map_op(size_t _ptr, InputOutputArray _dst, int height, int width, int type, Stream& stream, ptr_func_t func) {

    GpuMat& dst = _dst.getGpuMatRef();
    func(_ptr, dst, height, width, type, stream);

  }
}

void ptr2mat_gpu(const size_t _ptr, GpuMat& dst, int height, int width, int type, Stream& stream);

void cv::cuda::ptr2mat(size_t _ptr, InputOutputArray _dst, int height, int width, int type, Stream& stream)
{
  ptr2map_op(_ptr, _dst, height, width, type, stream, ptr2mat_gpu);
}

#endif // HAVE_CUDA

#include "opencv2/opencv_modules.hpp"

#ifndef HAVE_OPENCV_CUDEV

#error "opencv_cudev is required"

#else

#include "opencv2/cudev.hpp"

using namespace cv::cudev;

void ptr2mat_gpu(const size_t _ptr, GpuMat& _dst, int height, int width, int type, Stream& stream);

void ptr2mat_gpu(const size_t _ptr, GpuMat& _dst, int height, int width, int type, Stream& stream){

    // empty GpuMat we simply swap the pointer
    GpuMat _mat = GpuMat(height, width, type, (void *) _ptr);

    _mat.swap(_dst);

}

#endif

This would also require, Implementing a OpenCV contrib_module and rebuilding OpenCV with the new module. Not sure how familiar are you with this.

FYI — If you wanna try it out i can make our forked OpenCV image public, which already implements cv2.cuda.ptr2mat before you head down the cuda rabbit hole.

[Gnarlemagne]

Thank you for the detailed response!
I ultimately managed to figure it out on my own a few days after my comment by using PyCuda as an intermediary, since you can do direct device memory operations with it. It definitely felt a bit... Heretical to be doing any sort of memory manipulation in python, but ultimately it worked like a charm, didn't require any builds from source (other than PyCuda itself), and once I figured out how images are stored in device mem and how to convert a surface from NV12 to a cv2.cuda friendly format, the solution came out to less than 10 lines of code.

That said I think this is phenomenal information to have out there, so thank you for sharing it. I'm confident this will help many other people who come searching.

[sandhawalia]

Heya. Glad it worked out with going about the cv2 route. Do you mind sharing how you make the handshake between NV12/PyCuda and cv2.cuda array? I'm sure folks (myself included) would find it useful.