[BUG] {OAK-1 image pixels are displaced}

[aviteri2020]

Description

I am using depthai-core v2.20.2 with an OAK-1 camera (IMX378 sensor, SKU A00109) and I realized that the images have some displaced pixels (as seen in the image)

I have tested with three different cameras and I have a similar effect, just that the displaced pixels regions are not always the same. Do you have any clue of how to solve this issue or what is the cause? I am using depthai in a C++ application.

The distance at which the image was taken is around 45 cm and the displaced pixels zones are normally found on the right side of the image frame.

The used resolution is THE_4K_ at 30 FPS (it appears also at 20 FPS).

Minimal Reproducible Example

To save the image press the 'C' key . This is the minimum I can do since reducing something else from the code will not be representative or enough to reproduce the issue. The image recovered from the still node (2174_still.png) has not any image artifacts, but the image from the imagemanip node (2174_img_manip.png) always has it.

I use the ImageManip to avoid loading the whole 3 image channels into the buffer due to frame rate. This step is key since I use the OAK-1 camera connected to an Android Device and if I don't do this the frame rate goes from 20 (taking only one channel from imgmanip node) to 10(taking the 3 channels image from isp).

#!/usr/bin/env python3

import depthai as dai
import cv2
import numpy as np
from itertools import cycle

def clamp(num, v0, v1):
    return max(v0, min(num, v1))

print(dai.__version__)

# Create pipeline
pipeline = dai.Pipeline()

# Define sources and outputs
camRgb = pipeline.create(dai.node.ColorCamera)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K)
camRgb.setInterleaved(False)
#camRgb.setIspScale(3,3) # 1080P -> 720P
camRgb.setColorOrder(dai.ColorCameraProperties.ColorOrder.RGB)
camRgb.setImageOrientation(dai.CameraImageOrientation.NORMAL)

camRgb.initialControl.setSharpness(0)
camRgb.initialControl.setLumaDenoise(4)
camRgb.initialControl.setChromaDenoise(4)
camRgb.initialControl.setAutoExposureCompensation(-2)
camRgb.initialControl.setManualFocus(133)

manipVideo = pipeline.create(dai.node.ImageManip)
manipVideo.setMaxOutputFrameSize(3840*2160)
manipVideo.initialConfig.setFrameType(dai.ImgFrame.Type.YUV400p)

stillEncoder = pipeline.create(dai.node.VideoEncoder)

controlIn = pipeline.create(dai.node.XLinkIn)
configIn = pipeline.create(dai.node.XLinkIn)
ispOut = pipeline.create(dai.node.XLinkOut)
outVideo = pipeline.create(dai.node.XLinkOut) #########
videoOut = pipeline.create(dai.node.XLinkOut)
stillMjpegOut = pipeline.create(dai.node.XLinkOut)

controlIn.setStreamName('control')
configIn.setStreamName('config')
ispOut.setStreamName('isp')
videoOut.setStreamName('video')
stillMjpegOut.setStreamName('still')
outVideo.setStreamName('gray')  #########

# Properties
camRgb.setVideoSize(640,360)
stillEncoder.setDefaultProfilePreset(1, dai.VideoEncoderProperties.Profile.MJPEG)

# Linking
camRgb.isp.link(ispOut.input)
camRgb.still.link(stillEncoder.input)
camRgb.video.link(videoOut.input)
controlIn.out.link(camRgb.inputControl)
configIn.out.link(camRgb.inputConfig)
stillEncoder.bitstream.link(stillMjpegOut.input)

camRgb.isp.link(manipVideo.inputImage)   #########
manipVideo.out.link(outVideo.input)      #########

# Connect to device and start pipeline
with dai.Device(pipeline) as device:

    # Get data queues
    controlQueue = device.getInputQueue('control')
    configQueue = device.getInputQueue('config')
    ispQueue = device.getOutputQueue('isp')
    videoQueue = device.getOutputQueue('video')
    stillQueue = device.getOutputQueue('still')
    grayQueue = device.getOutputQueue('gray',3,False) #########

    
    while True:        
        vidFrames = videoQueue.tryGetAll()
        for vidFrame in vidFrames:
            cv2.imshow('video', vidFrame.getCvFrame())

        ispFrames = ispQueue.tryGetAll()
        for ispFrame in ispFrames:
            cv2.imshow('isp', ispFrame.getCvFrame())            

        stillFrames = stillQueue.tryGetAll()
        for stillFrame in stillFrames:
            # Decode JPEG
            frame = cv2.imdecode(stillFrame.getData(), cv2.IMREAD_UNCHANGED)
            # Display
            cv2.imshow('still', frame)
            cv2.imwrite("C:/Users/xxx/Desktop/artifact_images/2174_still.png",frame)

        grayFrames = grayQueue.tryGetAll()
        for grayFrame in grayFrames:
            gframe = np.array(grayFrame.getData(),np.uint8).reshape(2160,3840)
            cv2.imshow('gray', gframe)
        

        # Update screen (1ms pooling rate)
        key = cv2.waitKey(1)
        if key == ord('q'):
            break
        elif key == ord('c'):
            ctrl = dai.CameraControl()
            ctrl.setCaptureStill(True)
            controlQueue.send(ctrl)            
            cv2.imwrite("C:/Users/xxx/Desktop/artifact_images/2174_img_manip.png",gframe)

Expected behavior
The images SHOULD NOT have such artifacts

Additional context
The artifact is only produced when using the ImageManip node. About the ImageManip limitations I do not surpass the width of the image (3840 pixels which is also a multiple of 16). I've tried also with the GRAY8 image type and the artifact also appears with this configuration.

I've tested with the following versions of depthai-core:

2.17.4
2.18.0
2.19.1
2.20.2
2.21.2

The artifact appears in all the versions but for the 2.17.4. In this version the artifact is almost imperceptible, but it does appear when looking carefully (more evident in an image with checkerboard).

To give a little of background, I was working with the 2.17.4 since january 2022 (at that time it was the available release version), so that is why I have not noticed this artifact before. I was intending to update the depthai-core version for a new one to be able to recover the metadata from each image (lensPosition, ISO, exposureTime), and it may be related that from version 2.18.0 to 2.21.2 it is possible to obtain the imaga metada, but the artififact is also a lot more evident for those versions.

[Erol444]

Discussion here: https://discuss.luxonis.com/d/1307-oak-1-image-pixels-are-displaced/14