[Bug] Unable to run custom YOLOv5 with Raspberry PI and NCS2

[Averen19]

HI,
I'm trying to run my custom YOLOv5s model on a RPI4 with NCS2, I have converted the model using the default export.py file from Ultralytics, I tried to do inferencing on PC before moving to RPI and I get such an error. Do you think you can help me with this? Or maybe you can send a demo code to do inferencing with the YOLO model? The following is also the code I used to do detection.

test.zip

[brmarkus]

Not that long ago, in the Ultralytics repo support for OpenVINO was added:

ultralytics/yolov5#6057 "OpenVINO Export" (Dec 22, 2021)
ultralytics/yolov5#6179 "Add OpenVINO inference" (Jan 4, 2022)
ultralytics/yolov5#6739 "YOLOv5 v6.1 release" (Feb-22, 2022)

[Averen19]

Yes, I have tried running using the detect.py provided by Ultralytics using the xml file and it works. But I would like to use it with an NCS2, do you have any suggestion as to implement the integration with this intel hardware?

[brmarkus]

Can you use your NCS2/MyriadX successfully in your environment for other models - just checking whether everything is installed properly (like in Linux environments to install a specific UDEV-rule first).
Can you run any other model - it needs to be in FP16 precision - with your NCS2, using any of the OpenVINO or OpenModelZoo samples&demos? (like using benchmark_app [...] -m FP16/my_model.xml -d MYRIAD [...])

[Averen19]

Yes, I have tried using the demos provided by OpenVino, and it works. But I'm wondering if its possible to make yolov5s detect.py be compatible with ncs2? or do I have to put the xml file in the openvino2021 folder and create another script to run object detection?

[jgespino]

@Averen19 I was able to export the yolov5s model using python export.py --weights yolov5s.pt --include openvino from the ultralytics repository. Using the benchmark_app I was able to run inference on CPU and MYRIAD (Intel NCS 2).

The following may help you write your script to run inference.
https://github.com/violet17/yolov5_demo/blob/main/yolov5_demo_OV2021.3.py

[Averen19]

@jgespino Did you run this code in the openvino 2021.4 folder? Did you change anything from the script? Would it be possible to give a step by step guide on how you did it?

[Averen19]

@jgespino This is the result I get when i'm trying to run inferencing. Do you know whats the error?

[jgespino]

@Averen19 Couple of thigs:

• Converting with export.py from ultralytics repository will use the latest OpenVINO 2022.1 release which the IR will not work with OpenVINO 2021.4
• The demo code I shared expects the model to have a certain output as mentioned in the conversion steps.

Please install OpenVINO 2022 Runtime using the offline or online installer to include the MYRIAD plugin.

Steps to download/convert model and run demo:

"C:\Program Files (x86)\Intel\openvino_2022\setupvars.bat"
wget https://github.com/ultralytics/yolov5/releases/download/v6.1/yolov5s.pt
wget https://raw.githubusercontent.com/violet17/yolov5_demo/main/yolov5_demo_OV2021.3.py
git clone https://github.com/ultralytics/yolov5
pip install -r yolov5\requirements.txt
pip install onnx openvino-dev
python yolov5\export.py --weights yolov5s.pt --include onnx
mo --input_model yolov5s.onnx --scale 255 --reverse_input_channels --output Conv_198,Conv_217,Conv_236
python yolov5_demo_OV2021.3.py -m yolov5s.xml -i cam -d MYRIAD

[Averen19]

@jgespino by using the export.py with --include openvino, I no longer need to use the model optimizer right, or do i only convert to onnx using export.py and use openvino's model optimizer to convert to ir format.

What I have done is to install openvino_2022 and tried running yolov5s.xml directly with export.py. I put in the yolov5_demo_OV2021.3.py and the yolov5s.xml into the openvino file and get such a result:

If I was meant to use the model optimzer from openvino, I can't seem to find it in the openvino2022.

[jgespino]

@Averen19 Correct, you need to use export.py to convert pt model to onnx. Then use the model optimizer to convert to IR and specify the three outputs as expected by this demo code. If you use the IR file generated by the export.py the outputs are different and you will need to write your own inference script.

[Averen19]

@jgespino to use the model optimizer, I would need to get the dev tools build right instead of run time? I tried both and can't seem to find the mo.py file in to do model optimizing.

[Averen19]

@jgespino I realized that it might not be a .py file and just input what you wrote and got the following:

[Averen19]

Hi, I managed to run the yolov5.xml already thank you very much. However, I am unable to run it with the MYRIAD. My assumptions when installing openvino it comes preinstalled with ncs2 dependencies. This is the results I get when trying to run with Myriad. Also, if I were to implement this on a raspberry pi 4 would I also need openvino 2022? As far as I have found, openvino2022 is not compatible with Rpi.

[Averen19]

Also, I i managed to convert my pytorch models to openvino format in openvino 2021.4 and am able to run inferencing, although I was unable to use MYRIAD as the device. But for raspberry pi the latest version that I was able to find was openvino 2021.2 and when I try to run the code it shows that I have such an error:

I tried to downgrade to openvino 2021.2 on windows and run the model optimzer but I get such an error:

Any idea as to how to solve this issue?

[jgespino]

@Averen19 Looks like we missed uploading the OpenVINO 2022.1 package for Raspbian OS. I have reached out to my peers for assistance. In the meantime, you can download OpenVINO 2021.4.2 package for Rasbian OS. Please note that you need to match the OpenVINO version that is used to convert the model to IR and the runtime package installed on your pi. You can also build OpenVINO from source If you want to use the latest 2022.1 release on Raspberry Pi.

Let me know if you see the same error converting the model on 2021.4.2.

[Averen19]

@jgespino I have installed the 2021.4.2 for Raspbian and I have done the necessary installation for NCS2 dependencies but I get the following error. I have check on windows using the same codes and files it works.

It just gets stuck at preparing input, this is the same issue with -d MYRIAD as well.

[jgespino]

@Averen19 What is the output of lsusb? After running the install_NCS_udev_rules.sh did you logout or reboot the pi?

Could you verify your NCS 2 has been configured correctly by following the build and run object detection sample instructions in the documentation?

[Averen19]

I have rebooted the rpi and still have the same issue. Ive tried the example but I'm stuck at this step: _./armv7l/Release/object_detection_sample_ssd -m <path_to_model>/face-detection-adas-0001.xml -d MYRIAD -i <path_to_image>_

I can't seem to find where the path to model is.

[jgespino]

@Averen19 you have to download the model as described in step 3:

git clone --depth 1 https://github.com/openvinotoolkit/open_model_zoo
cd open_model_zoo/tools/model_tools
python3 -m pip install -r requirements.in
python3 downloader.py --name face-detection-adas-0001

or you can download it directly:

wget https://storage.openvinotoolkit.org/repositories/open_model_zoo/2021.4/models_bin/3/face-detection-adas-0001/FP16/face-detection-adas-0001.bin
wget https://storage.openvinotoolkit.org/repositories/open_model_zoo/2021.4/models_bin/3/face-detection-adas-0001/FP16/face-detection-adas-0001.xml

[Averen19]

@jgespino Ive tried to run the model and get such a problem.

[brmarkus]

Have you downloaded a model, or copied from another machine?
For an OpenVINO model (in IR-format) you should see at least two files, one XML and one BIN file - the same name only with two different file-extensions, like "face-detection-adas-0001.bin" and "face-detection-adas-0001.xml".
If these two files are in another folder than your application, just provide the full-path of the XML-file.
Example:

./object_detection_sample_ssd -m /path/to/downloaded/model/FP16/face-detection-adas-0001.xml -d MYRIAD -i /home/pi/openvino2021.4/fire_0001.jpg

[Averen19]

@jgespino Hi, I have tested with the example using MYRIAD and it works, I managed to get the out_0.bmp, proving that the MYRIAD plugin does work. I retried again with my yolov5 model but I still get stuck with the same result.

[brmarkus]

The YoloV5 model is much bigger than e.g. the "face-detection-adas-0001.xml", requiring likely more time to compile the OpenCL-kernels.
Can you create a writable folder like mkdir -p /tmp/opencl/ and export an ENV variable like export cl_cache_dir=/tmp/opencl/ and start the application again?
While the applications seems frozen - do you see the content of the folder "/tmp/opencl/" to grow, files being created? This is triggered by OpenVINO - and could sometimes take a lot of time... and requires sometimes a lot of space in the file system... and sometimes requires a lot of system memory for the OpenCL kernel compilation... and produces sometimes a lot of CPU-load while compiling these OpenCL kernels.

[Averen19]

I have given the yolo model to run for more than five minutes and its still stuck in preparing input. Tp test the writable folder, do you mean to create a folder like this? I've let it ran for five minutes as well and nothing seems to appear.

---

[brmarkus]

Hmm, ok, it's not stuck at loading the model (where the OpenCL kernels would be compiled), but at the previous step, "Preparing inputs".
Can you check the code after the log-message "Preparing inputs"?
Usually the code has loaded the network, was able to get the expected input requirements (like the expected input resolution of 640x640 pixels) - and then reads-in the given file (fire_001.jpg), scaling it to the expected resolution.
Can it be something like your environment cannot open and decode the JPG-file? Maybe a JPG-library is missing or your file has an exotic format (like "JPEG2000"), or exotic color-format? Or the file is very huge?
Can you use another picture file?

[Averen19]

Hi, I managed to get it to run thank you. However, there are still quite a few issues. I've tried with camera and it is able to detect albeit very laggy, but when I try with images it will show the following, where it says that it is able to run but no result is shown.

Also I tried to run it without MYRIAD and it comes with the following error, is it unable to detect the cpu in the RPI, making it unable to run?

[brmarkus]

hmm, not sure I understood your response...
You said you got it working? You see it working even with a camera? So, it's working when you use MYRIAD and a local file or with camera? It's working with YoloV5?
Yes, YoloV5 is complex, the throughput will be lower due to complexity - you will not get 30fps with MYRIAD or with a RPI CPU... you would need a more powerful system (or a powerful GPU).

But your additional notes are confusing... because then you say it's not working?

Sometimes simple test-applications don't handle termination not correctly... Sometimes when an application exists or get terminated the connection and libraries are not closed gracefully - and starting the application again will fail with initializing the MYRIADX device again. The simple applications usually don't have proper error-handling or shutdown-sequences.
Usually it helps to unplug the NCS2 and plugging -in again...

[Averen19]

What I mean is that no window will pop up to show the inference result with bounding boxes and confidence score as seen below. Also is there a way to save inference result when the detection is done? Lastly, I would like to change the class name of displayed from # to the actual class name i have made previously from yolo, where should i change that?

[brmarkus]

There are other OpenVINO samples, like "https://docs.openvino.ai/2021.4/openvino_inference_engine_ie_bridges_python_sample_object_detection_sample_ssd_README.html", supporting a "labels" file; the application is reading the file, putting the labels in an array/map at startup.

And then, instead of printing the integer "0" like in your example, "#0 77.4 %", using the integer "0" as index (array) or as a key (map) to look-up the label instead.

[Averen19]

Ok, I will look into that, but for now, it still doesn't create this result window when running the code on RPI as this was captured from windows.

[jgespino]

@Averen19 Are you using a custom trained model of yolov5s or the pre-trained model? I just tested on a fresh Raspberry Pi using the pre-trained model and everything is working as expected. Here are the steps I took:

Convert the model on my Windows system running OpenVINO 2021.4.2

wget https://github.com/ultralytics/yolov5/releases/download/v6.1/yolov5s.pt
git clone https://github.com/ultralytics/yolov5
pip install -r yolov5\requirements.txt
pip install onnx
python yolov5\export.py --weights yolov5s.pt --include onnx
python "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo.py" --input_model yolov5s.onnx --scale 255 --reverse_input_channels --output Conv_198,Conv_217,Conv_236 --data_type FP16

On Raspberry Pi, installed OpenVINO 2021.4.2 and configured per documentation.

source /opt/intel/openvino_2021/bin/setupvars.sh
"C:\Program Files (x86)\Intel\openvino_2022\setupvars.bat"
wget https://raw.githubusercontent.com/violet17/yolov5_demo/main/yolov5_demo_OV2021.3.py
python3 yolov5_demo_OV2021.3.py -m yolov5s.xml -i cam -d MYRIAD --labels labels.txt

The python code should show up a pop up window showing the camera feed with detections. For the labels I used the coco dataset labels and seem to line up to the detections.
https://github.com/amikelive/coco-labels/blob/master/coco-labels-2014_2017.txt

[Averen19]

Yes, I was using a custom pre-trained model, the only thing I changed was the output layers, which I followed Violet's method of setting the output layer above the transpose function.

When I run the code for camera, I am able to get a camera feed. Even thought it is unable to detect my object, I think it is due to the bad camera quality as I am able to detect the objects with Windows.

The problem is when I try to run on a sample image and it says the code is able to run but there's no pop up indicating the bounding boxes as shown below.

[jgespino]

@Averen19 Maybe a bug in the code because the following always returns -1 when passing an image.
https://github.com/violet17/yolov5_demo/blob/main/yolov5_demo_OV2021.3.py#L268

Try changing it to the following for testing purposes and you should see a window with the image.
number_input_frames = 1 #if number_input_frames != -1 and number_input_frames < 0 else number_input_frames

[Averen19]

@jgespino yeah, I'm not sure as to why it doesn't show the inferencing result with images. I'm using the same code on Windows and it works well but when I transfer over to RPi, only the real time camera works but not for images. Do you have any inclination for where the problem lies? Is there any software dependencies that may lead to such error?

I tied to comment out the section you mentioned but its still just stuck at preparing input.

[jgespino]

@Averen19 Is it possible for you to share your custom model and input image with me? Feel free to attach it here or share privately via email (email is on my profile).

[Averen19]

@jgespino Hello, I have sent you my custom files to your email address.

[jgespino]

@Averen19 Thank you for sharing your model. I was able to run and display the inference results (see attached screenshot) by quickly modifying a couple of lines in the yolov5_demo_OV2021.3.py. Please note, this was only done to check that your model was able to run on NCS 2. The changes will break some functionality of this code.

Please consider writing your own code to meet your projects needs by referencing the shared demo. You can also find additional information in the OpenVINO API documentation.

https://docs.openvino.ai/2021.4/api/ie_python_api/api.html

[Averen19]

@jgespino I cannot seem to find your screenshot. Did you run this using Windows or Raspberry Pi? I am able to run inference on images on Windows but not for Raspberry Pi, so I'm unsure if its a coding error or software error from OpenVINO.

[jgespino]

@Averen19 Sorry meant to attach it... I ran it on the Pi. The code is using OpenCV to read the video/image file, can you check if the version on your windows system is the same from the one on your pi?

[Averen19]

The version of OpenVINO on my Windows is 2021.4.752 while the one you provided was 2021.4.2, do you think that would affect the inferencing?

[jgespino]

@Averen19 That is the same release of OpenVINO 2021.4.2. I was referring to the OpenCV, did you install a different version or are you using the included OpenCV version?

[Averen19]

Im using the include OpenCV version, not custom installed.

[Averen19]

@jgespino The OpenCV version I'm currently using is OpenCV 4.5.3, is this support with this version of OpenVINO?

[jgespino]

@Averen19 Apologies for the delay, so the issue seems to be with OpenCV version 4.5.3-openvino which is part of OpenVINO 2021.4.2 release. However, building OpenCV version 4.5.5-openvino does not seem to have this issue. See below.

My recommendation is to build OpenCV 4.5.5-openvino from source and comment out line 69 to line 77 in the setupvars.sh script to not configure the pre-built OpenCV.

cd ~
git clone -b 4.5.5-openvino-2022.1.0 https://github.com/opencv/opencv.git
git clone -b 4.5.5 https://github.com/opencv/opencv_contrib.git
cd opencv && mkdir build && cd build
cmake DOPENCV_EXTRA_MODULES_PATH=/home/pi/opencv_contrib/modules –DCMAKE_BUILD_TYPE=Release –DCMAKE_INSTALL_PREFIX=/usr/local ..
make
make install

[maecky]

@jgespino could you please explain what --ouput Conv_198,Conv217,Conv_236 is doing in the command:
python "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo.py" --input_model yolov5s.onnx --scale 255 --reverse_input_channels --output Conv_198,Conv_217,Conv_236 --data_type FP16

I am trying to convert yolov5m / yolov5n / yolov5l for the NCS2 stick but these Conv_xxx layers do not exist and I fail to get it working.

Thanks!

[jgespino]

@maecky From the model optimzier:

--output OUTPUT       The name of the output operation of the model. For
                        TensorFlow*, do not add :0 to this name.The order of
                        outputs in converted model is the same as order of
                        specified operation names.

I am specifying the new outputs for this model, the ops/layer names can be found using a model visualizer like neutron.

You can find more information here: https://github.com/violet17/yolov5_demo#convert-onnx-file-to-ir-file

[Carson-Bay]

@Averen19 Looks like we missed uploading the OpenVINO 2022.1 package for Raspbian OS. I have reached out to my peers for assistance. In the meantime, you can download OpenVINO 2021.4.2 package for Rasbian OS. Please note that you need to match the OpenVINO version that is used to convert the model to IR and the runtime package installed on your pi. You can also build OpenVINO from source If you want to use the latest 2022.1 release on Raspberry Pi.

Let me know if you see the same error converting the model on 2021.4.2.

@jgespino Has the 2022.1 package been uploaded. I am having the same issues and have been trying to solve myself over the last few weeks. I got to the point of building everything from sources on Ubuntu 18.0.4 on the Pi and am still running into a bunch of errors.

[Carson-Bay]

@jgespino Just tried installing OpenVINO 2022.1 on Raspbian and came across this error.

`ERROR: Cannot install openvino-dev==2021.3.0, openvino-dev==2021.4.0, openvino-dev==2021.4.1, openvino-dev==2021.4.2 and openvino-dev==2022.1.0 because these package versions have conflicting dependencies.

The conflict is caused by:
openvino-dev 2022.1.0 depends on rawpy>=0.16.0
openvino-dev 2021.4.2 depends on openvino==2021.4.2
openvino-dev 2021.4.1 depends on rawpy>=0.16.0
openvino-dev 2021.4.0 depends on openvino==2021.4.0
openvino-dev 2021.3.0 depends on rawpy>=0.15

To fix this you could try to:

1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
   `

Please advise how to bypass these strict dependancies

[jgespino]

@Carson-Bay The OpenVINO for Raspbian OS package has not been uploaded yet. I recommend building the package from source. You can follow the instructions on this guide.

ERROR: Cannot install openvino-dev==2021.3.0, openvino-dev==2021.4.0, openvino-dev==2021.4.1, openvino-dev==2021.4.2 and openvino-dev==2022.1.0 because these package versions have conflicting dependencies.

The pip package is not compatible with Rasbpian OS, see the system requirements:

Supported Operating System	Python* Version (64-bit)
Ubuntu* 18.04 long-term support (LTS), 64-bit	3.6, 3.7, 3.8
Ubuntu* 20.04 long-term support (LTS), 64-bit	3.6, 3.7, 3.8, 3.9
Red Hat* Enterprise Linux* 8, 64-bit	3.6, 3.8
macOS* 10.15.x	3.6, 3.7, 3.8, 3.9
Windows 10*, 64-bit	3.6, 3.7, 3.8, 3.9

I am closing this discussion, please start a new issue if you need additional assistance.