Use custom Onnx model in place of Onnx model found on Object Detection sample

[masgh021]

I have a digit detection model in custom vision and it is detecting digit Very good at cloud https://www.customvision.ai/.
After i export it local (onnx) and i checked with ml.net
our accuracy decrease and it does not recognize object . in most images .
note : my objects are small , character size like ocr,
please help me.

[antoniovs1029]

Hi,

Can you share a .zip containing your Visual Studio project (code + onnx model + dataset)?

After i export it local (onnx) and i checked with ml.net our accuracy decrease

How did you evaluate the accuracy of the customvision model before evaluating it in ML.NET? did you use the same test dataset for doing the evaluation outside and inside ML.NET?

[masgh021]

@antoniovs1029
this is my model and 10 new image for test .

model address ==>> http://club.ses.co.ir/iter7ocrwin691pic.zip
10 image for test ==> http://club.ses.co.ir/10image.rar

i used this sample code .
https://github.com/dotnet/machinelearning-samples/tree/master/samples/csharp/end-to-end-apps/ObjectDetection-Onnx

in cloud this 10 images detected good
but in sample it doesn't work .

[antoniovs1029]

Hi, so if I understand correctly, you're simply replacing the onnx model found on the sample with your own onnx model without making changes to the code in the sample, is this correct?

If so, please notice that you would actually need to modify the sample to make it work for your own Onnx model, as each onnx model can have a different input/output. Looking into your onnx model, I see that it expects the same input as the onnx model on the sample, but the output of your onnx model is different.

For instance, as explained here in the sample, the onnx model on the sample detects 20 different classes of objects, and it encodes the output in a 125x13x13 output array. The classes that the sample works with were hard coded here:

https://github.com/dotnet/machinelearning-samples/blob/5021b7c8a6b7c3fe10ebe2070dc95b03cd69a72e/samples/csharp/end-to-end-apps/ObjectDetection-Onnx/OnnxObjectDetection/ML/DataModels/TinyYoloModel.cs#L10-L16

In your case, you'd need to modify the file above to match the labels you expect from your model. Your .zip containing your model, has also a Labels.txt file with the following text:

0
1
2
3
4
5
6
7
8
9

I believe this means your model was trained on 10 different classes, but there are no names for these classes, only digits (which I believe is what you want the model to detect, anyway).

So, for starters, you'd need to update the file I pointed above, to have these 10 labels. If that doesn't work, then there's another situation to consider: the onnx model from the sample outputs a Nonex125x13x13 array as seen bellow (where the None dimension is the batch dimension, so you can consider this 125x13x13):

Whereas your model has the following NonexNone2x13x13, which, ignoring the batch dimension, I think would be a Nonex13x13 array where the None means "variable length".

Because of this, it might be the case that your model doesn't encode its predictions in the same way as the model of the sample, and so you'll need to figure out how to further modify the sample to fit your scenario.

[justinormont]

@antoniovs1029: At runtime is there enough information to verify that the inputs/outputs exist and are the right shapes?

[masgh021]

@antoniovs1029
in sample project i checked label.txt file and it is correct (has 10 digit),
i am beginner in ml.net and
i dont know how change configuration for my model to work with this image in locall(my pc)
can you send me configuration ( sample code ) ??

[masgh021]

@antoniovs1029

see this is my result picture
my accuracy is baaaaad

https://pasteboard.co/JAkV70c.png

what is problem ??
please send configuration

[masgh021]

@antoniovs1029
@justinormont

please help i need this project .

[bknutson77]

I am having similar issues with an exported object detection from custom vision, definitely not just you @masgh021, I think this is happening to anyone who is exporting a model and trying to use it in this way. I've been through editing anchors and making any obviously needed changes, but my accuracy is also reduced. This issue needs to be elevated, @antoniovs1029 please work with us to find a solution, my email is benjk117@gmail.com if anyone would like to reach out.

[bknutson77]

I am able to see in the debugger that my model's output is a float array of size 30 x 13 x 13 = 5070 rather than the 125 x 13 x 13 = 21125 of the sample. What exactly to do about this.. I have no idea.

[cvixxt]

Hi all,
I also recently used Custom Vision's Onnx to export files, but under the C# WinForm I wrote, the output results are very different from the cloud results. The final output float[ 55x13x13 ] -> means that I have (6 various labels + 5 parameters) x Anchors(5) -> 55, but the predicted results are still very different. Will this question be updated later?

[michaelgsharp]

@cvixxt can you share a sample repro project so I can take a look? With the onnx model/code/data so we can get an easier repro. And also what results were you getting online so we know where the discrepancy is.

[cvixxt]

@cvixxt你可以分享一個示例複製項目，以便我看一下嗎？使用 onnx 模型/代碼/數據，我們可以更輕鬆地進行複制。還有你上網的結果是什麼，所以我們知道差異在哪裡。

Hi @michaelgsharp
See issue #6130. Thanks.

[michaelgsharp]

This issue has been resolved by issue #6130 and has the appropriate instructions there. If anyone else is still having issue after making the changes listed in #6130 please open a new issue to provide more info.