Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML analysis] Support independent classifier analysis #214

Open
nathanielrindlaub opened this issue Jun 12, 2024 · 0 comments
Open

[ML analysis] Support independent classifier analysis #214

nathanielrindlaub opened this issue Jun 12, 2024 · 0 comments
Labels

Comments

@nathanielrindlaub
Copy link
Member

nathanielrindlaub commented Jun 12, 2024

Right now both scripts/analyzeMLObjectLevel.js and scripts/analyzeMLSequenceLevel.js can analyze Megadetector independently, or they can analyze the results of using both Megadetector and a classifier together.

If assessing the latter, a true positive would mean that (a) the object detector correctly identified the object, and (b) the classifier correctly identified the class. So a false negative could mean that the object detector correctly identified the object, but the classifier incorrectly identified the class.

This is fine if you're interested in assessing how a pipeline is performing end-to-end, i.e. to answer questions like, "If a rodent trips a camera, how likely is it that both my object detector and classifier model will work as expected and I will receive an alert"?

However, it's not as useful for assessing the performance of the classifier independently from the object detector. To analyze classifier performance independently, you'd need to filter the objects we evaluate to only include those that had the Automation Rule conditions that they needed to meet to get run through the classifier. In most cases this would jut mean that the object had a megadetector prediction of an "animal".

@nathanielrindlaub nathanielrindlaub changed the title Support independent classifier analysis [ML analysis] Support independent classifier analysis Jun 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant