Is there a way to get the accuracy of this chatterBot

[csrgxtu]

In Machine Learning area, there always a test data set used for get the accuracy of the model, so, is there any similiar thing for ChatterBot?

[gunthercox]

ChatterBot uses logic adapters, which are essentially just modules that take input and return a response. Each response that is returned has a confidence value associated with it. The confidence value is a numeric indicator of how accurate the logic adapter thinks the response is.

A confidence score is kind of a metric for accuracy but it probably better reflects the breadth of a particular bot's knowledge than the accuracy of it's responses. Arguably a chat bot that responds more confidently may also be more accurate, but that is based on the assumption that the set of logic adapters the bot uses are all completely accurate in their ability to judge the precision of the response they generate.

Because ChatterBot's logic adapters each encompass a modular process for selecting a response, there isn't necessarily a common way to gauge the accuracy of all of them. On an individual basis this might be possible but I don't have anything in the code base right now for doing a full evaluation of accuracy.

Please let me know if I can clarify anything.

[kennetham]

@gunthercox how do I get the confidence level from get_response? Is there any way to get the confidence level from a statement?

[gunthercox]

Currently get_response does not return the confidence value. I can see how this would be useful and I would be happy to add it as a feature in one of the upcoming releases of ChatterBot.

[kennetham]

Cool. I am using a quick and dirty method now, I'm not sure if it was meant to be used this way.

from chatterbot.conversation import Statement
# initialization
...

statement = Statement(message)
confidence_level = bot.logic.process(statement)[0]
response = bot.logic.process(statement)[1].text

[gunthercox]

Hi @csrgxtu I've opened #565 which makes a change so that the confidence score can be accessed on the returned statement object. (response.confidence)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.