Metrics::f1

[muskuloes]

The f1 metric from the package seems to be wrongly implemented. The first two lines are already an indication (unique), making the rest of the code also wrong.

Metrics::f1
# function (actual, predicted) 
# {
#     act <- unique(actual)
#     pred <- unique(predicted)
#     tp <- length(intersect(act, pred))
#     fp <- length(setdiff(pred, act))
#     fn <- length(setdiff(act, pred))
#     if (tp == 0) {
#         return(0)
#     }
#     else {
#         precision <- tp/(tp + fp)
#         recall <- tp/(tp + fn)
#         return(2 * precision * recall/(precision + recall))
#     }
# }
# <bytecode: 0x55fa9abdfd98>
# <environment: namespace:Metrics>

Reproducible example:

actual <- c(0,1,1,0,1,1)
# [1] 0 1 1 0 1 1
predicted <- c(0,1,1,0,0,1)
# [1] 0 1 1 0 0 1
Metrics::f1(actual, predicted)
# [1] 1

This is wrong as one would expect (for binary classification) with 1 as the positive label, an F1 score of 0.86. If the positive label were 0 the F1 score would be 0.8.

[mfrasco]

Hi! Thank you for raising this issue. I agree that this function is confusing. In the documentation for the function, I explain that this is actually the f1 score in the context of information retrieval problems. It is not the f1 score in the context of binary classification problems. I recognize that the name f1 score is more commonly used in binary classification, which is why this function is confusing. I tried to make that clear in the documentation. What do you think about the documentation?

[muskuloes]

Hi,

Thanks for the quick reply. I have to admit that I didn't know exactly what "information retrieval context" meant. I was actually using this package indirectly from the iml package which uses it as a dependency.

Thanks for pointing that out and I also notice that you implemented an fbeta_score which works for my case for beta = 1.

As a suggestion, additional thing that would be great to see from the package would be an f1 score for multi-class cases (i.e., micro and macro cases).