Metrics::f1

[muskuloes]

Hello,

Thanks for the package.

I know that the Metrics package is not implemented by you but only exported by your package. But I encountered an issue trying to run FeatureImp using the f1 metric.

The f1 metric from the Metrics package seems to be wrongly implemented. The first two lines are already an indication (unique), making the rest of the code also wrong.

library("iml")
Metrics::f1
# function (actual, predicted) 
# {
#     act <- unique(actual)
#     pred <- unique(predicted)
#     tp <- length(intersect(act, pred))
#     fp <- length(setdiff(pred, act))
#     fn <- length(setdiff(act, pred))
#     if (tp == 0) {
#         return(0)
#     }
#     else {
#         precision <- tp/(tp + fp)
#         recall <- tp/(tp + fn)
#         return(2 * precision * recall/(precision + recall))
#     }
# }
# <bytecode: 0x55fa9abdfd98>
# <environment: namespace:Metrics>

Reproducible example:

actual <- c(0,1,1,0,1,1)
# [1] 0 1 1 0 1 1
predicted <- c(0,1,1,0,0,1)
# [1] 0 1 1 0 0 1
Metrics::f1(actual, predicted)
# [1] 1

This is wrong as one would expect (for binary classification) with 1 as the true label, an F1 score of 0.86. If the positive label were 0 the F1 score would be 0.8

[christophM]

This is indeed an issue in the Metrics package then.
Did you open an issue there?

[pat-s]

Unfortunately {Metrics} has not seen a commit since March 2019 and it looks like it has been abandoned.

[muskuloes]

This is indeed an issue in the Metrics package then.
Did you open an issue there?

Hi,

no I haven't opened an issue in the Metrics package repository, I can do that too.

Thankfully the feature importance interface allows for custom losses, so we implemented our own f1 measure for binary classification.

[muskuloes]

Unfortunately {Metrics} has not seen a commit since March 2019 and it looks like it has been abandoned.

I created the issue in the Metrics package repo, and he replied pretty past. I also noticed that the package provides an fbeta_score function which works for binary classification with 0/1 labels, but not for multi-class classification (e.g., micro and macro F1).

So I think the documentation for Feature Importance can be updated stating more precisely to use fbeta_score for binary classification (with 0/1 labels) and may be a custom implementation for multi-class classification.