[ASK] True/False positives on training data

[simonmandlik]

Description

Hi, I was trying to understand how cornac uses ranking metrics and after reading the code it seems to me that given a specific user, all items from the test set are ranked (regardless of whether (user, item) pair is in the train or test set), and the TPs on training samples are later evaluated as FPs.

If I evaluate, say, Precision@K, and the model ranks K positive training items first, which is a perfect ranking, evaluation considers these samples as FPs and computes Precision@K as 0.

Why are actual TPs on training samples considered FPs and why is the evaluation not done only using unseen (user, item) pairs?

Could anyone please clarify this for me?

Other Comments

To be more specific, here is ut_gt_pos filled only with testing positive samples, training positive samples are ignored. But the length of ut_gt_pos are all testing items, which in my case (using RatioSplit) is all items in the dataset. This means that all highly ranked items, which appeared in the train set, have ground truth label of zero. Wouldn't it be better to ignore all items that user has seen during training here?