Drastic bug on nDCG computation (method 0)

[tommasocarraro]

Hello everyone,
I found a critical bug in your nDCG (method 0) computation.

I was doing some experiments using the MindReader dataset. I implemented my version of nDCG. Even with a gradient-based Matrix Factorization model, I could not reach your results in terms of nDCG compared to your ALS Matrix Factorization.

Then, I went to your metrics.py file and discovered an issue with the nDCG (method 0) computation. The error is at line 42:

return r[0] + np.sum(r[1:] / np.log2(np.arange(2, r.size + 1)))

It should be:

return r[0] + np.sum(r[1:] / np.log2(np.arange(3, r.size + 2)))

According to wikipedia, this should be the right formula for the DCG:

As you can see, the LHS term is your method 1, while the RHS term is method 0. It is clear that the np.arange should begin from 3, not 2.

I hope this is helpful in some way. With this fix, I am now able to reach your results. The nDCG decreased to 0.17 for your ALS MF on the split 0 of the ntp-all-movies dataset.

[theisjendal]

Hi bmxitalia,

You are quite right. We actually took the code from elsewhere and after a bit of digging seems to be an error in multiple repositories.