Bug in bm25 implementation.

[se4u]

In line 52 of bm25.py the variable self.corpus_size should be replaced by len(document).

https://github.com/RaRe-Technologies/gensim/blob/82c394a9085d583e8a75c2bb32ecd37cf61236f0/gensim/summarization/bm25.py#L52

[menshikh-iv]

Nice catch @se4u, thanks for the report!

[souravsingh]

@se4u Can you tell us how the bug affects the execution of the summarization module? A small code sample reproducing the bug would be nice.

[se4u]

@sourav : Actually, I used the bm25 module independently of the summarization module just for IR. Basically, I wanted to quickly compare bm25 with tf-idf, pylucene's api seemed to be too cumbersome, and I did not want to reinvent the wheel. Gensim generally has pretty good coding standards that's why I decided to use bm25 from gensim instead of the various other BM25 implementations on github. But after implementation I noticed that the results from bm25 were worse than standard tf-idf and that's why I looked at the code in more detail. The data that I am working on comes from [https://www.ldc.upenn.edu ](LDC) so its hard to share it.

…

On Sat, Nov 18, 2017 at 11:37 AM, Sourav Singh ***@***.***> wrote: @se4u <https://github.com/se4u> Can you tell us how the bug affects the execution of the summarization module? A small code sample reproducing the bug would be nice. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1718 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACMdtH60tauzndGppW-36a6fMV4Jfvrtks5s3wfagaJpZM4QfXkL> .