NPMI scorer does not take into account min_count

[kikoaumond]

in phrases.py, npmi_scorer does not take into account min_count when scoring bigrams.

I suggest the following alternative for taking min_count into account when using NPMI scoring. This function has yielded good results for me; however, I have not explored other ways to use min_count:

# Custom scoring function to take into account min_counts when using npmi
def npmi_scorer_with_min_count(worda_count, wordb_count, bigram_count, len_vocab, min_count, corpus_word_count):

    if bigram_count < min_count:
        return -1

    pa = worda_count / corpus_word_count
    pb = wordb_count / corpus_word_count
    pab = bigram_count / corpus_word_count
    return log(pab / (pa * pb)) / -log(pab)

[EstevaoUyra]

The default score also does not takes into account properly. Your function sadly does not correct the problem. The behavior is very strange, I think the min_count should be implemented in the analyze_sentence instead of appearing in the score

[rafabr4]

In my tests, the default scorer does take into account the min_count parameter properly. I verified that by changing the parameter many times and looking at the length of the Phraser.phrasegrams dictionary. However, when using 'npmi' scorer, the length never changes by modifying the min_count, only with the threshold parameter.

Edit: after going through the phrases.py code I see why min_count is taken into account. The formula of the default scorer does (bigram_count - min_count) in the numerator. Bigrams with count less than min_count will always have a negative score. As long as the threshold supplied is positive, these will be filtered. This leads me to think that @kikoaumond solution should work, so long as the threshold supplied is greater than or equal to -1.

[lopusz]

A while ago, I also noticed that problem with NPMI and sumbmitted a pull request on that
#2072

[menshikh-iv]

Fixed in #2072