annoy.py conversion of cosine distance to cosine similarity is incorrect

[monash8]

in this function the code to calculate cosine similarity is incorrect

def most_similar(self, vector, num_neighbors):
    """Find `num_neighbors` most similar items.

    Parameters
    ----------
    vector : numpy.array
        Vector for word/document.
    num_neighbors : int
        Number of most similar items

    Returns
    -------
    list of (str, float)
        List of most similar items in format [(`item`, `cosine_distance`), ... ]

    """
    ids, distances = self.index.get_nns_by_vector(
        vector, num_neighbors, include_distances=True)

    return [(self.labels[ids[i]], 1 - distances[i] / 2) for i in range(len(ids))]

according to annoy documentation get_nns_by_vector with include_distances=True will return the distances and not the square power of the distance (this was changed since aug 2016):

a.get_distance(i, j) returns the distance between items i and j. NOTE: this used to return the squared distance, but has been changed as of Aug 2016.

link

also:
Annoy uses Euclidean distance of normalized vectors for its angular distance, which for two vectors u,v is equal to sqrt(2(1-cos(u,v)))
link

so this means that in order to calculate the cosine similarity correctly we should do this:
return [(self.labels[ids[i]], 1 - distances[i]^2 / 2) for i in range(len(ids))]

[gojomo]

As this Gensim code was last touched July 2016, & the Annoy library says it changed its behavior August 2016, I can see how this is out-of-sync.

An effect would be that the max distance returned by get_nns_by_vector() of 2.0 will be turned into a cosine-similarity of 0.0 instead of the correct -1.0 – squashed range of returned values. (However, in no case will the relative ranking of nearest-neighbors be incorrect – which may have contributed to this going unnnoticed for so long.)

The proposed fix looks correct.

[piskvorky]

Thanks @monash8 for the report and investigation. And @gojomo for confirming.

@monash8 are you able to open a PR with the fix?

[monash8]

Thanks @monash8 for the report and investigation. And @gojomo for confirming.

@monash8 are you able to open a PR with the fix?

sure! @piskvorky can you give me access to create a new branch?

[piskvorky]

@monash8 please create a branch in your fork & open a PR again gensim's develop branch from there.