Make most_similar accept topn of any integer type

[Witiko]

This PR continues #2356 and #2461 and closes #2496. The PR makes the following changes:

• d910fa9: This change is unrelated to SparseTermSimilarityMatrix - TypeError: 'numpy.float32' object is not iterable #2496, but it fixes one implementation of most_similar (PoincareKeyedVectors) that I missed in Fix WordEmbeddingsKeyedVectors.most_similar #2461.
• b34b06e: This change closes SparseTermSimilarityMatrix - TypeError: 'numpy.float32' object is not iterable #2496 by allowing any subtype of numbers.Integral to be accepted as the parameter topn in most_similar. This change also adds a unit test.
• b588bd7: This change is an additional defensive fix of SparseTermSimilarityMatrix - TypeError: 'numpy.float32' object is not iterable #2496 that patches the caller of most_similar not to use topn=0. This is because there are many implementations of most_similar and expecting all the current and future implementations to uphold the contract seems naive.
• 83455e2: This change makes the variable notation of the SparseTermSimilarityMatrix consistent and prevents a possible integer overflow.

[Witiko]

I tried to force-push a couple of times, but AppVeynor consistently fails to install dependencies.

[piskvorky]

CC @mpenkov – can you please assist Vitek with the CI (AppVeyor) problems?

[mpenkov]

Looks like appveyor is failing to install numpy under Py2:

distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('numpy<2.0.0,>=1.9.0')

@menshikh-iv Any idea why that could be?

On a related note, numpy will be dropping Py2 support in less than a year (https://www.numpy.org/neps/nep-0014-dropping-python2.7-proposal.html)... @piskvorky What does that mean for gensim and our plans for Py2 support in the future?

[menshikh-iv]

@mpenkov

Thanks to pyemd https://github.com/wmayner/pyemd/blob/develop/setup.py#L95
I recommend do not install it on windows with python 2 (just avoid this problem, if we are lucky, related tests will be skip).

upd: as an alternative way (if you looking to link above) - install numpy first (in separate command BEFORE pyemd install), this can help, but also can produce new issue (need to check).

[piskvorky]

@mpenkov I don't think it means anything for us. We're not using any functionality from the newer NumPy releases. That pyemd dependency is marginal, we could drop it altogether if it gives us trouble.

In other words, we can continue to require a fairly old version of NumPy (the latest that supports py2?). But users are free to install newer versions of NumPy too — we don't care.

[Witiko]

Please note that if we drop the pyemd dependency, 12 tests will be skipped.

[Witiko]

I removed pyemd from win_testenv in setup.py, but this does not appear to solve the issue. The issue could be with numpy, scipy, or any of the packages in .[test-win], i.e. in win_testenv:

deps =
    py37: numpy==1.14.5
    py37: scipy==1.1.0

    py27: numpy==1.11.3
    py27: scipy==0.18.1
    py35: numpy==1.11.3
    py35: scipy==0.18.1
    py36: numpy==1.11.3
    py36: scipy==0.18.1

    linux: .[test]
    win: .[test-win]

    extras_require={
        'distributed': distributed_env,
        'test-win': win_testenv,

win_testenv = [
    'pytest',
    'pytest-rerunfailures',
    'mock',
    'cython',
    'testfixtures',
    'scikit-learn',
    'Morfessor==2.0.2a4',
    'python-Levenshtein >= 0.10.2',
    'visdom >= 0.1.8, != 0.1.8.7',
]

[menshikh-iv]

So, now the issue with scikit-learn, because they don't support python2 more
Possible solutions

• use an older version of scikit-learn for windows (now: https://github.com/scikit-learn/scikit-learn/blob/master/setup.py#L231, one version before - https://github.com/scikit-learn/scikit-learn/blob/debian/0.17.0-4/setup.py)
• get rid sklearn from python2+win
• get rid python2 support from gensim (it's time)

[Witiko]

I reverted the pyemd removal, so that we can solve this general issue separately.

[mpenkov]

@menshikh-iv I'm +1 for removing Py2 support. We could do this in the next major release.

If people are keen to continue using Py2, then that's fine, they can keep using gensim 3.x. @piskvorky WDYT?

[piskvorky]

Yes, next major release (4.x) is OK. No clear release date nor milestones for 4.x though, so this is theoretical for now. Getting Gensim into a better shape is more important (pyemd, sklearn are largely irrelevant to Gensim; we do need numpy though).

Sorry for hijacking your PR @Witiko :)

[Witiko]

I don't mind, dropping the Python 2 support is an interesting topic. Please, let me know if I can help, or when I can rerun the tests for this PR.

[mpenkov]

@piskvorky @Witiko I fixed the AppVeyor problems. Is this ready to merge?

[Witiko]

@mpenkov Great work! Yes, this PR should be ready to merge.

[piskvorky]

@mpenkov ping. After fixing smart_open, perhaps we can make a bugfix release? Anything else you want in? Maybe remove that stillborn NMF code?

[mpenkov]

OK, merged. @Witiko Thank you for your contribution and your patience.

@piskvorky What do you think about the scope of this change? This seems to be new functionality: does it warrant a minor version bump?

[piskvorky]

Looking at the PR description: this looks like a textbook bug fix.

Which part do you see as substantial?

[mpenkov]

I think I got it mixed up with another PR. We'll treat this as a bugfix.