Add export_phrases function to Phrase model (TODO: cythonize the primitive loops)

[hanabi1224]

No description provided.

[hanabi1224]

The .h file under models may not be included in system C++ compiler,
This change is to avoid manually copying that file
But problem here is when models is imported in gensim.init, it starts compiling word2vec_inner before user call gensim.get_include (like numpy), so initialize pyximport in gensim.init or export get_include and stop importing models in gensim.init

[tmylk]

Please add tests and CHANGELOG for the new functionality. Thanks.

[gojomo]

The cython-related commits seem unrelated to the headline title?

[hanabi1224]

@tmylk @gojomo comments resolved, thanks!

[tmylk]

@gojomo Should this be merged?

[gojomo]

There's still two unrelated things here:

(1) Some sort of Windows-specific workaround for an issue with the cython code. I don't know who's been affected by that and have no way to evaluate if the change is a net improvement, since I don't build/use on Windows (and maybe this only affects some compilers/pythons, or it would have been an issue for others). It seems to me that change should get a separate PR.

(2) A utility method to yield just the phrases and their scores from a block of sentences (instead of the usual mix of phrase-ified content). I don't know the exact motivation for that need – was it a feature request? But, the method export_phrases() appears to be a straightforward way to dump that info, if someone needed it.

[hanabi1224]

@gojomo
for 1, I'm currently using windows and MSVC for python27 contains neither .h file under models nor under num.get_include out of box, what I originally thought was to expose an get_include method from gensim, however, models package is included in gensim.init, which directly triggers cython compile logic before caller can call gensim.get_include, and I don't think it's possible to remove models import in gensim.init (this is will make 'import gensim;gensim.models.Word2Vec' not working), and in such situation, the only work around is to copy .h file into compiler includes

for 2, yes, it is a feature request, I'd like to dump phrase, score pair to do some rough manual review

[piskvorky]

Re. item 1: we definitely don't want the pyximport parts, that's a regression. Gensim doesn't need cython (it ships with already-compiled C modules), and the dynamic pyximport doesn't work well in any case (it messes up imports of other, unrelated libraries that use cython; according to cython devs, it's an experimental/development feature, not meant for end use in libraries).

[hanabi1224]

@piskvorky good to know we have already-compiled C modules, How to make it work? Does below words in tutorials implies it will compile once during installation, what if user misses that point? Does he have to uninstall, make c compiler ready and install again? In my case, an on-demand runtime-compile does not seems to make things worse anyway :)

P.S. I have not tried gensim on linux yet, which may mitigate all the trouble I met on windows hopefully : )

Make sure you have a C compiler before installing gensim, to use optimized (compiled) word2vec training

[piskvorky]

Yes, the C code is compiled once, during gensim install. The install outputs a warning in case the compilation fails, notifying the user that some functionality will be slow as a result.

[hanabi1224]

@piskvorky got it, how about the feature request of export_phrases? If feasible, I can revert cython related change

[piskvorky]

Sure, that sounds good.

Incidentally, the phrases module is a great adept at cythonization = rewriting the primitive loops in cython, compiling an (optional) C module so that it all runs faster.

[tmylk]

@hanabi1224 Thanks for the PR!