-
-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't import gensim library - Python 3.8.5 + numpy 1.20.2 #3097
Comments
Thanks for reporting. That's weird – my understanding was that upgrading numpy should help. Related to #3095 and numpy/numpy#16938. @mpenkov could you have a look please? This Numpy's C API incompatibility fuckup seems critical, will hit a large percentage of Gensim users. |
@norbertgieruc What version of numpy did you upgrade to? Also, what O/S? |
Current: If you are asking what Numpy version were working previously I don't remember. I created that code like 1 year ago and just wanted to rerun it now and failed. |
@piskvorky Something really strange is happening with numpy. In this ticket, the user is still unable to import gensim even after upgrading numpy to 1.20.2, when using Python 3.8.5 on Windows. For that configuration, Appveyor built the wheels using numpy 1.20.1 (see https://ci.appveyor.com/project/piskvorky/gensim-wheels-2x1bk/build/job/7bhvjbjhcr9mivx7#L1372). If numpy version 1.20.2 is incompatible with wheels built against 1.20.1, then we're in trouble. |
Indeed. I'll open a ticket at numpy, but can you clarify one thing for me please? How come the wheels used numpy 1.20.1, when you switched to using Because if we're building against the latest version, that's against the official numpy recommendation, so I guess problems are to be expected. BTW, Numpy have a nice signpost page for new issues: https://github.com/numpy/numpy/issues/new/choose |
It could be a bug in oldest-supported-numpy. Its choice of versions for Windows builds is quite strange:
|
No, wait, the problem is that Windows builds happen using Appveyor, not Travis CI, and that uses a different build mechanism. This mechanism isn't picking up oldest-supported-numpy. I'll build new wheels and do a bugfix release. That should fix the wheels for Windows users (but, as I mentioned above, I'm not sure that will fix the ImportError that OP is having). Have non-Windows users reported similar problems? |
Oh crap, I just submitted the Numpy ticket :) |
I think the Numpy ticket is still valid. We built the wheel using 1.20.1, and the user is unable to use it with numpy 1.20.2 installed. Is that expected? We built with an older version than what they have, so things should work, right? |
I think so. @norbertgieruc how did you install Gensim? Was it a standard Because I see some |
@piskvorky Yes, it was standard pip install via anaconda prompt. |
We're also experiencing this error in one of our projects that uses gensim, but I don't think your lib is the issue here. Our source of the problem is coming from a scikit-learn package: hdbscan (https://github.com/scikit-learn-contrib/hdbscan) Problem occurs in both Windows (python 3.8.8) and macOS (python 3.8.5). I'm not able to disclose the full stack, but the snippet that matters: File "C:\Users\Brent\AppData\Local\Programs\Python\Python38\lib\site-packages\hdbscan\__init__.py", line 1, in <module>
from .hdbscan_ import HDBSCAN, hdbscan
File "C:\Users\Brent\AppData\Local\Programs\Python\Python38\lib\site-packages\hdbscan\hdbscan_.py", line 21, in <module>
from ._hdbscan_linkage import (single_linkage,
File "hdbscan/_hdbscan_linkage.pyx", line 1, in init hdbscan._hdbscan_linkage
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject In our particular case, we're also using tensorflow 2.4.1, which fails when using numpy 1.20.x, so we seem to be in a difficult spot atm. Dependencies are fun, right? |
@llunn it's possible the issue with HDBScan is a separate one. Do they use |
@piskvorky honestly I'm not sure what HDBScan does or how to check on that |
@llunn Please mention the numpy versions for each of your environments (Windows/Mac, Python version, etc) that exhibit the problem. I'm trying to pin down the cause, seeing a matrix with what works and what doesn't would be helpful. |
@mpenkov Full disclosure here, our project is unlikely be a good case study to use. We have had to drop our gensim version to < Completely recognize that our problem is completely outside of your area of concern and it is almost certainly unrelated to gensim. That being said, I'm not sure any of the information below can be considered relevant, so I defer to your expertise in this project and provide it for whatever value it might have. We tried today:
Based on the response over at NumPy, I don't find it surprising that none of these work since the hdbscan import is surely using the same numpy version in their pyc. What follows is unrelated to the gensim project, but for awareness of how I joined this conversation:
** Edit note** |
Seems fixed now with Gensim 4.0.1. @llunn I'd recommend you urge your dependencies (top2vec etc) to upgrade using the Gensim 4 Migration Guide. Because Gensim 4 is miles ahead of 3.8 in terms of performance and memory, and also fixed a number of important bugs. |
@norbertgieruc Can you please try gensim 4.0.1 and let us know whether the problem still persists? |
gensim 4.0.1 fails on python 3.9.5 with the same error when installed with |
Hm. Maybe something to do with the fact we don't distribute Python 3.9 wheels yet? Although I don't see how that could affect whether you use |
Did some testing. Changing packages order in Reproducing with docker: Works python3.8:
Works python3.9, install after numpy:
Works installing w/ numpy==1.20:
Doesn't work installing with numpy==1.19 on py3.9:
Installing without
|
I still have a compatibility issue when running : pip install numpy==1.19.4 pip install tensorflow=2.5.0 pip install gensim==4.0.1 On Ubuntu 18.04, with Python 3.9.5 (installs made inside docker container). I get the following exception when trying to import gensim: ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject. Not sure how i can make this work, I tried downgrading several libraries but still not achieved to make it work in Ubuntu. Edit : works on Python 3.8.10 |
Hi all - We came across the same issue using gensim 3.8.3 and numpy 1.22.2 I removed numpy from |
Gensim 3.x is not supported at this point – please upgrade to the latest version (4.1.2 currently). If something doesn't work there, report here. Thanks. |
…ompatibility. Expected 88 from C header, got 80 from PyObject See Similar to piskvorky/gensim#3097
…ompatibility. Expected 88 from C header, got 80 from PyObject (#61) See Similar to piskvorky/gensim#3097
I had the same error on Python 3.8.8 and gensim 4.2.0 on Jupyter Notebook. I fixed by upgrading Numpy to 1.22.4 and restarting run time. |
I am on Python 3.10.7, gensim 4.3.1, and Numpy 1.24.1 getting this error when running |
Hi there, I am using poetry, and installing the latest gensim forced a downgrade to numpy 1.26.4 from 2.1.2 This breaks some other things that were expecting numpy 2 (of course it's the other offending package for not properly specifying only numpy 2) but still, should gensim be officially depending on numpy 2 or not? |
Hi,
I was trying to run script that I created some time ago on Python 3.6.5 and it seems to don't work anymore. I can't import the library. Upgrading numpy didn't help. Can I ask for a solution to this problem?
Best regards,
Norbert
The text was updated successfully, but these errors were encountered: