Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Library importation bug in GCloud VM #29

Closed
ProfLeao opened this issue May 27, 2022 · 11 comments
Closed

Library importation bug in GCloud VM #29

ProfLeao opened this issue May 27, 2022 · 11 comments

Comments

@ProfLeao
Copy link

I've been experiencing the following error when importing the library when using it in a gCloud VM:

Python 3.9.12 (main, Apr  5 2022, 06:56:58) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import cantera
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 import cantera

File ~/anaconda3/envs/ct-env/lib/python3.9/site-packages/cantera/__init__.py:4, in <module>
      1 # This file is part of Cantera. See License.txt in the top-level directory or
      2 # at https://cantera.org/license.txt for license and copyright information.
----> 4 from ._cantera import *
      5 from ._cantera import __version__, __sundials_version__, __git_commit__
      6 from .composite import *

ImportError: libmkl_rt.so.2: cannot open shared object file: No such file or directory

The enviroment has following setup:

# packages in environment at /home/reginaldo/anaconda3/envs/ct-env:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
argon2-cffi               21.3.0             pyhd3eb1b0_0  
argon2-cffi-bindings      21.2.0           py39h7f8727e_0  
asttokens                 2.0.5              pyhd3eb1b0_0  
attrs                     21.4.0             pyhd3eb1b0_0  
backcall                  0.2.0              pyhd3eb1b0_0  
beautifulsoup4            4.11.1           py39h06a4308_0  
blas                      1.0                         mkl  
bleach                    4.1.0              pyhd3eb1b0_0  
brotli                    1.0.9                he6710b0_2  
ca-certificates           2022.4.26            h06a4308_0  
cantera                   2.6.0            py39h9a426be_0    cantera
certifi                   2022.5.18.1      py39h06a4308_0  
cffi                      1.15.0           py39hd667e15_1  
cycler                    0.11.0             pyhd3eb1b0_0  
dbus                      1.13.18              hb2f20db_0  
debugpy                   1.5.1            py39h295c915_0  
decorator                 5.1.1              pyhd3eb1b0_0  
defusedxml                0.7.1              pyhd3eb1b0_0  
entrypoints               0.4              py39h06a4308_0  
executing                 0.8.3              pyhd3eb1b0_0  
expat                     2.4.4                h295c915_0  
fontconfig                2.13.1               h6c09931_0  
fonttools                 4.25.0             pyhd3eb1b0_0  
freetype                  2.11.0               h70c0345_0  
giflib                    5.2.1                h7b6447c_0  
glib                      2.69.1               h4ff587b_1  
gst-plugins-base          1.14.0               h8213a91_2  
gstreamer                 1.14.0               h28cd5cc_2  
h5py                      3.6.0            py39ha0f2276_0  
hdf5                      1.10.6               hb1b8bf9_0  
icu                       58.2                 he6710b0_3  
intel-openmp              2021.4.0          h06a4308_3561  
ipykernel                 6.9.1            py39h06a4308_0  
ipython                   8.3.0            py39h06a4308_0  
ipython_genutils          0.2.0              pyhd3eb1b0_1  
ipywidgets                7.6.5              pyhd3eb1b0_1  
jedi                      0.18.1           py39h06a4308_1  
jinja2                    3.0.3              pyhd3eb1b0_0  
jpeg                      9e                   h7f8727e_0  
jsonschema                4.4.0            py39h06a4308_0  
jupyter                   1.0.0            py39h06a4308_7  
jupyter_client            7.2.2            py39h06a4308_0  
jupyter_console           6.4.3              pyhd3eb1b0_0  
jupyter_core              4.10.0           py39h06a4308_0  
jupyterlab_pygments       0.1.2                      py_0  
jupyterlab_widgets        1.0.0              pyhd3eb1b0_1  
kiwisolver                1.4.2            py39h295c915_0  
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.38                 h1181459_1  
libcantera                2.6.0                h3fd9d12_0    cantera
libffi                    3.3                  he6710b0_2  
libgcc-ng                 11.2.0               h1234567_0  
libgfortran-ng            7.5.0               ha8ba4b0_17  
libgfortran4              7.5.0               ha8ba4b0_17  
libgomp                   11.2.0               h1234567_0  
libpng                    1.6.37               hbc83047_0  
libsodium                 1.0.18               h7b6447c_0  
libstdcxx-ng              11.2.0               h1234567_0  
libtiff                   4.2.0                h2818925_1  
libuuid                   1.0.3                h7f8727e_2  
libwebp                   1.2.2                h55f646e_0  
libwebp-base              1.2.2                h7f8727e_0  
libxcb                    1.15                 h7f8727e_0  
libxml2                   2.9.14               h74e7548_0  
lz4-c                     1.9.3                h295c915_1  
markupsafe                2.0.1            py39h27cfd23_0  
matplotlib                3.5.1            py39h06a4308_1  
matplotlib-base           3.5.1            py39ha18d171_1  
matplotlib-inline         0.1.2              pyhd3eb1b0_2  
mistune                   0.8.4           py39h27cfd23_1000  
mkl                       2021.4.0           h06a4308_640  
mkl-service               2.4.0            py39h7f8727e_0  
mkl_fft                   1.3.1            py39hd3c417c_0  
mkl_random                1.2.2            py39h51133e4_0  
munkres                   1.1.4                      py_0  
nbclient                  0.5.13           py39h06a4308_0  
nbconvert                 6.4.4            py39h06a4308_0  
nbformat                  5.3.0            py39h06a4308_0  
ncurses                   6.3                  h7f8727e_2  
nest-asyncio              1.5.5            py39h06a4308_0  
notebook                  6.4.11           py39h06a4308_0  
numpy                     1.22.3           py39he7a7128_0  
numpy-base                1.22.3           py39hf524024_0  
openssl                   1.1.1o               h7f8727e_0  
packaging                 21.3               pyhd3eb1b0_0  
pandocfilters             1.5.0              pyhd3eb1b0_0  
parso                     0.8.3              pyhd3eb1b0_0  
pcre                      8.45                 h295c915_0  
pexpect                   4.8.0              pyhd3eb1b0_3  
pickleshare               0.7.5           pyhd3eb1b0_1003  
pillow                    9.0.1            py39h22f2fdc_0  
pip                       21.2.4           py39h06a4308_0  
prometheus_client         0.13.1             pyhd3eb1b0_0  
prompt-toolkit            3.0.20             pyhd3eb1b0_0  
prompt_toolkit            3.0.20               hd3eb1b0_0  
ptyprocess                0.7.0              pyhd3eb1b0_2  
pure_eval                 0.2.2              pyhd3eb1b0_0  
pycparser                 2.21               pyhd3eb1b0_0  
pygments                  2.11.2             pyhd3eb1b0_0  
pyparsing                 3.0.4              pyhd3eb1b0_0  
pyqt                      5.9.2            py39h2531618_6  
pyrsistent                0.18.0           py39heee7806_0  
python                    3.9.12               h12debd9_0  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python-fastjsonschema     2.15.1             pyhd3eb1b0_0  
pyzmq                     22.3.0           py39h295c915_2  
qt                        5.9.7                h5867ecd_1  
qtconsole                 5.3.0              pyhd3eb1b0_0  
qtpy                      2.0.1              pyhd3eb1b0_0  
readline                  8.1.2                h7f8727e_1  
ruamel.yaml               0.16.12          py39h27cfd23_1  
ruamel.yaml.clib          0.2.6            py39h7f8727e_0  
send2trash                1.8.0              pyhd3eb1b0_1  
setuptools                61.2.0           py39h06a4308_0  
sip                       4.19.13          py39h295c915_0  
six                       1.16.0             pyhd3eb1b0_1  
soupsieve                 2.3.1              pyhd3eb1b0_0  
sqlite                    3.38.3               hc218d9a_0  
stack_data                0.2.0              pyhd3eb1b0_0  
terminado                 0.13.1           py39h06a4308_0  
testpath                  0.5.0              pyhd3eb1b0_0  
tk                        8.6.11               h1ccaba5_1  
tornado                   6.1              py39h27cfd23_0  
traitlets                 5.1.1              pyhd3eb1b0_0  
typing-extensions         4.1.1                hd3eb1b0_0  
typing_extensions         4.1.1              pyh06a4308_0  
tzdata                    2022a                hda174b7_0  
wcwidth                   0.2.5              pyhd3eb1b0_0  
webencodings              0.5.1            py39h06a4308_1  
wheel                     0.37.1             pyhd3eb1b0_0  
widgetsnbextension        3.5.2            py39h06a4308_0  
xz                        5.2.5                h7f8727e_1  
zeromq                    4.3.4                h2531618_0  
zlib                      1.2.12               h7f8727e_2  
zstd                      1.5.2                ha4553b6_0

The OS is 20.04.1-Ubuntu.

I appreciate any help.
@ischoegl
Copy link
Member

Hi @ProfLeao ... thank you for reporting. As a workaround, you can install cantera from the conda-forge channel. I am also transferring this over to the repository that handles conda compilations.

@ischoegl ischoegl transferred this issue from Cantera/cantera May 27, 2022
@ProfLeao
Copy link
Author

Thanks @ischoegl Worked perfectly.

@speth
Copy link
Member

speth commented May 31, 2022

I was able to replicate this problem on Ubuntu 20.04. It seems the problem is that the Cantera extension module is linked specifically to libmkl_rt.so.2, but the mkl package with version 2021.4.0 provides libmkl_rt.so.1. I tried installing mkl version 2022.0.1 (which contains libmkl_rt.so.2), but it failed with the message:

package mkl_random-1.2.2-py310h00e6091_0 requires mkl >=2021.4.0,<2022.0a0, but none of the providers can be installed

I looked and saw that all the Python packages installed in my conda environment explicitly specify libmkl_rt.so.1, so I'm not really sure how the transition between these ABI versions is supposed to work. I'm wondering if we need to rebuild with mkl pinned to <2022.0a0.

@ischoegl
Copy link
Member

Fwiw, I ran into the exact same issues for

conda install -c cantera/label/dev cantera

on Fedora. The dev version (still 2.6.0b2) is obsolete, so I didn’t investigate further.

@bryanwweber
Copy link
Member

It's not clear to me how the tests can pass on CI and this doesn't work on user machines. I guess Conda is solving for the 2022 version of mkl on CI but 2021 ends up in the user environment... Do you have a sense for what requires mkl_random, since that seems to be the immediate culprit here?

@speth
Copy link
Member

speth commented May 31, 2022

It's required by numpy. Here is the info for the version of numpy that currently gets installed into a clean environment with Cantera:

numpy 1.22.3 py310hfa59a62_0
----------------------------
file name   : numpy-1.22.3-py310hfa59a62_0.conda
name        : numpy
version     : 1.22.3
build       : py310hfa59a62_0
build number: 0
size        : 10 KB
license     : BSD-3-Clause
subdir      : linux-64
url         : https://repo.anaconda.com/pkgs/main/linux-64/numpy-1.22.3-py310hfa59a62_0.conda
md5         : df1e8183d83e07f2c074d02a856d9757
timestamp   : 2022-05-17 15:50:38 UTC
dependencies: 
  - blas 1.0 mkl
  - libgcc-ng >=7.5.0
  - libstdcxx-ng >=7.5.0
  - mkl >=2021.4.0,<2022.0a0
  - mkl-service >=2.3.0,<3.0a0
  - mkl_fft
  - mkl_random
  - numpy-base 1.22.3 py310h9585f30_0
  - python >=3.10,<3.11.0a0

Note the explicit version limits on MKL.

The version that's being used for at least some of the conda builds is:

numpy 1.21.5 py310h4f1e569_2
----------------------------
file name   : numpy-1.21.5-py310h4f1e569_2.conda
name        : numpy
version     : 1.21.5
build       : py310h4f1e569_2
build number: 2
size        : 10 KB
license     : BSD-3-Clause
subdir      : linux-64
url         : https://repo.anaconda.com/pkgs/main/linux-64/numpy-1.21.5-py310h4f1e569_2.conda
md5         : 0aa8e5ef5ac3640325d58f79a457265d
timestamp   : 2022-05-03 07:44:35 UTC
dependencies: 
  - blas * openblas
  - libgcc-ng >=7.5.0
  - libopenblas >=0.3.3,<1.0a0
  - libstdcxx-ng >=7.5.0
  - numpy-base 1.21.5 py310hf2716ce_2
  - python >=3.10,<3.11.0a0

Which doesn't block MKL 2022.0.1, and that's what gets installed.

@bryanwweber bryanwweber mentioned this issue Jun 6, 2022
Merged
5 tasks
@dylanrubini dylanrubini mentioned this issue Jul 28, 2022
Closed
@ischoegl ischoegl mentioned this issue Aug 9, 2022
Merged
@speth
Copy link
Member

speth commented Aug 9, 2022

Is there a way for us to just depend on a shim library that links to either OpenBLAS or MKL, and doesn't require a specific version of either? The only functions we use are just standard LAPACK routines that are available in any implementation. I don't think it should be necessary for us have different packages for different versions of these libraries.

@bryanwweber
Copy link
Member

blas is that shim library, but we need to build separate packages for each BLAS system because the linking/ABI is different, at least to my understanding. Conda-forge has sorted a lot of this stuff out, since they mostly link to OpenBlas. It's just a matter of figuring out the right incantation to get it working.

@ischoegl
Copy link
Member

ischoegl commented Aug 9, 2022

@bryanwweber … good to know. I’ll pass on this one as I got stuck with MKL before (Cantera/cantera#1316) … fwiw, #33 temporarily disables MKL, although it should definitely be enabled again.

ischoegl added a commit to ischoegl/conda-recipes that referenced this issue Aug 10, 2022
@ischoegl
Disable MKL
e8aa1be 
MKL is disabled as a temporary fix of Cantera#29 to resolve Cantera#31. It should be
reactivated in a permanent fix.
bryanwweber pushed a commit that referenced this issue Aug 10, 2022
@ischoegl
Prevent installation of cantera from PyPI (#33)
68448f7 
* Prevent installation of cantera from PyPI

* Disable MKL

MKL is disabled as a temporary fix of #29 to resolve #31. It should be
reactivated in a permanent fix.

* Add yaml2ck script

* Update pytest

Starting with Cantera 3.0, the Python test suite is no longer packaged
with the Python module. This commit ensures that conda is still able
to run the tests.

* Replace deprecated SCons option

* Include share/cantera/data in libcantera
@bryanwweber bryanwweber mentioned this issue Aug 23, 2022
Closed
@ischoegl ischoegl mentioned this issue Sep 11, 2022
Closed
@bryanwweber bryanwweber mentioned this issue Sep 16, 2022
Closed
@speth
Copy link
Member

speth commented Aug 15, 2023

Recent builds of Numpy seem to have separate builds for different MKL versions. Partial output of mamba search --info -c defaults numpy=1.23.5 shows:

numpy 1.23.5 py310h5f9d8c6_1
----------------------------
file name   : numpy-1.23.5-py310h5f9d8c6_1.conda
name        : numpy
version     : 1.23.5
build       : py310h5f9d8c6_1
build number: 1
size        : 10 KB
license     : BSD-3-Clause
subdir      : linux-64
url         : https://repo.anaconda.com/pkgs/main/linux-64/numpy-1.23.5-py310h5f9d8c6_1.conda
md5         : 875377e83010e75870affa1797444ce7
timestamp   : 2023-05-01 15:03:28 UTC
dependencies: 
  - blas 1.0 mkl
  - libgcc-ng >=11.2.0
  - libstdcxx-ng >=11.2.0
  - mkl >=2023.1.0,<2024.0a0
  - mkl-service >=2.3.0,<3.0a0
  - mkl_fft
  - mkl_random
  - numpy-base 1.23.5 py310hb5e798b_1
  - python >=3.10,<3.11.0a0

numpy 1.23.5 py310hd5efca6_0
----------------------------
file name   : numpy-1.23.5-py310hd5efca6_0.conda
name        : numpy
version     : 1.23.5
build       : py310hd5efca6_0
build number: 0
size        : 10 KB
license     : BSD-3-Clause
subdir      : linux-64
url         : https://repo.anaconda.com/pkgs/main/linux-64/numpy-1.23.5-py310hd5efca6_0.conda
md5         : 14704ba32c38d4d8d78e5812f274a795
timestamp   : 2022-12-29 19:03:16 UTC
dependencies: 
  - blas 1.0 mkl
  - libgcc-ng >=11.2.0
  - libstdcxx-ng >=11.2.0
  - mkl >=2021.4.0,<2022.0a0
  - mkl-service >=2.3.0,<3.0a0
  - mkl_fft
  - mkl_random
  - numpy-base 1.23.5 py310h8e6c178_0
  - python >=3.10,<3.11.0a0

So, one build that specifically requires MKL 2021 and one that specifically requires MKL 2023, in addition to a build for OpenBLAS (and a conspicuous absence of a build for MKL 2022, which was the source of some of the original trouble reported here). I think for us, all the extra builds really aren't worth it, and we should just stick to using OpenBLAS for the packages on the cantera channel.

@ischoegl ischoegl mentioned this issue Aug 16, 2023
Open
@ischoegl
Copy link
Member

Superseded by #46

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
No open projects
Cantera 3.0 Release Planning
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix mkl Cantera/conda-recipes
4 participants
@speth @bryanwweber @ischoegl @ProfLeao