Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issues while installing mpi4py on linux based cluster #17

Open
rB080 opened this issue Sep 12, 2023 · 4 comments
Open

Issues while installing mpi4py on linux based cluster #17

rB080 opened this issue Sep 12, 2023 · 4 comments

Comments

@rB080
Copy link

rB080 commented Sep 12, 2023

Hi, I am trying to run a pip install mpi4py but it throws off with an error. The exact error is here:

Defaulting to user installation because normal site-packages is not writeable
Collecting mpi4py
  Downloading mpi4py-3.1.4.tar.gz (2.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 18.3 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: mpi4py
  Building wheel for mpi4py (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for mpi4py (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [156 lines of output]
      running bdist_wheel
      running build
      running build_src
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-39
      creating build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/run.py -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/__init__.py -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/bench.py -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/__main__.py -> build/lib.linux-x86_64-cpython-39/mpi4py
      creating build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/server.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/_base.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/_lib.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/__init__.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/_core.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/aplus.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/pool.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/__main__.py -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      creating build/lib.linux-x86_64-cpython-39/mpi4py/util
      copying src/mpi4py/util/__init__.py -> build/lib.linux-x86_64-cpython-39/mpi4py/util
      copying src/mpi4py/util/pkl5.py -> build/lib.linux-x86_64-cpython-39/mpi4py/util
      copying src/mpi4py/util/dtlib.py -> build/lib.linux-x86_64-cpython-39/mpi4py/util
      copying src/mpi4py/py.typed -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/dl.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/run.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/__main__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/__init__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/bench.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/MPI.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/MPI.pxd -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/__init__.pxd -> build/lib.linux-x86_64-cpython-39/mpi4py
      copying src/mpi4py/libmpi.pxd -> build/lib.linux-x86_64-cpython-39/mpi4py
      creating build/lib.linux-x86_64-cpython-39/mpi4py/include
      creating build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/mpi4py.h -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/mpi4py.MPI.h -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/mpi4py.MPI_api.h -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/mpi4py.i -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
      copying src/mpi4py/include/mpi4py/mpi.pxi -> build/lib.linux-x86_64-cpython-39/mpi4py/include/mpi4py
      copying src/mpi4py/futures/_lib.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/pool.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/server.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/__main__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/aplus.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/__init__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/futures/_core.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/futures
      copying src/mpi4py/util/pkl5.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/util
      copying src/mpi4py/util/__init__.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/util
      copying src/mpi4py/util/dtlib.pyi -> build/lib.linux-x86_64-cpython-39/mpi4py/util
      running build_clib
      MPI configuration: [mpi] from 'mpi.cfg'
      MPI C compiler:    /share/apps/anaconda3-2022.05/bin/mpicc
      MPI C++ compiler:  /share/apps/anaconda3-2022.05/bin/mpicxx
      MPI F compiler:    /share/apps/anaconda3-2022.05/bin/mpifort
      MPI F90 compiler:  /share/apps/anaconda3-2022.05/bin/mpif90
      MPI F77 compiler:  /share/apps/anaconda3-2022.05/bin/mpif77
      checking for library 'lmpe' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      building 'mpe' dylib library
      creating build/temp.linux-x86_64-cpython-39
      creating build/temp.linux-x86_64-cpython-39/src
      creating build/temp.linux-x86_64-cpython-39/src/lib-pmpi
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c src/lib-pmpi/mpe.c -o build/temp.linux-x86_64-cpython-39/src/lib-pmpi/mpe.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      warning: build_clib: command '/share/apps/anaconda3-2022.05/bin/mpicc' failed with exit code 127

      warning: build_clib: building optional library "mpe" failed

      checking for library 'vt-mpi' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      checking for library 'vt.mpi' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      building 'vt' dylib library
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c src/lib-pmpi/vt.c -o build/temp.linux-x86_64-cpython-39/src/lib-pmpi/vt.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      warning: build_clib: command '/share/apps/anaconda3-2022.05/bin/mpicc' failed with exit code 127

      warning: build_clib: building optional library "vt" failed

      checking for library 'vt-mpi' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      checking for library 'vt.mpi' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      building 'vt-mpi' dylib library
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c src/lib-pmpi/vt-mpi.c -o build/temp.linux-x86_64-cpython-39/src/lib-pmpi/vt-mpi.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      warning: build_clib: command '/share/apps/anaconda3-2022.05/bin/mpicc' failed with exit code 127

      warning: build_clib: building optional library "vt-mpi" failed

      checking for library 'vt-hyb' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      checking for library 'vt.ompi' ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      building 'vt-hyb' dylib library
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -c src/lib-pmpi/vt-hyb.c -o build/temp.linux-x86_64-cpython-39/src/lib-pmpi/vt-hyb.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      warning: build_clib: command '/share/apps/anaconda3-2022.05/bin/mpicc' failed with exit code 127

      warning: build_clib: building optional library "vt-hyb" failed

      running build_ext
      MPI configuration: [mpi] from 'mpi.cfg'
      MPI C compiler:    /share/apps/anaconda3-2022.05/bin/mpicc
      MPI C++ compiler:  /share/apps/anaconda3-2022.05/bin/mpicxx
      MPI F compiler:    /share/apps/anaconda3-2022.05/bin/mpifort
      MPI F90 compiler:  /share/apps/anaconda3-2022.05/bin/mpif90
      MPI F77 compiler:  /share/apps/anaconda3-2022.05/bin/mpif77
      checking for dlopen() availability ...
      checking for header 'dlfcn.h' ...
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -I/share/apps/anaconda3-2022.05/include/python3.9 -c _configtest.c -o _configtest.o
      success!
      removing: _configtest.c _configtest.o
      success!
      checking for library 'dl' ...
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -I/share/apps/anaconda3-2022.05/include/python3.9 -c _configtest.c -o _configtest.o
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat _configtest.o -Lbuild/temp.linux-x86_64-cpython-39 -ldl -o _configtest
      success!
      removing: _configtest.c _configtest.o _configtest
      checking for function 'dlopen' ...
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -I/share/apps/anaconda3-2022.05/include/python3.9 -c _configtest.c -o _configtest.o
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat _configtest.o -Lbuild/temp.linux-x86_64-cpython-39 -ldl -o _configtest
      success!
      removing: _configtest.c _configtest.o _configtest
      building 'mpi4py.dl' extension
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -DHAVE_DLFCN_H=1 -DHAVE_DLOPEN=1 -I/share/apps/anaconda3-2022.05/include/python3.9 -c src/dynload.c -o build/temp.linux-x86_64-cpython-39/src/dynload.o
      gcc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -shared -Wl,-rpath,/share/apps/anaconda3-2022.05/lib -Wl,-rpath-link,/share/apps/anaconda3-2022.05/lib -L/share/apps/anaconda3-2022.05/lib -L/share/apps/anaconda3-2022.05/lib -Wl,-rpath,/share/apps/anaconda3-2022.05/lib -Wl,-rpath-link,/share/apps/anaconda3-2022.05/lib -L/share/apps/anaconda3-2022.05/lib build/temp.linux-x86_64-cpython-39/src/dynload.o -Lbuild/temp.linux-x86_64-cpython-39 -ldl -o build/lib.linux-x86_64-cpython-39/mpi4py/dl.cpython-39-x86_64-linux-gnu.so
      checking for MPI compile and link ...
      /share/apps/anaconda3-2022.05/bin/mpicc -pthread -B /share/apps/anaconda3-2022.05/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -I/share/apps/anaconda3-2022.05/include -fPIC -O2 -isystem /share/apps/anaconda3-2022.05/include -fPIC -I/share/apps/anaconda3-2022.05/include/python3.9 -c _configtest.c -o _configtest.o
      /share/apps/anaconda3-2022.05/bin/mpicc: line 301: x86_64-conda_cos6-linux-gnu-cc: command not found
      failure.
      removing: _configtest.c _configtest.o
      error: Cannot compile MPI programs. Check your configuration!!!
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for mpi4py
Failed to build mpi4py
ERROR: Could not build wheels for mpi4py, which is required to install pyproject.toml-based projects

This however works on conda, but even if I can see the package on conda list, it doesn't show up when I try importing it from python. Is there a fix for this? Is this a python version issue? I'm using python 3.10.
@wangxiaobo007
Copy link

same question

@m-bain
Copy link
Owner

m-bain commented Oct 7, 2023

Install with video2dataset, it's much faster

https://github.com/iejMac/video2dataset

@LymanY
Copy link

LymanY commented Nov 28, 2023

u can use conda instead of pip

@shinever22
Copy link

Hi~We are researching a project, and the Webvid-2.5M dataset is very important to us. We found that the link is no longer working. Could you please provide us with a new link?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@LymanY @m-bain @wangxiaobo007 @rB080 @shinever22