Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can not build tensorflow-io from source on Apple M1 #1643

Closed
sun1638650145 opened this issue Feb 18, 2022 · 32 comments
Closed

Can not build tensorflow-io from source on Apple M1 #1643

sun1638650145 opened this issue Feb 18, 2022 · 32 comments

Comments

@sun1638650145
Copy link

sun1638650145 commented Feb 18, 2022

  1. First I tried using python setup.py bdist_wheel, which builds wheel, but the installation prompts me

    ERROR: Could not find a version that satisfies the requirement tensorflow-io-gcs-filesystem==0.24.0 (from tensorflow-io) (from versions: none)
    ERROR: No matching distribution found for tensorflow-io-gcs-filesystem==0.24.0

    So I changed it to install pip install dist/tensorflow_io-0.24.0-cp38-cp38-macosx_11_0_arm64.whl --no-deps without dependencies, so although it can be installed normally, it will prompt me that there is no dynamic library when running

    /Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
    caused by: ["[Errno 2] The file to load file system plugin from does not exist.: '/Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/libtensorflow_io_plugins.so'"]
    warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
    /Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/libtensorflow_io.so']
    caused by: ["dlopen(/Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/libtensorflow_io.so, 0x0006): tried: '/Users/sunruiqi/Desktop/io-0.24.0/tensorflow_io/python/ops/libtensorflow_io.so' (no such file), '/usr/local/lib/libtensorflow_io.so' (no such file), '/usr/lib/libtensorflow_io.so' (no such file)"]
    warnings.warn(f"file system plugins are not loaded: {e}")
  2. After the above method fails, I refer to the build method in the documentation, unfortunately, run this command bazel build -s --verbose_failures $BAZEL_OPTIMIZATION //tensorflow_io/... //tensorflow_io_gcs_filesystem/... will produce the following error:

    ERROR: While resolving toolchains for target //tensorflow_io/core:golang_ops: no matching toolchains found for types @io_bazel_rules_go//go:toolchain
    ERROR: Analysis of target '//tensorflow_io/core:golang_ops' failed; build aborted: no matching toolchains found for types @io_bazel_rules_go//go:toolchain

    So, what should I do?

@sun1638650145
Copy link
Author

截屏2022-02-18 上午11 18 28

There is some information about the development environment.

@ZaibanAli
Copy link

ZaibanAli commented Feb 18, 2022

First, goto to the tensorflow/io root folder.

Second, do:

rm -r dist/*
export MACOSX_DEPLOYMENT_TARGET="$(sw_vers -productVersion)"

Third, on the same bash shell do:

python3 setup.py -q bdist_wheel --project tensorflow_io_gcs_filesystem
pip install --no-dependencies dist/tensorflow_io_gcs_filesystem-0.24.0-cp38-cp38-macosx_12_0_arm64.whl

Fourth,

python3 setup.py -q bdist_wheel
pip install --no-dependencies dist/tensorflow_io-0.24.0-cp38-cp38-macosx_12_0_arm64.whl

What I know currently is that Bazel build is not supported for mac m1 (for tensorflow/io).

@sun1638650145
Copy link
Author

@ZaibanAli Thank you very much for your help, but still there is no dynamic library error.

@ZaibanAli
Copy link

@sun1638650145 Okay so I tested it again and I am getting this error too when importing tensorflow_io or when using object detection api.

>>> import tensorflow_io
/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
caused by: ["[Errno 2] The file to load file system plugin from does not exist.: '/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so'"]
  warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ["dlopen(/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/libtensorflow_io.so, 0x0006): tried: '/Users/zaiban.ali/miniforge3/envs/test1/lib/python3.8/site-packages/tensorflow_io/python/ops/libtensorflow_io.so' (no such file), '/usr/local/lib/libtensorflow_io.so' (no such file), '/usr/lib/libtensorflow_io.so' (no such file)"]
  warnings.warn(f"file system plugins are not loaded: {e}")
>>> print(tensorflow_io.__version__)
0.24.0

@yongtang
Copy link
Member

The tensorflow-io will need to have .so file complied from C++ code in order for file system plugins to work. But in order to compile C++ code it will need to have tensorflow installed. @ZaibanAli @sun1638650145 Which tensorflow package are you using on M1?

@ZaibanAli
Copy link

@yongtang I am using this https://github.com/apple/tensorflow_macos/releases/tag/v0.1alpha3

All other things are working correctly, the issues only occurs when I try to use tensorlfow-io. (in my case it is required by tensorflow object detection api)

@sun1638650145
Copy link
Author

@yongtang I'm using the stable version of tensorflow_macos.

@ZaibanAli
Copy link

Update: I am able to run object detection api even with this warning, the problem was that I had previously installed the old TensorFlow addon package. Updating the TensorFlow addon package to the newest resolved the issue.

@yongtang
Copy link
Member

@ZaibanAli @sun1638650145 My recent M1 laptop is having a firmware issue and is with apple repair now. I may only be able to take a look at building tensorflow-io after the laptop is back.

Before the firmware issue I was using pip to install tensorflow_macos though I remember I had an issue with hdf5 dependency. Do you encounter the same hdf5 dependencies?

@ZaibanAli
Copy link

@yongtang no dependency issues on my side.

@sun1638650145
Copy link
Author

@yongtang hdf5 build depends on numpy, luckily, numpy has provided precompiled packages since 1.21.0, and building from source is also very easy.

@dvaldivia
Copy link

dvaldivia commented Mar 10, 2022

trying the steps suggested to compile fails, using python 3.9, is it needed to use python 3.8?

(base)  io git:(master) ✗ 
➜ pip install --no-dependencies dist/tensorflow_io_gcs_filesystem-0.24.0-cp39-cp39-macosx_12_2_arm64.whl 
ERROR: tensorflow_io_gcs_filesystem-0.24.0-cp39-cp39-macosx_12_2_arm64.whl is not a supported wheel on this platform.
(base)  io git:(master) ✗ 
➜ pip install --no-dependencies dist/tensorflow_io-0.24.0-cp39-cp39-macosx_12_2_arm64.whl               
ERROR: tensorflow_io-0.24.0-cp39-cp39-macosx_12_2_arm64.whl is not a supported wheel on this platform.

@ZaibanAli
Copy link

@dvaldivia I had the same issue with python 3.9. Try it with python 3.8.

The reason I found out was that TensorFlow .whl files are available for cp38 only for macos (tensorflow_macos/v0.1alpha3).

@yongtang
Copy link
Member

I am able to get my M1 laptop back. However, while I successfully installed tensorflow_macos, during the bazel build the following issues popping up:

ERROR: /private/var/tmp/_bazel_yongtang/c33235605c366b16ba32c7f8a25ff251/external/local_config_cc/BUILD:48:19: in cc_toolchain_suite rule @local_config_cc//:toolchain: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'darwin_arm64'
ERROR: /private/var/tmp/_bazel_yongtang/c33235605c366b16ba32c7f8a25ff251/external/local_config_cc/BUILD:48:19: Analysis of target '@local_config_cc//:toolchain' failed
INFO: Repository curl instantiated at:

It looks like there is still some compatibility issue from bazel, either due to bazel itself, or due to some third party libraries (e.g., tensorflow)'s bazel BUILD. Will have to dig further.

@sun1638650145
Copy link
Author

@yongtang I also had type problems when building with bazel, but now bazel has native support for M1, and can build tf, tf-addons, tf-text perfectly. So why is this?

@yongtang
Copy link
Member

The only difference I can see, is that I haven't installed XCode (only use command line tools). I can try install xcode.

@leochoo
Copy link

leochoo commented May 29, 2022

@yongtang Any update yet? I'm running into this issue as well. It does not let me use tfio.audio as a result.

What would be a workaround you recommend?

@npanpaliya
Copy link

The tensorflow-io will need to have .so file complied from C++ code in order for file system plugins to work. But in order to compile C++ code it will need to have tensorflow installed. @ZaibanAli @sun1638650145 Which tensorflow package are you using on M1?

Hello, I'm also facing the same issue while importing tensorflow-io 0.27.0. What exactly needs to be done to resolve these warnings?

@guites
Copy link

guites commented Nov 29, 2022

Hello, I'm also facing the same issue while importing tensorflow-io 0.27.0. What exactly needs to be done to resolve these warnings?

I'm receiving the same warnings

(tfod-api-cpu) guites@macos io % python                   
Python 3.9.15 (main, Nov 24 2022, 08:28:41) 
[Clang 14.0.6 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow_io
~/another-test/io/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['~/another-test/io/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
caused by: ["[Errno 2] The file to load file system plugin from does not exist.: '~/another-test/io/tensorflow_io/python/ops/libtensorflow_io_plugins.so'"]
  warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
~/another-test/io/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['~/another-test/io/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ["dlopen(~/another-test/io/tensorflow_io/python/ops/libtensorflow_io.so, 0x0006): tried: '~/another-test/io/tensorflow_io/python/ops/libtensorflow_io.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/guites/another-test/io/tensorflow_io/python/ops/libtensorflow_io.so' (no such file), '~/another-test/io/tensorflow_io/python/ops/libtensorflow_io.so' (no such file)"]
  warnings.warn(f"file system plugins are not loaded: {e}")
>>> tensorflow_io.__version__
'0.26.0'

Tried using an arm64 tensorflow_addons wheel as provided by @sun1638650145, but the warnings persist.

still, I am able to run objecte detection jobs (https://www.tensorflow.org/hub/tutorials/tf2_object_detection).

@cmditch
Copy link

cmditch commented Dec 7, 2022

I'm seeing the same error you're seeing @guites. This is happening inside a docker container (dev container) on an M1 mac.

Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow_io
/home/vscode/.local/lib/python3.9/site-packages/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/home/vscode/.local/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
caused by: ["[Errno 2] The file to load file system plugin from does not exist.: '/home/vscode/.local/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so'"]
  warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
/home/vscode/.local/lib/python3.9/site-packages/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/home/vscode/.local/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ['/home/vscode/.local/lib/python3.9/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: cannot open shared object file: No such file or directory']
  warnings.warn(f"file system plugins are not loaded: {e}")
>>> tensorflow_io.__version__
'0.27.0'

@ALEEF02
Copy link

ALEEF02 commented Jan 27, 2023

Just wanted to chime in that I am also having these issues but on the base tensorflow. Seems to run fine, checking tensorflow.__version__ yields 2.11.0.

@sun1638650145
Copy link
Author

I built tensorflow from source, without using tensorflow-macos, but I'm still having issues building tensorflow-io.

@gopal-tathe
Copy link

Hi, any update on this issue, getting same error.

@LeoBlackCat
Copy link

the same error...

@TiagoGouvea
Copy link

I'm trying to make it fully work for three days, but still have those errors.

@TiagoGouvea
Copy link

Looking at many questions on stackoverflow, with 5k view, 3 answers, and 6 up votes on the best answer, make me convinced that it just doesn't work for the major of devs trying.

@kulinseth
Copy link

I built tensorflow from source, without using tensorflow-macos, but I'm still having issues building tensorflow-io.

Hi @sun1638650145 and @TiagoGouvea , can you please paste the build error ?

@TiagoGouvea
Copy link

I'm getting this message on every run, and it takes some like 5 seconds. After that the script follows.

/Users/tiagogouvea/python/io/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
caused by: ["[Errno 2] The file to load file system plugin from does not exist.: '/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io_plugins.so'"]
  warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
/Users/tiagogouvea/python/io/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ["dlopen(/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io.so, 0x0006): tried: '/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io.so' (no such file), '/Users/tiagogouvea/python/io/tensorflow_io/python/ops/libtensorflow_io.so' (no such file)"]
  warnings.warn(f"file system plugins are not loaded: {e}")

Using a conda environment, this is my build and install results:

> python setup.py build
Project: tensorflow-io
Exclude: ['tests', 'tests.*', 'tensorflow_io_gcs_filesystem', 'tensorflow_io_gcs_filesystem.*']
Install Requires: ['tensorflow-io-gcs-filesystem==0.32.0']
Project Rootpath: tensorflow_io
running build
running build_py
running build_ext
> ython setup.py install
Project: tensorflow-io
Exclude: ['tests', 'tests.*', 'tensorflow_io_gcs_filesystem', 'tensorflow_io_gcs_filesystem.*']
Install Requires: ['tensorflow-io-gcs-filesystem==0.32.0']
Project Rootpath: tensorflow_io
running install
/Users/tiagogouvea/anaconda3/envs/py310/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer, pypa/build or
        other standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
/Users/tiagogouvea/anaconda3/envs/py310/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer, pypa/build or
        other standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  self.initialize_options()
running bdist_egg
running egg_info
writing tensorflow_io.egg-info/PKG-INFO
writing dependency_links to tensorflow_io.egg-info/dependency_links.txt
writing requirements to tensorflow_io.egg-info/requires.txt
writing top-level names to tensorflow_io.egg-info/top_level.txt
reading manifest file 'tensorflow_io.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'tensorflow_io.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-11.1-arm64/egg
running install_lib
running build_py
running build_ext
creating build/bdist.macosx-11.1-arm64/egg
....
copying tensorflow_io.egg-info/top_level.txt -> build/bdist.macosx-11.1-arm64/egg/EGG-INFO
creating 'dist/tensorflow_io-0.32.0-py3.10-macosx-11.1-arm64.egg' and adding 'build/bdist.macosx-11.1-arm64/egg' to it
removing 'build/bdist.macosx-11.1-arm64/egg' (and everything under it)
Processing tensorflow_io-0.32.0-py3.10-macosx-11.1-arm64.egg
removing '/Users/tiagogouvea/anaconda3/envs/py310/lib/python3.10/site-packages/tensorflow_io-0.32.0-py3.10-macosx-11.1-arm64.egg' (and everything under it)
creating /Users/tiagogouvea/anaconda3/envs/py310/lib/python3.10/site-packages/tensorflow_io-0.32.0-py3.10-macosx-11.1-arm64.egg
Extracting tensorflow_io-0.32.0-py3.10-macosx-11.1-arm64.egg to /Users/tiagogouvea/anaconda3/envs/py310/lib/python3.10/site-packages
Adding tensorflow-io 0.32.0 to easy-install.pth file

Installed /Users/tiagogouvea/anaconda3/envs/py310/lib/python3.10/site-packages/tensorflow_io-0.32.0-py3.10-macosx-11.1-arm64.egg
Processing dependencies for tensorflow-io==0.32.0
Searching for tensorflow-io-gcs-filesystem==0.32.0
Reading https://pypi.org/simple/tensorflow-io-gcs-filesystem/
No local packages or working download links found for tensorflow-io-gcs-filesystem==0.32.0
error: Could not find suitable distribution for Requirement.parse('tensorflow-io-gcs-filesystem==0.32.0')

Thank you for your attention @kulinseth

@sun1638650145
Copy link
Author

sun1638650145 commented Sep 9, 2023

Now, io can be built perfectly on Apple silicon. If anyone needs it, you can find the tutorial here. I hope it can be helpful to everyone.

@flippinroo2
Copy link

flippinroo2 commented Jan 8, 2024

here

Any chance we can re-open this until a MR is made to fix this for anyone who has a MacBook after 2020...?

I really don't want to have to add anymore new brew packages and then modify a few lines in the source code too :/

@sun1638650145
Copy link
Author

@flippinroo2 tensorflow does not seem to be planning official support for Apple silicon. I have provided several tensorflow libraries whl files, and if you need them, you can download them directly from here.

@flippinroo2
Copy link

@flippinroo2 tensorflow does not seem to be planning official support for Apple silicon. I have provided several tensorflow libraries whl files, and if you need them, you can download them directly from here.

WOW! Thanks so much dude. This helps A LOT!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests