Python cannot use brew-installed libraries on Apple Silicon

[guneemwelloeux]

Hello team,

I recently got my hands on a M2 Macbook Pro.
I reinstalled brew, and figured out (should have read the instructions before) that /opt/homebrew is the new home on arm64.
I understand that decision, and in a lot of ways, it's a good thing. It makes it clearer what is x64 stuff (migrated from an intel Mac) vs arm64.

However, I found myself in a bit of a pickle.
One of the python packages I'm using is WeasyPrint, and it relies on libffi, fontconfig, pango and a couple of other things.
I installed these using brew, but I was still having the following error:

  File "/Users/wollivier/.pyenv/versions/3.9.13/envs/editor/lib/python3.9/site-packages/weasyprint/document.py", line 24, in <module>
    from .fonts import FontConfiguration
  File "/Users/wollivier/.pyenv/versions/3.9.13/envs/editor/lib/python3.9/site-packages/weasyprint/fonts.py", line 48, in <module>
    fontconfig = dlopen(ffi, 'fontconfig', 'libfontconfig',
  File "/Users/wollivier/.pyenv/versions/3.9.13/envs/editor/lib/python3.9/site-packages/weasyprint/text.py", line 255, in dlopen
    return ffi.dlopen(names[0])  # pragma: no cover
  File "/Users/wollivier/.pyenv/versions/3.9.13/envs/editor/lib/python3.9/site-packages/cffi/api.py", line 150, in dlopen
    lib, function_cache = _make_ffi_library(self, name, flags)
  File "/Users/wollivier/.pyenv/versions/3.9.13/envs/editor/lib/python3.9/site-packages/cffi/api.py", line 832, in _make_ffi_library
    backendlib = _load_backend_lib(backend, libname, flags)
  File "/Users/wollivier/.pyenv/versions/3.9.13/envs/editor/lib/python3.9/site-packages/cffi/api.py", line 827, in _load_backend_lib
    raise OSError(msg)
OSError: ctypes.util.find_library() did not manage to locate a library called 'fontconfig'

I tried setting LD_LIBRARY_PATH=/opt/homebrew/lib, to no avail.

According to the python documentation for ctypes, on MacOS, the ctypes.util.find_library function looks at a predetermined set of paths to resolve dynamic libraries, and does not use LD_LIBRARY_PATH.

I resorted to create a symlink to /opt/homebrew/Cellar in /usr/local, and that was enough to solve things.
The links from my x64 brew install were still in /usr/local/lib, and those point to the Cellar, so the link above restored things.

So in a nutshell, the issue is:

• brew installs in /opt/homebrew on arm64
• Python expects dyn libraries in /usr/lib or /usr/local/lib (maybe others, I couldn't find an exact list)
• Therefore python cannot find the brew installed libraries unless things are linked in /usr/local on Mac arm64 platforms

Not sure what the best solution would be at this point, but symbolic links do the trick, and are quite easy to setup.
Problem is that they interfere with a potential previous x64 installation that would need to be removed beforehand.

edit

After looking at the code for ctypes in Python 3.9, I found that the DYLD_LIBRARY_PATH environment variable is actually taken into account as well when resolving libraries.
Setting DYLD_LIBRARY_PATH=/opt/homebrew/lib in my .zshenv solved the issue, so that's what I'm going to stick with.

The solution might just be good documentation on that topic.

[SMillerDev]

Which python are you using? OS python? Arm homebrew python? X64 homebrew python? Python installed from a pkg?

How can we reproduce the issue?

[guneemwelloeux]

I'm using Python 3.9 installed with pyenv, itself installed with arm64 brew.

Reproduction steps:

1. Install arm homebrew (using instructions from brew.sh)
2. Install pyenv: brew install pyenv pyenv-virtualenv
3. Install python 3.9: pyenv install 3.9.13
4. pyenv virtualenv 3.9.13 somename
5. activate the virtual env pyenv activate somename
6. install weasyprint dependencies: brew install fontconfig pango cairo gdk-pixbuf libffi
7. Install weasyprint pip install weasyprint [Note: depending on which packages need compiling from scratch, some LDFLAGS and CPPFLAGS directives might be required to point to the installed libraries above]
8. Try importing weasyprint python -m weasyprint

The last step above should fail with:

Traceback (most recent call last):
  File "/Users/wollivier/.pyenv/versions/3.9.13/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/Users/wollivier/.pyenv/versions/3.9.13/lib/python3.9/runpy.py", line 147, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/Users/wollivier/.pyenv/versions/3.9.13/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/Users/wollivier/.pyenv/versions/ss/lib/python3.9/site-packages/weasyprint/__init__.py", line 336, in <module>
    from .css import preprocess_stylesheet  # noqa isort:skip
  File "/Users/wollivier/.pyenv/versions/ss/lib/python3.9/site-packages/weasyprint/css/__init__.py", line 25, in <module>
    from . import computed_values, counters, media_queries
  File "/Users/wollivier/.pyenv/versions/ss/lib/python3.9/site-packages/weasyprint/css/computed_values.py", line 9, in <module>
    from ..text.ffi import ffi, pango, units_to_double
  File "/Users/wollivier/.pyenv/versions/ss/lib/python3.9/site-packages/weasyprint/text/ffi.py", line 408, in <module>
    fontconfig = _dlopen(
  File "/Users/wollivier/.pyenv/versions/ss/lib/python3.9/site-packages/weasyprint/text/ffi.py", line 385, in _dlopen
    return ffi.dlopen(names[0])  # pragma: no cover
  File "/Users/wollivier/.pyenv/versions/ss/lib/python3.9/site-packages/cffi/api.py", line 150, in dlopen
    lib, function_cache = _make_ffi_library(self, name, flags)
  File "/Users/wollivier/.pyenv/versions/ss/lib/python3.9/site-packages/cffi/api.py", line 832, in _make_ffi_library
    backendlib = _load_backend_lib(backend, libname, flags)
  File "/Users/wollivier/.pyenv/versions/ss/lib/python3.9/site-packages/cffi/api.py", line 827, in _load_backend_lib
    raise OSError(msg)
OSError: ctypes.util.find_library() did not manage to locate a library called 'fontconfig-1'

[SMillerDev]

That really sounds more like a pyenv configuration issue than a homebrew one.

[carlocab]

I found that the DYLD_LIBRARY_PATH environment variable is actually taken into account as well when resolving libraries.

Note that Apple considers DYLD_LIBRARY_PATH a security risk, and clears it from the environment before executing programs protected by SIP:

❯ FOO=foo /bin/bash -c 'echo ${FOO-unset}'
foo
❯ DYLD_LIBRARY_PATH=foo /bin/bash -c 'echo ${FOO-unset}'
unset

Setting DYLD_LIBRARY_PATH is also likely to cause programs you have installed yourself to misbehave. From man dyld:

       DYLD_LIBRARY_PATH
              This is a colon separated list of directories that contain libraries. The dynamic linker searches these directories
              before it searches the default locations for libraries. It allows you to test new versions of existing libraries.

This means that setting DYLD_LIBRARY_PATH will override the default search path and can result in loading libraries that are incompatible with the programs that you run.

If you must rely on DYLD_* environment variables to influence dyld, consider setting DYLD_FALLBACK_LIBRARY_PATH instead. It's still cleared from the environment by SIP, but you at least avoid issues from overriding the default search path by using it as a fallback path instead.

[carlocab]

Incidentally, if you use a Homebrew-built Python, you will find that ctypes.util.find_library will be able to find libraries installed by Homebrew.

❯ /opt/homebrew/bin/python3 -c 'import ctypes.util; print(ctypes.util.find_library("libuv"))'
/opt/homebrew/lib/libuv.dylib

[guneemwelloeux]

That sounds great, but I can't really do without pyenv these days, it makes a lot of things soo much simpler.
I have tried to have pyenv use the homebrew python install, but couldn't get it to work (missing pip).

[Bo98]

Yeah we patch find_library in our Python: https://github.com/Homebrew/homebrew-core/blob/ed6d3f73a6cd3d779d0254f4ae5ba39b99d3217b/Formula/python@3.10.rb#L185-L192. There's not exactly much we can do beyond that really. DYLD_FALLBACK_LIBRARY_PATH/DYLD_FALLBACK_FRAMEWORK_PATH seems like a reasonable workaround where patching is not possible.

Just be aware that scripts which use /usr/bin/env python3 might not pick up DYLD_ variables. This is a security restriction Apple have put in place.