[1.8.0-rc2][macOS] could not load library "@rpath/libcurl.4.dylib" error

[SaschaMann]

Since ~3 days ago, 1.8.0-rc2 and the latest nightlies from release-1.8 are erroring on the GHA macOS runner. I'm not sure if this is caused by an issue with the runner or an issue with Julia.

It was still working as of 6368fdc aka 1.8.0-rc1. 381eac3 is the first commit that I know to have this issue but the tests only run every 24h, so this isn't necessarily the commit that introduced it.

Possibly related: #45933, #45967

---

Full error

julia --compile=min -O0 -e using InteractiveUtils; versioninfo()
[...]
fatal: error thrown and no exception handler available.
InitError(mod=:LibCURL_jll, error=ErrorException("could not load library "@rpath/libcurl.4.dylib"
dlopen(@rpath/libcurl.4.dylib, 1): Library not loaded: @rpath/libnghttp2.14.dylib
  Referenced from: /Users/runner/hostedtoolcache/julia/1.8-nightly/x64/lib/julia/libcurl.4.dylib
  Reason: Incompatible library version: libcurl.4.dylib requires version 36.0.0 or later, but libnghttp2.14.dylib provides version 35.0.0"))
ijl_errorf at /Users/runner/hostedtoolcache/julia/1.8-nightly/x64/lib/julia/libjulia-internal.1.8.dylib (unknown line)
ijl_load_dynamic_library at /Users/runner/hostedtoolcache/julia/1.8-nightly/x64/lib/julia/libjulia-internal.1.8.dylib (unknown line)
#dlopen#3 at ./libdl.jl:117
dlopen at ./libdl.jl:116 [inlined]
dlopen at ./libdl.jl:116 [inlined]
__init__ at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-macmini-x64-6.0/build/default-macmini-x64-6-0/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/LibCURL_jll/src/LibCURL_jll.jl:30
jfptr___init___70049.clone_1 at /Users/runner/hostedtoolcache/julia/1.8-nightly/x64/lib/julia/sys.dylib (unknown line)
ijl_apply_generic at /Users/runner/hostedtoolcache/julia/1.8-nightly/x64/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_module_run_initializer at /Users/runner/hostedtoolcache/julia/1.8-nightly/x64/lib/julia/libjulia-internal.1.8.dylib (unknown line)
_finish_julia_init at /Users/runner/hostedtoolcache/julia/1.8-nightly/x64/lib/julia/libjulia-internal.1.8.dylib (unknown line)
julia_init at /Users/runner/hostedtoolcache/julia/1.8-nightly/x64/lib/julia/libjulia-internal.1.8.dylib (unknown line)
jl_repl_entrypoint at /Users/runner/hostedtoolcache/julia/1.8-nightly/x64/lib/julia/libjulia-internal.1.8.dylib (unknown line)
Error: The process 'julia' failed with exit code 1

Environment info:

Operating System
  macOS
  11.6.6
  20G624
Virtual Environment
  Environment: macos-11
  Version: 20220627.1
  Included Software: https://github.com/actions/virtual-environments/blob/macOS-11/20220627.1/images/macos/macos-11-Readme.md
  Image Release: https://github.com/actions/virtual-environments/releases/tag/macOS-11%2F20220627.1

Some failing builds:

• release-1.8: https://github.com/julia-actions/setup-julia/actions/workflows/example-builds-nightly.yml
• 1.8.0-rc2: https://github.com/julia-actions/setup-julia/runs/7266108728?check_suite_focus=true

[giordano]

How can #45967 possibly be related if it isn't merged yet?

[SaschaMann]

How can #45967 possibly be related if it isn't merged yet?

it could be a fix for it, but I don't know if it is.

[giordano]

I don't think so. The error message is

but libnghttp2.14.dylib provides version 35.0.0

and nghttp2 1.47.0 instead has

sandbox:${WORKSPACE} # otool -L libnghttp2.dylib 
libnghttp2.dylib:
        @rpath/libnghttp2.14.dylib (compatibility version 36.0.0, current version 36.2.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

It'd look like there is another libnghttp2.14.dylib around being picked up.

[giordano]

Wait, that's not in release-1.8 though:

julia/stdlib/nghttp2_jll/Project.toml

Line 3 in ce5d15e

version = "1.41.0+1"

#45408 has to be backported.

[KristofferC]

Why wasn't this caught in CI?

[DilumAluthge]

Why wasn't this caught in CI?

@staticfloat @gbaraldi Any ideas?

[DilumAluthge]

I can reproduce this locally with:

cd $(mktemp -d)

curl https://julialangnightlies-s3.julialang.org/bin/mac/x64/1.8/julia-latest-mac64.tar.gz | tar xzv

./julia-477324a171/bin/julia -e 'println("hello world")'

This is on an Intel Mac (x86_64) running macOS 11.6.7 (20G630).

[DilumAluthge]

I believe this is a 1.8 release blocker, as macOS x86_64 is a Tier 1 platform.

[giordano]

#45408 has been already marked for backport.

[giordano]

@SaschaMann out of curiosity, can you try running on macos-12 instead of macos-latest?

[SaschaMann]

@giordano That seems to work: https://github.com/julia-actions/setup-julia/runs/7271781363

 julia --compile=min -O0 -e using InteractiveUtils; versioninfo()
[...]
Julia Version 1.8.0-rc1.51
Commit 477324a1711 (2022-07-08 10:53 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin21.4.0)
  CPU: 3 × Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, ivybridge)
  Threads: 1 on 3 virtual cores

[giordano]

Ok, so it does look like loading the incompatible library works on macOS 12 (probably what's used here in CI) but not 11 (that's what macos-latest points to in github actions). Thanks!

I've been trying to create a small reproducer in C at https://github.com/giordano/macos-compatibility-version, but so far with little success: tests pass on both macOS 11 and 12, I can't trigger the compatibility check at all, not even on macOS 11.

[giordano]

Why wasn't this caught in CI?

Ok, I finally managed to figure out this and I have updated the repository at https://github.com/giordano/macos-compatibility-version, adding also a README file to explain what's going on.

The gist is that the check for the compatibility is run only when setting a sufficiently old minimum version of macOS (that is, before 10.14, in BinaryBuilder we use 10.12 when building for x86_64-apple-darwin) and apparently only on macOS up to 11. It doesn't look like this check is ever triggered on macOS 12 anymore, and we're probably using macOS 12 here on CI, which would explain why this problem wasn't caught.

[KristofferC]

Should be fixed