Asterisk crashes when homebrew is installed somewhere other than /usr/local

[leedm777]

Lots of great info Homebrew/legacy-homebrew#46126 regarding what happens when you install Asterisk somewhere other than /usr or /usr/local. Thanks to @afh for the detailed report in that thread.

[leedm777]

@afh

Where would I look in asterisk's compile setup (configure et al) in order to find out/change which paths are checked in which order?

Unfortunately, the Asterisk build system is built using Dark Magic™. When it works, it's real nice and easy. When it breaks, debugging it is a pain.

The link step for the executable is the $(MAIN_TGT) target in main/Makefile. It's a big mess of variables from other Makefiles, and it's usually a scavenger hunt tracking down what comes from where.

Asterisk also builds a number of module. I have a feeling if the main exe is linked incorrectly, then a number of the modules will also be linked incorrectly.

If you brew install --verbose asterisk, you'll see the output from ./configure and make. The link step for Asterisk is toward the end. This is what it looks like for my brew.

g++-5 -o asterisk -mmacosx-version-min=10.6 -Wl,-undefined,dynamic_lookup
 -force_flat_namespace /usr/lib/bundle1.o -L/usr/local/lib
 -Wl,-rpath,/usr/local/Cellar/asterisk/13.6.0/lib abstract_jb.o acl.o adsi.o
 alaw.o aoc.o app.o ast_expr2.o ast_expr2f.o asterisk.o astfd.o astmm.o
 astobj2.o astobj2_container.o astobj2_hash.o astobj2_rbtree.o audiohook.o
 autochan.o autoservice.o backtrace.o bridge.o bridge_after.o bridge_basic.o
 bridge_channel.o bridge_roles.o bucket.o callerid.o ccss.o cdr.o cel.o
 channel.o channel_internal_api.o chanvars.o cli.o codec.o codec_builtin.o
 config.o config_options.o core_local.o core_unreal.o crypt.o data.o datastore.o
 db.o devicestate.o dial.o dns.o dnsmgr.o dsp.o endpoints.o enum.o event.o
 features.o features_config.o file.o fixedjitterbuf.o format.o format_cache.o
 format_cap.o format_compatibility.o frame.o framehook.o fskmodem.o
 global_datastores.o hashtab.o heap.o http.o image.o indications.o io.o
 jitterbuf.o json.o loader.o lock.o logger.o manager.o manager_bridges.o
 manager_channels.o manager_endpoints.o manager_mwi.o manager_system.o
 max_forwards.o md5.o media_index.o message.o mixmonitor.o named_acl.o netsock.o
 netsock2.o optional_api.o parking.o pbx.o pickup.o plc.o poll.o presencestate.o
 privacy.o rtp_engine.o say.o sched.o sdp_srtp.o security_events.o sem.o sha1.o
 sip_api.o slinfactory.o smoother.o sorcery.o sounds_index.o srv.o stasis.o
 stasis_bridges.o stasis_cache.o stasis_cache_pattern.o stasis_channels.o
 stasis_endpoints.o stasis_message.o stasis_message_router.o stasis_system.o
 stdtime/localtime.o strcompat.o strings.o stun.o syslog.o taskprocessor.o
 tcptls.o tdd.o term.o test.o threadpool.o threadstorage.o timing.o translate.o
 udptl.o ulaw.o uri.o utils.o uuid.o version.o xml.o xmldoc.o -L. -lasteriskssl
 buildinfo.o -lc
 -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib
 -lxml2 -lz -lpthread -licucore -lm -lxslt
 -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/lib
 -lxml2 -lz -lpthread -licucore -lm -L/usr/local/opt/sqlite/lib -lsqlite3
 -L/usr/local/opt/openssl/lib -lssl -lcrypto -ljansson -luuid -ltermcap -lm
 -lresolv -L/usr/local/lib -ledit -lcurses

I'll do some debugging on my own, but I'm not quite sure when I'll have the chance to dig into it.

[leedm777]

@afh - I finally was able to setup a fresh El Capitan VM, but I could not reproduce your problem :-(

dlee-osx-elcapitan:~ dlee$ otool -L $(which asterisk)
/opt/homebrew/sbin/asterisk:
    /opt/homebrew/Cellar/asterisk/13.6.0/lib/libasteriskssl.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version 10.9.0)
    /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
    /usr/lib/libicucore.A.dylib (compatibility version 1.0.0, current version 55.1.0)
    /usr/lib/libxslt.1.dylib (compatibility version 3.0.0, current version 3.26.0)
    /usr/lib/libsqlite3.dylib (compatibility version 9.0.0, current version 216.4.0)
    /opt/homebrew/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /opt/homebrew/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /opt/homebrew/opt/jansson/lib/libjansson.4.dylib (compatibility version 12.0.0, current version 12.0.0)
    /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
    /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)
    /opt/homebrew/opt/gcc/lib/gcc/5/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.21.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1225.1.1)
    /opt/homebrew/lib/gcc/5/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

I have made a few changes recently, but it seems extremely unlikely that one of those accidentally fixed the issue.

It's also possible that you have some dependency in /usr that the configure script is picking up and putting in before /opt/homebrew/opt/openssl. Can you brew reinstall --verbose asterisk 2>&1 | tee asterisk.log, and pastebin the output? Hopefully there's some clue in the output that can elaborate the differences between our builds.

[afh]

@leedm777 I'm quite clueless at the moment.

I've updated to the latest version of the formula from you tap and as requested I've gathered all relevant files in the following gist: https://gist.github.com/afh/0db81c018136f2d40598

(Personally I like working with log files locally on my workstation, so here's a link to download a ZIP archive of the gist: https://gist.github.com/afh/0db81c018136f2d40598/archive/f31103c834aa6ea2f260e3a89ba65709beefeaba.zip)

I wonder what the issue could be?

[leedm777]

@afh Thanks for the feedback!

I finally got a setup to reproduce it.

1. Clean VM
2. Install Xcode
3. Install command-line-tools
4. Install brew into /opt/homebrew
5. brew install asterisk --with-clang

(FWIW, the scenario above where I couldn't reproduce the issue, I was using the CLT without Xcode, and did not use --with-clang when building Asterisk).

I'm trying to narrow it down further, but for whatever reason the linker is sometimes searching /usr/lib prior to /opt/homebrew/opt/openssl. That path isn't explicitly passed into the linker (in fact, the linker arguments are basically the same for working vs. broken builds), so I'm not sure why it would be doing that. My guess is that there is probably some crazy rpath thing happening at link time, but I'm not sure yet.

[leedm777]

@afh I've confirmed that Xcode+CLT+GCC does not have the link problem you've discovered, at least on my machine

Can you try re-installing without the --with-clang option, and capture the verbose output from that? If it's consistent with what I'm seeing, that should avoid the problem.

On my regular dev machine, --with-clang seems to do the right thing if Homebrew is installed in /usr/local. And GCC does the right thing when Homebrew is installed into /opt/homebrew.

So maybe it's the combination of --with-clang and the non-standard Homebrew directory.

[afh]

@leedm777 Compiling gcc takes a while… ⏳ I'll post updates as soon as it's done

[afh]

@leedm777 I can confirm that compiling asterisk without the --with-clang option using GCC asterisk is then correctly linked to the homebrewed OpenSSL and the segfaults do not occur. 😄

Unfortunately I find GCC quite heavy as a dependency and would really like to see asterisk compile correctly with LLVM. Yet I understand that it could be of rather little interest to you to see this edge case fixed.

I'm uncertain as to whether the different behaviour in GCC and LLVM when linking OpenSSL is within the compilers itself or within the asterisk specific black magic 🔮 configure; make; make install.

[leedm777]

I may poke around with it some more, but this is getting into the blackest of the Asterisk build magic. I doubt it's even in clang specifically, but something about Xcode's distribution of clang. Or even in some library distributed with Xcode.

FWIW, if you could run homebrew in /usr/local, GCC could be poured from a bottle instead of compiled. It's still a heavy dependency, but not having to build it lightens it up a lot.