macOS build source crashes the OS (for both julia-1.8.0-beta3 and master)

[fxcoudert]

Now that #44584 is fixed (thanks @giordano !), we tried to build julia-1.8.0-beta3 from source as part of Homebrew again. It crashed our CI nodes, and does the same locally on my laptop: forced reboot was the only solution.

The crash occurs in this step:

cd /private/tmp/julia-20220430-38583-144297r/julia-1.8.0-beta3/base && /private/tmp/julia-20220430-38583-144297r/julia-1.8.0-beta3/usr/bin/julia -C "generic;cortex-a57;thunderx2t99;armv8.2-a,crypto,fullfp16,lse,rdm" --output-ji /private/tmp/julia-20220430-38583-144297r/julia-1.8.0-beta3/usr/lib/julia/corecompiler.ji.tmp --startup-file=no --warn-overwrite=yes -g0 -O0 compiler/compiler.jl

When this is running, we have the following:

• one julia process, using ~100% CPU
• kernel_task is using about ~200% CPU, and increasing fast
• analyticsd and logd are also furiously active
• the Console is flooded with messages (several thousands per second)

This is 100% reproducible 😭

The build command-line is:

make VERBOSE=1 USE_BINARYBUILDER=0 prefix=/opt/homebrew/Cellar/julia/1.8.0-beta3 sysconfdir=/opt/homebrew/etc USE_SYSTEM_CSL=1 USE_SYSTEM_LIBUNWIND=1 USE_SYSTEM_PCRE=1 USE_SYSTEM_OPENLIBM=1 USE_SYSTEM_BLAS=1 USE_SYSTEM_LAPACK=1 USE_SYSTEM_GMP=1 USE_SYSTEM_MPFR=1 USE_SYSTEM_LIBSUITESPARSE=1 USE_SYSTEM_UTF8PROC=1 USE_SYSTEM_MBEDTLS=1 USE_SYSTEM_LIBSSH2=1 USE_SYSTEM_NGHTTP2=1 USE_SYSTEM_CURL=1 USE_SYSTEM_LIBGIT2=1 USE_SYSTEM_PATCHELF=1 USE_SYSTEM_ZLIB=1 USE_SYSTEM_P7ZIP=1 LIBBLAS=-lopenblas LIBBLASNAME=libopenblas LIBLAPACK=-lopenblas LIBLAPACKNAME=libopenblas USE_BLAS64=0 PYTHON=python3 MACOSX_VERSION_MIN=12 JULIA_CPU_TARGET=generic;cortex-a57;thunderx2t99;armv8.2-a,crypto,fullfp16,lse,rdm TAGGED_RELEASE_BANNER=Built by Homebrew (v1.8.0-beta3) install

The install is from julia-1.8.0-beta3.tar.gz source, with this patch applied: 5e74df5f

[giordano]

Is this only with M1 or also Intel?

[fxcoudert]

The console logs are full of this:

14:24:49.319953+0200	kernel	IOReturn CoreAnalyticsHub::sendUpEventAndPayload(OSString *, OSObject *)::405:userClient->sendDataToUser failed general error
14:24:49.319972+0200	kernel	IOReturn CoreAnalyticsUserClient::sendDataToUser(uint8_t *, size_t)::291: queue full, dropping

and when I said thousands per second, it's actually tens of thousands per second. And even when the Julia process is killed in time, it takes a while for the system to process all that and recover.

[fxcoudert]

If I run julia inside lldb, I get stuck there:

(lldb) r
Process 41267 launched: '/private/tmp/julia-20220430-2066-oj7ig5/julia-1.8.0-beta3/usr/bin/julia' (arm64)
warning: (arm64) /private/tmp/julia-20220430-2066-oj7ig5/julia-1.8.0-beta3/usr/lib/libjulia-internal.1.8.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
warning: (arm64) /private/tmp/julia-20220430-2066-oj7ig5/julia-1.8.0-beta3/usr/lib/libjulia-codegen.1.8.dylib empty dSYM file detected, dSYM was created with an executable with no debug info.
Process 41267 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BREAKPOINT (code=1, subcode=0x1b8c57ba0)
    frame #0: 0x00000001b8c57ba0 libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::step() + 1744
libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::step:
->  0x1b8c57ba0 <+1744>: brk    #0xc471
    0x1b8c57ba4 <+1748>: pacib  x16, x8
    0x1b8c57ba8 <+1752>: str    x16, [sp, #0x128]
    0x1b8c57bac <+1756>: add    x1, sp, #0x28
Target 0: (julia) stopped.

The top of the backtrace:

  * frame #0: 0x00000001b8c57ba0 libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::step() + 1744
    frame #1: 0x0000000101857ae8 libjulia-internal.1.8.dylib`jl_unw_stepn + 248
    frame #2: 0x0000000101857d24 libjulia-internal.1.8.dylib`rec_backtrace + 132
    frame #3: 0x0000000101834b88 libjulia-internal.1.8.dylib`record_backtrace + 36
    frame #4: 0x0000000101834b38 libjulia-internal.1.8.dylib`ijl_throw + 72
    frame #5: 0x000000014622409c
    frame #6: 0x00000001018260a4 libjulia-internal.1.8.dylib`jl_f__call_in_world_total + 192
    frame #7: 0x000000010181ab34 libjulia-internal.1.8.dylib`ijl_apply_generic + 1684
    frame #8: 0x0000000101825cfc libjulia-internal.1.8.dylib`do_apply + 1944

Even if I tell lldb to continue, it gets stuck at that same address forever. I suppose it's this loop that triggers the crash data collection in macOS, in a loop, sending the OS down the spiral.

[giordano]

So this happens with system unwind? This is what we're currently using in the official binaries: https://github.com/JuliaPackaging/Yggdrasil/tree/cc0b0caa2df2857cc9c00527732cb79c97ed35a1/L/LLVMLibUnwind/LLVMLibUnwind%4012.0.1. There are some patches there, I don't know if anything is relevant here. Might try not using system libunwind?

[fxcoudert]

• Re. Intel: I do not know, I've started a build there, we will see.
• Re. using Julia's patched libunwind, it does not currently build from source: Build failure due to missing patch llvm-libunwind-prologue-epilogue.patch #44499
• The bug is also present on master

[ViralBShah]

Please reopen if still an issue.