-
-
Notifications
You must be signed in to change notification settings - Fork 188
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
macOS: System image file fails consistency check on binary startup #738
Comments
Similar issue here on
the app execute smoothly when it is in the folder where it has been compiled but fail with > Code/Julia/PackageCompiler.jl/examples/MyAppCompiled/bin/MyApp
Downloaded artifact: MKL
ARGS = String[]
Base.PROGRAM_FILE = "Code/Julia/PackageCompiler.jl/examples/MyAppCompiled/bin/MyApp"
...
> cp -r Code/Julia/PackageCompiler.jl/examples/MyAppCompiled .
> MyAppCompiled/bin/MyApp
ERROR: System image file failed consistency check: maybe opened the wrong version? |
I'm running into this issue as well, just trying to build any sysimage, e.g.
then
|
Same issue, tried on Julia 1.9.3, 1.9.2, 1.10.0-beta2, all with PackageCompiler version 2.1.9, also tried on Julia 1.9.2 with PackageCompiler version 2.1.7 (the mix I had when I last had it running a few weeks ago), this versioninfo() below is from this latest combination.
|
I'm seeing the same error when building GSEA-MSigDB/GSEA2.jl@79364ca |
This happens on macOS only? |
I have only tried on macOS Sonoma 14.0. |
I'm still getting the error on PackageCompiler 2.1.11 (which specifically addresses this issue) on macOS Ventura (13.6). Running Julia 1.8.5, not sure if 1.9 is required for the fix. |
Thanks for reporting. Just to clarify: There is no fix available as of yet (also not in PC v2.1.11) |
Thanks for clarifying! I assumed that the closed issue indicated that it was fixed. Please let me know if I can contribute to a fix in some way. |
At the moment we are lacking an understanding of why the consistency check fails. If you manage to find out about this, we could start looking for a remedy 🙂 |
I was able to build apps using Julia 1.9.0, PackageCompiler 2.1.7, and macOS 13. Recently I upgraded Julia to 1.9.3, PackageCompiler to 2.1.2, and macOS to 14, and the error occurred. I tried to restore Julia to 1.9.0 and PackageCompiler to 2.1.7, and the error still occurs. So I guess macOS should cause it since Xcode Command Line Tools was also upgraded to 15, which is required by PackageCompiler to compile executables. Unfortunately, when I tried to downgrade Xcode Command Line Tools, it warned that older version Xcode C. L. T. can not be installed on macOS 14... And I can not downgrade the OS... :( I would recommend anyone (@jayscook ?) still using macOS 13 try to install an older version (v14.3?)of Xcode. It can be downloaded from: https://developer.apple.com/download/all/ |
i can't build sysimages on apple silicon either. please let me know how i can help fix it. |
I can confirm it is related to |
it really works! but remember to set the compiler to you can specify the compiler by running an export JULIA_CC="gcc-13" |
Great catch! Could you please report which version of clang/gcc you are each using by default on your systems that works (using |
clang v15 from Xcode CLT doesn't work
(clang v14 should work, but older version Xcode CLT can not be installed on latest macOS) gcc v13 from brew works
versioninfo
|
Good catch! I've installed gcc-13 with
|
Since the versions of Julia and LLVM are different, maybe you should try older versions of gcc or upgrade Julia to 1.9. |
Hm, this is unfortunate. It also means we're back to square one, since I don't understand now (again) what the necessary conditions are to trigger the error. |
This is expected as per JuliaLang/julia#49581. You should pick an earlier version of GCC/Clang or newer version of Julia. |
How can we make this work with the latest mac gcc? I would be nice to not have to downgrade. |
TBH, I do not know, since I neither know the rationale behind this check nor do I understand what it actually checks 😅 It seems this check was introduced in JuliaLang/julia@4e7ec79, specifically here. @vtjnash can you maybe shed some light on this and help us understand
|
It simply checks if the symbol |
Thanks for the explanation! So you mean we should set Sorry for the many questions, we just don't have any enough experience to tackle this at the moment. |
See |
I've been trying to reproduce the problem on my M1 Macbook, but without success so far - all my sysimages or apps I build seem to work. @KwatMDPhD @bjarthur @hycakir @ctarn Can someone who had this problem please provide an MWE with a sysimage/app as small as possible and the exact commands you use to build it? In addition, please also post the output of |
I can confirm this works for me too on MacBook Air M1 with Julia Version 1.9.4
Commit 8e5136fa297 (2023-11-14 08:46 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: macOS (arm64-apple-darwin22.4.0)
CPU: 8 × Apple M1
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
Threads: 1 on 4 virtual cores and λ gcc --version
Apple clang version 15.0.0 (clang-1500.0.40.1)
Target: arm64-apple-darwin23.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin But as @PhilReinhold stated, that is one heck of a hack! |
#738 (comment) works on my Intel macOS :D using PackageCompiler
using PackageCompiler: get_extra_linker_flags, julia_libdir, julia_private_libdir, ldlibs, bitflag, march, run_compiler
# https://github.com/JuliaLang/PackageCompiler.jl/issues/738#issuecomment-1838901893
# I know this causes type piracy, which is a bad solution, but it will save life for those who want to comple for macOS somehow
function PackageCompiler.create_sysimg_from_object_file(object_files::Vector{String},
sysimage_path::String;
version,
compat_level::String,
soname::Union{Nothing, String})
if soname === nothing && (Sys.isunix() && !Sys.isapple())
soname = basename(sysimage_path)
end
mkpath(dirname(sysimage_path))
# Prevent compiler from stripping all symbols from the shared lib.
o_file_flags = Sys.isapple() ? `-Wl,-all_load $object_files -Wl,-ld_classic` : `-Wl,--whole-archive $object_files -Wl,--no-whole-archive`
extra = get_extra_linker_flags(version, compat_level, soname)
cmd = `$(bitflag()) $(march()) -shared -L$(julia_libdir()) -L$(julia_private_libdir()) -o $sysimage_path $o_file_flags $(Base.shell_split(ldlibs())) $extra`
run_compiler(cmd; cplusplus=true)
return nothing
end
using MyExe
create_app(pkgdir(MyExe), "build", force=true) See also my gist |
Hello @PhilReinhold @dokie @terasakisatoshi, Could you please explain a little more on where should I implement these code #738 I am totally new to julia and am using it as a dependency of a dependency of a snakemake pipeline. Some clue will be very helpful! Thanks!! |
@PhilReinhold , I can also confirm your hack worked for my use case. Definitely would prefer a cleaner solution but thank all of you for your efforts and for sharing. |
Here are my tests on macOS Sonoma 14.2.1 M1 (for aarch64) with PackageCompiler v2.1.17: clang 15.0.0 + Julia 1.9.4 = GOOD Based on this, I would be more inclined to think that this is an issue with Julia 1.10 rather than a difference between gcc and clang, but I am noob in this. |
Brand new machine M3 (today) on Sonoma 14.3 and install from scratch. None of options work: Just observing the generated Sysimage.dylib one easily can see that it is empty. That's the reason at least for me it not passing the consistency check. Size is 17KB for a project I usually get 400MB *.dylib. Do note that macos uses macos linker: Homebrew/homebrew-core#17794 Check the notes for XCode 15 linker section: https://developer.apple.com/documentation/xcode-release-notes/xcode-15-release-notes#Linking "A new linker has been written to significantly speed up static linking. It’s the default for all macOS, iOS, tvOS and visionOS binaries and anyone using the “Mergeable Libraries” feature. The classic linker can still be explicitly requested using -ld64, and will be removed in a future release." Some discussion also here: https://www.scivision.dev/xcode-ld_classic/, but setting LDFLAGS in environment variable seem not get passed to PlatformCompiler. Only thing which helped for me was manual modification of Hope these observations point to correct direction. Perhaps as an intermediate solution adding |
With "helped" do you mean that it works properly when using the workaround? Or that it generated "non-empty" .dylib files but still failed the consistency check? |
Let me try to be clearer: Just using different combinations of clang, gcc and julia "out-of-the-box" did not produce complete sysimage. Some combination did produce slightly larger (2MB vs 17kB), but not complete sysimage (~400MB). At the time I did not document exactly. But I can try to reproduce if needed. In all of the combinations, the compilation works fine, it takes time and compiles all the dependencies, but fails to link the complete *.dylib. And none of the produced libraries pass the consistency check. Setting just the LDFLAGS environment variable as suggested in one of the links did not help. It does not have any impact to Julia compiler as it has to CMake based C or C++ compilation. If I manually modify the PackageCompiler.jl as suggested at least Apple Clang 15.0.0 + Julia 1.10.0 combination works. It produces complete Sysimage and passes consistency check. I stopped there as I was happy to do some work. I can try other combinations as well, but I imagine they work as well if the culprit is the new XCode 15 linker or how it is used. In an another machine two months back after an upgrade from MacOS 13 to 14, a Mac Mini M2, just setting Still have access to both of the machines for further testing. |
FYI: Have to add that in all of these cases below the precompilation, excution of the program and then the compilation take approximately the same time. Only thing which I see is different is the failed or working linking of dylib. Apple Clang (15.0.0) and Julia 1.10:This does not work out-of-the-box:
gcc-13 (13.2.0) and Julia 1.10:This does not work out-of-the-box:
Manual edit of PackageCompiler.jl w Apple Clang (15.0.0) and Julia 1.10... that is addition of This works:
|
others have this problem too and used the same |
can someone with xcode version 14 or less try the |
Unfortunately I've seen this flag break compilation with xcode 14 since I think it's unrecognized. I've used the following locally to detect xcode version and conditionally apply the flag: function PackageCompiler.create_sysimg_from_object_file(object_files::Vector{String},
sysimage_path::String;
version,
compat_level::String,
soname::Union{Nothing, String})
if soname === nothing && (Sys.isunix() && !Sys.isapple())
soname = basename(sysimage_path)
end
mkpath(dirname(sysimage_path))
# Prevent compiler from stripping all symbols from the shared lib.
if Sys.isapple()
cltools_version_cmd = `pkgutil --pkg-info=com.apple.pkg.CLTools_Executables`
cltools_version = match(r"version: (.*)\n", readchomp(cltools_version_cmd))[1]
if startswith(cltools_version, "15")
o_file_flags = `-Wl,-all_load $object_files -Wl,-ld_classic`
else
o_file_flags = `-Wl,-all_load $object_files`
end
else
o_file_flags = `-Wl,--whole-archive $object_files -Wl,--no-whole-archive`
end
extra = get_extra_linker_flags(version, compat_level, soname)
cmd = `$(bitflag()) $(march()) -shared -L$(julia_libdir()) -L$(julia_private_libdir()) -o $sysimage_path $o_file_flags $(Base.shell_split(ldlibs())) $extra`
run_compiler(cmd; cplusplus=true)
return nothing
end |
@PhilReinhold submit a PR! is there any reason not to? you've found a nice solution and everyone here has verified it works for them. |
Can confirm that this worked for me too. Julia 1.9.4 and gcc-13. |
Hey folks, I went ahead and opened a PR for @PhilReinhold's fix. I'd love it if someone can review! I haven't contributed here before so please let me know if I'm missing any conventions, considerations, etc. |
I was experiencing the same problem and I can confirm the above fix works on
|
@FerreolS did you ever get resolution for your problem? It seems to be closely-related-yet-distinct from the workarounds mentioned in this thread. After using @PhilReinhold fix I am able to build an app and run it in its build location; but now I am facing your problem where relocating the app causes the I've created a new issue: #943 |
I run into the same problem again... I previously set o_file_flags = Sys.isapple() ? `-Wl,-ld_classic,-all_load $object_files` : `-Wl,--whole-archive $object_files -Wl,--no-whole-archive` Hope we can solve the issue soon.😵💫 |
I have face the same issue, hope it can be fix soon. |
Facing the same issue on x64 machine |
Upon creating an app, the process goes on smoothly, untill it is time to actually call the binary.
I get the error:
ERROR: System image file failed consistency check: maybe opened the wrong version?
I installed julia through
juliaup
. the juliaup folder is in~/.local/juliaup
and the julia versions in~/.local/julia/juliaup
.the binaries in
~/.local/juliaup/bin
are symlinked to~/.local/bin
which is in thePATH
.I also have julia aliased to
julia --project=@.
Could it be that the symlinking throws some pathing off somewhere?
The text was updated successfully, but these errors were encountered: