Limit libraries to be included in the bundle

[simonbyrne]

If julia_libdir() == "/usr/lib" then it will attempt to bundle the whole directory into the app. Fixes #16.

@jlapeyre, @OliverEvans96, @feanor12, @alexriss: would you please be able to try out this branch (]add PackageCompiler#sb/limit-bundle) and see if it fixes the problem?

[alexriss]

Thank you! This seems to fix the bundle_julia_libraries function. But now I get an error in bundle_cert:

IOError: open("/usr/bin/../share/julia/cert.pem", 0, 0): no such file or directory (ENOENT)

at PackageCompiler.jl:1206

I am somehow not able to find a cert.pem belonging to Julia on my system (Manjaro 22, Julia installed via the package manager).

[simonbyrne]

Thanks @alexriss: I'm not sure how the certificates are used, so don't know what the fix is (can we just drop them?)

FWIW, the recommendation is to use the binaries from https://julialang.org/downloads/, as system packages often don't include patches or other fixes to upstream libraries that Julia requires.

[codecov]

Codecov Report

Merging #742 (59b6528) into master (376e3bf) will decrease coverage by 0.57%.
The diff coverage is 87.50%.

@@            Coverage Diff             @@
##           master     #742      +/-   ##
==========================================
- Coverage   93.46%   92.88%   -0.58%     
==========================================
  Files           2        2              
  Lines         520      520              
==========================================
- Hits          486      483       -3     
- Misses         34       37       +3     

Impacted Files	Coverage Δ
src/PackageCompiler.jl	93.29% <87.50%> (-0.42%)	⬇️
src/juliaconfig.jl	88.37% <0.00%> (-2.33%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[feanor12]

I tried to compile a small demo project with aur/juliaup, aur/julia-bin and the community/julia package.

With the new branch of PackageCompiler aur/juliaup and aur/julia-bin work.

the community/julia package was throwing an error:

ERROR: IOError: open("/usr/bin/../share/julia/cert.pem", 0, 0): no such file or directory (ENOENT)
Stacktrace:
 [1] cp
   @ ./file.jl:376 [inlined]
 [2] bundle_cert
   @ ~/.julia/packages/PackageCompiler/Lz0Nj/src/PackageCompiler.jl:1211 [inlined]
 [3] create_app(package_dir::String, app_dir::String; executables::Nothing, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, force::Bool, c_driver_program::String, cpu_target::String, include_lazy_artifacts::Bool, sysimage_build_args::Cmd, include_transitive_dependencies::Bool)
   @ PackageCompiler ~/.julia/packages/PackageCompiler/Lz0Nj/src/PackageCompiler.jl:712
 [4] create_app(package_dir::String, app_dir::String)
   @ PackageCompiler ~/.julia/packages/PackageCompiler/Lz0Nj/src/PackageCompiler.jl:685
 [5] top-level scope
   @ REPL[3]:1

The community/julia package, however, does not ship cert.pem at all.

[alexriss]

I have done some more tests.

If I install the official binaries on my Manjaro distro, everything works, even with the previous version without any changes.

With the version from the package manager, everything is more complicated. I have adapted the bundle_cert function as follows:

function bundle_cert(dest_dir)
    cert_path = joinpath(Sys.BINDIR, "..", "share", "julia", "cert.pem")
    if Sys.isunix() && !isfile(cert_path)
        cert_path = "/etc/ssl/cert.pem"
    end
    share_path = joinpath(dest_dir, "share", "julia")
    mkpath(share_path)
    isfile(cert_path) && cp(cert_path, joinpath(share_path, "cert.pem"))
end

I am not quite sure if this is the right cert.pem file. However, I get a different error now for some packages that rely on these certs:

Failed to precompile Blink [ad839575-38b3-5650-b840-f874b8c74a25] to /home/riss/.julia/compiled/v1.8/Blink/jl_z6KvWg.
ERROR: LoadError: InitError: SystemError: opening file "/usr/share/julia/cert.pem": No such file or directory

I have traced the problem down further, it seems to come from the package MozillaCACerts_jll.jl, which seems to be generated by JLLWrappers.jl.

Now somewhere there is a function MozillaCACerts_jll.find_artifact_dir() which does basically this:

global artifact_dir = dirname(Sys.BINDIR)
global cacert = normpath(Sys.BINDIR, Base.DATAROOTDIR, "julia", "cert.pem")

and it ends up with:

julia> Base.DATAROOTDIR
"../share"

julia> Sys.BINDIR
"/usr/bin"

julia> MozillaCACerts_jll.artifact_dir
"/usr"

So this does not seem to be a PackageCompiler problem anymore, since I cannot compile these packages directly under my Julia instance, see also this thread. And yes, this should have been the first thing to try, would have saved quite some time.

tl;dr: Seems like the solution is to not use the community/julia package for now.