[HSL] Compile v1.0.0 #6549

amontoison · 2023-04-07T22:37:32Z

With the HSL team, we developed a JuliaHSL package that contains all the HSL packages together.
Because the package JuliaHSL requires an academic license, we can't precompile it here with Yggdrasil.
To ease the use of it in the Julia ecosystem, we compile a dummy version here such that we can add it as a dependency of any Julia package.
The dummy version only has one C routine and the users will be able to call it to check if they have the dummy version or not and potentially call other routines.

using HSL_jll

# Function also available in the Julia interface HSL.jl
function JULIAHSL_isfunctional()
    @ccall libhsl.JULIAHSL_isfunctional()::Bool
end

if JULIAHSL_isfunctional()
  ...  # use an HSL linear solver
else
  ...  # use a SuiteSparse linear solver
end

The users will be able to download the JuliaHSL package (source code) with the "real" HSL_jll.jl Julia package (for Julia >= 1.9) from https://licences.stfc.ac.uk/.
It requires Julia v1.9 because we want to use LBT on all platforms.

The real HSL_jll has the same products than the dummy HSL_jll.jl package generated here. The main difference is that we have an override folder with the content of the "real" tarballs precompiled with BinaryBuilder.jl at the Rutherford Appleton Laboratory (RAL).

Thus, the RAL will provide these archives:

JuliaHSL.zip (source code)
HSL_jll_x86_64-apple-darwin-libgfortran4.zip (Julia package)
HSL_jll_x86_64-linux-gnu-libgfortran5.zip (Julia package)
HSL_jll_x86_64-w64-mingw32-libgfortran5.zip (Julia package)
HSL_jll_aarch64-apple-darwin-libgfortran5 (Julia package)
...

The user will choose the archive based on the result of Base.BinaryPlatforms.host_triplet() and install it with just a dev ..

Comments:

We also need a dummy version because Julia must downloads an artifact from the Artifacts.toml even if we have an override folder. Julia will not use it but it returns an error if it can't download an archive;
The hsl_ma57 solver is used for sparse \ in MATLAB.

H/HSL/build_tarballs.jl

vchuravy · 2023-04-08T05:14:43Z

I am not sure why a jll package is necessary? You could just handle this in a normal Julia package and allow the user to set the path to the library they obtained from you?

amontoison · 2023-04-08T06:01:00Z

@vchuravy The shared library that we precompile requires some dependencies (METIS, BLAS/LAPACK, MPI). By using a jll package we are sure that they are installed and loaded on all platforms before any call to libhsl.${dlext}.

vchuravy · 2023-04-08T06:59:46Z

Your non JLL package can also depend on these dependencies. JLL packages are not special in that way.

amontoison · 2023-04-08T08:17:48Z

Ok so an alternative solution is to add METIS_jll, LBT_jll, MicrosoftMPI_jll and MPICH_jll in the Project.toml of HSL.jl and also an option to provide the path to the shared library libhsl.

The drawbacks that I have in mind are:

The Windows users must add an environment variable and some of them don't know how to do that;
If we compile the tarballs with different compat entries of the dependencies or Julia in the future, a Project.toml for each new jll is preferable otherwise we will not be able to support multiple version of precompiled JuliaHSL in HSL.jl;
We can't use the include directory of HSL_jll to regenerate or update the Julia wrappers with Clang.jl;
One special application of HSL_jll is with Ipopt.jl that can dynamically load it to use HSL solvers (Linking full hsl library to Ipopt.jl on ubuntu jump-dev/Ipopt.jl#342) and the jll package is probably the most user-friendly form. We don't want to use HSL_jll in a Julia code for this case but with Ipopt_jll.

@odow Can you give your point of view too?

odow · 2023-04-08T08:45:11Z

I think @vchuravy's point was that we don't need this to be a jll with a C dependency.

The actual HSL_jll.jl that you download can be different structure to the dummy package.

We register an HSL_jll.jl dummy package without needing to compile via Yggdrasil. There's nothing special about the _jll suffix in the package manager.

odow · 2023-04-08T08:46:00Z

The drawbacks that I have in mind are:

The Windows users must add an environment variable and some of them don't know how to do that;

I don't understand this. Can you explain why it's needed?

amontoison · 2023-04-08T19:44:36Z

The drawbacks that I have in mind are:

The Windows users must add an environment variable and some of them don't know how to do that;

I don't understand this. Can you explain why it's needed?

I thought that Valentin wanted that we give a path to the precompiled shared library libhsl with an environment variable like:

amontoison · 2023-04-08T20:37:45Z

I think @vchuravy's point was that we don't need this to be a jll with a C dependency.

The actual HSL_jll.jl that you download can be different structure to the dummy package.

We register an HSL_jll.jl dummy package without needing to compile via Yggdrasil. There's nothing special about the _jll suffix in the package manager.

Ok, I misunderstood the point of @vchuravy.
He probably wanted to say that we don't need an HSL_jll.jl package with the current form.
I thought that he said that we didn't need a dummy HSL_jll.jl at all.

For the current use of HSL_jll.jl in HSL.jl and Ipopt.jl, we could do what you suggest.
My last worry is for the future GALAHAD_jll, GALAHAD is a package developed by the HSL team (like SPRAL) that we plan to precompile with Ygddrasil.
It uses the HSL solvers as optional dependencies and we are working on a new release with a Julia interface.

GALAHAD doesn't provide an option to load a libhsl library externally like Ipopt.
What they did is different, they developed dummy Fortran files (cc @jfowkes) :

such that we can compile them here in the shared library of the dummy HSL_jll and link the future GALAHAD_jll with it.
In this way, if the user has the "real" HSL_jll.jl, he will be able to use the HSL solvers automatically in GALAHAD.

jfowkes · 2023-04-12T15:54:19Z

The HSL team is happy to have this dummy JuliaHSL v1.0.0 package.

This can then be bumped to v2.0.0 for GALAHAD in future once its Julia interfaces are ready.

ViralBShah · 2023-04-14T03:48:41Z

@amontoison Please merge when you feel this is ready.

amontoison · 2023-04-14T04:13:29Z

@ViralBShah Can you merge it?
I'm not authorized to merge pull requests for this repository.

ViralBShah · 2023-04-16T14:20:13Z

@amontoison I have invited you to have Yggdrasil commit access.

* master: (38 commits) Bump LLVM to 14.0.6-3 (JuliaPackaging#6589) [llama_cpp] Bump version (JuliaPackaging#6588) [GMT] Let libcurl_jll compat have 7.7s and 8.0.1 (JuliaPackaging#6586) Make autotools more relocatable (JuliaPackaging#6585) [Libtool] Clean up, add more executable products (JuliaPackaging#6581) Bump MParT_jll to 2.0.2 (JuliaPackaging#6584) [automake] Add Automake `v1.16` (JuliaPackaging#6583) [autoconf] Add `autoconf` `v2.71` (JuliaPackaging#6582) [Bison] ensure the package does not hardcode paths (JuliaPackaging#6580) [gawk] Add v5.2.1 (JuliaPackaging#6579) New Recipe: Kira v2.3.0 (JuliaPackaging#6485) Regenerate PTSCOTCH_jll with a compat entry for SCOTCH_jll (JuliaPackaging#6575) [file] Add GNU file `v5.44` (JuliaPackaging#6573) [HSL] Compile v1.0.0 (JuliaPackaging#6549) Regenerate MUMPS_jll with a compat entry for SCOTCH_jll (JuliaPackaging#6576) Regenerate MMG_jll with a compat entry for SCOTCH_jll (JuliaPackaging#6571) Regenerate neper_jll with a compat entry for SCOTCH_jll (JuliaPackaging#6572) [Singular] update to 4.3.2-0 + depth pathches (JuliaPackaging#6577) Regenerate qr_mumps_jll with a compat entry for SCOTCH_jll (JuliaPackaging#6574) [patch] Add Patch `v2.7.6` (JuliaPackaging#6570) ...

amontoison · 2023-04-17T23:51:42Z

Thanks @ViralBShah and @giordano for the invitations!

I found a way to have an unique HSL_jll.jl package for all platforms. I modified the files in HSL_jll.jl/src/wrapper such that the shared libraries of all platforms coexist in the override folder.

function __init__()
    JLLWrappers.@generate_init_header(CompilerSupportLibraries_jll, MPICH_jll, METIS_jll, libblastrampoline_jll)
    JLLWrappers.@init_library_product(
        libhsl,
        "lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib",
        RTLD_LAZY | RTLD_DEEPBIND,
    )

    JLLWrappers.@generate_init_footer()
end  # __init__()

function __init__()
    JLLWrappers.@generate_init_header(CompilerSupportLibraries_jll, MicrosoftMPI_jll, METIS_jll, libblastrampoline_jll)
    JLLWrappers.@init_library_product(
        libhsl,
        "bin\\x86_64-w64-mingw32-libgfortran5\\libhsl.dll",
        RTLD_LAZY | RTLD_DEEPBIND,
    )

    JLLWrappers.@generate_init_footer()
end  # __init__()

function __init__()
    JLLWrappers.@generate_init_header(CompilerSupportLibraries_jll, MPICH_jll, METIS_jll, libblastrampoline_jll)
    JLLWrappers.@init_library_product(
        libhsl,
        "lib/x86_64-linux-gnu-libgfortran5/libhsl.so",
        RTLD_LAZY | RTLD_DEEPBIND,
    )

    JLLWrappers.@generate_init_footer()
end  # __init__()

it works perfectly except for Apple platforms,

julia> using HSL_jll
ERROR: InitError: could not load library "/Users/alexis/Downloads/HSL_jll/override/lib/x86_64-apple-darwin-libgfortran4/libhsl.dylib"
dlopen(/Users/alexis/Downloads/HSL_jll/override/lib/x86_64-apple-darwin-libgfortran4/libhsl.dylib, 1): no suitable image found.  Did find:
	/Users/alexis/Downloads/HSL_jll/override/lib/x86_64-apple-darwin-libgfortran4/libhsl.dylib: code signature in (/Users/alexis/Downloads/HSL_jll/override/lib/x86_64-apple-darwin-libgfortran4/libhsl.dylib) not valid for use in process using Library Validation: library load disallowed by system policy

I generated the tarballs with BinaryBuilder.jl on my Linux (Ubuntu 22.04 -- x86_64-linux-gnu-libgfortran5).
Are you using a Mac to sign the tarballs with Yggdrasil?

giordano · 2023-04-18T00:09:38Z

Are you using a Mac to sign the tarballs with Yggdrasil?

No, we sign on Linux, like the rest of the building pipeline. But I'm not sure this answers your question.

amontoison · 2023-04-18T00:19:28Z

It's a good news that you sign the tarballs on Linux.
It probably means that the tarballs are not signed on my computer for an unknown reason.
Is it possible that the tarballs are not signed if we use the option --deploy="local"?

We use the command

julia --color=yes build_tarballs.jl aarch64-apple-darwin-libgfortran5,aarch64-linux-gnu-libgfortran3,aarch64-linux-gnu-libgfortran4,aarch64-linux-gnu-libgfortran5,aarch64-linux-musl-libgfortran3,aarch64-linux-musl-libgfortran4,aarch64-linux-musl-libgfortran5,powerpc64le-linux-gnu-libgfortran3,powerpc64le-linux-gnu-libgfortran4,powerpc64le-linux-gnu-libgfortran5,x86_64-apple-darwin-libgfortran3,x86_64-apple-darwin-libgfortran4,x86_64-apple-darwin-libgfortran5,x86_64-linux-gnu-libgfortran3,x86_64-linux-gnu-libgfortran4,x86_64-linux-gnu-libgfortran5,x86_64-linux-musl-libgfortran3,x86_64-linux-musl-libgfortran4,x86_64-linux-musl-libgfortran5,x86_64-unknown-freebsd-libgfortran3,x86_64-unknown-freebsd-libgfortran4,x86_64-unknown-freebsd-libgfortran5,x86_64-w64-mingw32-libgfortran3,x86_64-w64-mingw32-libgfortran4,x86_64-w64-mingw32-libgfortran5 --debug --verbose --deploy="local"

to generate the tarballs for all 64-bits platforms.

amontoison · 2023-04-25T15:09:52Z

@giordano @staticfloat
I investigated and the shared libraries are correctly signed on my platform.
Do you use a special command to zip / unzip the tarballs?
I create the HSL_jll.jl package with the tarballs that I unzip with tar -xzf and at the end I zip the Julia package with tar -czf.
Another possibility is that we didn't download the package like Julia downloads the tarballs. I think that Julia uses the curl command and it could avoid the archive to be somehow quarantined.

An alternative solution is to ask the users to resign the shared libraries.

using ldid_jll

path_hsljll = Base.find_package("HSL_jll")
path_override = joinpath(dirname(path_hsljll), "..", "override")

run(`$(ldid()) -S -d $(path_override)/lib/aarch64-apple-darwin-libgfortran5/libhsl.dylib`)
run(`$(ldid()) -S -d $(path_override)/lib/x86_64-apple-darwin-libgfortran3/libhsl.dylib`)
run(`$(ldid()) -S -d $(path_override)/lib/x86_64-apple-darwin-libgfortran4/libhsl.dylib`)
run(`$(ldid()) -S -d $(path_override)/lib/x86_64-apple-darwin-libgfortran5/libhsl.dylib`)

or

if Sys.isapple()
  path_hsl_jll = Base.find_package("HSL_jll")
  libdir = joinpath(dirname(path_hsl_jll), "..", "override", "lib")

  run(`codesign -f -s - $(libdir)/aarch64-apple-darwin-libgfortran5/libhsl.dylib`)
  run(`codesign -f -s - $(libdir)/x86_64-apple-darwin-libgfortran3/libhsl.dylib`)
  run(`codesign -f -s - $(libdir)/x86_64-apple-darwin-libgfortran4/libhsl.dylib`)
  run(`codesign -f -s - $(libdir)/x86_64-apple-darwin-libgfortran5/libhsl.dylib`)
end

giordano · 2023-04-25T15:16:14Z

Honestly I don't understand the problem. The library is already signed during audit as far as I can tell: https://buildkite.com/julialang/yggdrasil/builds/2175#01876299-9fbf-439c-8655-972f62f972e1/638-770

amontoison commented Apr 8, 2023

View reviewed changes

H/HSL/build_tarballs.jl Outdated Show resolved Hide resolved

H/HSL/build_tarballs.jl Outdated Show resolved Hide resolved

amontoison force-pushed the HSL_jll branch from fd1cc45 to 2d9c6c0 Compare April 8, 2023 20:39

amontoison added 2 commits April 8, 2023 16:39

[HSL] Compile v1.0.0

1a2ad46

Remove the dependencies not used by the dummy version

aa371e1

amontoison force-pushed the HSL_jll branch from 2d9c6c0 to aa371e1 Compare April 8, 2023 20:39

Merge branch 'master' into HSL_jll

f5a3035

ViralBShah merged commit faf9d83 into JuliaPackaging:master Apr 14, 2023

amontoison deleted the HSL_jll branch April 14, 2023 15:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HSL] Compile v1.0.0 #6549

[HSL] Compile v1.0.0 #6549

amontoison commented Apr 7, 2023 •

edited

Loading

vchuravy commented Apr 8, 2023

amontoison commented Apr 8, 2023 •

edited

Loading

vchuravy commented Apr 8, 2023

amontoison commented Apr 8, 2023 •

edited

Loading

odow commented Apr 8, 2023

odow commented Apr 8, 2023

amontoison commented Apr 8, 2023

amontoison commented Apr 8, 2023 •

edited

Loading

jfowkes commented Apr 12, 2023

ViralBShah commented Apr 14, 2023

amontoison commented Apr 14, 2023 •

edited

Loading

ViralBShah commented Apr 16, 2023

amontoison commented Apr 17, 2023 •

edited

Loading

giordano commented Apr 18, 2023

amontoison commented Apr 18, 2023 •

edited

Loading

amontoison commented Apr 25, 2023 •

edited

Loading

giordano commented Apr 25, 2023

[HSL] Compile v1.0.0 #6549

[HSL] Compile v1.0.0 #6549

Conversation

amontoison commented Apr 7, 2023 • edited Loading

vchuravy commented Apr 8, 2023

amontoison commented Apr 8, 2023 • edited Loading

vchuravy commented Apr 8, 2023

amontoison commented Apr 8, 2023 • edited Loading

odow commented Apr 8, 2023

odow commented Apr 8, 2023

amontoison commented Apr 8, 2023

amontoison commented Apr 8, 2023 • edited Loading

jfowkes commented Apr 12, 2023

ViralBShah commented Apr 14, 2023

amontoison commented Apr 14, 2023 • edited Loading

ViralBShah commented Apr 16, 2023

amontoison commented Apr 17, 2023 • edited Loading

giordano commented Apr 18, 2023

amontoison commented Apr 18, 2023 • edited Loading

amontoison commented Apr 25, 2023 • edited Loading

giordano commented Apr 25, 2023

amontoison commented Apr 7, 2023 •

edited

Loading

amontoison commented Apr 8, 2023 •

edited

Loading

amontoison commented Apr 8, 2023 •

edited

Loading

amontoison commented Apr 8, 2023 •

edited

Loading

amontoison commented Apr 14, 2023 •

edited

Loading

amontoison commented Apr 17, 2023 •

edited

Loading

amontoison commented Apr 18, 2023 •

edited

Loading

amontoison commented Apr 25, 2023 •

edited

Loading