Add SCIP_jll as dependency

[rschwarz]

Fix #165 .

[rschwarz]

I might have introduced too many changes in the CI now, let's see...

[rschwarz]

It looks OK now, except for this nasty error on AppVeyor for Windows:

Exception: EXCEPTION_ACCESS_VIOLATION at 0xffffffffc5160000 -- unknown function (ip: FFFFFFFFC5160000)
in expression starting at C:\projects\scip-jl\test\direct_library_calls.jl:3
unknown function (ip: FFFFFFFFC5160000)
Allocations: 5949752 (Pool: 5947818; Big: 1934); GC: 6
ERROR: Package SCIP errored during testing
Stacktrace:
 [1] pkgerror(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Types.jl:52
 [2] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:1566
 [3] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:328
 [4] #test#61 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:67 [inlined]
 [5] test(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:coverage,),Tuple{Bool}}}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:80
 [6] top-level scope at none:1

I wouldn't know how to debug this, except for running more variations on AppVeyor.

[rschwarz]

I tried reproducing the failing tests on Windows and succeeded: I get the same error message, but no additional information.
When I try to include the test file in a Julia REPL, it simply closes on me, burying the output.

When I run the lines of the first test one-by-one, I find that the first couple of ccalls actually work OK. It's only the call to SCIPsolve that crashes.

This makes me think that the compilation of SCIP itself might be fine, but that there is a problem with one of the linked libraries, such as SoPlex or Ipopt, which are only used at this point. Maybe the SoPlex library should be listed explicitly as a LibraryProduct after all? Then again, why does it work on all OSs?

[matbesancon]

Does SCIP use the shared library or the executable? When building, make install only seems to install the SoPlex executable and not library

[rschwarz]

Actually, I believe that the SoPlex code is already contained in libscip.so. Here's what I get for the scip executable in the artifact on my Linux:

# rs@t440p:~/.julia/artifacts/d94fb1efe141bd5787829453f44129800a77305d/bin [14:59:25]
$ ldd scip 
	linux-vdso.so.1 (0x00007ffda0d7c000)
	libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f78fa3ea000)
	libgmpxx.so.4 => /usr/lib/x86_64-linux-gnu/libgmpxx.so.4 (0x00007f78fa3e1000)
	libbliss.so => not found
	libipopt.so.3 => not found
	libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f78fa39f000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f78fa21b000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f78fa096000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f78fa07c000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f78f9ebb000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f78fa4ac000)

What is surprising here, though, is the reference of libgmp. This is supposed to be disabled via the cmake flag.

[matbesancon]

libipopt.so.3 => not found

also surprising is the Ipopt not being there?

[rschwarz]

No, I think that's fine. There is Julia code in the SCIP_jll package that will take care of that.

It might be worth trying to add GMP back to the dependencies. Or else, apply more force to cmake to make sure it's not found during the build either.

But I'd rather find out which call is causing the error (within SCIPsolve). Not sure whether that would be visible if we compiled SCIP in debug mode and/or used gdb to run Julia.

[matbesancon]

So I re-added GMP as a dependency on this branch: https://github.com/matbesancon/Yggdrasil/tree/scip-up
But I can't test for now, no access to a powerful enough machine

[rschwarz]

So, I do have a notebook with Win 10 Pro on it, and could do some testing locally. Not sure if I will find the time before the holidays...

[matbesancon]

I have access to a windows machine I'll be able to take a look

…

On Sat, Nov 28, 2020, 14:40 Robert Schwarz ***@***.***> wrote: So, I do have a notebook with Win 10 Pro on it, and could do some testing locally. Not sure if I will find the time before the holidays... — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#177 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2FDMUQTQ6NBSWCNQ2CPETSSD4V7ANCNFSM4PW5N6NQ> .

[matbesancon]

I wonder if we should switch the URLs to scipopt.org from scip.zib.de, not sure if the older one will be kept

[rschwarz]

Yes, that's a good idea.

I planned to make a separate PR to switch from Travis to GitHub actions and update some other minor things.

[matbesancon]

Yes I also have to do it for other repos

[GregPlowman]

“So if you use ...”

[matbesancon]

So I had forgotten how awful all the tooling can be on windows. For some reason the error message I get is in French:

That happens on ]add SCIP_jll

[matbesancon]

And when testing SCIP with the branch:

[matbesancon]

the error message is awfully unhelpful here

[matbesancon]

well that's new

[rschwarz]

Oh my 😢

Well, the stacktrace of the failure on Ubuntu with Julia 1.3 is deep within IPOPT. And IPOPT wasn't included in SCIP_jll before, right?

[matbesancon]

Yes exactly

[matbesancon]

Could it be that SCIP requires a specific version of Ipopt that is not compatible with the one we have through Ipopt_jll?

[matbesancon]

In the tests, this happens for me on 1.5 with the following:

@testset "MOI Quadratic Constraint" begin
    # needs objective bridge (MOI/#529)
    # MOIT.qptest(BRIDGED, CONFIG)

--->     MOIT.qcptest(BRIDGED, CONFIG)
end

julia> MOIT.qcptest(BRIDGED, CONFIG)

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit http://projects.coin-or.org/Ipopt
******************************************************************************

free(): invalid next size (normal)

signal (6): Aborted
in expression starting at REPL[9]:1

[rschwarz]

Could it be that SCIP requires a specific version of Ipopt that is not compatible with the one we have through Ipopt_jll?

Not that I know of. I used to simply compile IPOPT from the tarball available from COIN-OR (latest version).

[matbesancon]

The weird part is that Ipopt seems to be loaded? I can't figure out at which point this is triggered

[matbesancon]

@odow have you experienced something similar with Ipopt? We used the JLL here

[rschwarz]

Yeah, it's not that IPOPT is misisng, but it tries to make some kind of syscalls(?) that are not supported. Maybe the IPOPT_jll and SCIP_jll are compiled in some incompatible way?

[matbesancon]

Just cross-posted to slack. The biggest smell right now is that the bug only occurs on linux

[matbesancon]

OK there was something strange with Ipopt 3.13.1 breaking, this is fixed on 3.13.2

[matbesancon]

If it doesn't pick it up, I added a dependency of SCIP.jl on Ipopt_jll with an explicit requirement

[matbesancon]

Windows still failing "as expected", but we now have Ipopt integration

[rschwarz]

OK, great. I'll try to build in the Windows exception in the build.jl some time soon now.

[odow]

There was an issue with MUMPS: jump-dev/Ipopt.jl#246

Edit: no idea on the Windows side.

[matbesancon]

Edit: no idea on the Windows side.

the Windows issue is on SCIP, so Windows users will have to provide a SCIP install, while unix users can get the package shipped with a default SCIP (they can but don't have to provide one)

[matbesancon]

There was an issue with MUMPS: jump-dev/Ipopt.jl#246

it might have been that, combined with the fact that the package manager had not picked the latest Ipopt_jll for reasons I didn't get

[rschwarz]

I wonder whether deps/build.jl and src/init.jl should still be modified to print some warning for Windows users.