Ubuntu 18.04 LTS linking error

[zbroyar]

Hi,

Some time ago I decided to give Owl a try and installed it on my MacBook. It work like a charm. But when I try to build my project with Owl on the server I get a bunch of linking errors:

/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_391_LAPACKE_slagge':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:15413: undefined reference to `LAPACKE_slagge'
/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_392_LAPACKE_dlagge':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:15446: undefined reference to `LAPACKE_dlagge'
/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_393_LAPACKE_clagge':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:15479: undefined reference to `LAPACKE_clagge'
/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_394_LAPACKE_zlagge':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:15512: undefined reference to `LAPACKE_zlagge'
/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_429_LAPACKE_slatms':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:16509: undefined reference to `LAPACKE_slatms'
/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_430_LAPACKE_dlatms':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:16558: undefined reference to `LAPACKE_dlatms'
/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_431_LAPACKE_clatms':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:16607: undefined reference to `LAPACKE_clatms'
/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_432_LAPACKE_zlatms':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:16656: undefined reference to `LAPACKE_zlatms'
/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_923_LAPACKE_claghe':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:33633: undefined reference to `LAPACKE_claghe'
/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_924_LAPACKE_zlaghe':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:33661: undefined reference to `LAPACKE_zlaghe'
/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_925_LAPACKE_slagsy':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:33689: undefined reference to `LAPACKE_slagsy'
/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_926_LAPACKE_dlagsy':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:33717: undefined reference to `LAPACKE_dlagsy'
/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_927_LAPACKE_clagsy':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:33745: undefined reference to `LAPACKE_clagsy'
/home/nick/.opam/4.07.0/lib/owl/libowl_stubs.a(owl_lapacke_generated_stub.o): In function `owl_stub_928_LAPACKE_zlagsy':
/home/nick/.opam/4.07.0/.opam-switch/build/owl.0.6.0/_build/default/src/owl/src/owl/lapacke/owl_lapacke_generated_stub.c:33773: undefined reference to `LAPACKE_zlagsy'

And it looks like my LAPACKE library really doesn't have those symbols:
objdump -T /usr/lib/x86_64-linux-gnu/liblapacke.so.3.7.1 | grep -i 'slatms' gives nothing.

What am I doing wrong?

[mseri]

Unfortunately we know. It is a known issue in how openblas is complied in Ubuntu. You need to recompile openblas and install it manually. You can see how we deal with it in the docket files in the repository: https://github.com/owlbarn/owl/blob/master/docker/Dockerfile.ubuntu

EDIT: this is no longer the correct file to look at, it has been changed with an internal customization that should not be used as template. The correct file was: https://github.com/owlbarn/owl/blob/c832b0ac1459b7694d8d666f245c18d6ae1a0430/docker/Dockerfile.ubuntu

[zbroyar]

Thanks. Probably, It's worth mentioning in the documentation

[mseri]

That seems a good idea. I don't have time to touch it at the moment, but a PR to https://github.com/owlbarn/owl_guide/blob/master/book/chapter/install.rst would be really welcome

[kkirstein]

Ubuntu 20.04, which installs lapack 3.9.0 per default, doesn't show this issue anymore.

[mseri]

@kkirstein thanks for the heads up. Great news!

[UnixJunkie]

My god... that makes the software so unportable... I am bailing out of owl.

[mseri]

@UnixJunkie What in particular? All other libraries have similar problem, that's why practically all of them ship their binary version of openblas bundled with the library itself (there was a long discussion for example in the numpy's mailing list a number of years ago).

Recompiling blas has always been a solution, and the maintainers of ubuntu's openblas did not want to address this problem when we asked. At least now you can install owl without recompiling openblas also on ubuntu. In most other distributions and on osx it has always worked without problems.

[UnixJunkie]

Yes, I can install owl, but then I cannot link against it on Ubuntu 18.04 ...

[UnixJunkie]

@mseri Then, the proper fix would be for owl to ship its own binary version of openblas and not rely on the system-wide-installed one.

[mseri]

You can if you use a re-compiled openblas. This has been an issue for ages, it was broken also in trusty, we contacted the maintainers with no success, there was an issue opened by other people hit by the same problem in their libraries, but it has been unaddressed ever since.

We had also thought to vendor openblas, but it was affecting the compilation time of owl massively, and it was only an issue in ubuntu. Once we have a good strategy to deliver binary system-dependent blobs, we can ship owl with its own precompiled openblas, but for now we have not found a way.

[mseri]

@mseri Then, the proper fix would be for owl to ship its own binary version of openblas and not rely on the system-wide-installed one.

This was discussed openly in the past we don't have a good strategy for the binary blobs. I am very open to suggestions though.

[UnixJunkie]

Having owl being more portable is more important than having owl compiling faster.
"Just an issue in Ubuntu", well, that's the most widely used linux distro by software developers. :(

[UnixJunkie]

@Chris00 might know because of his https://github.com/Chris00/L-BFGS-ocaml

[mseri]

Having owl being more portable is more important than having owl compiling faster.
"Just an issue in Ubuntu", well, that's the most widely used linux distro by software developers. :(

Fair enough, but we need help with that. Over 10 minutes of compilation time on each update and during development was unbearable.

Maybe it is no-longer an issue on recent hardware but I still think the ideal would be the precompiled binary (not affecting performances too much and working on all hardware supported by ocaml and openblas) with an option to link a custom install. For that we need help because we were not able to find a way to do it properly

[mseri]

@Chris00 might know because of his https://github.com/Chris00/L-BFGS-ocaml

Awesome, I am all ears!

[bluddy]

It might be useful to create a PPA for this version of openblas and include instructions for adding it. I wonder if we can add it to the ocaml PPA @avsm runs.

[mseri]

Isn't that nearly the same as compiling openblas manually though?

[bluddy]

No PPA is binary downloads AFAIK.

[mseri]

Yes, but users will still need to add the custom repository and manually install the special openblas. We must also make sure to not affect any other openblas installation they might have.

[bluddy]

a. Is this custom openblas incompatible with other uses?
b. It's possible to give the package a slightly different name, (ocaml-openblas), and make sure the .so files are differently named, so there's no conflict.
c. Adding PPAs is really fast and generally much easier than compiling by hand.

[mseri]

a. Don't know.
b. We need to be careful, otherwise it will make owl build even more complicated just for that usecase. My feeling is that it would be enough to install it in a nonstandard position, and ask the users to pass the right PKG_CONFIG_FLAG to opam.
c. Fair enough

But since we need to go with an extra complication just for ubuntu, I start thinking it would make sense to steer the effort to retry to find a decent way to vendor openblas in owl directly (and, if it is improved, accept to pay the price for the first compilation time).

[mseri]

Just to clarify: I will be very happy if somebody knows how to do it and does it. It is probably easy if you already know what to do: my only experience creating debian packages is from something like two decades ago, with checkinstall doing it for me, when I had just moved from slackware and wasn't bothered to learn how to package for debian :P

[bluddy]

So the other option is just to literally include the binary/ies in the owl source tree -- at least until 20.04+ becomes the standard. It's enough to just support AMD64 as that's the most used platform at this point -- other platforms are more sophisticated and their users can figure out how to recompile.

[mseri]

That's right. One downside of it is that we will increase building time. Of course, we could bundle openblas into a separate owl-openblas package so that the number of recompilations will be kept to a minimum. The upside is that, since we have just one configuration, we could greatly simplify owl's internal build rules and, in principle, fix all installation issues -- including many potential future ones -- at once...

[bluddy]

I don't follow -- why would build times increase? There's no need to build openblas every time -- build it once on a single machine and add the vendored binaries to the repo (you can perhaps use a submodule if you want to keep this repo clean).

[mseri]

Because I was thinking of vendoring sources instead: that would avoid the need to build binaries for each platform and have a way to make opam download the right one.

[UnixJunkie]

Maybe an optional dependency to the owl opam package could do the source install of the required version of openblas (for people who want/need it).
A PPA is not a solution, that requires some sysadmin on each computer using owl (i.e. that's not automatic enough, unlike an opam install in user-space).