[clojure-package] On modern Ubuntu, scala package cannot be loaded without legacy libcurl

[kohr-h]

Description

The Scala package as pulled in by the Clojure package has a dependency on legacy libcurl (on Ubuntu: libcurl3) and doesn't work with the current version (on Ubuntu: libcurl4).

Environment info (Required)

Only relevant part:

----------System Info----------
Platform     : Linux-4.15.0-36-generic-x86_64-with-Ubuntu-18.04-bionic
system       : Linux
node         : pop-os
release      : 4.15.0-36-generic
version      : #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018

Package used (Python/R/Scala/Julia): Clojure

For Scala user, please provide: (Okay, just in case)

1. Java version:
   openjdk version "10.0.2" 2018-07-17
   OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.2)
   OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.2, mixed mode)

2. Maven version: nil

3. Scala runtime if applicable: nil

Error Message:

INFO  MXNetJVM: Try loading mxnet-scala from native path.
INFO  MXNetJVM: Try loading mxnet-scala-linux-x86_64-gpu from native path.
INFO  MXNetJVM: Try loading mxnet-scala-linux-x86_64-cpu from native path.
WARN  MXNetJVM: MXNet Scala native library not found in path. Copying native library from the archive. Consider installing the library somewhere in the path (for Windows: PATH, for Linux: LD_LIBRARY_PATH), or specifying by Java cmd option -Djava.library.path=[lib path].
INFO  org.apache.mxnet.util.NativeLibraryLoader: Loading libmxnet-scala.so from /lib/native/ copying to mxnet-scala
ERROR org.apache.mxnet.util.NativeLibraryLoader: Couldn't load copied link file: java.lang.UnsatisfiedLinkError: /tmp/mxnet3266536299040978061/mxnet-scala: /usr/lib/x86_64-linux-gnu/libcurl.so.4: version `CURL_OPENSSL_3' not found (required by /tmp/mxnet3266536299040978061/mxnet-scala)
ERROR MXNetJVM: Couldn't find native library mxnet-scala

Steps to reproduce

Follow the installation instructions in the clojure package README.

What have you tried to solve it?

For projects that use Leiningen, there's a workaround that nicely integrates with the CIDER REPL in Emacs:

0. Make a Leiningen project with clojure-mxnet dependency.
1. In the top-level project directory, download the legacy libcurl3 package:

   wget http://archive.ubuntu.com/ubuntu/pool/universe/c/curl3/libcurl3_7.58.0-2ubuntu2_amd64.deb

2. Extract into a temporary directory:

   mkdir -p tmp && dpkg-deb -R libcurl3_7.58.0-2ubuntu2_amd64.deb tmp/

3. Copy the shared library to a directory of your choice, here resources, and remove the temporary directory:

   cp tmp/usr/lib/x86_64-linux-gnu/libcurl.so.4 resources && rm tmp -r

4. In the project.clj file, add the following:

     :plugins [[lein-with-env-vars "0.2.0"]]
     :env-vars {:LD_LIBRARY_PATH "./resources"}
     :hooks [leiningen.with-env-vars/auto-inject]

This works better than

• installing libcurl3 system-wide because it blocks libcurl4, which is a dependency of the curl package
• globally setting LD_LIBRARY_PATH since it might break other packages that now find the wrong shared library
• manually setting the environment variable because then, only stuff run from that console session will work correctly (including emacs).

However, it would be much better if this problem didn't exist in the first place. What would it take to use libcurl4 instead? Maybe the Scala package Dockerfile needs an update?

[marcoabreu]

If you got a second, you can have a look at https://github.com/apache/incubator-mxnet/blob/c9a9db6ba04ecc1ee395e74524ccdcb82b528bc3/ci/docker/runtime_functions.sh#L774 https://github.com/apache/incubator-mxnet/blob/master/ci/docker/Dockerfile.build.ubuntu_cpu to see how we are building and running our CI environment. We should also be installing libcurl there. Maybe you can check which version we are using there and whether it suits your need.

Please note that the CI and the distributed scala/clojure package are different.

[piyushghai]

@kohr-h Thank you for posting your issue. We will look into the libcurl4 compatibility issue.
@mxnet-label-bot [Clojure, Scala]

[kohr-h]

@marcoabreu I checked that file and also looked around a bit. Lots of scripts install curl, and on Ubuntu 16.04 that pulls in libcurl3-gnutls, but not libcurl3. Nowhere in the repo is libcurl3 explicitly installed, and in lots of places, libcurl4 is explicitly used, which blocks libcurl3. So I don't quite understand how the Scala package becomes dependent on the legacy version.

[marcoabreu]

Interesting analysis, thank you! @nswamy can you take it from here?

[lanking520]

Thanks for your question @kohr-h. One question, how do you build your package? From Maven or build from source?

If you grab the package (Scala only) from Maven, we use the following configuration for publishing

 "libcurl4-openssl-dev"
 "libcurl-devel"

@gigasquid for visibility 😄

[kohr-h]

One question, how do you build your package? From Maven or build from source?

For now I just let lein do its job, and I assume that it pulls the Scala package from Maven.

libcurl-devel

Hm, that's the RPM name of the package, so does that mean that you build with CentOS or something? Depending on the distro version, that might refer to the legacy version of the curl libraries.

[gigasquid]

Hi @kohr-h - trying to catch up here, are you using the clojure-mxnet jar on maven that has the native deps baked in https://search.maven.org/search?q=clojure-mxnet or are you building from source and then running (meaning doing make scalapkg and make scalainstall)

[gigasquid]

@lanking520 I think this issue is related gigasquid/clojure-mxnet#2 - it lists libucurl3 as a dep needed

libopenblas-base
libatlas3-base
libcurl3
libopencv-imgcodecs3.4

[kohr-h]

Hi @gigasquid, I used the prebuilt jar for everything, no custom builds were involved.

[kohr-h]

To be precise, here's the relevant part of my :dependencies:

[org.apache.mxnet.contrib.clojure/clojure-mxnet-linux-cpu "1.3.0"]

Reminds me, the package spec on in the instructions seems to be outdated. I can file a PR.

[zachgk]

@kohr-h With the 1.4 release, we statically link the dependencies such as curl that are used by the maven scala package. You shouldn't have this problem with the new version

[kohr-h]

Sounds good @zachgk. Unfortunately I don't have time to test it myself, so I'll just close the issue.