Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new preset for libpostal #499

Closed
Maurice-Betzel opened this issue Dec 14, 2017 · 32 comments
Closed

new preset for libpostal #499

Maurice-Betzel opened this issue Dec 14, 2017 · 32 comments

Comments

@Maurice-Betzel
Copy link
Contributor

Maurice-Betzel commented Dec 14, 2017

I have started working on a libpostal preset. libpostal is a C library for parsing/normalizing street addresses around the world using statistical NLP and open data. I have created a branch but i got stuck pretty fast on the compile part, see mailinglist. Libpostal also carries huge blobs of trainingdata that is in need of a javacpp strategy. The lib is functional on my msys win10 x64 default console setup. I am hoping for somebody to provide a similar example i can study, or even some support.

@saudet
Copy link
Member

saudet commented Dec 15, 2017

Could you try to follow these instructions?
https://github.com/bytedeco/javacpp-presets/wiki/Build-Environments#windows-x86-and-x86_64

@Maurice-Betzel
Copy link
Contributor Author

Maurice-Betzel commented Dec 15, 2017

Yes i did, but i am missing deeper knowlledge on what a autoreconfig does. I need help if the default build instructions from libpostal do not work within the preset setup. I see forgot to push my latest state to the branch yesterday.

@saudet
Copy link
Member

saudet commented Dec 15, 2017

cppbuild.sh is just a bash script, it doesn't do anything special. We don't even need it. You could install libpostal on your system and provide those paths in the pom.xml file, as done for CUDA, for example:
https://github.com/bytedeco/javacpp-presets/blob/master/cuda/pom.xml

@Maurice-Betzel
Copy link
Contributor Author

If i use these methods in the JavaCPP generated class from my branch for libpostal the software locks up or a access violation gets thrown:

public static native @Cast("char**") @ByPtrPtr byte[] libpostal_expand_address(@Cast("char*") byte[] input, @ByVal libpostal_normalize_options_t options, @Cast("size_t*") SizeTPointer n); public static native libpostal_address_parser_response_t libpostal_parse_address(@Cast("char*") byte[] address, @ByVal libpostal_address_parser_options_t options);

Trying to convert these methods from byte[] to String type with the following Info:

.put(new Info("char").cast().valueTypes("byte").pointerTypes("BytePointer", "String"))

resulted in:

C:\msys64\home\betzm\javacpp-presets-libpostal\libpostal\target\classes\org\bytedeco\javacpp\jnilibpostal.cpp(2606): error C2440: '=': cannot convert from 'char **' to 'const char *'
C:\msys64\home\betzm\javacpp-presets-libpostal\libpostal\target\classes\org\bytedeco\javacpp\jnilibpostal.cpp(2606): note: Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast

Is it possible to do this?

@saudet
Copy link
Member

saudet commented Dec 21, 2017

Thanks for reporting! I've fixed this issue in the commit above, so make sure to use JavaCPP 1.3.4-SNAPSHOT.

You also had an issue running Autoconf through JavaCPP on Windows, right? Could you give me something to reproduce that here? Thanks!

@Maurice-Betzel
Copy link
Contributor Author

Maurice-Betzel commented Dec 21, 2017

Doing the libpostal Windows mingw64 shell c build as explaind here via the console, all goes well. As soon as i do the Maven preset build with the same build.sh instructions it blows up on executing the bootstrap.sh containing autoreconf -fi --warning=no-portability with the following result:
...aclocal-1.15: found macro AM_PROG_AR in /usr/share/aclocal-1.15/ar-lib.m4: 13
aclocal-1.15: found macro AM_AUTOMAKE_VERSION in /usr/share/aclocal-1.15/amversion.m4: 14
aclocal-1.15: found macro AM_SET_CURRENT_AUTOMAKE_VERSION in /usr/share/aclocal-1.15/amversion.m4: 33

aclocal-1.15: error: aclocal: file '/msys64/usr/share/aclocal/xsize.m4' does not exist
autoreconf: aclocal failed with exit status: 1
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:764)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:711)
at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:289)

The file xsize.m4 is located at where the above path is pointing.
The same Maven build from msys shell works, but produces a POSIX dependend dll file. New msys and pacman installs on the msys shell or the mingw shell resulted in the same errornous behaviour. I am tipping on shell specific behaviour, but thats beyond my knowlledge.

@saudet
Copy link
Member

saudet commented Dec 22, 2017 via email

@Maurice-Betzel
Copy link
Contributor Author

@Maurice-Betzel
Copy link
Contributor Author

Maurice-Betzel commented Dec 22, 2017

This put me on the right track resulting in:

if [[ -z "$PLATFORM" ]]; then
    pushd ..
    bash cppbuild.sh "$@" libpostal
    popd
    exit
fi

echo "downloading libpostal"

LIBPOSTAL_VERSION=master
download https://github.com/openvenues/libpostal/archive/$LIBPOSTAL_VERSION.tar.gz libpostal-$LIBPOSTAL_VERSION.tar.gz

echo "unpacking libpostal"

mkdir -p $PLATFORM
cd $PLATFORM
tar -xzvf ../libpostal-$LIBPOSTAL_VERSION.tar.gz
cd libpostal-$LIBPOSTAL_VERSION


case $PLATFORM in
    windows-x86_64)
    echo "Starting windows-x86_64 build"
    cp -rf windows/* ./
    cwd=$(pwd)
    bash -lc "cd $cwd && autoreconf -fi --warning=no-portability"
    #./configure --datadir=/c
    ./configure --disable-data-download
    make -j4
    make install
        ;;
    *)
        echo "Error: Platform \"$PLATFORM\" is not supported"
;;

esac

cd ../..

@saudet
Copy link
Member

saudet commented Dec 23, 2017

Thanks! Yeah, it's happening here as well. Not sure why, seems to be a bug in MSYS2. Running it with bash -lc works, so let's just do that.

@Maurice-Betzel
Copy link
Contributor Author

I got MacOS 10 running on my vmware machine today, maybee i can provide and test libpostal mac builds as well.

@saudet
Copy link
Member

saudet commented Dec 27, 2017

BTW, when you're ready to send a pull request, please rebase your fork on the master branch (currently 1.3.4-SNAPSHOT), thanks!

@Maurice-Betzel
Copy link
Contributor Author

Of course, did that yesterday already, now i am having issues with the Centos 7 libpostal build. How do i deal with pkg-config and JavaCPP?

@saudet
Copy link
Member

saudet commented Dec 27, 2017 via email

@Maurice-Betzel
Copy link
Contributor Author

Package libpostal was not found in the pkg-config search path.
Perhaps you should add the directory containing `libpostal.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libpostal' found
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:33.632s
[INFO] Finished at: Wed Dec 27 08:29:25 CET 2017
[INFO] Final Memory: 26M/430M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build (javacpp.cppbuild.install) on project libpostal: Execution javacpp.cppbuild.install of goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build failed: Process exited with an error: 1 -> [Help 1]
[ERROR]

@saudet
Copy link
Member

saudet commented Dec 27, 2017

Well, we're supposed to be building it from source in cppbuild.sh, so just do that...

@Maurice-Betzel
Copy link
Contributor Author

That is what i am trying to do:

case $PLATFORM in
linux-x86_64)
./bootstrap.sh
#./configure --datadir=[...some dir with a few GB of space...]
./configure --disable-data-download
make -j4
sudo make install
sudo ldconfig

gives me:

[INFO] g++ -I/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/ -I/usr/java/jdk1.8.0_121/include -I/usr/java/jdk1.8.0_121/include/linux /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp -march=x86-64 -m64 -O3 -s -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjnilibpostal.so -llibpostal
[ERROR] Failed to execute JavaCPP Builder: Cannot run program "g++" (in directory "/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/linux-x86_64"): error=2, No such file or directory
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:17.802s
[INFO] Finished at: Wed Dec 27 08:52:35 CET 2017
[INFO] Final Memory: 26M/360M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build (javacpp.compiler) on project libpostal: Failed to execute JavaCPP Builder: Cannot run program "g++" (in directory "/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/linux-x86_64"): error=2, No such file or directory -> [Help 1]

@Maurice-Betzel
Copy link
Contributor Author

The default build from the console gives me working libpostal software. What does JavaCPP need to wrap this?

@saudet
Copy link
Member

saudet commented Dec 27, 2017

You'll need to install the C++ compiler of GCC as well:
https://github.com/bytedeco/javacpp-presets/wiki/Build-Environments#linux-x86-and-x86_64

@Maurice-Betzel
Copy link
Contributor Author

Oh boy, i did follow all of the instructions, but g++ was still missing somehow. A clear case of RTFE (read the f***** exception), trying to run too fast. Next issue:

[INFO] g++ -I/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/ -I/usr/java/jdk1.8.0_121/include -I/usr/java/jdk1.8.0_121/include/linux /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp -march=x86-64 -m64 -O3 -s -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjnilibpostal.so -llibpostal
/usr/bin/ld: cannot find -llibpostal
collect2: error: ld returned 1 exit status

@Maurice-Betzel
Copy link
Contributor Author

Switched off ldconfig, results in

[INFO] Building for platform "linux-x86_64"
[INFO] Targeting /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java
[ERROR] Failed to execute JavaCPP Builder: Could not parse "libpostal.h": File does not exist
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2:50.565s
[INFO] Finished at: Wed Dec 27 10:06:43 CET 2017
[INFO] Final Memory: 26M/430M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build (javacpp.parser) on project libpostal: Failed to execute JavaCPP Builder: Could not parse "libpostal.h": File does not exist -> [Help 1]

I am way out of my comfort zone here...

@Maurice-Betzel
Copy link
Contributor Author

Switched back on include mkdir and libpostal.h copy to it, resulted in:

[INFO] --- javacpp:1.3.4-SNAPSHOT:build (javacpp.parser) @ libpostal ---
[INFO] Detected platform "linux-x86_64"
[INFO] Building for platform "linux-x86_64"
[INFO] Targeting /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java
[INFO] Parsing /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/libpostal.h
[INFO]
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ libpostal ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ libpostal ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes
[INFO] /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java: /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java uses unchecked or unsafe operations.
[INFO] /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java: Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- javacpp:1.3.4-SNAPSHOT:build (javacpp.compiler) @ libpostal ---
[INFO] Detected platform "linux-x86_64"
[INFO] Building for platform "linux-x86_64"
[INFO] Generating /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp
[INFO] Generating /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp
[INFO] Compiling /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/linux-x86_64/libjnilibpostal.so
[INFO] g++ -I/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/ -I/usr/java/jdk1.8.0_121/include -I/usr/java/jdk1.8.0_121/include/linux /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp -march=x86-64 -m64 -O3 -s -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjnilibpostal.so -llibpostal
/usr/bin/ld: cannot find -llibpostal
collect2: error: ld returned 1 exit status
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2:45.644s
[INFO] Finished at: Wed Dec 27 10:16:08 CET 2017
[INFO] Final Memory: 26M/360M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build (javacpp.compiler) on project libpostal: Execution javacpp.compiler of goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build failed: Process exited with an error: 1 -> [Help 1]

@Maurice-Betzel
Copy link
Contributor Author

Am i missing the file for linking? And where do i get it from?

@Maurice-Betzel
Copy link
Contributor Author

i copy the libpostal.a in the lib folder but with the same result. What am i missing?

@Maurice-Betzel
Copy link
Contributor Author

Trying to link against the so file from the .libs folder....

@Maurice-Betzel
Copy link
Contributor Author

I copied the whole generated .libs folder into the linux-x86_64/lib folder, with the same result:

/IdeaProjects/javacpp-presets
[INFO]
[INFO] --- javacpp:1.3.4-SNAPSHOT:build (javacpp.parser) @ libpostal ---
[INFO] Detected platform "linux-x86_64"
[INFO] Building for platform "linux-x86_64"
[INFO] Targeting /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java
[INFO] Parsing /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/libpostal.h
[INFO]
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ libpostal ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ libpostal ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes
[INFO] /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java: /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java uses unchecked or unsafe operations.
[INFO] /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java: Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- javacpp:1.3.4-SNAPSHOT:build (javacpp.compiler) @ libpostal ---
[INFO] Detected platform "linux-x86_64"
[INFO] Building for platform "linux-x86_64"
[INFO] Generating /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp
[INFO] Generating /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp
[INFO] Compiling /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/linux-x86_64/libjnilibpostal.so
[INFO] g++ -I/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/ -I/usr/java/jdk1.8.0_121/include -I/usr/java/jdk1.8.0_121/include/linux /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp -march=x86-64 -m64 -O3 -s -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjnilibpostal.so -L/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/lib/ -Wl,-rpath,/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/lib/ -llibpostal
/usr/bin/ld: cannot find -llibpostal
collect2: error: ld returned 1 exit status
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:02.658s
[INFO] Finished at: Wed Dec 27 12:25:28 CET 2017
[INFO] Final Memory: 26M/360M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build (javacpp.compiler) on project libpostal: Execution javacpp.compiler of goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build failed: Process exited with an error: 1 -> [Help 1]

@Maurice-Betzel
Copy link
Contributor Author

If i manually compile like this i get my libjnilibpostal.so:

g++ -I/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/ -I/usr/java/jdk1.8.0_121/include -I/usr/java/jdk1.8.0_121/include/linux /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp -march=x86-64 -m64 -O3 -s -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjnilibpostal.so -L/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/lib/ -Wl,-rpath,/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/lib/ -lpostal

@Maurice-Betzel
Copy link
Contributor Author

lovin this one:
cp src/.libs/libpostal.so ../lib/liblibpostal.so

@saudet
Copy link
Member

saudet commented Dec 27, 2017

Usually "make install" will do what we need. I designed it like that. Check the other presets, for example:
https://github.com/bytedeco/javacpp-presets/blob/master/libdc1394/cppbuild.sh

@Maurice-Betzel
Copy link
Contributor Author

Maurice-Betzel commented Dec 27, 2017

I have studied the default setup, but it was not working. A working and tested build is now up. Hoping the mac build will be nearly the same :)

@Maurice-Betzel
Copy link
Contributor Author

Maurice-Betzel commented Dec 28, 2017

I have added and tested the MacOS build and i am ready for a pull request. Could you take a look at the setup and report anything missing?

@saudet
Copy link
Member

saudet commented Jan 17, 2018

Released!
http://search.maven.org/#search%7Cga%7C1%7Clibpostal
Thanks again for the contribution.

@saudet saudet closed this as completed Jan 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants