undefined symbol: clock_gettime #606
Comments
|
Interesting. We do set -lrt when we compile: https://github.com/facebook/rocksdb/blob/master/build_tools/build_detect_platform#L125 Do we need to set something other than |
|
I'm experiencing the same issue using the rocksdbjni 3.10.2 jar from Maven, running on Centos 6.6. Using I was using version 3.9.1 previously, and had no issues. This is the nm results: I build the library from the latest, this is the results: ** To save others from possible frustration contrary to the INSTALL.md details...
When running 'make shared_lib' it does appear to include the -lrt flag, but clock_gettime is still undefined: I'm not sure what to check next, however if you have suggestions I'm happy to try. |
|
One thing I don't understand is how clock_gettime was not included at all in Rocksdb 3.6, but now it shows up unlinked. I don't have time to try and build but one suggestion I found online is to use From ld man: |
|
@igorcanadi if you can explain how clock_gettime was not included at all in Rocksdb 3.6 or 3.9.1, but now it shows up unlinked, that would be very helpful. Also, it looks like the code in older Rocksdb versions did not reference clock_gettime at all. I looked at build_detect_platform, but I don't see a big change that may have caused the code to start using clock_gettime. Any idea? |
|
I found the same article, and I tried to change the build_detect_platform and set the following: But the output of nm is still: I also see that clock_gettime was not included in the 3.9.1 release, so it looks new to 3.10. It's worth noting that setting the workaround you defined |
|
@DerekSchenk What are you running that you get the clock_gettime error? I didn't see the error in our production env. I only noticed it when I tried to write a million dummy KVs to a dummy db to see the effect of Snappy compression vs NO_COMPRESSION. Do you know if it happens during read, write, or is it the statistics gathering piece of code that is causing this issue? |
|
@pshareghi I'm not 100% certain, but it seems to happen when I initialize the database. I think it's caused because I use ".createStatistics()", but I do have compression enabled as well. I'd need to try a few scenarios to see for sure. If it's in fact the statistics I can disable that and work without it, but at the moment I can't upgrade at all. |
|
I think it is statistics calculation that triggers the issue. I checked my LOG file when the problem happened, and it was in the middle of printing out the statistics. |
|
@pshareghi Thanks for the tip - I can confirm that the issue is triggered when including statistics. I removed the call to .createStatistics() and I can use the 3.10 version. This was not an issue in the 3.9 release. While I do use the stats I can live without them for know. |
RocksDB 3.6 did call clock_gettime: https://github.com/facebook/rocksdb/blob/rocksdb-3.6.1/util/env_posix.cc#L1362 so I have no idea why it's showing up as unlinked right now and it worked before :( |
|
@DerekSchenk thanks for the report that our compilation doesn't work on gcc 4.7. We fixed this with 74f3832. We should probably set up contbuild using 4.7 to make sure this doesn't happen again. |
|
Are you both guys running with shared library? |
|
@igorcanadi Something that occurred to me - is it possible that the newer build running against gcc 4.8 has linked a newer version of glibc (ie 2.2.5), and all my Centos 6.6 machines are runing glibc 2.12? That may explain how it worked in 3.9 (using gcc 4.7), and why it works on other platforms? I am pulling the jar from Maven, and it uses the shared library. When I built it myself I also built shared. |
|
@DerekSchenk can you try building with gcc 4.7 on master now? If this works, it will partially verify your hypothesis. |
|
what does |
|
|
|
|
This is the output: Version 3.9 Version 3.10 And my attempt to build from head last week I just updated on my build machine and will try building again using 4.7. |
|
@igorcanadi The master build 'make shared_lib' now works on gcc 4.7, however it still looks as though it's not linked properly. I should have probably know this already as the gcc version 4.7 is not fully installed either. The most up to date version of gcc you can get for Centos 6 through yum is 4.6, so I've got 4.7 and 4.8 installed in alternate locations. The yum command `yum install gcc47-c++' just gets your hopes up, then yum lets you down ;) This is the output of the version I just built: I wanted to run the test cases to see if it still fails, but 'make check' fails: |
|
@DerekSchenk I'm not sure why our compiler doesn't complain about that. I fixed it here: 04feaee |
|
@igorcanadi Thanks - that fix worked. I was able to compile and run this time. I was hoping one of the tests would it the clock_gettime call and confirm if it was working or not. Do you know what test would be a good one to confirm? Off topic, not sure if you want this in a different issue, but the check run fails on one of the environment tests. |
|
Yes, please create the different issue for this one.
|
|
@igorcanadi I am building using Also, I am using GCC 4.9 |
|
@DerekSchenk try this super simple test. I shamelessly modified RocksDbSample.java. Even though I commented out createStatistics(), it still dies halfway. |
|
@pshareghi I pulled down the RocksDbSample.java, and using the copy I built it worked (completed the test) if I removed the call to .createStatistics(), as well as the Statistics creation 'Statistics stats = options.statisticsPtr();', and the lines following 'db.getProperty("rocksdb.stats");'. With those lines included the test fails every time with the missing symbol error. I got looking at the build script for rocksdbjavastatic which doesn't work, and I realized the the LDFLAGS, notably -lrt isn't being passed when the library is created. @igorcanadi Not sure if this is correct or not, but when I modified the Makefile, and changed the compile to this below line I'm able to build the Java library, and run the RocksDbSample.java file unmodified and it works properly
Original Makefile output Output with the LDFLAGS modification |
|
Oh interesting. I'm actually not familiar with RocksJava compilation, but this change looks good to me. @yhchiang @fyrz @adamretter can you please evaluate the change? @DerekSchenk feel free to send the PR :) |
Includes the LDFLAGS so that the correct libraries will be linked. This links rt to resolve the issue facebook#606.
|
@igorcanadi I've created a PR (should have probably waited until morning :) ) for this issue. Hope everything is fine. |
|
Created a pull request to document the changes discussed above. |
|
Hello, I just built and uploaded the 3.13.1 package once again using the fix by @jwlent55, and it seems OSSRH has updated the file. Can I know whether the package work now? |
|
Just tested the build and @dmittendorf is correct. The dependencies look much better, but, because the Snappy header files were not available at compile time the Snappy code was not linked in. voldemort.store.StorageInitializationException: org.rocksdb.RocksDBException: Invalid argument: Compression type Snappy is not linked with the binary. $ ldd ~/Downloads/librocksdbjni-linux64.so |
|
Thanks @jwlent55 and @dmittendorf, I will cherry-pick the change and build again. |
|
Have rebuilt and republished once again, can I know whether the new build solve the issue? |
|
Thanks. Ran one quick test on a linux-64 box and it worked for me. Some output that verifies it: 2015/10/10-23:31:30.443170 7f3ffc553700 Compression algorithms supported: Note that LZ4 has the same issue. It could be solved the same way, but, this approach has issues:
I don't think this needs to be addressed in 3.13.1, but, I have coded up a Makefile change that I think eliminates both these issues. I am no Makefile expert so there may be better ways to handle it and/or issues with the approach. I will send you another pull request just to document the approach and get feedback. Again I have only tested it on a linux-64 box. |
|
Thanks for verifying the build, @jwlent55! We are looking forward to your pull request! |
|
Having OS X (portability?) issues with the latest build 3.13.1 build (not sure about earlier builds). When @dmittendorf tested it (via Voldemort) on his Mac we get: @dmittendorf rebuilt the 3.13.1 on his Mac under the following conditions:
That build worked on his Mac. Here is some shared library dependency info: Here is an abbreviated stack trace: Having problems attaching a full core dump, but, I can provide one. @dmittendorf is busy on other more pressing issues and I don't have access to a Mac so we are not able to make a lot more progress on our end. Right now I am just searching the web for issues like this. Searches on "mixed mode bsd-amd64 compressed oops" have turned up some interesting reading. Are there any known OS X portability issues with RocksDB? |
|
I am not sure if the Snappy compression library has been linked into the linux-64 library included in the static jar for a long time now. What appears to have changed is that selecting a compression algorithm that is not supported is now an error in 3.13.1. I ran some tests where I wrote 500,000 1024 byte identical records to a RocksDB database configured as follows: I then tested the Snappy vs No Compression and 3.10.1 vs 3.13.1 (both downloaded from Maven Central). Only in the 3.13.1 case did I see a significant difference in DB size. I then extracted the dynamic library from 5 different RocksDB jars files (including an old 3.13.1 I still had) and looked at the symbol table. Only the 3.13.1 library seemed to contain Snappy code: Some of the Snappy symbols in 3.13.1: I have never used the "nm" tool before so I may be misinterpreting the results and perhaps there is another explanation for the DB sizes (RocksDB is new to me) so take this all with a grain of salt. |
|
Summarizing here since there has been a lot of churn from @jwlent55 and I...
My suspicion is that when @yhchiang created the latest 3.13.1 build, there was an old librocksdbjni-*.jnilib lying around in the /java directory on the build machine which masked number 7. and caused 6. My recommendation is to do the following: |
|
Hey @dmittendorf, thanks for the summarization and the fix! I will rebuild the package from clean 3.13.fb with the two fixes. |
|
Hello @dmittendorf, I've republished the package using the steps you suggested in https://oss.sonatype.org/#nexus-search;quick~rocksdb. Can I know whether the package work better now? |
|
Thanks @yhchiang! Based on initial testing the new JAR looks good. I tested locally on my Mac with some ad-hoc testing as well as running the Samza test suite which uses Snappy compression, and all tests passed. |
|
Also forgot to mention...the new JAR still hasn't synced to maven central, but I'm assuming that happens automatically, right? |
|
Nevermind...must just be the search results...the mod date within maven central is up-to-date. |
|
Just tested linux-64 with all 4 compression algorithms and no compression (using Voldemort). All ran OK and the compression db sizes were significantly smaller than the no compression db. |
|
That sounds great! Thank you so much for the great help, @jwlent55 and @dmittendorf! |
|
@yhchiang I am still seeing the same exception on my linux box. In fact, I tried it yesterday and I get the snappy not linked exception even on my Mac :( Could I be doing something wrong? The 3.13.1 rocksdb jni in maven central is the latest with the above suggested changes, right ? @dmittendorf I noticed you tested the new rocksdb version with the samza test suite. I am curious if you changed anything in samza other than the rocksdb version in gradle dependency? |
|
Hi Navina, The only change I made to Samza was the bump the version of RocksDB to 3.13.1. I saw the same thing initially when running the Samza tests against the latest build, and realized that there was a cached version of the JAR in my local Gradle cache. You can run the following to ensure that the dependencies are refreshed: |
|
refresh! doh! I have been clearing out my maven cache, instead of gradle.. #facepalm |
|
Getting below error when opening DB in c. |
|
@deepeshj it's weird that the open fails when you try with no compression. In any case, this should work if you install snappy library before compiling RocksDB (we automatically detect the presence of snappy) |
|
Just for reference: similar issue in my project, fixed by adding |
and others
Using
Centos 6
GCC 4.9 using devtools-3
Java 7
Kernel 2.6.32-504.12.2.el6.x86_64
I am getting the following error after running a test for a while.
/usr/java/latest/bin/java: symbol lookup error: /tmp/librocksdbjni2974434001434564758..so: undefined symbol: clock_gettime
The same test runs fine on RocksDb 3.6. My investigation shows that librt.so is not linked with RocksDb correctly. My test (with 3.10.2) worked correctly after I ran
export LD_PRELOAD=/lib64/rtkaio/librt.so.1
To investigate the 3.10.2 library, I ran nm
$nm /tmp/librocksdbjni2974434001434564758..so | grep clock
0000000000332260 T _ZNSt6chrono3_V212steady_clock3nowEv
000000000037e33c R _ZNSt6chrono3_V212steady_clock9is_steadyE
0000000000332230 T _ZNSt6chrono3_V212system_clock3nowEv
000000000037e33d R _ZNSt6chrono3_V212system_clock9is_steadyE
U clock_gettime
I ran nm on the older rocksdb-3.6
nm /tmp/librocksdbjni1323312933457066341..so |grep clock
0000000000287390 T _ZNSt6chrono3_V212steady_clock3nowEv
00000000002c783c R _ZNSt6chrono3_V212steady_clock9is_steadyE
0000000000287360 T _ZNSt6chrono3_V212system_clock3nowEv
00000000002c783d R _ZNSt6chrono3_V212system_clock9is_steadyE
You can see that clock_gettime is undefined in 3.10.2 highlighted in the result of first nm command. Looking at the code, the single call to this function is only included in the C++ code if OS_LINUX or OS_FREEBSD is defined.
Judging from the above nm results, do you think, in 3.6, none of the two flags above were set, where as in 3.10, somehow at least one gets set?
The text was updated successfully, but these errors were encountered: