Undefined Symbols at Runtime

[Christian-Prather]

Issue

Hello again,
After working through all of my build issues I was finally able to get a successful rust workspace build using Corrosion! However upon trying to run some of my rust bins I was greeted with symbol lookup errors such as what you see below.

root@Christian-dev:/app/test-code/src/test_robot/test_rust/build/cargo/build/x86_64-unknown-linux-gnu
/debug# ./test-bin
./test-bin: symbol lookup error: ./test-bin: undefined symbol: _ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE

This stumped me and for the last day I have been messing with any possible combination of linker / settings I could think but nothing seems to work. I am wondering if anyone has seen this and if anyone working on Corrosion has any gut feel about this issue. I'm SO close to a working build I really want to get across that finish line!

Situation

Essentially I am using Corrosion not to add rust code to C++ but to support building rust code in a larger cmake based system.

I have a cmake file for a project called test-rust and in this cmake I am importing a workspace cargo.toml with Corrosion. This workspace consists of binaries I am trying to build and some dylib dependencies for said bins.

Currently the Cmake compilers for C/C++ are using GNU and are the drivers for the rustc linker (since I am NOT setting NO_LINKER_OVERRIDE) however I am passing -fuse-ld=lld to rust through a cargo config.toml file. Without this the cxx_build I have in some of my build.rs files fails to link.

If I turn on NO_LINKER_OVERRIDE I get a full build same as without and the same error

Summary

I am using Corrosion to integrate the building of Rust bins and libs into a much larger legacy Cmake system. I have a rust workspace and while everything is building and linking at run time all of the (~20) bins have symbol lookup errors.
Again I want to make clear this is all building but at run time it fails.

Code Snippets

test-rust/CMakeLists.txt

corrosion_import_crate(
  MANIFEST_PATH ${CMAKE_CURRENT_SOURCE_DIR}/Cargo.toml
  IMPORTED_CRATES CORROSION_IMPORTED
  # NO_LINKER_OVERRIDE
  FLAGS "--timings"
  PROFILE Debug
)

test-rust/.cargo/config.toml

[build]
rustflags = ["-C", "prefer-dynamic=yes", "-C", "debuginfo=1", "-C", "link-arg=-fuse-ld=lld"]

Thank you in advance for any input, advice, or guidance!

[jschwe]

To clarify, you are using cxx build from build.rs buildscripts and not the function that corrosion provides for generating bindings with cxx, right?

[Christian-Prather]

Correct, these particular crates use the build.rs method. If possible I would like to keep it in there, I know that it is a black box to Corrosion but as I am only trying to link the CPP into the rust binary and the rustc linker should be handling bin crates I figured it wouldn't matter.

It worth also noting since I forgot to mention it I am able to build this workspace with just cargo and items run correctly (not sure the use of that info but wanted to rule out the obvious of its just my stuff breaking).

[jschwe]

Do you also get the error if you remove the "prefer-dynamic" option?

[jschwe]

could you set the LD_DEBUG environment variable when running the bin and check if libtest_rust.so and libstd-d2ef02247056996e.so get loaded?

[Christian-Prather]

Yes, give me just a few minutes I had cleared my WS to test enabling rpath in my Cargo.toml and need to rebuild.
Thank you for such fast feedback it is VERY appreciated.

[Christian-Prather]

Okay sorry for the delay a fresh build was brutal.

Running LD_DEBUG=all LD_DEBUG_OUTPUT=../rust-code/debug-file testbin produces the following. It appears both those libs are being loaded. I have the entire output as a txt file as well if that is useful.

       100:	file=libtest_rust.so [0];  needed by testbin [0]
       100:	find library=libtest_rust.so [0]; searching
       100:	 search path=/app/ws/install_debug/lib/tls/haswell/x86_64:/app/ws/install_debug/lib/tls/haswell:/app/ws/install_debug/lib/tls/x86_64:/app/ws/install_debug/lib/tls:/app/ws/install_debug/lib/haswell/x86_64:/app/ws/install_debug/lib/haswell:/app/ws/install_debug/lib/x86_64:/app/ws/install_debug/lib:/opt/ros/melodic/lib/tls/haswell/x86_64:/opt/ros/melodic/lib/tls/haswell:/opt/ros/melodic/lib/tls/x86_64:/opt/ros/melodic/lib/tls:/opt/ros/melodic/lib/haswell/x86_64:/opt/ros/melodic/lib/haswell:/opt/ros/melodic/lib/x86_64:/opt/ros/melodic/lib:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/tls/haswell/x86_64:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/tls/haswell:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/tls/x86_64:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/tls:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/haswell/x86_64:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/haswell:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/x86_64:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug:/usr/local/cuda/lib64/tls/haswell/x86_64:/usr/local/cuda/lib64/tls/haswell:/usr/local/cuda/lib64/tls/x86_64:/usr/local/cuda/lib64/tls:/usr/local/cuda/lib64/haswell/x86_64:/usr/local/cuda/lib64/haswell:/usr/local/cuda/lib64/x86_64:/usr/local/cuda/lib64		(LD_LIBRARY_PATH)
       100:	  trying file=/app/ws/install_debug/lib/tls/haswell/x86_64/libtest_rust.so
       100:	  trying file=/app/ws/install_debug/lib/tls/haswell/libtest_rust.so
       100:	  trying file=/app/ws/install_debug/lib/tls/x86_64/libtest_rust.so
       100:	  trying file=/app/ws/install_debug/lib/tls/libtest_rust.so
       100:	  trying file=/app/ws/install_debug/lib/haswell/x86_64/libtest_rust.so
       100:	  trying file=/app/ws/install_debug/lib/haswell/libtest_rust.so
       100:	  trying file=/app/ws/install_debug/lib/x86_64/libtest_rust.so
       100:	  trying file=/app/ws/install_debug/lib/libtest_rust.so
       100:	
       100:	file=libtest_rust.so [0];  generating link map
       100:	  dynamic: 0x000073d0fb049970  base: 0x000073d0f9600000   size: 0x0000000001a5b210
       100:	    entry: 0x000073d0fa106000  phdr: 0x000073d0f9600040  phnum:                 11
       100:	
       100:	
       100:	file=libstd-d2ef02247056996e.so [0];  needed by testbin [0]
       100:	find library=libstd-d2ef02247056996e.so [0]; searching
       100:	 search path=/app/ws/install_debug/lib:/opt/ros/melodic/lib/tls/haswell/x86_64:/opt/ros/melodic/lib/tls/haswell:/opt/ros/melodic/lib/tls/x86_64:/opt/ros/melodic/lib/tls:/opt/ros/melodic/lib/haswell/x86_64:/opt/ros/melodic/lib/haswell:/opt/ros/melodic/lib/x86_64:/opt/ros/melodic/lib:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/tls/haswell/x86_64:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/tls/haswell:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/tls/x86_64:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/tls:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/haswell/x86_64:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/haswell:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug/x86_64:/app/ws/build_debug/test_rust/cargo/build/x86_64-unknown-linux-gnu/debug:/usr/local/cuda/lib64/tls/haswell/x86_64:/usr/local/cuda/lib64/tls/haswell:/usr/local/cuda/lib64/tls/x86_64:/usr/local/cuda/lib64/tls:/usr/local/cuda/lib64/haswell/x86_64:/usr/local/cuda/lib64/haswell:/usr/local/cuda/lib64/x86_64:/usr/local/cuda/lib64		(LD_LIBRARY_PATH)
       100:	  trying file=/app/ws/install_debug/lib/libstd-d2ef02247056996e.so
       100:	
       100:	file=libstd-d2ef02247056996e.so [0];  generating link map
       100:	  dynamic: 0x000073d0fb3f5150  base: 0x000073d0fb2d0000   size: 0x0000000000127420
       100:	    entry: 0x000073d0fb2d0000  phdr: 0x000073d0fb2d0040  phnum:                 10
       100:	
       100:	
       100:	file=libdl.so.2 [0];  needed by testbin [0]

Here is the ending of the file:

       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=testbin [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/app/ws/install_debug/lib/libtest_rust.so [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/app/ws/install_debug/lib/libstd-d2ef02247056996e.so [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/lib/x86_64-linux-gnu/libz.so.1 [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/lib/x86_64-linux-gnu/libgcc_s.so.1 [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/lib/x86_64-linux-gnu/librt.so.1 [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/app/ws/install_debug/lib/libglog.so.0 [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/usr/lib/x86_64-linux-gnu/libstdc++.so.6 [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/lib/x86_64-linux-gnu/libudev.so.1 [0]
       100:	symbol=_ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE;  lookup in file=/app/ws/install_debug/lib/libgflags.so.2.2 [0]
       100:	testbin: error: symbol lookup error: undefined symbol: _ZN61_$LT$serde_json..error..Error$u20$as$u20$core..fmt..Debug$GT$3fmt17h93639d39d470efadE (fatal)

[jschwe]

Could you verify with objdump or nm if there is any serde_json related code in libtest_rust.so (similar to the missing function)?
Could you also use ldd to get a trace of which dynamic libraries libtest_rust.so might in turn depend on? I'm guessing that the missing function is in some dependency which is again a shared library.

[Christian-Prather]

Absolutely:

For the objdump I am not sure the command for that exactly, if I run objdump -d libtest_rust.so -M x86-64| grep serde_json
I do get a large number of output, I think indicating it is in fact used?

ldd

root@Christian:/app/ws/install_debug/lib# ldd libtest_rust.so 
	linux-vdso.so.1 (0x00007ffc86f89000)
	libglog.so.0 => /app/ws/install_debug/lib/libglog.so.0 (0x00007ec9aca00000)
	libstd-d2ef02247056996e.so => /app/ws/install_debug/lib/libstd-d2ef02247056996e.so (0x00007ec9aeb0a000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ec9ac677000)
	libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007ec9ac459000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ec9ac241000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ec9ac039000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ec9abe1a000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ec9aba7c000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ec9ab68b000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ec9aea0b000)
	libgflags.so.2.2 => /app/ws/install_debug/lib/libgflags.so.2.2 (0x00007ec9ab400000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ec9ab1fc000)
root@Christian:/app/ws/install_debug/lib#