Linking error (test_atomics_executable) on Raspbian/armv6l

[christianrauch]

I am trying to compile ROS dashing from source on Raspbian Buster (gcc (Raspbian 8.3.0-6+rpi1) 8.3.0) running on armv6l and encounter a linking error for target test_atomics_executable:

/usr/bin/ld: CMakeFiles/test_atomics_executable.dir/test/test_atomics.c.o: in function `main':
test_atomics.c:(.text+0x11a8): undefined reference to `__atomic_store_8'
/usr/bin/ld: test_atomics.c:(.text+0x11c0): undefined reference to `__atomic_load_8'
/usr/bin/ld: test_atomics.c:(.text+0x1268): undefined reference to `__atomic_store_8'
/usr/bin/ld: test_atomics.c:(.text+0x12b8): undefined reference to `__atomic_exchange_8'
[...]

I am using the standard repo at https://raw.githubusercontent.com/ros2/ros2/release-latest/ros2.repos, excluding rviz2 and rqt.

Edit: The problem can be resolved by manually adding -latomic, i.e. target_link_libraries(test_atomics_executable ${PROJECT_NAME} -latomic).

[clalancette]

Edit: The problem can be resolved by manually adding -latomic, i.e. target_link_libraries(test_atomics_executable ${PROJECT_NAME} -latomic).

Please open a PR with that change; we'll gladly review it. Thanks.

[christianrauch]

I discovered that I get the same linking issues in a couple of other packages (tlsf_cpp, rcl, rclcpp, demo_nodes_cpp, ...) which can all be resolved by linking against libatomic.
Before sending PRs for all of them, adding -latomic, I would like to hear back if there might be a better solution. After all, this seems to happen either with never GCC (>8) or specifically on Raspbian.

[sloretz]

Here's an example that sets CMAKE_CXX_STANDARD_LIBRARIES to conditionally include -latomic, which it seems is a variable not meant to be set by a project. CMake itself sets CMAKE_..._STANDARD_LIBRARIES to include -latomic on android static.

It seems reasonable to target_link_libraries(... -latomic) as a workaround, but it will need to be conditional since that won't be available on all platforms. Maybe it's also worth checking if this is something that can be fixed upstream?

[sloretz]

Reopening since issue was reverted in #177

[nkueng]

I am using the standard repo at https://raw.githubusercontent.com/ros2/ros2/release-latest/ros2.repos, excluding rviz2 and rqt.

@christianrauch what's the best practice to exclude certain packages (i.e., heavy stuff like rviz2 and everything else with a gui) from the build? Have you just added the AMENT_IGNORE files or is there another way?

[christianrauch]

I am using the standard repo at https://raw.githubusercontent.com/ros2/ros2/release-latest/ros2.repos, excluding rviz2 and rqt.

@christianrauch what's the best practice to exclude certain packages (i.e., heavy stuff like rviz2 and everything else with a gui) from the build? Have you just added the AMENT_IGNORE files or is there another way?

In this case, I just removed the packages from the ros2.repos yaml file such that they are not downloaded as part of the workspace.

Apart from this obvious solution, you can add a COLCON_IGNORE file to a package to skip over it during the build (see https://index.ros.org/doc/ros2/Tutorials/Colcon-Tutorial/#tips). This is useful if you only want to build a subset of packages from the same repo.

[BrettRD]

This has been reverted again without a permanent fix and needs re-opening.

For those looking for minimal arm builds, you can limit the packages selected for download with ros2/ros2_documentation#870 (comment)
summary, use rosinstall_generator ros_base --deps --rosdistro foxy | vcs import src/ instead of pulling wget ... ros2.repos

and then limit what colcon deals with using colcon build --packages-up-to <packages you care about>
https://colcon.readthedocs.io/en/released/reference/package-selection-arguments.html

For anyone finding this issue on this or other packages (like fastrtps), you can add --cmake-args "-DCMAKE_SHARED_LINKER_FLAGS='-latomic'" "-DCMAKE_EXE_LINKER_FLAGS='-latomic'" to your colcon arguments.
ros2/ros2#418 (comment)

[clalancette]

I'll encourage you to open a new issue (or even better, PR) if this is something that needs to be fixed in code. We're happy to review it.

[sloretz]

This has been reverted again without a permanent fix and needs re-opening.

@BrettRD what PR reverted it? I still see the fix from #178 on master:

rcutils/CMakeLists.txt

Line 162 in 1570091

ament_export_link_flags("-latomic")

rcutils/CMakeLists.txt

Line 261 in 1570091

target_link_libraries(test_atomics_executable atomic)

[BrettRD]

@BrettRD what PR reverted it? I still see the fix from #178 on master:

My mistake, The fixes are all still in place.
The tests were added since then, and they didn't use the same conditional link. I just put a patch together, pull request coming shortly.

[Ecuashungo]

I have the same problem with rcutils and the atomic library with ros galactic on Raspberry Pi OS 64-bit (bullseye). The fixes from #178 can still be seen in the respective files, but I still get the following error message:

/usr/bin/ld: librcutils.so: undefined reference to `__atomic_compare_exchange_8'
/usr/bin/ld: librcutils.so: undefined reference to `__atomic_load_8'
/usr/bin/ld: librcutils.so: undefined reference to `__atomic_store_8'
collect2: error: ld returned 1 exit status
gmake[2]: *** [CMakeFiles/test_logging_long_messages.dir/build.make:104: test_logging_long_messages] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1229: CMakeFiles/test_logging_long_messages.dir/all] Error 2
gmake: *** [Makefile:160: all] Error 2
---
Failed   <<< rcutils [57.0s, exited with code 2]

Can someone point me to a resource where I can find a possible fix?

[BrettRD]

This is an issue with raspberry pi os as built by the raspberry pi foundation.
You need to pass -latomic to cmake via colcon for every package.
the easiest way is to add a colcon defaults file:

/home/pi/.colcon/defaults.yaml

{
  "build": {
    "cmake-args":[ "-DCMAKE_SHARED_LINKER_FLAGS='-latomic'","-DCMAKE_EXE_LINKER_FLAGS='-latomic'" ],
  }
}

if you don't fee like adding the defaults file, you can use the arguments on the command line instead:
colcon build --cmake-args "-DCMAKE_SHARED_LINKER_FLAGS='-latomic'" "-DCMAKE_EXE_LINKER_FLAGS='-latomic'"

I've tried to patch this lib and others, this is just the tip of the iceberg. Every ROS package needs -latomic but only on Raspberry OSes.

The raspberry pi is an important part of the ecosystem, but this is a Raspbian bug and the workaround above is the sane solution

[Ecuashungo]

Thanks for the prompt answer.
This actually worked for me. I added the file as @BrettRD suggested:

/home/pi/.colcon/defaults.yaml

{
  "build": {
    "cmake-args":[ "-DCMAKE_SHARED_LINKER_FLAGS='-latomic'","-DCMAKE_EXE_LINKER_FLAGS='-latomic'" ],
  }
}

And I modified the CMakeLists.txt as in #341. Basically adding target_link_libraries(XXX atomic) after every library linking to ${PROJECT_NAME}. Just like this:

target_link_libraries(XXX ${PROJECT_NAME})
if(HAVE_LIBATOMICS)
  target_link_libraries(XXX atomic)
endif()

But as mentioned by @BrettRD, after rcutils the next package failed with the same error. So it is really just the tip of the iceberg.
What would have to be done to solve this systematically?

[sloretz]

googling raspberry pi latomic brings up a lot of results in a lot more than ROS. These two looked interesting.

• https://gitlab.kitware.com/cmake/cmake/-/issues/21174
• https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81358

I can't say I understand well enough to suggest a fix, but @BrettRD's workaround looks reasonable.

[BrettRD]

What would have to be done to solve this systematically?

Two options:
Either

• Every ROS package and cmake package in existence needs to have a specific provision for the raspberry pi, even when the developers don't own a raspberry pi

Or

• Raspberry Pi Foundation can fix their build environments like Armbian and everyone else has (this is an old enough RPi bug that people are making up excuses as to why it should be and remain this way)

For now, colcon defaults.yaml lets RPi users use ROS as normal

[Ecuashungo]

I think I have misunderstood.
@BrettRD are you saying that the only thing that I need to do is to add the colcon defaults.yaml and then I can install everything normally?
Because right now it only worked if I added the defaults.yaml file and additionally I have added the following snippets for all target_link_libraries calls following the same pattern you have used in #341:

target_link_libraries(XXX ${PROJECT_NAME})
if(HAVE_LIBATOMICS)
  target_link_libraries(XXX atomic)
endif()

Only by doing both I got rcutils to build.

Should rcutils build just with the defaults.yaml fix alone?

Unfortunately rcpputils and rosidl_runtime_c still fail to link the atomics library even with the two above steps implemented.

[BrettRD]

@BrettRD are you saying that the only thing that I need to do is to add the colcon defaults.yaml and then I can install everything normally?

Yes.
Cmake caches build flags, you need a clean build rm -r install build

target_link_libraries(XXX atomic) is more or less equivalent to --cmake-args "-DCMAKE_SHARED_LINKER_FLAGS='-latomic'" "-DCMAKE_EXE_LINKER_FLAGS='-latomic'"

I did the run around last year for the rpi trying to get automated builds out of https://github.com/BrettRD/ros2-playbooks
It worked last year, right down to the pi zero.

I did have to send some patches upstream (armv6 support), but trying to patch RPi atomics into ROS (#341) was a mistake.