behavior_planning_container dies while driving straight.

[Kim-mins]

Checklist

• I've read the contribution guidelines.
• I've searched other issues and no duplicate issues were found.
• I'm convinced that this is not my fault but a bug.

Description

Hi team,

I'm currently running Autoware with Carla using bridge implementation here, and I found a situation that the behavior_planning_container diles while driving straight, and it makes the ego vehicle stop before reaching the goal.

Here's the video of the situation: [rviz]
In the video, the vehicle drive well, but it suddenly stops at 0:30.

I also checked a corresponding [launch.log] file and I could find the error as below:

...
1721395398.4731574 [component_container_mt-75] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
1721395398.4732425 [component_container_mt-75]   what():  failed to add guard condition to wait set: guard condition implementation is invalid, at ./src/rcl/guard_condition.c:172, at ./src/rcl/wait.c:460
1721395398.4733043 [component_container_mt-75] *** Aborted at 1721395398 (unix time) try "date -d @1721395398" if you are using GNU date ***
1721395398.4819667 [component_container_mt-75] PC: @                0x0 (unknown)
1721395398.4821553 [component_container_mt-75] *** SIGABRT (@0x3e80000136d) received by PID 4973 (TID 0x7fc3933da640) from PID 4973; stack trace: ***
1721395398.4822626 [component_container_mt-75]     @     0x7fc304040a86 google::(anonymous namespace)::FailureSignalHandler()
1721395398.4823201 [component_container_mt-75]     @     0x7fc39d27b520 (unknown)
1721395398.4841051 [component_container_mt-75]     @     0x7fc39d2cf9fc pthread_kill
1721395398.4870968 [component_container_mt-75]     @     0x7fc39d27b476 raise
1721395398.4893513 [component_container_mt-75]     @     0x7fc39d2617f3 abort
1721395398.4922843 [component_container_mt-75]     @     0x7fc39d524b9e (unknown)
1721395398.4951425 [component_container_mt-75]     @     0x7fc39d53020c (unknown)
1721395398.4973755 [component_container_mt-75]     @     0x7fc39d530277 std::terminate()
1721395398.4996758 [component_container_mt-75]     @     0x7fc39d5301fe std::rethrow_exception()
1721395398.5018551 [component_container_mt-75]     @     0x7fc39d7cf119 rclcpp::exceptions::throw_from_rcl_error()
1721395398.5039811 [component_container_mt-75]     @     0x7fc39d7d2e9c rclcpp::detail::add_guard_condition_to_rcl_wait_set()
1721395398.5060802 [component_container_mt-75]     @     0x7fc39d7e55d7 (unknown)
1721395398.5080755 [component_container_mt-75]     @     0x7fc39d7d2685 rclcpp::Executor::wait_for_work()
1721395398.5100472 [component_container_mt-75]     @     0x7fc39d7d53c3 rclcpp::Executor::get_next_executable()
1721395398.5119836 [component_container_mt-75]     @     0x7fc39d7dc252 rclcpp::executors::MultiThreadedExecutor::run()
1721395398.5138829 [component_container_mt-75]     @     0x7fc39d55e253 (unknown)
1721395398.5156958 [component_container_mt-75]     @     0x7fc39d2cdac3 (unknown)
1721395398.5174739 [component_container_mt-75]     @     0x7fc39d35f850 (unknown)
1721395398.5193157 [component_container_mt-75]     @                0x0 (unknown)
1721395398.6333418 [ERROR] [component_container_mt-75]: process has died [pid 4973, exit code -6, cmd '/opt/ros/humble/lib/rclcpp_components/component_container_mt --ros-args -r __node:=behavior_planning_container -r __ns:=/planning/scenario_planning/lane_driving/behavior_planning -p use_sim_time:=True -p wheel_radius:=0.383 -p wheel_width:=0.235 -p wheel_base:=2.79 -p wheel_tread:=1.64 -p front_overhang:=1.0 -p rear_overhang:=1.1 -p left_overhang:=0.128 -p right_overhang:=0.128 -p vehicle_height:=2.5 -p max_steer_angle:=0.7'].

Expected behavior

I hope the ego vehicle to reach to the destination.

Actual behavior

But the vehicle stops in the middle of the road, and does not move at all.

Steps to reproduce

Here's the ros2bag file: [ros2bag]
I used the lane2let map here, and the initial/goal point as below (excerpted from launch.log file):

# Initial pose
x: 120.057753, 
y: -8.898988, 
z: 0.034443
qx: -0.000467, 
qy: -0.000028, 
qz: -0.007445, 
qw: 0.999972
# Goal pose
x: 209.993347, 
y: -9.808371, 
z: 0.275307
qx: 0.000000, 
qy: 0.000000, 
qz: -0.007468, 
qw: 0.999972

Versions

• autoware.universe: 7877192cb5700c1db15109dcd959785fc381951d
• Carla: 0.9.15

Possible causes

No response

Additional context

No response

[VRichardJP]

Unfortunatelly, your logs just show the main thread crashed at a random place. Most likely the root cause of the crash comes from another thread.

If you are able to reproduce the issue, could you try to catch it with gdb? You just need to recompile with -CMAKE_BUILD_TYPE=RelWithDebInfo and add launch-prefix='gnome-terminal -- gdb -ex=r --args' at the end of <node_container ...> in the behavior_planning.launch.xml. When the program crashes in the new terminal, GDB should automatically switch to the problematic thread and you should be able to get a backtrace with bt

[maxime-clem]

I have not been able to reproduce the issue with the bag.
I was also unable to get the map you showed. It looks different from the Town03 whose link you shared. Unless you are using a custom map_config.yaml ?

[Kim-mins]

Sorry for late reply @VRichardJP, @maxime-clem.. I was busy for other tasks..

For @VRichardJP,
Actually, I failed to reproduce the issue..
I tried to compile Autoware with the option -DCMAKE_BUILD_TYPE=RelWithDebInfo and added launch-prefix='gnome-terminal -- gdb -ex=run --args at the end of <node_container ...> in the behavior_planning.launch.xml, I checked the gnome-terminal is loaded, but I could not debug it more as I failed to reproduce.

For @maxime-clem,
Sorry for providing little information.
Here are the map files and map_projector_info.yaml I used: [.osm] [.pcd] [map_projector_info.yaml]

I'm really sorry for delayed response, and thank you for the answers.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity.

[shamanqing1]

I encountered the same problem.

#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140736773228096) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=140736773228096) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=140736773228096, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff7442476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007ffff74287f3 in __GI_abort () at ./stdlib/abort.c:79
#5 0x00007ffff78a2b9e in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff78ae20c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007ffff78ae277 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007ffff78ae1fe in std::rethrow_exception(std::__exception_ptr::exception_ptr) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007ffff7d5cc59 in rclcpp::exceptions::throw_from_rcl_error (ret=ret@entry=1, prefix="failed to add guard condition to wait set", error_state=error_state@entry=0x0, reset_error=)
at /home/heqing/autoware/src/rclcpp/rclcpp/src/rclcpp/exceptions/exceptions.cpp:90
#10 0x00007ffff7d58e2e in rclcpp::detail::add_guard_condition_to_rcl_wait_set (wait_set=..., guard_condition=...) at /home/heqing/autoware/src/rclcpp/rclcpp/src/rclcpp/detail/add_guard_condition_to_rcl_wait_set.cpp:33
#11 0x00007ffff7d7c9f7 in rclcpp::memory_strategies::allocator_memory_strategy::AllocatorMemoryStrategy<std::allocator >::add_handles_to_wait_set (this=0x555555580a10, wait_set=0x5555555cd898)
at /home/heqing/autoware/src/rclcpp/rclcpp/include/rclcpp/strategies/allocator_memory_strategy.hpp:236
#12 0x00007ffff7d65567 in rclcpp::Executor::wait_for_work (this=0x5555555cd7f0, timeout=...) at /home/heqing/autoware/src/rclcpp/rclcpp/src/rclcpp/executor.cpp:755
#13 0x00007ffff7d65ba3 in rclcpp::Executor::get_next_executable (this=0x5555555cd7f0, any_executable=..., timeout=...) at /home/heqing/autoware/src/rclcpp/rclcpp/src/rclcpp/executor.cpp:917
#14 0x00007ffff7d69912 in rclcpp::executors::MultiThreadedExecutor::run (this=0x5555555cd7f0, this_thread_number=) at /home/heqing/autoware/src/rclcpp/rclcpp/src/rclcpp/executors/multi_threaded_executor.cpp:85
#15 0x00007ffff78dc253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#16 0x00007ffff7494ac3 in start_thread (arg=) at ./nptl/pthread_create.c:442
#17 0x00007ffff7526850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

[maxime-clem]

To prevent this error some fixes are needed on rclcpp.
You can try this custom branch for example: https://github.com/tier4/rclcpp/tree/t4-main
I think the required commit is this one.

[shamanqing1]

To prevent this error some fixes are needed on rclcpp. You can try this custom branch for example: https://github.com/tier4/rclcpp/tree/t4-main I think the required commit is this one.

This commit seams already merge into official humble branch, I used the latest humble branch，and error still exist