Regarding slow output rate in ROS2

[ajay1606]

Hello,

First of all thanks for the ROS2 version of Autoware, pleased to see Autoware newer versions.
I tried with running Autoware ROS2 version, main branch from https://github.com/autowarefoundation /[autoware] (https://github.com/autowarefoundation/autoware) following the instruction given https://autowarefoundation.github.io/autoware-documentation/main/tutorials/ad-hoc-simulation/rosbag-replay-simulation/ .

There were no errors at any stage, but while running rosbag simulation observed topic output frequency drops.

RVIZ : Visualization also shows delay in LiDAR data

Rate : ros2 topic hz /sensing/lidar/top/velodyne_packets returns avg 8 hz & os2 topic hz /sensing/lidar/top/pointcloud_raw_ex returns avg 5 hz

While running only bag file: ros2 topic hz /sensing/lidar/top/velodyne_packets returns avg 9.5 hz (seems to be no issues), but running with autoware drops output rate.

Is this problem associated with ROS2 DDS or Autoware algorithms processing burden?

Greatly appreciate any advice to tune this rate of frequency.

Working Platform:
Ubuntu 20.04 / ROS2 Galactic / Cyclone DDS /

Regards,
Ajay

[storrrrrrrrm]

you can open system monitor of Ubuntu to see if the network load reaches the upper limit

[storrrrrrrrm]

I don't know what the upper limit of the transmission capacity of your switch, You can disconnect the network and test again to see if problem still exists.

besides,this is the cyclonedds.xml we use on our real car.

<CycloneDDS>
    <Domain id="any">
        <General>
            <NetworkInterfaceAddress>auto</NetworkInterfaceAddress>
            <AllowMulticast>spdp</AllowMulticast>
        </General>
    </Domain>
</CycloneDDS>

i do not quite understand this since i am not doing dds related stuff, it works on our car,u can have a try.

[Sharrrrk]

I suggest you firstly tune cyclone DDS according to https://docs.ros.org/en/foxy/How-To-Guides/DDS-tuning.html.

Also you could refer to the instruction given by TakaHoribe:
https://github.com/orgs/autowarefoundation/discussions/204#discussioncomment-2608081

[ajay1606]

@storrrrrrrrm & @Sharrrrk Thanks so much for your suggestions, I will try out as you suggested and get back to you as soon as possible.

[ajay1606]

cyclonedds.xml

Hello @storrrrrrrrm , Would you please let me know, where can I find this cyclonedds.xml on my computer?

[storrrrrrrrm]

cyclonedds.xml

Hello @storrrrrrrrm , Would you please let me know, where can I find this cyclonedds.xml on my computer?

you should create this file manually. then export CYCLONEDDS_URI to your file path

export CYCLONEDDS_URI=/autoware/cyclonedds.xml

[mitsudome-r]

FYI: In the past, I had a case where ros2 topic hz had lower output compared to the actual topic frequency for large messages.
I'm doubting it's due to the difference between the python vs cpp implementation of rcl API.

I'm not sure if it is fixed in newest galactic, but you can use the following topic frequency check tool that use cpp to get the actual topic frequency.
https://github.com/tier4/topic_tools

ros2 launch topic_tools hz_checker.launch.xml t:=topic_name

[Sharrrrk]

FYI: In the past, I had a case where ros2 topic hz had lower output compared to the actual topic frequency for large messages. I'm doubting it's due to the difference between the python vs cpp implementation of rcl API.

I also met this before using foxy. Good to know this tool.

[ajay1606]

FYI: In the past, I had a case where ros2 topic hz had lower output compared to the actual topic frequency for large messages. I'm doubting it's due to the difference between the python vs cpp implementation of rcl API.

I'm not sure if it is fixed in newest galactic, but you can use the following topic frequency check tool that use cpp to get the actual topic frequency. https://github.com/tier4/topic_tools

ros2 launch topic_tools hz_checker.launch.xml t:=topic_name

Thank you so much, Indeed very useful information.

[storrrrrrrrm]

FYI: In the past, I had a case where ros2 topic hz had lower output compared to the actual topic frequency for large messages. I'm doubting it's due to the difference between the python vs cpp implementation of rcl API.

I'm not sure if it is fixed in newest galactic, but you can use the following topic frequency check tool that use cpp to get the actual topic frequency. https://github.com/tier4/topic_tools

ros2 launch topic_tools hz_checker.launch.xml t:=topic_name

thanks! very helpful.

[ajay1606]

@storrrrrrrrm & @Sharrrrk I have tried with as you suggested, exporting CYCLONEDDS_URI to cyclonedds.xml file and then Also instruction given by TakaHoribe:here. And observed that,
output rate of topic: /sensing/lidar/top/velodyne_packets some what maintaining on avg 9.5hz, but output rate of /sensing/lidar/top/pointcloud_raw getting dropped.

And CPU stress looks as below: Where velodyne container node which is responsible for publishing /sensing/lidar/top/pointcloud_raw topic consuming much of CPU resources ,

Are there any possible ways to overcome this issue?

Regarding ros2 topic hz tool i have tried with the @mitsudome-r recommended package. And observed that, the tool shows very consistent output rate for topic /sensing/lidar/top/velodyne_packets, and a very inconsistent topic rate for /sensing/lidar/top/pointcloud_raw .

Example, /sensing/lidar/top/velodyne_packets topic rat e was very consistent with showing rate as 9.5~ to 9.6hz, but for the topic /sensing/lidar/top/pointcloud_raw topic rate very inconsistent, there are sudden jumps from 6hz to 9hz and 9 hz 6hz in consecutive readings.

In the case of ros2 topic hz it was very consistent with both the topic data like /sensing/lidar/top/velodyne_packetsmaintained at avg of 9 hz while/sensing/lidar/top/pointcloud_raw` drops to 6hz gradually not suddenly.

And after referring to this link, still, it is unknown that, for what reason does topic rate getting slows down for /sensing/lidar/top/pointcloud_raw , Any hint greatly appreciated.

Regards,
Ajay

[nainaigetuide]

@ajay1606 Did you solve this problem please?

[hect95]

@ajay1606 Did you find any solution for this?

[ajay1606]

@hect95 Here is the very useful discussion related to slow rate. Indeed, it worked well for me. Please check it once.
autowarefoundation/autoware.universe#2597 (comment)

Make sure to delete previous build (install, log, build) directory before build workspace using following command.
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=1 --continue-on-error

[hect95]

Hey @ajay1606, this seems to work, so thank you very much for this!

However, when I try to record a rosbag with LIDAR pointlcouds, rates becomes slower.

I am using two lidars, one that is defined as Reliable QoS and the second one as best effort, do you think the problem can be related to this subject?

[ajay1606]

@hect95 I would suggest to check cyclone DDS configuration once here, and increase the buffer size as it mentioned.

Also, for real time sensor data, it is strongly recommended to set best effort in QoS profile.

By the way is there any specific reason to choose different modes (Reliable , Best effort) for 2 Lidars ?