QoS / middleware configuration: modifying buffer stream size

[anaelle-sw]

Hi! I am trying to configure a publisher's QoS policy by modifying the colcon.meta file of micro_ros_arduino library. I rather not use labels and .refs files for now.

Setup

• Computer: Ubuntu 20.04 / ROS2 Foxy
• Board: Teensy 3.1 / Arduino IDE 1.8.13 / Teensyduino 1.53

Use case description

I got one std_msgs__msg__Range type publisher running on my Teensy board. I can perfectly get the published messages via ros2 topic echo /my/range/topic, if the messages' frame_id is less than 6 char long. If not, the following error raises when checking out the topic:

terminate called after throwing an instance of 'eprosima::fastcdr::exception::NotEnoughMemoryException'
  what():  Not enough memory in the buffer stream
Aborted (core dumped)

As I got nine different frames ID which should be strings between 19 and 23 char long, I need to increase the stream buffer size. To do so, I tried to follow the micro-ROS tutorials about middleware configuration

Steps to reproduce

As I use a Teensy 3.1 board, the file which should be modified is colcon_lowmem.meta. After any modification of this file, I build the micro_ros_arduino library (only for Teensy 3 type boards) with command lines:
cd /Arduino/libraries/micro_ros_arduino
sudo docker pull microros/micro_ros_arduino_builder:latest
sudo docker run -it --rm -v $(pwd):/arduino_project microros/micro_ros_arduino_builder:latest -p teensy3

• First, I tried to add the option UCLIENT_SERIAL_TRANSPORT_MTU, which is set at 12 by default, according to the micro-ROS tutorial. So, I modified the colcon_lowmem.meta file to get this:

[...]
        "microxrcedds_client": {
            "cmake-args": [
                [...]
                "-DUCLIENT_SERIAL_TRANSPORT_MTU=24"
            ]
[...]

Once the library is built, I am able to upload the .ino source file on the board. But the application seems "stuck" right after initialization, as when running the micro-ROS agent, the only output is:

[1604914091.300743] info     | TermiosAgentLinux.cpp | init                     | running...             | fd: 3
[1604914091.801606] info     | TermiosAgentLinux.cpp | fini                     | server stopped         | fd: 3
[1604914091.801736] info     | TermiosAgentLinux.cpp | init                     | running...             | fd: 3
[1604914091.802472] info     | TermiosAgentLinux.cpp | fini                     | server stopped         | fd: 3
[1604914091.802565] info     | TermiosAgentLinux.cpp | init                     | running...             | fd: 3
[1604914091.802576] info     | TermiosAgentLinux.cpp | fini                     | server stopped         | fd: 3
[1604914091.802642] info     | TermiosAgentLinux.cpp | init                     | running...             | fd: 3
[1604914092.981950] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x50E95322, len: 24, data: 
0000: 80 00 00 00 00 01 10 00 58 52 43 45 01 00 01 0F 50 E9 53 22 81 00 14 00
[1604914092.982153] info     | SessionManager.hpp | establish_session        | session re-established | client_key: 0x1357468450, address: 1
[1604914092.982287] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x50E95322, len: 19, data: 
0000: 81 00 00 00 04 01 0B 00 00 00 58 52 43 45 01 00 01 0F 00

... and then stops.

I got the same result when trying to set this option to its default value: "-DUCLIENT_SERIAL_TRANSPORT_MTU=12".
So I guess this is not the right way to set up this option.

• Then I tried to add the option "RMW_UXRCE_STREAM_HISTORY" instead:

[...]
        "rmw_microxrcedds": {
            "cmake-args": [
                [...]
                "-DRMW_UXRCE_STREAM_HISTORY=10"
            ]
[...]

I tried with different values between 5 and 20.
With this option set up at 5 (which I guess is the default value), I get the same results as before configuring: the buffer stream error is raised.
With any else tested values (example: 20), I cannot upload my .ino code to the board.

Questions

Using custom QoS with .refs files and labels requires to define QoS for every publisher/subscriber that you declare, right? If so, I would really rather configure the QoS without using labels. So, I need to modify my colcon.meta file. Did I miss something about this? It seems that if I modify anything in this file, my application stops working.

Do you have any documentation or example colcon.meta file I could base mine on? It seems to me that micro-ROS tutorial does not list all settings (with definition, boundaries, etc, for each) for QoS/middleware configuration, and I was not able to find more exhaustive (and relevant) documentation about it.

Thank you for your awesome micro_ros_arduino and support!

[pablogs9]

Hello @anaelle-sw, some points:

---

1. I have seen this error before and it is not related to colcon.meta configuration. The way a string component of a type should be inited is something like:

// Init
std_msgs__msg__String msg;
msg.data.data = (char*)malloc(200*sizeof(char));
msg.data.size = 0;
msg.data.capacity = 200;

// Use
char my_str[] = "TEST";
memcpy(msg.data.data, my_str, strlen(my_str));
msg.data.size = strlen(msg.data.data);
rcl_publish(&publisher, &msg, NULL);

Please let me know how do you initialize your string and if this approach works for you.

---

2. Regarding the documentation for the parameters in the colcon.meta, there is no complete documentation about them because it just sets CMake args for each of the Colcon packages included in the micro-ROS stack. For example, the ones for the micro-ROS RMW are listed here and the ones related to XRCE-DDS are listed here.

I think that is important to clarify the usage of UCLIENT_SERIAL_TRANSPORT_MTU, RMW_UXRCE_STREAM_HISTORY, RMW_UXRCE_MAX_HISTORY:

• The micro-ROS XRCE middleware uses an internal buffer where all your incoming and outcoming data are serialized and deserialized. The size of this buffer is UCLIENT_SERIAL_TRANSPORT_MTU * RMW_UXRCE_STREAM_HISTORY. The RMW_UXRCE_STREAM_HISTORY is the number of slots in which the buffer is divided. These slots are used for message fragmentation. In general, you have to ensure that the biggest message incoming or outcoming message you have must fit in one buffer.

• The micro-ROS RMW handles a historic of an incoming topic, requests, and replies. This is due to the async nature of some of the internal ROS 2 layers. RMW_UXRCE_MAX_HISTORY is the size of this historic. For example, by default, these parameters is 4, so the middleware can retrieve up to 4 topic subscriptions and store them before the user takes the information. The size of each of this historic slots is a complete buffer, so you will have UCLIENT_SERIAL_TRANSPORT_MTU * RMW_UXRCE_STREAM_HISTORY * RMW_UXRCE_MAX_HISTORY.

---

3. When you are writing a refs file you have to follow the DDS-XML standard. Here you have an example. And here you have complete documentation of all the options available in these XML files. REMIND to use the ROS 2 name mangling (see things like std_msgs::msg::dds_::Float32_ or weather_station/humidity__dw in the example I linked)

[anaelle-sw]

About the string initialization, I am using a char list, just like this one:

unsigned char* frame_id[5] = {"sensor_1_link",  "sensor_2_link", "sensor_3_link",  "sensor_4_link",  "sensor_5_link"};

Then I was just putting the right frame_id in my range-type message, before publishing it:

sensor_msgs__msg__Range range_msg;
rcl_publisher_t range_msg;
[...]
range_msg.header.frame_id.data = frame_id[i];
[...]
rcl_publish(&range_pub, &range_msg, NULL);

This worked well, but only for frame_id less than 6 char long.
Your initialization method works perfectly in our case, even with "big" frame_id strings.
So actually, I no more need to configure the middleware options (at least for now)

Thanks for the explanations and documentations! I have checked out these, and we will sure find them to be really useful very soon. So, I will close this issue for now.

Thanks for your awesome support!

[anaelle-sw]

Sorry to re-open this, but I have one more question.

Our complete application that we need to run on our Teensy board features:

• 21 publishers
• 3 subscribers
• 12 service servers
• 3 service clients

When having the middleware configuration set to default (except for RMW_UXRCE_MAX_PUBLISHERS RMW_UXRCE_MAX_SUBSCRIPTIONS, RMW_UXRCE_MAX_SERVICES, RMW_UXRCE_MAX_CLIENTS that I had to set up), I cannot upload the code to the board (the board RAM would be overflowed). I have to "remove" at least 6 of the subscribers (or services, or clients) to get an acceptable used dynamic memory (around 83%).

I have read this (really nice!) document about memory profiling for micro-ROS. So I understand that the memory for subscribers/publishers/services/clients is pre-allocated, and defined at build time by the parameters set in the colcon.meta file.

My questionings here are about the reserved memory for subscribers/services/clients. If you use .refs files to configure the profiles, will the pre-allocated memory be reduced? I mean, in the profiles, you can (have to?) specify the topics' types. So in this case, is the memory part allocated for each subscribers/services/clients depending on the specified topic type?
Also, when using .refs files, are the options RMW_UXRCE_MAX_SUBSCRIPTIONS, RMW_UXRCE_MAX_SERVICES, and RMW_UXRCE_MAX_CLIENTS required to be set up too?
Last thing, in the document it is written about memory optimization: "Our most recent considerations are directed towards reducing this number by using a shared RMW history pool for every entity." Is this also considered for micro_ros_arduino (or only for micro_ros)? Do you know about when we can expect this optimization feature?

Thanks!

[pablogs9]

Hello @anaelle-sw, if you use .refs the client memory usage will be almost the same.

Regarding the memory pools optimizations, we are planning it, maybe in a couple of weeks, we have some results. Of course, it will apply also for micro-ROS for Arduino.

Now your problem, I think that I can help you to tune your configuration in order to fit in reasonable memory usage. Can you detail more information about these entities? I would need to know you if they are best_effort or you need reliability. And the most important thing: how big are your topics?

Knowing these two things we can tune UCLIENT_SERIAL_TRANSPORT_MTU, RMW_UXRCE_STREAM_HISTORY, RMW_UXRCE_MAX_HISTORY in order to make the application works.

[anaelle-sw]

Regarding the memory pools optimizations, we are planning it, maybe in a couple of weeks, we have some results. Of course, it will apply also for micro-ROS for Arduino.

Really nice!

Thanks for helping! So these are the entities I need:

Entities type	Topic size (Bytes)	Number of entities
Publisher	1	6
Publisher	2	3
Publisher	13	1
Publisher	129	9
Publisher	132	1
Publisher	347	1
-------------	-------------	-------------
Subscriber	1	2
Subscriber	24	1
-------------	-------------	-------------
Service client	101	2
Service client	133	1
-------------	-------------	-------------
Service server	102	4
Service server	105	4
Service server	106	1
Service server	125	1
Service server	136	1
Service server	152	1

I hope I didn't do huge mistakes computing these. I considered strings to be max. 100 bytes sized. Best effort is fine for all.

[pablogs9]

Try to set UCLIENT_SERIAL_TRANSPORT_MTU (under microxrcedds_client) to 128 and RMW_UXRCE_STREAM_HISTORY (under rmw_microxrcedds) to 4. Keep RMW_UXRCE_MAX_HISTORY to 1.

This way you will have RMW buffer of 512 B for each sub, service and client. Publishers does not have RMW buffers. UCLIENT_SERIAL_TRANSPORT_MTU (128) * RMW_UXRCE_STREAM_HISTORY (4) * RMW_UXRCE_MAX_HISTORY (1).
You have to ensure that your biggest message fits in UCLIENT_SERIAL_TRANSPORT_MTU * RMW_UXRCE_STREAM_HISTORY

With the default configuration, you have 2048 B buffer in RMW for each sub, service and client entity. UCLIENT_SERIAL_TRANSPORT_MTU (512) * RMW_UXRCE_STREAM_HISTORY (4) * RMW_UXRCE_MAX_HISTORY (1 in lowmem_colcon.meta).

I'm leaving for today, but if this does not work on Monday I will reproduce your use case and try fit the memory requirements here.

[anaelle-sw]

Your idea seems great, but some problems occurred...

Setting "-DUCLIENT_SERIAL_TRANSPORT_MTU=128" allows me to upload my application on a board, since memory usage is only 52% (with all RMW_UXRCE_MAX_[entities] options set to what I need). I confirm that all our messages should be fine with a 512B-sized buffer. But with this configuration, my application seems to crash right after initialization, as the micro-ROS agent is "stuck". The same problem occurs when running your example publisher or subscriber codes.

Then I tried "-UCLIENT_SERIAL_TRANSPORT_MTU=128" (without the "D" in front of the option name, contrary to the rest of the options in the colcon.meta file). If the RMW_UXRCE_MAX_[entities] options are set to default, a custom publisher node of mine works (as well as your example publisher/subscriber nodes). But if the RMW_UXRCE_MAX_[entities] options are set up to what my application requires, the memory usage overflows the board RAM for my custom publisher node (as if the UCLIENT_SERIAL_TRANSPORT_MTU option was actually not set up to 128), so the code cannot be uploaded to the board.

What does the "D" in front of the other options stands for?( "Default", may be?) Why does it make the application crash? I guess the issue is coming from my colcon.meta file, which looks like this:

    "names": {
        "tracetools": {
            "cmake-args": [
                "-DTRACETOOLS_DISABLED=ON",
                "-DTRACETOOLS_STATUS_CHECKING_TOOL=OFF"
            ]
        },
        "rosidl_typesupport": {
            "cmake-args": [
                "-DROSIDL_TYPESUPPORT_SINGLE_TYPESUPPORT=ON"
            ]
        },
        "rcl": {
            "cmake-args": [
                "-DBUILD_TESTING=OFF",
                "-DRCL_COMMAND_LINE_ENABLED=OFF",
                "-DRCL_LOGGING_ENABLED=OFF"
            ]
        }, 
        "rcutils": {
            "cmake-args": [
                "-DENABLE_TESTING=OFF",
                "-DRCUTILS_NO_FILESYSTEM=ON",
                "-DRCUTILS_NO_THREAD_SUPPORT=ON",
                "-DRCUTILS_NO_64_ATOMIC=ON",
                "-DRCUTILS_AVOID_DYNAMIC_ALLOCATION=ON"
            ]
        },
        "microxrcedds_client": {
            "cmake-args": [
                "-DUCLIENT_PIC=OFF",
                "-DUCLIENT_PROFILE_UDP=OFF",
                "-DUCLIENT_PROFILE_DISCOVERY=OFF",
                "-DUCLIENT_PROFILE_SERIAL=ON",
                "-DUCLIENT_EXTERNAL_SERIAL=ON",
                "-DUCLIENT_SERIAL_TRANSPORT_MTU=128" 
            ]
        },
        "rmw_microxrcedds": {
            "cmake-args": [
                "-DRMW_UXRCE_MAX_NODES=1",
                "-DRMW_UXRCE_MAX_PUBLISHERS=21",
                "-DRMW_UXRCE_MAX_SUBSCRIPTIONS=3",
                "-DRMW_UXRCE_MAX_SERVICES=12",
                "-DRMW_UXRCE_MAX_CLIENTS=3",
                "-DRMW_UXRCE_MAX_HISTORY=1",
                "-DRMW_UXRCE_TRANSPORT=custom_serial"
            ]
        }
    }
}

As you can see, it is similar to the default file, except I set up the options DRMW_UXRCE_MAX_[entities] to what we need, and add the option "-DUCLIENT_SERIAL_TRANSPORT_MTU=128".
I also tried to add the option "-DRMW_UXRCE_STREAM_HISTORY=4" (which is the default value), as you suggested. Results are the same.
I cannot figure out where these issues could come from, I am probably missing something here.

Thanks for helping out on this! Have a nice week-end

[pablogs9]

Hello @anaelle-sw, I forgot to mention something. When using XML you have to ensure that the XML strings that the micro-ROS client sends to the agent in order to create the entities fit in the buffer. These XML strings are generated for example here.

So the point is that with MTU = 128 B and XRCE history = 4 some of these XML doesn't fit in the buffer. We have a couple of options:

1. Increase a bit the MTU, you can try increasing it a bit and see which is the ideal size.
2. Use refs. This option is harder since you have to write all your entities QoS on the agent side but the memory usage of the buffer will be just the data you send/receive and the reference labels (just a bunch of characters).

[anaelle-sw]

Hi!
Could you please confirm that my colcon.meta file, as copied in the previous message, is not malformed?

I tried to double the buffer size by setting "-DUCLIENT_SERIAL_TRANSPORT_MTU=256". This way the memory usage reaches 71%, which is approximately the acceptable limit for us. But the micro agent is still "stuck" after initialization. So I guess the strings generated by XML still don't fit in the buffer. I also tested this config with simple examples, and the same problem occurs.

Using refs is not really a good fit for us, at least for now. I will maybe test this for optimization later on, but now we would like to make our application run without needing to configure each entity, as some clients/servers might be changed in the next weeks/months.

After discussing with colleagues, we think it might be better to slightly re-design our application and reduce the number of services, while keeping the default middleware configuration. This way, we prevent to have memory usage conflicts between what we optimized for our application, and what the micro-ros lib requires. I will let you know if I still encounter memory usage issues even with a reduction of entities. By the way, we are still looking forward to be able to use the history pool optimization you are working on.

Thanks for support!

[pablogs9]

Hello @anaelle-sw, investigating this we have found a couple of bugs. I'm going to update the library this afternoon. I'll notify you here in a while.

[pablogs9]

Ok, we have merged micro-ROS/rmw_microxrcedds#84 and eProsima/Micro-XRCE-DDS-Client#167.

I just have updated this library to v0.0.3 and I have tested the publisher example with the following colcon.meta:

{
    "names": {
        "tracetools": {
            "cmake-args": [
                "-DTRACETOOLS_DISABLED=ON",
                "-DTRACETOOLS_STATUS_CHECKING_TOOL=OFF"
            ]
        },
        "rosidl_typesupport": {
            "cmake-args": [
                "-DROSIDL_TYPESUPPORT_SINGLE_TYPESUPPORT=ON"
            ]
        },
        "rcl": {
            "cmake-args": [
                "-DBUILD_TESTING=OFF",
                "-DRCL_COMMAND_LINE_ENABLED=OFF",
                "-DRCL_LOGGING_ENABLED=OFF"
            ]
        }, 
        "rcutils": {
            "cmake-args": [
                "-DENABLE_TESTING=OFF",
                "-DRCUTILS_NO_FILESYSTEM=ON",
                "-DRCUTILS_NO_THREAD_SUPPORT=ON",
                "-DRCUTILS_NO_64_ATOMIC=ON",
                "-DRCUTILS_AVOID_DYNAMIC_ALLOCATION=ON"
            ]
        },
        "microxrcedds_client": {
            "cmake-args": [
                "-DUCLIENT_PIC=OFF",
                "-DUCLIENT_PROFILE_UDP=OFF",
                "-DUCLIENT_PROFILE_DISCOVERY=OFF",
                "-DUCLIENT_PROFILE_SERIAL=ON",
                "-DUCLIENT_EXTERNAL_SERIAL=ON",
                "-DUCLIENT_SERIAL_TRANSPORT_MTU=128" 
            ]
        },
        "rmw_microxrcedds": {
            "cmake-args": [
                "-DRMW_UXRCE_MAX_NODES=1",
                "-DRMW_UXRCE_MAX_PUBLISHERS=21",
                "-DRMW_UXRCE_MAX_SUBSCRIPTIONS=3",
                "-DRMW_UXRCE_MAX_SERVICES=12",
                "-DRMW_UXRCE_MAX_CLIENTS=3",
                "-DRMW_UXRCE_MAX_HISTORY=1",
                "-DRMW_UXRCE_STREAM_HISTORY=5",
                "-DRMW_UXRCE_TRANSPORT=custom_serial"
            ]
        }
    }
}

Before testing it with your code, could you tell me if the example with this meta works for you?

PD: I also have updated the library generation Docker, do a docker pull ...

[anaelle-sw]

The example publisher works nicely for me with this config and the new release, as well as our complete application! It allow us to keep our application as it is, while having 53% memory usage: it is perfect for us! I will close this. Thanks a lot!

[asukiaaa]

I also faced to NotEnoughMemoryException.
Updating docker container of the agent solved the problem.

docker pull microros/micro-ros-agent:$ROS_DISTRO

My environment
OS: ubuntu22.04
ROS2: rolling
agent container: ea344502d8af 14 months ago 1.7GB -> b8389e854ce5 4 weeks ago 502MB