This repository contains the performance results and comparisons obtained by eProsima using different testing tools. Our aim is to gather performance results for different DDS implementations and versions so their performance can be easily compared.
- Fast-RTPS Automated Benchmark
- DDS Vendors Comparison
- Fast-RTPS Cross-Version Comparison
- iRobot Performance Tests
Fast-RTPS Automated Benchmark provides a framework to automate the execution of performance tests for Fast-RTPS that is easy to deploy and integrate with Continuos Integration (CI) pipelines. The main goal is to ease the deployment and execution of performance test runs, generating reports that can be shared and used to evaluate Fast-RTPS' performance from various angles.
The following is an example of one of the many plots generated for the case of Latency tests.
It shows the evolution of Fast-RTPS' latency median performance when sending messages of 512 Bytes between a publisher and a subscriber in the same process when the reliability is set to BEST_EFFORT.
We execute and compare performance test for different DDS implementations (Fast-RPTS, CycloneDDS, and OpenSplice), using the performance testing tools provided by each vendor. We look at performance in terms of:
With the purpose of repeatability, each test case contains a description of the executed experiments, detailing the testing environment and configuration, as well as the software versions and any other information required to replicate the experiment.
An article on the latency performance of Fast-RTPS, CycloneDDS, and OpenSplice can be found here. The following constitute an example of such reports: one for latency and another one for throughput, both in the localhost test case (publisher and subscriber share machine but not process).
Latency comparison between Fast-RTPS (010ac53), CycloneDDS (801c4b1) and OpenSplice (v6.9)
Throughput comparison between Fast-RTPS (0bcafbd), CycloneDDS (aa5236d) and OpenSplice (v6.9)
To keep track of our own library's progress, we execute cross-version performance comparisons in terms of Latency.
iRobot has created a benchmark application to test the performance of ROS 2 in terms of latency, memory consumption, and CPU usages. This repository, not only contains tool for performance testing, but also a detail guide on how to cross-compile ROS 2 for a RaspberryPi 3b+ with Raspbian. We use that tool to characterize the different vendors' performance on a RaspberryPi 3b+. Here, you can find the experiments log, with a detail description of testing environment and configuration, as well as software versions tested. To ease the configuration of the experiments for different test-cases, a set of utilities for the iRobot performance tests has been created.
