initial version of plugin based storage api (ros2#7)

* initial version of plugin based storage api

* Add readable and writable storage interfaces

* Fix build and uncrustify

* Delete first storage interface proposal and adapt storage factory to new one

* Modify test to work with new storage interfaces

* Adapt sqlite3 plugin to new interface and extract rosbag2 part to own project

* Adapt read() and write() methods signature

* Prevent pluginlib from using boost

* Add plugin development documentation

* Remove Sqlite dependencies from rosbag2 tests

* Add tests to rosbag2_storage_default_plugins

* Add visibility control for Windows in rosbag_storage

* Rename visibility_control.h to visibility_control.hpp

* Cleanup CMakeLists in rosbag2_storage

* Use void * instead of char * in rosbag_storage

* Update plugin_description.xml and write() method

* Introduce better logging using rcutils in rosbag_storage

* Adapt interface and introduce better logging

* Fix package.xml in rosbag2_storage

* Add storage facade for plugins which are both readable and writable

* Extract bag_info struct to own file

* Change storage interface to have read/write access

* Adapt copyright and use copyright linter

* rosbag2 serialized message

* remove colcon ignores

* Add visibility to template specializations

* Remove no longer necessary File install from CMakeLists.txt

* Refactor storage_factory_impl.hpp

* Minor refactoring

* Add COLCON_IGNORE files to irrelevant projects

* Fix Windows warning

* Simpler class hierarchy without WritableStorage

* Use exceptions instead of bool returns everywhere in interface

* Change rosbag2_storage interface

* storage interfaces

* linters

* a bit of refactoring

* expose opening and closing

* take messages as shared ptr

* linters

* rename to open, unique_ptr for pimpl

* remove obsolete api

* comply with new interfaces

* change templated open to explicit open_ro and open_rw

* Delete superfluous classes + polishing

* Adapt SerializedBagMessage format

* Let sqlite3 storage use new interface

* Fix tests in rosbag2

* Write and read only data

* Replace creation of shared instance by unmanaged instance

* Add pragma for windows

* Add visibility control for Windows

* Expose template definitions

* Move const to better location

* Replace strcpy

* Delete superfluous methods

* Use visibility control in rosbag2

* Minor cleanup

* test for nullptr when opening storage

docs/plugin_development.md

@@ -0,0 +1,55 @@
+# Writing storage plugins
+
+There are different interfaces for storage plugins depending on your need: The general `ReadWriteStorage` and the more specific `ReadableStorage`.
+
+## Writing a general plugin
+
+Assume you write a plugin `MyStorage` which can both save messages and read messages.
+Its header file could be `my_storage.hpp` and it would derive from `rosbag2_storage::ReadWriteStorage`.
+While implementing the interface provided by `rosbag2_storage::ReadWriteStorage`, make sure that all resources such as file handles or database connections are closed or destroyed in the destructor, no additional `close` call should be necessary.
+
+In order to find the plugin at runtime, it needs to be exported to the pluginlib.
+Add the following lines to `my_storage.cpp`:
+
+```
+#include "pluginlib/class_list_macros.hpp"
+PLUGINLIB_EXPORT_CLASS(MyStorage, rosbag2_storage::ReadWriteStorage)
+```
+
+Furthermore, we need some meta-information in the form of a `plugin_description.xml` file.
+Here, it contains
+
+```
+<library path="my_storage_lib">
+  <class name="my_storage" type="MyStorage" base_class_type="rosbag2_storage::ReadWriteStorage">
+  </class>
+</library>
+```
+Here, `my_storage_lib` is the name of the library while `my_storage` is an identifier used by the pluginlib to load it.
+
+In addition, in the `CMakeLists.txt` plugins and `plugin_description` file need to be added to the index to be found at runtime:
+
+`pluginlib_export_plugin_description_file(rosbag2_storage plugin_description.xml)`
+
+The first argument `rosbag2_storage` denotes the library we add our plugin to, while the second argument is the path to the plugin description file.
+
+## Writing a plugin for reading only
+
+When writing plugins to only provide functionality for reading, derive from `rosbag2_storage::ReadableStorage`.
+
+If the read-only plugin is called `my_readonly_storage` in a library `my_storage_lib`, it will be registered using 
+
+```
+#include "pluginlib/class_list_macros.hpp"
+PLUGINLIB_EXPORT_CLASS(MyReadonlyStorage, rosbag2_storage::ReadableStorage)
+```
+with the plugin description
+```
+<library path="my_storage_lib">
+  <class name="my_readonly_storage" type="MyReadonlyStorage" base_class_type="rosbag2_storage::ReadableStorage">
+  </class>
+</library>
+```
+and the usual pluginlib export in the CMakeLists:
+
+`pluginlib_export_plugin_description_file(rosbag2_storage plugin_description.xml)`

rosbag2/CMakeLists.txt

@@ -12,24 +12,67 @@ if(NOT CMAKE_CXX_STANDARD)
 endif()
 
 if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
-  add_compile_options(-Wall -Wextra -Wpedantic)
+  add_compile_options(-Wall -Wextra -Wpedantic -Werror)
 endif()
 
-# find dependencies
 find_package(ament_cmake REQUIRED)
-# uncomment the following section in order to fill in
-# further dependencies manually.
-# find_package(<dependency> REQUIRED)
+find_package(rclcpp REQUIRED)
+find_package(rcutils REQUIRED)
+find_package(std_msgs REQUIRED)
+find_package(rosbag2_storage REQUIRED)
+
+add_library(
+  librosbag2
+  SHARED
+  src/rosbag2/rosbag2.cpp)
+
+ament_target_dependencies(librosbag2 rclcpp rcutils std_msgs rosbag2_storage)
+target_include_directories(librosbag2
+  PUBLIC
+  $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
+  $<INSTALL_INTERFACE:include>
+)
+
+# Causes the visibility macros to use dllexport rather than dllimport,
+# which is appropriate when building the dll but not consuming it.
+target_compile_definitions(librosbag2 PRIVATE "ROSBAG2_BUILDING_DLL")
+
+add_executable(${PROJECT_NAME}_record src/rosbag2/demo_record.cpp)
+target_link_libraries(${PROJECT_NAME}_record librosbag2)
+
+add_executable(${PROJECT_NAME}_play src/rosbag2/demo_play.cpp)
+target_link_libraries(${PROJECT_NAME}_play librosbag2)
+
+install(
+  DIRECTORY include/
+  DESTINATION include)
+
+install(
+  TARGETS librosbag2
+  ARCHIVE DESTINATION lib
+  LIBRARY DESTINATION lib)
 
 if(BUILD_TESTING)
+  find_package(ament_cmake_gtest REQUIRED)
+  find_package(ament_cmake_gmock REQUIRED)
   find_package(ament_lint_auto REQUIRED)
-  # the following line skips the linter which checks for copyrights
-  # remove the line when a copyright and license is present in all source files
-  set(ament_cmake_copyright_FOUND TRUE)
-  # the following line skips cpplint (only works in a git repo)
-  # remove the line when this package is a git repo
-  set(ament_cmake_cpplint_FOUND TRUE)
   ament_lint_auto_find_test_dependencies()
+
+  ament_add_gmock(rosbag2_write_integration_test
+    test/rosbag2/rosbag2_write_integration_test.cpp
+    test/rosbag2/rosbag2_test_fixture.hpp
+    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
+  if(TARGET rosbag2_write_integration_test)
+    target_link_libraries(rosbag2_write_integration_test librosbag2)
+  endif()
+
+  ament_add_gmock(rosbag2_read_integration_test
+    test/rosbag2/rosbag2_read_integration_test.cpp
+    test/rosbag2/rosbag2_test_fixture.hpp
+    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
+  if(TARGET rosbag2_read_integration_test)
+    target_link_libraries(rosbag2_read_integration_test librosbag2)
+  endif()
 endif()
 
 ament_package()

rosbag2/include/rosbag2/rosbag2.hpp

@@ -0,0 +1,41 @@
+// Copyright 2018, Bosch Software Innovations GmbH.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#ifndef ROSBAG2__ROSBAG2_HPP_
+#define ROSBAG2__ROSBAG2_HPP_
+
+#include <functional>
+#include <string>
+
+#include "rosbag2/visibility_control.hpp"
+
+namespace rosbag2
+{
+
+class Rosbag2
+{
+public:
+  ROSBAG2_PUBLIC
+  void record(
+    const std::string & file_name,
+    const std::string & topic_name,
+    std::function<void(void)> after_write_action = nullptr);
+
+  ROSBAG2_PUBLIC
+  void play(const std::string & file_name, const std::string & topic_name);
+};
+
+}  // namespace rosbag2
+
+#endif  // ROSBAG2__ROSBAG2_HPP_

rosbag2/include/rosbag2/visibility_control.h → rosbag2/include/rosbag2/visibility_control.hpp

@@ -12,8 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-#ifndef ROSBAG2__VISIBILITY_CONTROL_H_
-#define ROSBAG2__VISIBILITY_CONTROL_H_
+#ifndef ROSBAG2__VISIBILITY_CONTROL_HPP_
+#define ROSBAG2__VISIBILITY_CONTROL_HPP_
 
 #ifdef __cplusplus
 extern "C"
@@ -55,4 +55,4 @@ extern "C"
 }
 #endif
 
-#endif  // ROSBAG2__VISIBILITY_CONTROL_H_
+#endif  // ROSBAG2__VISIBILITY_CONTROL_HPP_

rosbag2/package.xml

@@ -9,6 +9,12 @@
 
   <buildtool_depend>ament_cmake</buildtool_depend>
 
+  <build_depend>rosbag2_storage</build_depend>
+  <build_depend>rcutils</build_depend>
+
+  <exec_depend>rosbag2_storage</exec_depend>
+  <exec_depend>rcutils</exec_depend>
+
   <test_depend>ament_lint_auto</test_depend>
   <test_depend>ament_lint_common</test_depend>
 

rosbag2/src/rosbag2/demo_play.cpp

@@ -0,0 +1,29 @@
+// Copyright 2018, Bosch Software Innovations GmbH.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "rclcpp/rclcpp.hpp"
+
+#include "rosbag2/rosbag2.hpp"
+
+int main(int argc, const char ** argv)
+{
+  rclcpp::init(argc, argv);
+
+  rosbag2::Rosbag2 rosbag2;
+  rosbag2.play("test.bag", "string_topic");
+
+  rclcpp::shutdown();
+
+  return 0;
+}

rosbag2/src/rosbag2/demo_record.cpp

@@ -0,0 +1,37 @@
+// Copyright 2018, Bosch Software Innovations GmbH.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include <cstdio>
+#include <string>
+
+#include "rclcpp/rclcpp.hpp"
+
+#include "rosbag2/rosbag2.hpp"
+
+int main(int argc, const char ** argv)
+{
+  // TODO(anhosi): allow output file to be specified by cli argument and do proper checking if
+  // file already exists
+  std::string filename("test.bag");
+  std::remove(filename.c_str());
+
+  rclcpp::init(argc, argv);
+
+  rosbag2::Rosbag2 rosbag2;
+  rosbag2.record(filename, "string_topic");
+
+  rclcpp::shutdown();
+
+  return 0;
+}

rosbag2/src/rosbag2/rosbag2.cpp

@@ -0,0 +1,87 @@
+// Copyright 2018, Bosch Software Innovations GmbH.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "rosbag2/rosbag2.hpp"
+
+#include <memory>
+#include <string>
+
+#include "rclcpp/rclcpp.hpp"
+
+#include "rcutils/logging_macros.h"
+
+#include "std_msgs/msg/string.hpp"
+
+#include "rosbag2_storage/serialized_bag_message.hpp"
+#include "rosbag2_storage/storage_interfaces/read_only_interface.hpp"
+#include "rosbag2_storage/storage_interfaces/read_write_interface.hpp"
+#include "rosbag2_storage/storage_factory.hpp"
+
+namespace rosbag2
+{
+
+const char * ROS_PACKAGE_NAME = "rosbag2";
+
+void Rosbag2::record(
+  const std::string & file_name,
+  const std::string & topic_name,
+  std::function<void(void)> after_write_action)
+{
+  rosbag2_storage::StorageFactory factory;
+  auto storage =
+    factory.open_read_write(file_name, "sqlite3");
+  storage->create_topic();
+
+  if (storage) {
+    auto node = std::make_shared<rclcpp::Node>("rosbag_node");
+    auto subscription = node->create_subscription<std_msgs::msg::String>(
+      topic_name,
+      [&storage, after_write_action](std::shared_ptr<rmw_serialized_message_t> msg) {
+        auto bag_msg = std::make_shared<rosbag2_storage::SerializedBagMessage>();
+        bag_msg->serialized_data = msg;
+        storage->write(bag_msg);
+        if (after_write_action) {
+          after_write_action();
+        }
+      });
+
+    RCUTILS_LOG_INFO_NAMED(ROS_PACKAGE_NAME, "Waiting for messages...");
+    rclcpp::spin(node);
+  }
+}
+
+void Rosbag2::play(const std::string & file_name, const std::string & topic_name)
+{
+  rosbag2_storage::StorageFactory factory;
+  auto storage =
+    factory.open_read_only(file_name, "sqlite3");
+
+  if (storage) {
+    auto node = std::make_shared<rclcpp::Node>("rosbag_publisher_node");
+    auto publisher = node->create_publisher<std_msgs::msg::String>(topic_name);
+    while (storage->has_next()) {
+      auto message = storage->read_next();
+
+      std_msgs::msg::String string_message;
+      // TODO(Martin-Idel-SI): We don't write a correct serialized_data in sqlite3_storage for now
+      // Change once available
+      string_message.data = std::string(message->serialized_data->buffer);
+      // without the sleep_for() many messages are lost.
+      std::this_thread::sleep_for(std::chrono::milliseconds(500));
+      publisher->publish(string_message);
+    }
+  }
+}
+
+}  // namespace rosbag2