docs(image_projection_based_fusion): add algorithm description of sen…

…sor synchronization (#2705)

* ci: add sync-upstream.yaml (#4)

Signed-off-by: Kenji Miyake <kenji.miyake@tier4.jp>

* ci(sync-upstream): update settings (#19)

Signed-off-by: Kenji Miyake <kenji.miyake@tier4.jp>

* chore: sync files (#629)

* chore: sync files

Signed-off-by: GitHub <noreply@github.com>

* ci(pre-commit): autofix

Co-authored-by: kenji-miyake <kenji-miyake@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* chore: sync files (#637)

Signed-off-by: GitHub <noreply@github.com>

Co-authored-by: kenji-miyake <kenji-miyake@users.noreply.github.com>

* fix(dummy_diag_publisher): use anon to make unique node name instead of diag name (#639)

* chore: sync files (#648)

* chore: sync files

Signed-off-by: GitHub <noreply@github.com>

* Revert "chore: sync files"

This reverts commit b24f530.

Signed-off-by: Kenji Miyake <kenji.miyake@tier4.jp>

* sync codecov.yaml

Signed-off-by: Kenji Miyake <kenji.miyake@tier4.jp>

Co-authored-by: kenji-miyake <kenji-miyake@users.noreply.github.com>
Co-authored-by: Kenji Miyake <kenji.miyake@tier4.jp>

* fix(autoware_state_panel): fix message type for /api/autoware/get/engage (#666)

* fix(autoware_state_panel): fix message type for /api/autoware/get/engage

Signed-off-by: h-ohta <hiroki.ota@tier4.jp>

* ci(pre-commit): autofix

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix(behavior_velocity): avoid insert same point on trajectory utils (#834)

Signed-off-by: tanaka3 <ttatcoder@outlook.jp>

* refactor(behavior_velocity_planner): simplify CMakeLists.txt (#855)

Signed-off-by: Kenji Miyake <kenji.miyake@tier4.jp>

* fix(system_monitor): fix build error on tegra platform (#869)

* fix(system_monitor): fix build error on tegra platform

Signed-off-by: Shark Liu <shark.liu@autocore.ai>

* ci(pre-commit): autofix

* Update system/system_monitor/src/gpu_monitor/tegra_gpu_monitor.cpp

Co-authored-by: Shark Liu <shark.liu@autocore.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daisuke Nishimatsu <42202095+wep21@users.noreply.github.com>

* feat(ad_service_state_monitor): limit odometry buffer size (#514)

* feat(ad_service_state_monitor): limit odometry buffer size 40

Signed-off-by: Takayuki Murooka <takayuki5168@gmail.com>

* Update system/ad_service_state_monitor/src/ad_service_state_monitor_node/ad_service_state_monitor_node.cpp

Co-authored-by: Kenji Miyake <31987104+kenji-miyake@users.noreply.github.com>

* Update ad_service_state_monitor_node.cpp

Co-authored-by: Kenji Miyake <31987104+kenji-miyake@users.noreply.github.com>

* docs: fix 404 error caused by typo in url (#871)

* docs: fix 404 error caused by typo in url

Signed-off-by: Shin-kyoto <aquashin0202@gmail.com>

* docs: fix typo in url for yolov4

Signed-off-by: Shin-kyoto <aquashin0202@gmail.com>

* fix(image_projection_based_fusion): set imagebuffersize (#820)

* fix: set imagebuffersize configured

Signed-off-by: suchang <chang.su@autocore.ai>

* ci(pre-commit): autofix

Co-authored-by: suchang <chang.su@autocore.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* chore(avoidance_module): fix spell check (#732)

Signed-off-by: Muhammad Zulfaqar Azmi <zulfaqar.azmi@tier4.jp>

* feat: isolate gtests in all packages (#693)

Signed-off-by: Maxime CLEMENT <maxime.clement@tier4.jp>

* docs(virtual traffic light): add documentation (#245)

* doc(behavior_velocity): add graph and fix link

* doc(behavior_velocity): update virtual traffic light doc

Signed-off-by: tanaka3 <ttatcoder@outlook.jp>

* doc(behavior_velocity): minor fix

Signed-off-by: tanaka3 <ttatcoder@outlook.jp>

* doc : mediate to coordinate

Signed-off-by: tanaka3 <ttatcoder@outlook.jp>

* doc: minor update

Signed-off-by: tanaka3 <ttatcoder@outlook.jp>

* doc: fix pre-commit

* doc: update docs

Signed-off-by: tanaka3 <ttatcoder@outlook.jp>

* apply suggestion

* doc: to intersection-coordination

Co-authored-by: Kenji Miyake <31987104+kenji-miyake@users.noreply.github.com>

* feat(surround_obstacle_checker): separate surround_obstacle_checker from hierarchical planning flow (#830)

* fix(surroud_obstacle_checker): use alias

Signed-off-by: satoshi-ota <satoshi.ota928@gmail.com>

* feat(surround_obstacle_checker): use velocity limit

Signed-off-by: satoshi-ota <satoshi.ota928@gmail.com>

* chore(surround_obstacle_checker): rename publisher, subscriber and callback functions

Signed-off-by: satoshi-ota <satoshi.ota928@gmail.com>

* refactor(surround_obstacle_checker): use parameter struct

Signed-off-by: satoshi-ota <satoshi.ota928@gmail.com>

* fix(surround_obstacle_checker): use alias

Signed-off-by: satoshi-ota <satoshi.ota928@gmail.com>

* refactor(surround_obstacle_checker): cleanup member functions

Signed-off-by: satoshi-ota <satoshi.ota928@gmail.com>

* refactor(surround_obstacle_checker): cleanup polygon handling

Signed-off-by: satoshi-ota <satoshi.ota928@gmail.com>

* refactor(surround_obstacle_checker): use marker helper

Signed-off-by: satoshi-ota <satoshi.ota928@gmail.com>

* feat(planning_launch): separate surround_obstacle_checker from hierarchical motion planning flow

Signed-off-by: satoshi-ota <satoshi.ota928@gmail.com>

* feat: remove deprecated package in prediction launch (#875)

Signed-off-by: Yukihiro Saito <yukky.saito@gmail.com>

* fix(surround_obstacle_checker): fix ego footprint polygon (#877)

Signed-off-by: satoshi-ota <satoshi.ota928@gmail.com>

* fix: update nvinfer api (#863)

* fix(lidar_centerpoint): update nvinfer api

Signed-off-by: Daisuke Nishimatsu <border_goldenmarket@yahoo.co.jp>

* fix(tensorrt_yolo): update nvinfer api

Signed-off-by: Daisuke Nishimatsu <border_goldenmarket@yahoo.co.jp>

* fix(lidar_apollo_instance_segmentation): update nvinfer api

Signed-off-by: Daisuke Nishimatsu <border_goldenmarket@yahoo.co.jp>

* fix(traffic_light_classifier): update nvinfer api

Signed-off-by: Daisuke Nishimatsu <border_goldenmarket@yahoo.co.jp>

* fix(traffic_light_ssd_fine_detector): update nvinfer api

Signed-off-by: Daisuke Nishimatsu <border_goldenmarket@yahoo.co.jp>

* pre-commit run

Signed-off-by: Daisuke Nishimatsu <border_goldenmarket@yahoo.co.jp>

* fix(avoidance_module): ignore object instead of creating zero shift (#731)

* fix: ignore object instead of creating zero shift

instead of creating zero shift point, the object will be ignored.
no behavior changes should be observed.

Signed-off-by: Muhammad Zulfaqar Azmi <zulfaqar.azmi@tier4.jp>

* refactor: sync continue with upstream

Signed-off-by: Muhammad Zulfaqar Azmi <zulfaqar.azmi@tier4.jp>

* fix: fix debug message for insufficient lateral margin

Signed-off-by: Muhammad Zulfaqar Azmi <zulfaqar.azmi@tier4.jp>

* fix(motion_velocity_smoother): curve deceleration not working with a specific parameter set (#738)

Signed-off-by: Takamasa Horibe <horibe.takamasa@gmail.com>

* test(autoware_testing): fix smoke_test (#479)

* fix(autoware_testing): fix smoke_test

Signed-off-by: Keisuke Shima <19993104+KeisukeShima@users.noreply.github.com>

* restore smoke_test for trajectory_follower_nodes

Signed-off-by: Keisuke Shima <19993104+KeisukeShima@users.noreply.github.com>

* add support multiple parameter files

Signed-off-by: Keisuke Shima <19993104+KeisukeShima@users.noreply.github.com>

* ci(pre-commit): autofix

* minor fix

Signed-off-by: Keisuke Shima <19993104+KeisukeShima@users.noreply.github.com>

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* feat(rviz_plugins): add velocity limit to autoware state panel (#879)

* feat(rviz_plugins): add velocity limit to autoware state panel

Signed-off-by: tanaka3 <ttatcoder@outlook.jp>

* chore(rviz_plugin): change ms to kmh

Signed-off-by: tanaka3 <ttatcoder@outlook.jp>

* fix(dummy_perception_publisher): publish multiple layers of pointcloud (#882)

* fix: single -> multiple layers pointcloud

Signed-off-by: Hirokazu Ishida <h-ishida@jsk.imi.i.u-tokyo.ac.jp>

* refactor: share common among different pcloud creators

Signed-off-by: Hirokazu Ishida <h-ishida@jsk.imi.i.u-tokyo.ac.jp>

* feat(vehicle_info_util): add max_steer_angle (#740)

* feat(vehicle_info_util): add max_steer_angle

Signed-off-by: Takayuki Murooka <takayuki5168@gmail.com>

* applied pre-commit

Signed-off-by: Takayuki Murooka <takayuki5168@gmail.com>

* Added max_steer_angle in test config

Signed-off-by: Takayuki Murooka <takayuki5168@gmail.com>

Co-authored-by: Tomoya Kimura <tomoya.kimura@tier4.jp>

* ci(deploy-docs): remove mdx_unimoji (#883)

Signed-off-by: Shumpei Wakabayashi <shumpei.wakabayashi@tier4.jp>

* chore: sync files (#884)

Signed-off-by: GitHub <noreply@github.com>

Co-authored-by: kenji-miyake <kenji-miyake@users.noreply.github.com>

* fix(lidar_centerpoint): fix google drive url to avoid 404 (#889)

* fix(lidar_centerpoint): fix google drive url to avoid 404

* Update CMakeLists.txt

Co-authored-by: Kenji Miyake <31987104+kenji-miyake@users.noreply.github.com>

* chore: fix typos (#886)

Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>

* feat(state_rviz_plugin): add GateMode and PathChangeApproval Button (#894)

* feat(state_rviz_plugin): add GateMode and PathChangeApproval Button

* ci(pre-commit): autofix

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* feat(map_tf_generator): accelerate the 'viewer' coordinate calculation (#890)

* add random point sampling function to quickly calculate the 'viewer' coordinate

Signed-off-by: IshitaTakeshi <ishitah.takeshi@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Kenji Miyake <31987104+kenji-miyake@users.noreply.github.com>

* docs(obstacle_stop_planner): update documentation (#880)

Signed-off-by: satoshi-ota <satoshi.ota928@gmail.com>

* docs(tier4_traffic_light_rviz_plugin): update documentation (#905)

Signed-off-by: satoshi-ota <satoshi.ota928@gmail.com>

* fix(accel_brake_map_calibrator): rviz panel type (#895)

* fixed panel type

Signed-off-by: Mamoru Sobue <mamoru.sobue@tier4.jp>

* modified instruction for rosbag replay case

Signed-off-by: Mamoru Sobue <mamoru.sobue@tier4.jp>

* modified update_map_dir service name

Signed-off-by: Mamoru Sobue <mamoru.sobue@tier4.jp>

* fix(behavior velocity planner): skipping emplace back stop reason if it is empty (#898)

* skipping emplace back stop reason if it is empty

Signed-off-by: TakumiKozaka-T4 <takumi.kozaka@tier4.jp>

* add braces

Signed-off-by: TakumiKozaka-T4 <takumi.kozaka@tier4.jp>

* ci(pre-commit): autofix

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Takagi, Isamu <43976882+isamu-takagi@users.noreply.github.com>

* feat(behavior_path_planner): weakened noise filtering of drivable area (#838)

* feat(behavior_path_planner): Weakened noise filtering of drivable area

Signed-off-by: Takayuki Murooka <takayuki5168@gmail.com>

* fix lanelet's longitudinal disconnection

Signed-off-by: Takayuki Murooka <takayuki5168@gmail.com>

* add comments of erode/dilate process

Signed-off-by: Takayuki Murooka <takayuki5168@gmail.com>

* ci: add slack-send.yaml (#133)

Signed-off-by: Kenji Miyake <kenji.miyake@tier4.jp>

Signed-off-by: Kenji Miyake <kenji.miyake@tier4.jp>

* add: update readme

Signed-off-by: tzhong518 <sworgun@gmail.com>

* ci(pre-commit): autofix

* fix: remove unneeded file

Signed-off-by: tzhong518 <sworgun@gmail.com>

* fix: table format

Signed-off-by: tzhong518 <sworgun@gmail.com>

* ci(pre-commit): autofix

* Update perception/image_projection_based_fusion/README.md

Co-authored-by: Shunsuke Miura <37187849+miursh@users.noreply.github.com>

* Update README.md

* Update README.md

* Update README.md

---------

Signed-off-by: Kenji Miyake <kenji.miyake@tier4.jp>
Signed-off-by: tanaka3 <ttatcoder@outlook.jp>
Signed-off-by: Shin-kyoto <aquashin0202@gmail.com>
Signed-off-by: Muhammad Zulfaqar Azmi <zulfaqar.azmi@tier4.jp>
Signed-off-by: Maxime CLEMENT <maxime.clement@tier4.jp>
Signed-off-by: satoshi-ota <satoshi.ota928@gmail.com>
Signed-off-by: Yukihiro Saito <yukky.saito@gmail.com>
Signed-off-by: Daisuke Nishimatsu <border_goldenmarket@yahoo.co.jp>
Signed-off-by: Takamasa Horibe <horibe.takamasa@gmail.com>
Signed-off-by: Hirokazu Ishida <h-ishida@jsk.imi.i.u-tokyo.ac.jp>
Signed-off-by: Shumpei Wakabayashi <shumpei.wakabayashi@tier4.jp>
Signed-off-by: badai-nguyen <dai.nguyen@tier4.jp>
Signed-off-by: IshitaTakeshi <ishitah.takeshi@gmail.com>
Signed-off-by: Mamoru Sobue <mamoru.sobue@tier4.jp>
Signed-off-by: Takayuki Murooka <takayuki5168@gmail.com>
Signed-off-by: tzhong518 <sworgun@gmail.com>
Co-authored-by: Kenji Miyake <31987104+kenji-miyake@users.noreply.github.com>
Co-authored-by: tier4-autoware-public-bot[bot] <98652886+tier4-autoware-public-bot[bot]@users.noreply.github.com>
Co-authored-by: awf-autoware-bot[bot] <94889083+awf-autoware-bot[bot]@users.noreply.github.com>
Co-authored-by: kenji-miyake <kenji-miyake@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Hiroki OTA <hiroki.ota@tier4.jp>
Co-authored-by: Yuma Nihei <yuma.nihei@tier4.jp>
Co-authored-by: Kenji Miyake <kenji.miyake@tier4.jp>
Co-authored-by: taikitanaka3 <65527974+taikitanaka3@users.noreply.github.com>
Co-authored-by: Shark <71419791+Sharrrrk@users.noreply.github.com>
Co-authored-by: Shark Liu <shark.liu@autocore.ai>
Co-authored-by: Daisuke Nishimatsu <42202095+wep21@users.noreply.github.com>
Co-authored-by: Takayuki Murooka <takayuki5168@gmail.com>
Co-authored-by: Shintaro Tomie <58775300+Shin-kyoto@users.noreply.github.com>
Co-authored-by: storrrrrrrrm <103425473+storrrrrrrrm@users.noreply.github.com>
Co-authored-by: suchang <chang.su@autocore.ai>
Co-authored-by: Zulfaqar Azmi <93502286+zulfaqar-azmi-t4@users.noreply.github.com>
Co-authored-by: Maxime CLEMENT <78338830+maxime-clem@users.noreply.github.com>
Co-authored-by: Satoshi OTA <44889564+satoshi-ota@users.noreply.github.com>
Co-authored-by: Yukihiro Saito <yukky.saito@gmail.com>
Co-authored-by: Takamasa Horibe <horibe.takamasa@gmail.com>
Co-authored-by: Keisuke Shima <19993104+KeisukeShima@users.noreply.github.com>
Co-authored-by: Hirokazu Ishida <38597814+HiroIshida@users.noreply.github.com>
Co-authored-by: Tomoya Kimura <tomoya.kimura@tier4.jp>
Co-authored-by: Shumpei Wakabayashi <42209144+shmpwk@users.noreply.github.com>
Co-authored-by: badai nguyen <94814556+badai-nguyen@users.noreply.github.com>
Co-authored-by: Takeshi Ishita <ishitah.takeshi@gmail.com>
Co-authored-by: Mamoru Sobue <hilo.soblin@gmail.com>
Co-authored-by: TakumiKozaka-T4 <70260442+TakumiKozaka-T4@users.noreply.github.com>
Co-authored-by: Takagi, Isamu <43976882+isamu-takagi@users.noreply.github.com>
Co-authored-by: Yohei Mishina <66298900+YoheiMishina@users.noreply.github.com>
Co-authored-by: naokimatsunawa <naoki.matsunawa@tier4.jp>
Co-authored-by: Tomohito ANDO <tomohito.ando@tier4.jp>
Co-authored-by: Shunsuke Miura <37187849+miursh@users.noreply.github.com>

perception/image_projection_based_fusion/README.md

@@ -6,9 +6,55 @@ The `image_projection_based_fusion` is a package to fuse detected obstacles (bou
 
 ## Inner-workings / Algorithms
 
-Detail description of each fusion's algorithm is in the following links.
+### Sync Algorithm
+
+#### matching
+
+The offset between each camera and the lidar is set according to their shutter timing.
+After applying the offset to the timestamp, if the interval between the timestamp of pointcloud topic and the roi message is less than the match threshold, the two messages are matched.
+
+![roi_sync_image1](./docs/images/roi_sync_1.png)
+
+current default value at autoware.universe for TIER IV Robotaxi are: - input_offset_ms: [61.67, 111.67, 45.0, 28.33, 78.33, 95.0] - match_threshold_ms: 30.0
+
+#### fusion and timer
+
+![roi_sync_image2](./docs/images/roi_sync_2.png)
+
+The subscription status of the message is signed with 'O'.
+1.if a pointcloud message is subscribed under the below condition:
+| | pointcloud | roi msg 1 | roi msg 2 | roi msg 3 |
+| :-----------------: | :--------: | :-------: | :-------: | :-------: |
+| subscription status | | O | O | O |
+
+If the roi msgs can be matched, fuse them and postprocess the pointcloud message.
+Otherwise, fuse the matched roi msgs and cache the pointcloud.
+2.if a pointcloud message is subscribed under the below condition:
+| | pointcloud | roi msg 1 | roi msg 2 | roi msg 3 |
+| :-----------------: | :--------: | :-------: | :-------: | :-------: |
+| subscription status | | O | O | |
+
+if the roi msgs can be matched, fuse them and cache the pointcloud.
+3.if a pointcloud message is subscribed under the below condition:
+| | pointcloud | roi msg 1 | roi msg 2 | roi msg 3 |
+| :-----------------: | :--------: | :-------: | :-------: | :-------: |
+| subscription status | O | O | O | |
+
+If the roi msg 3 is subscribed before the next pointcloud messge coming or timeout, fuse it if matched, otherwise wait for the next roi msg 3.
+If the roi msg 3 is not subscribed before the next pointcloud messge coming or timeout, postprocess the pointcloud messege as it is.
+
+The timeout threshold should be set according to the postprocessing time.
+E.g, if the postprocessing time is around 50ms, the timeout threshold should be set smaller than 50ms, so that the whole processing time could be less than 100ms.
+current default value at autoware.universe for XX1: - timeout_ms: 50.0
+
+#### Known Limits
+
+The rclcpp::TimerBase timer could not break a for loop, therefore even if time is out when fusing a roi msg at the middle, the program will run until all msgs are fused.
+
+### Detail description of each fusion's algorithm is in the following links
 
 | Fusion Name                | Description                                                                                     | Detail                                       |
 | -------------------------- | ----------------------------------------------------------------------------------------------- | -------------------------------------------- |
 | roi_cluster_fusion         | Overwrite a classification label of clusters by that of ROIs from a 2D object detector.         | [link](./docs/roi-cluster-fusion.md)         |
 | roi_detected_object_fusion | Overwrite a classification label of detected objects by that of ROIs from a 2D object detector. | [link](./docs/roi-detected-object-fusion.md) |
+| pointpainting_fusion       | Paint the point cloud with the ROIs from a 2D object detector and feed to a 3D object detector. | [link](./docs/pointpainting-fusion.md)       |