feat(perception_benchmark_tool): add perception benchmark tool

[kaancolak]

Signed-off-by: kaancolak kcolak@leodrive.ai

Description

Resolves #565
Builds on top of #565

Related links

Tests performed

Notes for reviewers

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

• I've confirmed the contribution guidelines.
• The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

• The PR follows the pull request guidelines.
• The PR has been properly tested.
• The PR has been reviewed by the code owners.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

• There are no open discussions or they are tracked via tickets.
• The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

[kaancolak]

@1222-takeshi I approved your review request, but this pull request is still a draft.

[kaancolak]

I shared the initial 3D tracking benchmark results in the README file of the PR link. It contains only the results of the lidar-only pipeline. For running the camera-lidar-fusion pipeline, we need to run it with multiple GPUs, the Waymo dataset contains 5 cameras. After this PR #736, I will add its result.

For vehicles, there is no blockage. But for pedestrians, We have some problems. We are giving a constant length and width size to the pedestrian bounding boxes in Autoware.Universe, it's equal to 1 meter. But, Waymo Dataset has so strict 3D IoU scores for matching tracked ground truth objects and tracked object predictions. When we give a fixed size to pedestrians, it falls below the cutoff score. (Vehicle: 0.7 , Pedestrian and Cyclist: 0.5). For this reason, pedestrian scores are almost equal to zero.

We can use bounding boxes directly coming from 3D detection nodes just for evaluation but it needs an external change of the perception stack and reproducing benchmarking results could be really hard, or we can change the score threshold from the dataset config file but in this way, we can't compare our results with other. I think there are only 2 options.

If you have any suggestions or advice please share them.

[kaancolak]

@aohsato @yukkysaito @miursh @yukke42 @1222-takeshi @mitsudome-r @xmfcx

[xmfcx]

But for pedestrians, We have some problems. We are giving a constant length and width size to the pedestrian bounding boxes in Autoware.Universe, it's equal to 1 meter. But, Waymo Dataset has so strict 3D IoU scores for matching tracked ground truth objects and tracked object predictions. When we give a fixed size to pedestrians, it falls below the cutoff score.

If the current stack is getting 0 points for pedestrians, it's ok. We can work on improving the detection pipeline to increase it later on.

@1222-takeshi when can you review this? Will you be able to reproduce the results by following instructions in the readme file?

[miursh]

@kaancolak I believe this benchmark should be placed under "perception" directory. Is there any reason for making benchmark directly at top-level ?

[kaancolak]

Thanks, @miursh for your feedback.

Actually, I talked to Fatih when I first started this tool, he advised this directory structure. It can also take place in Perception, but some of the developers from the Robotec working on generic evaluation tools, which will contain multiple packages for metric calculation (localization, control, etc. ). We can collect all benchmarking and evaluation tools under a single top-level folder. But this is optional.

For now, I think, reproducing benchmarking result of this package is enough. Our goal was the compare our tracking result with other tracking submissions in the Waymo 3D Tracking Challenge.

And also, after the generic evaluation tools are finished, I am planning to connect the generic evaluation tools and perception benchmark tools for a more generic evaluator / benchmarking tools.

[mitsudome-r]

@ktro2828 Are you done with the review of the PR? If so, I would like you to give approval so that we can merge.

[ktro2828]

@mitsudome-r Sorry not yet. As mentioned above(his reply at 15 days ago), @.kaancolak is trying to update codes. After he gets ready, I will re-review.

[kaancolak]

@ktro2828, Sorry for the delay, I was dealing with other issues about BUS ODD field tests.

I made the updates, the package is now ready to be reviewed.

[ktro2828]

@kaancolak sorry for late to reaction. I'm reviewing, but build-and-test is failed in CI/CD. Can you fix it??
Also, please add copyrights to top of the codes.

[kaancolak]

Thanks, I added the license agreement. Currently, Autoware docker doesn't contain TensorFlow but but all data in the Waymo dataset is in TensorFlow "tfrecord" format. For this reason, it doesn't pass the CI/CD pipeline at the moment.

[xmfcx]

@kenji-miyake What is the right way to have pip packages like tensorflow, waymo_open_dataset?

• Should we put them to ansible?
• Or should we skip this test?
• Is it possible to add pip package dependencies to package.xml?

https://github.com/autowarefoundation/autoware.universe/actions/runs/3088532428/jobs/5011962436 here I suspect launch-testing-ros causes this error.

[kenji-miyake]

Is it possible to add pip package dependencies to package.xml?

@xmfcx Yes, it is possible. You can send pull requests to add your dependencies here.
https://github.com/ros/rosdistro/tree/master/rosdep

If you have some troubles with rosdistro, I think we can consider using Ansible instead.

And I believe skipping tests is only acceptable as a tentative workaround. In this case, you should create a follow-up issue.

[kaancolak]

I create PR for waymo_open_dataset, a related link. After this PR is merged, waymo_open_dataset will install the relevant TensorFlow version.

[ktro2828]

@kaancolak It seems the PR has been merged, so please update package.xml to install waymo-open-dataset with rosdep.

[kaancolak]

waymo-open-dataset-tf-2-6-0 depends on protobuf==3.9.2 but the system has protobuf==4.21.7 for some reason.

And this causes python unit tests to fail.

I couldn't find what installs the protobuf==4.21.7.

But if I install waymo with (without sudo) pip install waymo-open-dataset-tf-2-6-0, it installs protobuf to ~/.local/lib/python3.8/site-packages/google/protobuf/ and it overrides it and everything works.

@kenji-miyake Is there a clean way to skip colcon test for only this package? (without needing to modify github actions)

If so, I can update the README instructions to install it that way.

If not, I will close this PR and add it as a separate repository in autowarefoundation org.

[kenji-miyake]

@kaancolak I guess the dependency of waymo-open-dataset-tf-2-6-0 isn't written properly.

$ docker run --rm -it ubuntu:20.04 /bin/bash
$ apt update && apt install -y python3-pip
$ pip3 install -U -q waymo-open-dataset-tf-2-6-0
$ pip list | grep protobuf
protobuf                    4.21.7

Also, build fails on Humble. In that case, you can't merge this PR.
https://github.com/autowarefoundation/autoware.universe/actions/runs/3181365356/jobs/5186030427#step:6:118

$ docker run --rm -it ubuntu:20.04 /bin/bash
$ apt update && apt install -y python3-pip
$ pip3 install -U -q waymo-open-dataset-tf-2-6-0
ERROR: Could not find a version that satisfies the requirement waymo-open-dataset-tf-2-6-0 (from version
s: none)                                                                                                ERROR: No matching distribution found for waymo-open-dataset-tf-2-6-0

[xmfcx]

You can move the contents of this package to https://github.com/autowarefoundation/perception_benchmark_tool

[kaancolak]

this PR moved to autowarefoundation/perception_benchmark_tool#1