This repo implements the paper FUTR3D: A Unified Sensor Fusion Framework for 3D Detection. Paper - project page
We built our implementation upon MMdetection3D 1.0.0rc6. The major part of the code is in the directory plugin/futr3d.
- mmcv-full>=1.5.2, <=1.7.0
- mmdet>=2.24.0, <=3.0.0
- mmseg>=0.20.0, <=1.0.0
- nuscenes-devkit
There is no neccesary to install mmdet3d separately, please install based on this repo:
cd futr3d
pip3 install -v -e .
Please follow the mmdet3d to process the data. mmdet3d_nuscenes_guidance
Notably, we have modified the nuscenes_converter.py to add the radar infomation, so the infos.pkl generated by our code is different from the original code. The other infomation except the radar infos is the same with the original infos.pkl.
For example, to train FUTR3D with LiDAR only on 8 GPUs, please use
bash tools/dist_train.sh plugin/futr3d/configs/lidar_only/lidar_0075_900q.py 8
For LiDAR-Cam and Cam-Radar version, we need pre-trained model.
The Cam-Radar uses DETR3D model as pre-trained model, please check DETR3D.
The LiDAR-Cam uses fused LiDAR-only and Cam-only model as pre-trained model. You can use
python tools/fuse_model.py --img <cam checkpoint path> --lidar <lidar checkpoint path> --out <out model path>
to fuse cam-only and lidar-only models.
For example, to evalaute FUTR3D with LiDAR-cam on 8 GPUs, please use
bash tools/dist_train.sh plugin/futr3d/configs/lidar_cam/lidar_0075_cam_res101.py ../lidar_cam.pth 8 --eval bbox
| models | mAP | NDS | Link |
|---|---|---|---|
| Res101 + VoxelNet | 67.4 | 70.9 | model |
| VoVNet + VoxelNet | 70.3 | 73.1 | model |
| models | mAP | NDS | Link |
|---|---|---|---|
| Res101 + Radar | 39.9 | 50.8 | model |
| models | mAP | NDS | Link |
|---|---|---|---|
| 32 beam VoxelNet | 63.3 | 68.9 | model |
| 4 beam VoxelNet | 44.3 | 56.4 | |
| 1 beam VoxelNet | 16.9 | 39.2 |
The camera-only version of FUTR3D is the same as DETR3D. Please check DETR3D for detail implementation.
For the implementation, we rely heavily on MMCV, MMDetection, MMDetection3D, and DETR3D
- DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries
- MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries
- For more projects on Autonomous Driving, check out our Visual-Centric Autonomous Driving (VCAD) project page webpage
@article{chen2022futr3d,
title={FUTR3D: A Unified Sensor Fusion Framework for 3D Detection},
author={Chen, Xuanyao and Zhang, Tianyuan and Wang, Yue and Wang, Yilun and Zhao, Hang},
journal={arXiv preprint arXiv:2203.10642},
year={2022}
}
Contact: Xuanyao Chen at: xuanyaochen19@fudan.edu.cn or ixyaochen@gmail.com