Abstract
Roadside perception is playing an increasingly important role in intelligent transportation systems (ITS), which can provide global object detection information for traffic dispatching and expand the sensing range of individual vehicles. However, existing approaches mainly focus on single-view or single-sensor perception, resulting in low perception accuracy and limited field of view in complex traffic scenarios. To solve such issues, this paper proposes a novel roadside perception framework for 3D object detection (RP3D) via multi-view cameras and LiDARs fusion. Firstly, a feature attention-guided lightweight 3D object detector is constructed for real-time object detection with multiple roadside LiDARs. Then, a 2D detector based on the efficient model NanoDet is adopted for multi-view image recognition. Moreover, a modified Hungarian algorithm is introduced to flexibly fuse multi-view and multi-sensor heterogeneous perception information from 2D and 3D detectors and improve detection accuracy. Furthermore, our proposal is deployed on a real-world V2X test field with four cameras and two LiDARs mounted on two roadside platforms. Experiments on DAIR-V2X-I and SCUT-V2R datasets demonstrate that the proposed method performs well in object detection accuracy and real-time performance in roadside perception scenes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Arnold, E., Dianati, M., de Temple, R., Fallah, S.: Cooperative perception for 3D object detection in driving scenarios using infrastructure sensors. IEEE Trans. Intell. Transp. Syst. 23(3), 1852–1864 (2022). https://doi.org/10.1109/TITS.2020.3028424
Bai, Z., et al.: Pillar attention encoder for adaptive cooperative perception. IEEE Internet Things J. (2024). https://doi.org/10.1109/JIOT.2024.3390552
Carrillo, J., Waslander, S.: UrbanNet: leveraging urban maps for long range 3D object detection. In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 3799–3806. IEEE (2021). https://doi.org/10.1109/ITSC48978.2021.9564840
Chen, M., Wu, Y., Gao, B., Zheng, K.: Vehicle target oriented bidirectional matching handover method for multi camera on roadside. Autom. Eng. 43(10), 1435–1441 (2019). https://doi.org/10.19562/j.chinasae.qcgc.2021.10.003
Chen, Y., Liu, J., Zhang, X., Qi, X., Jia, J.: VoxelNeXt: fully sparse VoxelNet for 3D object detection and tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21674–21683 (2023). https://doi.org/10.1109/CVPR52729.2023.02076
Contributors, M.: MMDetection3D: OpenMMLab next-generation platform for general 3D object detection (2020). https://github.com/open-mmlab/mmdetection3d
Deng, J., Shi, S., Li, P., Zhou, W., Zhang, Y., Li, H.: Voxel R-CNN: towards high performance voxel-based 3D object detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 1201–1209 (2021). https://doi.org/10.1609/aaai.v35i2.16207
Fleck, T., Ochs, S., Zofka, M.R., Zollner, J.M.: Robust tracking of reference trajectories for autonomous driving in intelligent roadside infrastructure. In: 2020 IEEE Intelligent Vehicles Symposium (IV), pp. 1337–1342 (2020). https://doi.org/10.1109/IV47402.2020.9304620
Hao, R., et al.: RCooper: a real-world large-scale dataset for roadside cooperative perception. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22347–22357 (2024). https://doi.org/10.48550/arXiv.2403.10145
Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Res. Logist. Q. 2(1–2), 83–97 (1955). https://doi.org/10.1002/nav.3800020109
Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: PointPillars: fast encoders for object detection from point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12697–12705 (2019). https://doi.org/10.1109/CVPR.2019.01298
Li, X., Yin, J., Li, W., Xu, C., Yang, R., Shen, J.: DI-V2X: learning domain-invariant representation for vehicle-infrastructure collaborative 3d object detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 3208–3215 (2024). https://doi.org/10.1609/aaai.v38i4.28105
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell. 42(02), 318–327 (2020). https://doi.org/10.1109/TPAMI.2018.2858826
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., Berg, A.C.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2
Masi, S., Xu, P., Bonnifait, P., Ieng, S.: Augmented perception with cooperative roadside vision systems for autonomous driving in complex scenarios. In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 1140–1146 (2021). https://doi.org/10.1109/ITSC48978.2021.9564833
RangiLyu: Nanodet-plus: Super fast and high accuracy lightweight anchor-free object detection model. https://github.com/RangiLyu/nanodet/. Accessed 9 Sept 2022
Shi, G., Li, R., Ma, C.: PillarNet: Real-time and high-performance pillar-based 3D object detection. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol 13670, pp. 35–52. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20080-9_3
Shi, S., et al.: PV-RCNN: point-voxel feature set abstraction for 3D object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10529–10538 (2020). https://doi.org/10.1109/CVPR42600.2020.01054
Shi, S., et al.: PV-RCNN++: point-voxel feature set abstraction with local vector representation for 3d object detection. Int. J. Comput. Vision 131(2), 531–551 (2023). https://doi.org/10.1007/s11263-022-01710-9
Shi, S., Wang, X., Li, H.: PointRCNN: 3D object proposal generation and detection from point cloud. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770–779 (2019). https://doi.org/10.1109/CVPR.2019.00086
Shi, S., Wang, Z., Wang, X., Li, H.: Part-A2 net: 3D part-aware and aggregation neural network for object detection from point cloud. arXiv:1907.03670, vol. 3, no. 2 (2019). https://doi.org/10.48550/arXiv.1907.03670
Sun, P., Sun, C., Wang, R., Zhao, X.: Object detection based on roadside LiDAR for cooperative driving automation: a review. Sensors 22(23) (2022). https://doi.org/10.3390/s22239316
Team, O.D.: OpenPCDet: an open-source toolbox for 3D object detection from point clouds (2020). https://github.com/open-mmlab/OpenPCDet
Wen, T., Xiao, Z., Jiang, K., Yang, M., Li, K., Yang, D.: High precision target positioning method for RSU in cooperative perception. In: 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP), pp. 1–6 (2019). https://doi.org/10.1109/MMSP.2019.8901755
Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: CBAM: convolutional block attention module. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_1
Yan, Y., Mao, Y., Li, B.: SECOND: sparsely embedded convolutional detection. Sensors 18(10), 3337 (2018). https://doi.org/10.3390/s18103337
Yang, H., et al.: GD-MAE: generative decoder for MAE pre-training on lidar point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9403–9414 (2023). https://doi.org/10.48550/arXiv.2212.03010
Yang, L., et al.: BEVHeight: a robust framework for vision-based roadside 3D object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21611–21620 (2023). https://doi.org/10.48550/arXiv.2303.08498
Ye, X., et al.: Rope3D: the roadside perception dataset for autonomous driving and monocular 3D object detection task. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21341–21350 (2022). https://doi.org/10.1109/CVPR52688.2022.02065
Yin, T., Zhou, X., Krahenbuhl, P.: Center-based 3D object detection and tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11784–11793 (2021). https://doi.org/10.1109/CVPR46437.2021.01161
Yu, H., et al.: DAIR-V2X: a large-scale dataset for vehicle-infrastructure cooperative 3D object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21361–21370 (2022). https://doi.org/10.1109/CVPR52688.2022.02067
Yu, H., et al.: V2X-seq: a large-scale sequential dataset for vehicle-infrastructure cooperative perception and forecasting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5486–5495 (2023). https://doi.org/10.1109/CVPR52729.2023.00531
Zhang, C., Wei, J., Hu, A.S., Fu, P.: A novel method for calibration and verification of roadside millimetre-wave radar. IET Intel. Transp. Syst. 16(3), 408–419 (2022). https://doi.org/10.1049/itr2.12151
Zhang, C., et al.: A roadside cooperative perception system with multi-camera fusion at an intersection. In: 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), pp. 642–649 (2023). https://doi.org/10.1109/ITSC57777.2023.10422029
Zheng, S., Xie, C., Huang, R., Yu, S., Ye, M., Li, W.: V2X-based cooperative positioning for connected vehicles in GNSS-denied environments. In: 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), pp. 4953–4959 (2023). https://doi.org/10.1109/ITSC57777.2023.10422687
Zheng, S., Xie, C., Yu, S., Ye, M., Huang, R., Li, W.: A robust strategy for roadside cooperative perception based on multi-sensor fusion. In: 2022 International Conference on Sensing, Measurement and Data Analytics in the era of Artificial Intelligence (ICSMD), pp. 1–6 (2022). https://doi.org/10.1109/ICSMD57530.2022.10058282
Zimmer, W., et al.: InfraDet3D: multi-modal 3D object detection based on roadside infrastructure camera and lidar sensors. In: 2023 IEEE Intelligent Vehicles Symposium (IV), pp. 1–8 (2023). https://doi.org/10.1109/IV55152.2023.10186723
Zimmer, W., Wardana, G.A., Sritharan, S., Zhou, X., Song, R., Knoll, A.C.: TUMTraf V2X cooperative perception dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 10 (2024). https://doi.org/10.48550/arXiv.2403.01316
Acknowledgments
This work is supported in part by the Key-Area Research and Development Program of Guangzhou City under Grant 202206030005, and in part by the Key-Area Research and Development Program of Guangdong Province under Grant 2019B090912001. The authors thank Guangzhou Huagong Automobile Inspection Technology Co., Ltd. for providing the test site and financial support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zheng, S., Huang, R., Ji, Y., Ye, M., Li, W. (2025). RP3D: A Roadside Perception Framework for 3D Object Detection via Multi-view Sensor Fusion. In: Del Bue, A., Canton, C., Pont-Tuset, J., Tommasi, T. (eds) Computer Vision – ECCV 2024 Workshops. ECCV 2024. Lecture Notes in Computer Science, vol 15630. Springer, Cham. https://doi.org/10.1007/978-3-031-91813-1_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-91813-1_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-91812-4
Online ISBN: 978-3-031-91813-1
eBook Packages: Computer ScienceComputer Science (R0)