簡易檢索 / 詳目顯示

回結果列表
研究生： 張芷瑀
Chang, Chih-Yu
論文名稱： 應用深度學習進行 MMS 點雲語意分割及影像分類產製高精地圖
Applying Deep Learning to MMS Point Cloud Semantic Segmentation and Image Classification for HD Map Generation
指導教授： 曾義星
Tseng, Yi-Hsing
學位類別： 碩士
Master
系所名稱： 工學院 - 測量及空間資訊學系
Department of Geomatics
論文出版年： 2021
畢業學年度： 109
語文別： 英文
論文頁數： 80
中文關鍵詞： 高精地圖 、移動式測繪系統 、深度學習 、語意分割 、影像分類
外文關鍵詞： HD Maps, MMS, Point cloud segmentation, Image Classification
相關次數： 點閱：248  下載：30 
分享至:
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報

    • 為實現自動架駛(Autonomous Driving)，車輛對於駕駛環境認知的需求促成了高精地圖(High Definition Map, HD Map)的發展。高精地圖是一種提供更高精度、詳細道路環境資訊的三維電子地圖，其組成要素包含精確的空間座標和語意屬性，能夠用來輔助自駕車定位、導航、避險等，以確保安全性。然而目前建置高精地圖的方式多仰賴人工數化，不僅消耗大量的人力資源，製作時間及成本也非常高，進而影響自駕車產業的發展。因此，要如何更自動化的從移動式測繪系統(Mobile Mapping System, MMS)蒐集來的大量資訊中萃取出高精地圖的必要數據是一大課題。
    本研究應用深度學習(Deep Learning, DL)的技術提出了一套完整可行的工作流程以自動化萃取資訊並建立交通標誌高精地圖。首先，利用PointNet 深度模型架構自動化分割點雲資料，並萃取出交通島(Traffic island)、標誌(Sign)、號誌(Signal)、桿狀物體(Pole)等必要之地上物。接著，將被分割成標誌的點雲資料透過基於密度聚類演算法(DBSCAN)分群，以便取得各獨立交通標誌的幾何屬性；語意屬性則透過投影及影像分類的方式取得。其中，語意之給定係藉由 GoogLeNet深度學習架構對多張影像進行分類並篩選完成。本研究另外提出了類似於訊噪比的篩選機制對影像分類結果進行評估，以確保語意屬性之正確性及代表性。最終，以沙崙自駕車場域做為測試場域，產製交通標誌高精地圖，成功提取出測試場域內12面標誌中的11面，且標誌分類結果皆正確，座標誤差亦皆符合規範要求。然而，沙崙場域之規模相較於國外大型數據集尚小了許多，並且資料的多樣性不足，因此若要實際應用於現實世界高精地圖產製流程的自動化，有待更多的訓練和測試，以期能更滿足應用需求。

    The ongoing race toward an autonomous era results in the development of High Definition (HD) Maps. To help extend the vision of self-driving vehicles and guarantee safety, HD maps provide detailed information about on-road environments with precise location and semantic meaning. However, one main challenge when making such a map is that it requires a massive amount of manual annotation, which is time-consuming and laborious. As such, to fulfill automation in extracting information from the sheer amount of data collected by mobile LiDAR scanners and cameras is at most concern. In this study, a workflow is proposed to provide a feasible alternative to building traffic sign HD maps automatically. To be mentioned, the entire workflow demonstrates the generation of HD map to a specific on-road inventory, that is, traffic sign. First, necessary components from LiDAR point clouds, including traffic islands, traffic signs, signals, and poles are extracted through PointNet. Then, point clouds of traffic signs are clustered by the DBSCAN algorithm so that the geometric information of each traffic sign can be obtained. An evaluation is performed to assess the accuracy of geolocation in the final stage. Next, point clouds in each traffic sign cluster are projected onto corresponding MMS images for classification purposes. Images of interest are cropped and then input to a traffic sign classifier based on GoogLeNet. Finally, the semantic attribute can be obtained based on the classification result and determined by a proposed mechanism, i.e. modified SNR ratio, which ensures the class with the most classified images is significant enough for that cluster to be considered as that specific type. An output text file including precise placement of traffic sign in 3D coordinate (geolocation), the position of both bottom-left and top-right of the traffic sign bounding box (bboxMin and bboxMax), as well as the type (code) is generated for further use in HD maps.

    Abstract i 摘要 ii 誌謝 iii Contents iv List of Tables vi List of Figures vii Chapter 1 Introduction 1 1.1 Background 1 1.2 Objectives 4 1.3 Literature Review 5 1.4 Thesis Structure 6 Chapter 2 Methodology 8 2.1 Deep Learning on 2D images 8 2.1.1 Convolution Neural Network (CNN) 9 2.1.2 GoogLeNet 12 2.2 Deep Learning on 3D Point Clouds 15 2.2.1 Point Cloud Semantic Segmentation 16 2.2.2 PointNet 18 2.3 Alignment of Point Clouds and Images 21 2.3.1 Clustering 22 2.3.2 Transformation definition in Photogrammetry and CV 23 2.3.3 Point Cloud to Image Projection 29 2.4 Inference and Assessment 30 2.4.1 Learning Curve 31 2.4.2 Confusion Matrix 33 2.4.3 Modified SNR 35 2.4.4 Coordinates Accuracy 37 Chapter 3 Neural Network Training Process 38 3.1 Testing Datasets 39 3.1.1 Test Field 39 3.1.2 Point Cloud Dataset 39 3.1.3 Image Dataset 40 3.2 Point Cloud Segmentation Training 42 3.2.1 Point Cloud Annotation 42 3.2.2 Point Cloud Pre-processing 43 3.2.3 Point Cloud Training Process 47 3.2.4 Point Cloud Augmentation 49 3.3 Image Classification Training 53 3.3.1 Image Annotation 53 3.3.2 Image Training Process 54 Chapter 4 Results and Discussion 57 4.1 Point Cloud Segmentation Results 57 4.2 Clustering and Projection Results 61 4.2.1 Clustering Results 61 4.2.2 Projection Results 64 4.3 Traffic Sign Classification Results 65 4.4 HD Map Results 72 Chapter 5 Summary and Future Work 74 References 76

    1. Seif, H.G. and X. Hu, Autonomous driving in the iCity—HD maps as a key challenge of the automotive industry. Engineering, 2016. 2(2): p. 159-162.
    2. Schreiber, M., C. Knöppel, and U. Franke. Laneloc: Lane marking based localization using highly accurate maps. in 2013 IEEE Intelligent Vehicles Symposium (IV). 2013. IEEE.
    3. Jiao, J. Machine learning assisted high-definition map creation. in 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC). 2018. IEEE.
    4. Ma, L., et al., Mobile laser scanned point-clouds for road object detection and extraction: A review. Remote Sensing, 2018. 10(10): p. 1531-1531.
    5. Staff, F. BMW China and Beijing-based Mapping Company NavInfo to Develop HD Maps for Autonomous Driving. 2019; Available from: https://m.futurecar.com/3346/BMW-China-and-Beijing-based-Mapping-Company-NavInfo-to-Develop-HD-Maps-for-Autonomous-Driving.
    6. Chellapilla, K. Rethinking Maps for Self-Driving. 2018.
    7. Yoshida, J. What HD mapping brings to autonomous vehicles. 2019.
    8. Yu, Y., et al., Automated extraction of urban road facilities using mobile laser scanning data. IEEE Transactions on intelligent transportation systems, 2015. 16(4): p. 2167–2181-2167–2181.
    9. Gargoum, S. and K. El-Basyouny, Effects of LiDAR point density on extraction of traffic signs: a sensitivity study. Transportation research record, 2019. 2673(1): p. 41–51-41–51.
    10. Che, E., J. Jung, and M.J. Olsen, Object recognition, segmentation, and classification of mobile laser scanning point clouds: A state of the art review. Sensors, 2019. 19(4): p. 810-810.
    11. Elhousni, M., et al. Automatic Building and Labeling of HD Maps with Deep Learning. in Proceedings of the AAAI Conference on Artificial Intelligence. 2020.
    12. Corp., R.S.a.G. International Conference on HD Maps for Autonomous Vehicle Handbook. 2019; Available from: http://www.hdm.geomatics.ncku.edu.tw/SpecialIssue/files/2019%20Annual%20Autonomous%20Vehicle%20Achievements%20Manual.pdf.
    13. Guan, H., et al., Using mobile laser scanning data for automated extraction of road markings. ISPRS Journal of Photogrammetry and Remote Sensing, 2014. 87: p. 93–107-93–107.
    14. Teo, T.-A. and C.-M. Chiu, Pole-like road object detection from mobile lidar system using a coarse-to-fine approach. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015. 8(10): p. 4805–4818-4805–4818.
    15. Riveiro, B., et al., Automatic segmentation and shape-based classification of retro-reflective traffic signs from mobile LiDAR data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015. 9(1): p. 295–303-295–303.
    16. Fukano, K. and H. Masuda, DETECTION AND CLASSIFICATION OF POLE-LIKE OBJECTS FROM MOBILE MAPPING DATA. ISPRS Annals of Photogrammetry, Remote Sensing & Spatial Information Sciences, 2015. 2.
    17. Yu, Y., et al., Semiautomated extraction of street light poles from mobile LiDAR point-clouds. IEEE Transactions on Geoscience and Remote Sensing, 2014. 53(3): p. 1374-1386.
    18. Zhuang, Z.-Y., Semantic Classification of LiDAR Point Cloud with 3D Deep Learning. 2020.
    19. Qin, F., B. Fang, and H. Zhao. Traffic sign segmentation and recognition in scene images. in 2010 Chinese Conference on Pattern Recognition (CCPR).
    20. Qian, R., et al. Robust Chinese traffic sign detection and recognition with deep convolutional neural network. in 2015 11th International Conference on Natural Computation (ICNC).
    21. Bascón, S.M., et al., An optimization on pictogram identification for the road-sign recognition task using SVMs. Computer Vision and Image Understanding, 2010. 114(3): p. 373-383.
    22. Zaklouta, F., B. Stanciulescu, and O. Hamdoun. Traffic sign classification using kd trees and random forests. in The 2011 international joint conference on neural networks. 2011. IEEE.
    23. Zaklouta, F. and B. Stanciulescu, Real-time traffic sign recognition in three stages. Robotics and autonomous systems, 2014. 62(1): p. 16-24.
    24. Dhar, P., et al. Traffic sign detection—A new approach and recognition using convolution neural network. in 2017 IEEE Region 10 Humanitarian Technology Conference (R10-HTC). 2017. IEEE.
    25. Balado, J., et al., Novel approach to automatic traffic sign inventory based on mobile mapping system data and deep learning. Remote Sensing, 2020. 12(3): p. 442-442.
    26. Gudigar, A., S. Chokkadi, and U. Raghavendra, A review on automatic detection and recognition of traffic sign. Multimedia Tools and Applications, 2016. 75(1): p. 333–364-333–364.
    27. Sermanet, P. and Y. LeCun. Traffic sign recognition with multi-scale convolutional networks. in The 2011 International Joint Conference on Neural Networks. 2011. IEEE.
    28. Cireşan, D., et al. A committee of neural networks for traffic sign classification. in The 2011 international joint conference on neural networks. 2011. IEEE.
    29. Zhu, Z., et al. Traffic-sign detection and classification in the wild. in Proceedings of the IEEE conference on computer vision and pattern recognition.
    30. Yamashita, R., et al., Convolutional neural networks: an overview and application in radiology. Insights into imaging, 2018. 9(4): p. 611-629.
    31. Rawat, W. and Z. Wang, Deep convolutional neural networks for image classification: A comprehensive review. Neural computation, 2017. 29(9): p. 2352-2449.
    32. Griffiths, D. and J. Boehm, A review on deep learning techniques for 3D sensed data classification. Remote Sensing, 2019. 11(12): p. 1499-1499.
    33. SHARMA, S. Activation Functions in Neural Networks. 2017; Available from: https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6.
    34. Szegedy, C., et al. Going deeper with convolutions. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
    35. Haloi, M., Traffic sign classification using deep inception based convolutional networks. arXiv preprint arXiv:1511.02992, 2015.
    36. Zhong, Z., L. Jin, and Z. Xie. High performance offline handwritten chinese character recognition using googlenet and directional feature maps. in 2015 13th International Conference on Document Analysis and Recognition (ICDAR). 2015. IEEE.
    37. liferlisiqi. Traffic-Sign-Classifier. 2018; Available from: https://github.com/liferlisiqi/Traffic-Sign-Classifier?fbclid=IwAR0-4AUjkD4u9wLATgkbBTK2npfpSgyjKltwY30dapKTEFSmyCij-NjcuQw.
    38. datahacker.rs. CNN Inception Network. 2018; Available from: http://datahacker.rs/deep-learning-inception-network/.
    39. Bello, S.A., et al., deep learning on 3D point clouds. Remote Sensing, 2020. 12(11): p. 1729-1729.
    40. Qi, C.R., et al. Volumetric and multi-view cnns for object classification on 3d data. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
    41. Feng, Y., et al. GVCNN: Group-view convolutional neural networks for 3D shape recognition. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
    42. Wei, X., R. Yu, and J. Sun. View-GCN: View-based graph convolutional network for 3D shape analysis. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
    43. Wang, C., et al., NormalNet: A voxel-based CNN for 3D object classification and retrieval. Neurocomputing, 2019. 323: p. 139-147.
    44. Zhou, Y. and O. Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
    45. Zhao, H., et al. Pointweb: Enhancing local neighborhood features for point cloud processing. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
    46. Hu, Q., et al. Randla-net: Efficient semantic segmentation of large-scale point clouds. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
    47. Guo, Y., et al., Deep learning for 3d point clouds: A survey. IEEE transactions on pattern analysis and machine intelligence, 2020.
    48. Li, Y., et al., Pointcnn: Convolution on x-transformed points. Advances in neural information processing systems, 2018. 31: p. 820–830-820–830.
    49. Thomas, H., et al. Kpconv: Flexible and deformable convolution for point clouds. in Proceedings of the IEEE/CVF International Conference on Computer Vision.
    50. Shi, W. and R. Rajkumar. Point-gnn: Graph neural network for 3d object detection in a point cloud. in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.
    51. Qi, C.R., et al., Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.02413, 2017.
    52. Hua, B.-S., M.-K. Tran, and S.-K. Yeung. Pointwise convolutional neural networks. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
    53. Siam, M., et al. Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges. in 2017 IEEE 20th international conference on intelligent transportation systems (ITSC).
    54. Tatarchenko, M., et al. Tangent convolutions for dense prediction in 3d. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
    55. Huang, J. and S. You. Point cloud labeling using 3d convolutional neural network. in 2016 23rd International Conference on Pattern Recognition (ICPR). 2016. IEEE.
    56. Dai, A., et al. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
    57. Dai, A. and M. Nießner. 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation. in Proceedings of the European Conference on Computer Vision (ECCV). 2018.
    58. Qi, C.R., et al. Pointnet: Deep learning on point sets for 3d classification and segmentation. in Proceedings of the IEEE conference on computer vision and pattern recognition.
    59. Liu, W., et al., Deep learning on point clouds and its application: A survey. Sensors, 2019. 19(19): p. 4188.
    60. Guan, H., et al., Robust traffic-sign detection and classification using mobile LiDAR data with digital images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018. 11(5): p. 1715–1724-1715–1724.
    61. Soilán, M., et al., Traffic sign detection in MLS acquired point clouds for geometric and image-based semantic inventory. ISPRS Journal of Photogrammetry and Remote Sensing, 2016. 114: p. 92–101-92–101.
    62. Arcos-Garcia, A., et al., Exploiting synergies of mobile mapping sensors and deep learning for traffic sign recognition systems. Expert Systems with Applications, 2017. 89: p. 286-295.
    63. Brownlee, J. 10 Clustering Algorithms With Python. 2020; Available from: https://machinelearningmastery.com/clustering-algorithms-with-python/.
    64. Ester, M., et al. A density-based algorithm for discovering clusters in large spatial databases with noise. in kdd. 1996.
    65. SHARMA, A., How to Master the Popular DBSCAN Clustering Algorithm for Machine Learning. 2020.
    66. (NLSC), N.L.S.a.M.C. Coordinate System. 2020; Available from: https://www.nlsc.gov.tw/en/cp.aspx?n=2122.
    67. DERENYI, E.E. PHOTOGRAMMETRY: THE CONCEPTS. 1996; Available from: http://www2.unb.ca/gge/Pubs/LN57.pdf.
    68. Ozcakir, E. Camera Calibration with OpenCV. 2020; Available from: https://medium.com/@elifozcakiir/camera-calibration-with-opencv-9fb104fdf879.
    69. Brownlee, J. How to use Learning Curves to Diagnose Machine Learning Model Performance. 2019; Available from: https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/.
    70. Shorten, C. and T.M. Khoshgoftaar, A survey on image data augmentation for deep learning. Journal of Big Data, 2019. 6(1): p. 1-48.
    71. Li, R., et al. Pointaugment: an auto-augmentation framework for point cloud classification. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
    72. Malinverni, E.S., et al., Deep learning for semantic segmentation of 3D point cloud. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, 2019.
    73. Kowalczuk, Z. and K. Szymański, Classification of objects in the LIDAR point clouds using Deep Neural Networks based on the PointNet model. IFAC-PapersOnLine, 2019. 52(8): p. 416-421.

    下載圖示
    校外：立即公開
    QR CODE