簡易檢索 / 詳目顯示

研究生: 顧又榮
Gu, You Rong
論文名稱: 在分散式環境下設計高效率管理LiDAR資料的地理查詢系統
An efficient geospatial data management system for LiDAR in a distributed environment
指導教授: 李強
Lee, Chiang
學位類別: 碩士
Master
系所名稱: 電機資訊學院 - 資訊工程學系
Department of Computer Science and Information Engineering
論文出版年: 2016
畢業學年度: 104
語文別: 英文
論文頁數: 64
中文關鍵詞: 空載光達分散式系統點雲大數據地表最鄰近範圍查詢地理查詢系統
外文關鍵詞: LiDAR, distributed system, point cloud, big data, surface NN, range query, GIS
相關次數: 點閱:94下載:3
分享至:
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報
  • 我們實作了一套專為地理研究人員使用的地理資訊系統,其主要管理的資料為LiDAR資料,統稱為點雲資料 (point cloud),而且其資料量相當的龐大,這也使得我們在設計系統上有很大的挑戰。過去的研究都採單機環境管理,其運算資源和儲存空間都有限,實在不適合用來管理點雲資料,因此我們改採用master/slave分散式架構。在系統上的設計,主要有三個重要原件,第一,使用者介面,我們提供了使用者下達查詢的使用介面,方便使用者操作。第二,資料索引的設計,我們提出兩層式的索引架構,能讓查詢的處理加快。第三,我們採用資料庫的形式來儲存資料,因為相較於檔案系統,沒有檔案大小的限制,和支援多人同時查詢的好處。查詢處理的部份,我們主要處理的是range query和surface NN的查詢。其中較為困難的是surface NN的處理,我們是第一篇work探討在分散式環境下處理surface NN的問題。由於資料處理量相當龐大,礙於有限的記憶體和運算資源,我們提出了heuristic surface NN的solution。另外針對跨機器的表面距離計算,我們也提出了較為rough的heuristic surface distance的計算方法。而且在後面的實驗中也顯示了我們提出的heuristic solution的準確度可以高達80%以上。

    We design a geographic information system (GIS) for those professional researchers in Geography. This system is implemented to manage the LiDAR data called point clouds, and the data size is huge. Therefore, it makes our system design more challenge. However, lots of the past works adapt the one-machine system to manage the point clouds. Due to its limited computation resources and limited storage, it is not appropriate for managing such huge data. So, we adapt the distributed system which is master/slave architecture. In our system design, there are three components. First, it’s the user interface. We design the interface for users to apply the query request easily. Second, it’s the indexing design. We propose the two level indexing to manage the data and make the query processing more efficient. Third, we use the database to store the huge data. Compared with the FileSystem, it doesn’t have the limitation of the storage size and support multiple queries concurrently. For the query processing, we mainly work on the range query processing and surface NN query processing. Among these two queries, the surface NN is more difficult. Moreover we are the first one to propose the algorithm of surface NN query in distributed system. Due to the huge size, limited memory space, and complex computation, we propose the solution of heuristic surface NN, and we also propose the rough method to do the heuristic surface distance computation when the two points are on the different machines. In addition, the experiments show that the accuracy of our heuristic solution can be more than 80%.

    Chinese Abstract i Abstract ii Acknowledgements iii List of Contents iv List of Figures vii 1 Introduction 1 2 Related work 7 2.1 LiDAR technology introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 LiDAR data management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 Query processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4 Distributed system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3 Problem de nition 12 3.1 Metrics and Problem de nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.1.1 Range query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.1.2 Surface NN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.2 Shortest Surface Path Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4 System structure 17 4.1 System overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 4.2 Indexing design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 4.2.1 First level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 4.2.2 Second level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.3 Machines update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.3.1 Slave join . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.3.2 Slave departure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.3.3 Slave failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5 Algorithm 28 5.1 Range query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 5.2 Surface NN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 6 Experiment 43 6.1 Experiment setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6.2 System implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6.3 Experiment overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 6.4 Parameters in experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.5 Data storage eciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.6 Query processing eciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 6.6.1 Range query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 6.6.2 Surface NN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 6.7 Network IO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 6.7.1 Range query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 6.7.2 Surface NN query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 6.8 Grid size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 6.9 Answer correct rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 6.10 surface distance di erential rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 7 Conclusion and future work 61 Bibliography 62

    [1] Carter, J., Keil, S., Kirk, W., Lindy, B., Brian, H., Rebecca, M., and Jennifer, H. (2012). Lidar 101:
    An introduction to lidar technology, data, and applications.
    [2] Chen, J. and Han, Y. (1990). Shortest paths on a polyhedron. In Proceedings of the sixth annual
    symposium on Computational geometry, pages 360-369. ACM.
    [3] Deng, K., Shen, H. T., Xu, K., and Lin, X. (2006). Surface k-nn query processing. In 22nd Interna-
    tional Conference on Data Engineering (ICDE'06), pages 78-78. IEEE.
    [4] DeWitt, D. and Stonebraker, M. (2008). Mapreduce: A major step backwards. The Database Column,
    1:23.
    [5] DTM. http:==ngis:moea:gov:tw=moeaweb=Rinfo=KMSubItem:aspx?ID=20.
    [6] Du Mouza, C., Litwin, W., and Rigaux, P. (2009). Large-scale indexing of spatial data in distributed
    repositories: the sd-rtree. The VLDB Journal-The International Journal on Very Large Data Bases,
    18(4):933-958.
    [7] Elseberg, J., Borrmann, D., and Nuchter, A. (2013). One billion points in the cloud-an octree for
    e cient processing of 3d laser scans. ISPRS Journal of Photogrammetry and Remote Sensing, 76:76-88.
    [8] GeoHash. https:==en:wikipedia:org=wiki=Geohash.
    [9] GIS. https:==en:wikipedia:org=wiki=Geographicn informationn system.
    [10] Hinks, T., Carr, H., and Laefer, D. F. (2009). Flight optimization algorithms for aerial lidar capture
    for urban infrastructure model generation. Journal of Computing in Civil Engineering, 23(6):330-339.
    [11] Hsieh, Y.-C. Introduction to DTM. http:==twgeoref:moeacgs:gov:tw=GipOpenWeb=wSite=ct?
    xItem=140861&ctNode=1233&mp=105.
    [12] Koudas, N., Ooi, B. C., Tan, K.-L., and Zhang, R. (2004). Approximate nn queries on streams
    with guaranteed error/performance bounds. In Proceedings of the Thirtieth international conference
    on Very large data bases-Volume 30, pages 804-815. VLDB Endowment.
    [13] Laefer, D. F. and Pradhan, A. R. (2006). Evacuation route selection based on tree-based hazards
    using light detection and ranging and gis. Journal of transportation engineering, 132(4):312-320.
    [14] MpichCluster. https:==help:ubuntu:com=community=MpichCluster.
    [15] Otepka, J., Mandlburger, G., and Karel, W. (2012). The opals data manager-e cient data management
    for processing large airborne laser scanning projects. Proceedings of the ISPRS Annals of the
    Photogrammetry, Melbourne, Australia, 25:153-159.
    [16] RAID. https:==zh:wikipedia:org=wiki=RAID.
    [17] Sabo, N., Beaulieu, A., B elanger, D., Belzile, Y., and Pich e, B. (2014). The geohashtree: a multiresolution
    data structure for the management of point clouds. Technical notes, 4.
    [18] SchoN, B., Mosa, A. S. M., Laefer, D. F., and Bertolotto, M. (2013). Octree-based indexing for 3d
    pointclouds within an oracle spatial dbms. Computers & Geosciences, 51:430-438.
    [19] Shahabi, C., Tang, L.-A., and Xing, S. (2008). Indexing land surface for e cient knn query. Pro-
    ceedings of the VLDB Endowment, 1(1):1020-1031.
    [20] Sulebak, J. (2000). Applications of digital elevation models. DYNAMAP Project, page 11.
    [21] Tsatsanifos, G., Sacharidis, D., and Sellis, T. (2013). Index-based query processing on distributed
    multidimensional data. GeoInformatica, 17(3):489-519.
    [22] Xing, S., Shahabi, C., and Pan, B. (2009). Continuous monitoring of nearest neighbors on land
    surface. Proceedings of the VLDB Endowment, 2(1):1114-1125.
    [23] Xu, K., Zhou, X., and Lin, X. (2004). Direct mesh: a multiresolution approach to terrain visualization.
    In Data Engineering, 2004. Proceedings. 20th International Conference on, pages 766-776.
    IEEE.
    [24] Zhu, Q., Gong, J., and Zhang, Y. (2007). An e cient 3d r-tree spatial index method for virtual
    geographic environments. ISPRS Journal of Photogrammetry and Remote Sensing, 62(3):217-224.

    下載圖示 校內:2021-07-21公開
    校外:2021-07-21公開
    QR CODE