| 研究生: |
林修平 LIN, HSIU PING |
|---|---|
| 論文名稱: |
大數據樣本選取方法及評估指標之比較 A Comparison of Subsampling Methods and Evaluation Metrics for Big Data |
| 指導教授: |
馬瀰嘉
Ma, Mi-Chia |
| 學位類別: |
碩士 Master |
| 系所名稱: |
管理學院 - 統計學系 Department of Statistics |
| 論文出版年: | 2026 |
| 畢業學年度: | 114 |
| 語文別: | 中文 |
| 論文頁數: | 66 |
| 中文關鍵詞: | 大數據 、樣本選取 、降維方法 、IBOSS 、TSF-u(PCA) 、資訊矩陣 |
| 外文關鍵詞: | Big data, subsample selection, dimensionality reduction methods, IBOSS, TSF-u(PCA), information matrix |
| 相關次數: | 點閱:12 下載:1 |
| 分享至: |
| 查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報 |
隨著大數據分析技術的普及,在有限的計算資源下,完整的資料在使用上經常備受挑戰。因此,如何從龐大的母體中選取具有代表性的子樣本,成為提升分析效率的重要課題。現有文獻基於最大化資訊矩陣行列式值提出資訊導向最佳子數據選擇法 (Information-BasedOptimal Subdata Selection; IBOSS),與結合主成份分析降維技術和空間填充抽樣概念的主成分分析修剪空間填滿抽樣法 Trimmed Space-Filling Sampling on PCA; TSF-u(PCA) ,它們雖然能處理大規模數據,但針對高維度下各種空間非對稱資料,是否仍有優化的空間,以及可兼顧各方面的評估指標。
首先,本研究提出偏態感知樣本選取法 (SkewAware Sampling; SA),為比較 IBOSS、TSF-u(PCA) 和 SA 三種子樣本選取方法的表現,本研究透過模擬資料和分析 UCI 機器學習資料庫化學氣體感測資料,提出單維度中位數誤差 (Absolute Deviation of Medians, ADM)和單維度四分位數絕對誤差 (Absolute Deviation of Quartiles; ADQ) 統計量,並使用馬氏距離(Mahalanobis Distance) 和 F 範數 (Frobenius Norm) 來評估 IBOSS、TSF-u(PCA) 和 SA 三種方法在多維空間 (2D、3D、4D) 投影的優劣。在評估指標上,我們採用資訊矩陣行列式值、ADM、ADQ 、馬氏距離、F 範數,以評估子樣本在中心位置估計效率和變異矩陣結構上與母體資料的差距。馬氏距離可偵測子樣本中心的偏移程度,其優點是考慮了變數間的相關性以確保抽樣無偏,但缺點則是計算上高度依賴逆矩陣 (Inverse Matrix) 的穩定性。F 範數可衡量共變異結構,其優點在於能直觀量化整體矩陣差異,但由於其平方和的計算特性,容易受變異數較大的變數或離群值所影響,故本研究也考慮相關係數矩陣範數。
本研究透過模擬研究證實 TSF-u(PCA) 仍是維持全域統計特徵的首選方法;然而,SA 方法提供一個在不需高耗能降維運算下,能更有效處理非對稱資料且優於傳統 IBOSS 的穩健抽樣方案。
Due to resource constraints in big data analytics, selecting representative subsamples is essential. Existing methods, such as IBOSS and TSF-u(PCA), effectively process large-scale data; however, their performance on high-dimensional, spatially asymmetric datasets and across multiple evaluation criteria remains unverified.
This study proposes an improved SA (SkewAware) method and compares it with IBOSS and TSF-u(PCA). Using simulated datasets and chemical gas sensor data from the UCI Machine Learning Repository, we construct the Absolute Deviation of Medians (ADM) and Absolute Deviation of Quartiles (ADQ) statistic for one-dimensional assessment and employ the Mahalanobis distance and Frobenius norm to evaluate performance in multi-dimensional projections (2D, 3D, and 4D). Evaluation metrics include the determinant of the information matrix, ADM, ADQ, Mahalanobis distance, and Frobenius norm to measure differences between subsamples and the full population in terms of center estimation efficiency and covariance structure. Mahalanobis distance captures shifts in subsample centers while accounting for variable correlations, though it depends on the stability of the inverse matrix. The Frobenius norm provides an intuitive measure of covariance differences but is sensitive to variables with large variances and outliers; thus, the correlation matrix norm is also considered.
Results show that TSF-u(PCA) best preserves global statistical characteristics. However, the proposed SA method offers a robust alternative that effectively handles asymmetric data without computationally intensive dimensionality reduction and outperforms the traditional IBOSS approach.
1. Cai, T. T., Zhang, C.-H., & Zhou, H. H. (2010). Optimal rates of convergence for covariance matrix estimation. Annals of Statistics, 38(4), 2118-2144.
2. Deng, J., Zhu, Z., Yang, C.-C., & Wang, H. (2024). Big Data Model Building Using Dimension Reduction and Sample Selection. Journal of Computational and Graphical Statistics, 33(3), 1013-1031.
3. Fan, J., Sun, Q., Zhou, W.-X., & Zhu, Z. (2018). Principal Component Analysis for Big Data. Wiley StatsRef: Statistics Reference Online, 1-13.
4. Fonollosa, J., Sheik, S., Huerta, R., & Marco, S. (2015). Reservoir Computing compensates slow response of chemosensor arrays exposed to fast varying gas concentrations in continuous monitoring. Sensors and Actuators B: Chemical, 215, 618-629.
5. Johnson, R. A., & Wichern, D. W. (2007). Applied Multivariate Statistical Analysis (6th ed.). Upper Saddle River, NJ: Pearson Prentice Hall.
6. Kulis, B. (2012). Metric Learning: A Survey. Foundations and Trends in Machine Learning, 5(4), 287-364.
7. Mahalanobis, P. C. (1936). On the Generalised Distance in Statistics. Proceedings of the National Institute of Sciences of India, 2, 49-55.
8. Montgomery, D. C. (2013). Design and Analysis of Experiments (8th ed.). Hoboken, NJ: John Wiley & Sons.
9. St. John, R. C., & Draper, N. R. (1975). D-Optimality for Regression Designs: A Review. Technometrics, 17(1), 15-23.
10. Thompson, S. K. (2012). Sampling (3rd ed.). Hoboken, NJ: John Wiley & Sons.
11. Wang, H., Yang, M., & Stufken, J. (2019). Information-based optimal subdata selection for big data linear regression. Journal of the American Statistical Association, 114(526), 393-405.