在機器學習領域中，「訓練資料量越大，模型效能越好」的基礎假設正日益受到挑戰。實證研究指出，約有 20% 的案例會出現「非單調學習曲線（Non-monotonic learning curves）」的現象，亦即增加資料量反而導致模型效能發生 1% 至 15% 不等的衰退。然而，過去相關文獻多依賴觀察性資料，難以確立因果關係。

本研究首度將「實驗設計（Design of Experiments, DOE）」框架引入學習曲線分析。第一階段採用 3^4 全因子設計，系統性地探討四項干擾因子（標記雜訊、分佈偏移、特徵雜訊與特徵相關性）對六種常見演算法（羅吉斯迴歸、隨機森林、XGBoost、K-近鄰演算法、支持向量機及多層感知器）的影響，共計執行 14,580 組控制實驗。第二階段則採用真實世界的 PhysioNet 敗血症（Sepsis）資料集進行驗證。透過模擬跨醫療機構間「盲目資料合併（blind data aggregation）」的消融實驗，本研究提供強烈的實證證據指出，隱藏的醫療常規差異實為一種分佈偏移；在此情況下，即使納入更多的訓練資料，仍會觸發模型效能的退化。

本研究旨在探討兩大核心問題：(1) 各項干擾因子及其交互作用對非單調學習曲線的因果效應為何？(2) 研究發現如何為實務上的資料收集決策提供指引？變異數分析（ANOVA）的結果顯示，非單調退化行為主要源自於複雜的交互作用，尤其是「標記雜訊」與「分佈偏移」兩者的結合。

本論文之主要貢獻包含：提出一套應用於機器學習因果推論的 DOE 框架、量化導致效能退化因子間的交互作用效應，並為「何時持續收集資料將適得其反」提供基於實證證據的決策參考。整體而言，本研究證實，在特定且可識別的條件下，盲目擴增資料量將系統性地損害模型預測效能。

The assumption that larger training datasets always improve machine learning performance is increasingly being challenged. Empirical studies have shown that non-monotonic learning curves, where additional data actually degrades performance, occur in roughly 20% of cases with drops ranging from 1% to 15%. However, most existing work relies on observational data and cannot establish clear causal relationships.

This thesis applies Design of Experiments (DOE) for the first time to learning curve analysis. In the first stage, we use a $3^4$ full factorial design to systematically examine four factors (label noise, distribution shift, feature noise, and feature correlation) across six algorithms: logistic regression, random forest, XGBoost, k-nearest neighbors, support vector machine, and multi-layer perceptron. This resulted in 14,580 controlled experiments. In the second stage, we validate these causal findings using the real-world PhysioNet Sepsis dataset. Through an ablation study simulating blind data aggregation between medical institutions, we provide strong empirical evidence suggesting that hidden protocol differences act as distribution shifts that trigger model degradation despite the inclusion of additional data.

Two research questions guide the study: (1) What are the causal effects of these factors and their interactions on non-monotonic learning curves? (2) How can the findings inform practical data collection decisions? ANOVA results reveal that non-monotonic behavior arises primarily from complex interactions, particularly between label noise and distribution shift.

The key contributions include a DOE-based framework for causal inference in machine learning, quantified interaction effects among degradation factors, and evidence-based thresholds for when collecting more data becomes counterproductive. Overall, this work demonstrates that under identifiable conditions, adding more data can systematically harm model performance.

Banko, M., & Brill, E. (2001). Scaling to very very large corpora for natural language disambiguation. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics (pp. 26–33).

Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32), 15849–15854.

Besbes, O., & Mouchtaki, O. (2023). How big should your data be? Data-driven newsvendor: Learning one sample at a time. Management Science, 69(11), 6667–6686.

Beyer, K., Goldstein, J., Ramakrishnan, R., & Shaft, U. (1999). When is “nearest neighbor” meaningful? In International Conference on Database Theory (pp. 217–235). Springer.

Cohen, J. P., Hashir, M., Brooks, R., & Bertrand, H. (2020). On the limits of cross-domain generalization in automated X-ray prediction. In Medical Imaging with Deep Learning (pp. 136–155). PMLR.

Fisher, R. A. (1935). The design of experiments. Oliver and Boyd.

Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys (CSUR), 46(4), 1–37.

Goldberger, A. L., Amaral, L. A., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C. K., & Stanley, H. E. (2000). PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation, 101(23), e215–e220.

Halevy, A., Norvig, P., & Pereira, F. (2009). The unreasonable effectiveness of data. IEEE Intelligent Systems, 24(2), 8–12.

Hand, D. J. (2006). Classifier technology and the illusion of progress. Statistical Science, 21(1), 1–14.

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer.

Hughes, G. (1968). On the mean accuracy of statistical pattern recognizers. IEEE Transactions on Information Theory, 14(1), 55–63.

Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2019). Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations (ICLR).

Nettleton, D. F., Orriols-Puig, A., & Fornells, A. (2010). A study of the effect of different types of noise on the precision of supervised learning techniques. Artificial Intelligence Review, 33(4), 275–306.

Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., & Lawrence, N. D. (Eds.). (2009). Dataset shift in machine learning. MIT Press.

Raudys, S. J., & Jain, A. K. (1991). Small sample size effects in statistical pattern recognition: Recommendations for practitioners. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(3), 252–264.

Reyna, M. A., Josef, C. S., Jeter, R., Shashikumar, S. P., Westover, M. B., Nemati, S., Clifford, G. D., & Sharma, A. (2020). Early prediction of sepsis from clinical data: the PhysioNet/Computing in Cardiology Challenge 2019. Critical Care Medicine, 48(2), 210–217.

Skurichina, M., & Duin, R. P. W. (2002). Bagging, boosting and the random subspace method for linear classifiers. Pattern Recognition, 35(6), 1215–1228.

Valiant, L. G. (1984). A theory of the learnable. Communications of the ACM, 27(11), 1134–1142.

Vapnik, V. N. (1999). An overview of statistical learning theory. IEEE Transactions on Neural Networks, 10(5), 988–999.

Viering, T., & Loog, M. (2022). A survey of learning curves with bad behavior: or how more data need not lead to better performance. arXiv preprint arXiv:2211.14061.

Viering, T., & Loog, M. (2023). The shape of learning curves: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6), 7799–7819.