在資料探勘領域中，常使用K等分交叉驗證法作為分類演算法之評估方法，但以此方法所獲得之分類正確率變異數通常較大，導致在後續的統計檢定中只能定下較保守的結論，因此，部分文獻建議重複執行交叉驗證法，並宣稱此法可以得到變異數較低的估計量，然而多數的文獻未提及每一輪交叉驗證之間的相關性，若每一輪交叉驗證間存在高度相關，忽略相關性將產生對變異數過度低估的情形，且高度相關的存在也代表由任一輪交叉驗證皆能獲得相似的結果，表示不一定需要耗費運算能量來重複執行K等分交叉驗證法。因此本研究的目的在於探討重複執行K等分交叉驗證法的合適性，考慮由每一輪交叉驗證所得之分類正確率是否符合獨立假設對分類正確率變異數計算的影響；以k近鄰演算法，令k為一，應用其較為簡單的特性來探討預測結果在不同輪間為相依或獨立之情形，並提出判斷每輪間是否達高度相關之統計檢定方法，以探討高度相關的情形在實際資料檔中發生的頻率。根據理論與實際驗證之結果，變異數之計算不應假設不同輪交叉驗證間為獨立，而相關性會隨著等分數的增加而變強，多數資料檔在10等分交叉驗證法下，重複執行將會有超過70%的資料分類結果相同；在大樣本條件成立下，若使用平均每輪變異數的算法來做為變異數的估計則較耗費運算能力。最後，本研究探討如何合適的使用重複執行K等分交叉驗證法，建議使用低重複高等分數之K等分交叉驗證法。

K-fold cross validation is a popular approach for evaluating the performance of classification algorithms. The variance of accuracy estimate resulting from this approach is generally relatively large for conservative inference. Several studies therefore suggested to repeatedly perform K-fold cross validation for reducing the variance. Most of them did not consider the correlation among the repetitions of K-fold cross validation, and hence the variance could be underestimated. The purpose of this thesis is to study the appropriateness of repeating K-fold cross validation. We first investigate whether the accuracy estimates obtained from the repetitions of K-fold cross validation can be assumed to be independent. K-Nearest Neighbor algorithm with K = 1 is used to analyze the dependency relationships among the predictions of two repetitions of K-fold cross validation. Statistical methods are also proposed to test the strength of the dependency relationships. The experimental results on twenty data sets show that the predictions in two repetitions of K-fold cross validation are generally highly correlated, and the correlation will be higher as the number of folds increases. The results of a simulation study suggest that the K-fold cross validation with a small number of repetitions and a large number of folds should be adopted.

Alpaydm, E. (1999). Combined 5× 2 cv F test for comparing supervised classification learning algorithms. Neural Computation, 11(8), 1885-1892.
Bengio, Y. & Grandvalet, Y. (2004). No unbiased estimator of the variance of k-fold cross-validation. Journal of Machine Learning Research, 5(Sep), 1089-1105.
Bouckaert, R. R. (2003). Choosing between two learning algorithms based on calibrated tests. Proceedings of the 20th International Conference on Machine Learning, Washington, DC, USA(ICML-03).
Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7), 1895–1923.
Filzmoser, P., Liebmann, B., & Varmuza, K. (2009). Repeated double cross validation. Journal of Chemometrics, 23(4), 160-171.
Grandvalet, Y. & Bengio, Y. (2006). Hypothesis testing for cross-validation. Montreal Universite de Montreal, Operationnelle DdIeR, 1285.
Kim, J. H. (2009). Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap. Computational Statistics & Data Analysis, 53(11), 3735-3745.
Lichman, M. (2013). UCI machine learning repository http://www.ics.uci.edu/~mlearn/MLRepository.html.
Raeder, T., Hoens, T. R., & Chawla, N. V. (2010). Consequences of variability in classifier performance estimates. Proceedings of 2011 IEEE International Conference on Data Mining, Sydney, Australia.
Rodriguez, J. D., Perez, A., & Lozano, J. A. (2010). Sensitivity analysis of k-fold cross validation in prediction error estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(3), 569-575.
Rodriguez, J. D., Perez, A., & Lozano, J. A. (2013). A general framework for the statistical analysis of the sources of variance for classification error estimators. Pattern Recognition, 46(3), 855-864
Santafe, G., Inza, I., & Lozano, J. A. (2015). Dealing with the evaluation of supervised classification algorithms. Artificial Intelligence Review, 44(4), 467-508.
Schmetterer, L. (1974). Introduction to Mathematical Statistics , Springer Science & Business Media.
Simon, R. M., Subramanian, J., Li, M.-C., & Menezes, S. (2011). Using cross-validation to evaluate predictive accuracy of survival risk classifiers based on high-dimensional data. Briefings in Bioinformatics, 12(3), 203-214.
Vanwinckelen, G. & Blockeel, H. (2012). On estimating model accuracy with repeated cross-validation. Proceedings of the 21st Belgian-Dutch Conference on Machine Learning, Ghent, Belgium.
Wang, Y., Li, J., & Li, Y. (2015). Measure for data partitioning in m× 2 cross-validation. Pattern Recognition Letters, 65, 211-217.
Wong, T.-T. (2015). Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation. Pattern Recognition, 48(9), 2839-2846.
Wang, Y., Li, J., Wang, R, & Jia, H. (2014). Blocked 3× 2 cross-validated t-test for comparing supervised classification learning algorithms. Neural Computation, 26(1), 208-235.