應用機器學習於小樣本研究：巢套交叉驗證之倡議

Applying Machine Learning to Small Sample Research: A Proposal for Nested Cross-Validation

近年來，機器學習（machine learning, ML）不只在電腦視覺、自然語言處理等科學領域取得前所未有的成就，生成式人工智慧的商品化也讓ML進入了大眾生活。儘管心理學研究主要仰賴線性統計模型，但仍有部分指標性的研究開始將ML運用於行為資料分析。然而，心理學研究常伴隨的小樣本問題對ML之應用造成了挑戰。本文透過偏誤變異權衡（bias-variance tradeoff）來說明小樣本對預測誤差之影響，以及ML慣用之交叉驗證與測試（cross-validation and testing, CVT）會如何導致所謂「預測誤差樂透（prediction error lottery）」之現象。為了減緩前述的問題，本研究提出一巢套交叉驗證（nested cross-validation, NCV）策略，預期NCV能得到更穩定的預測誤差估計，並有效避免預測誤差樂透之發生。本研究透過模擬實驗來檢驗前述假設，並了解CVT、NCV、以及其衍生策略的實徵表現。模擬的結果支持了我們的假設，並根據這些結果，我們為打算應用ML之研究者提供了使用策略之建議。

In recent years, machine learning (ML) has not only achieved unprecedented success in scientific domains such as computer vision and natural language processing but also, through the commercialization of generative AI, has infiltrated the everyday lives of the lay public. Psychological research, traditionally dependent on linear statistical models, is now incorporating ML to analyze behavioral data. This shift, however, is hampered by the typically small sample sizes in psychological studies, which challenge the robust application of ML techniques. This paper elucidates the effects of small sample sizes on prediction error estimates through an analysis of the bias-variance tradeoff and illustrates how the prevalent cross-validation and testing (CVT) strategy may inadvertently instigate a“prediction error lottery.”To counteract these issues, we introduce a nested cross-validation (NCV) strategy, which is posited to yield more stable prediction error estimates and to circumvent the prediction error lottery phenomenon effectively. We performed simulations to assess the empirical performance of CVT, NCV, and their variants, thereby validating our hypotheses. The outcomes of these simulations corroborate our conjectures and lead us to offer strategic recommendations for researchers poised to leverage ML in their work.