以多層面Rasch 分析的角度來評估標準設定之變異性

Evaluating the Variability in Standard Setting Using Many Faceted Rasch Model

執行標準設定時，研究者常需檢視標準設定成員之間的變異性。研究者希望標準設定成員在設定切斷分數時，彼此之間對於決斷分數的判斷能夠達到一致性，也就是成員之間的變異性越小越好。此外，在舉行標準設定時，成員之間為了要能夠達到共識，必須經過好幾輪的討論。因此，如何提供有用的訊息回饋，供標準設定成員參考，以節省成員的討論時間，也是研究的重點。在本文中，將多層面Rasch 分析應用於標準設定上，並使用Yes/No Angoff 的方法來進行操作，研究發現透過多層面Rasch 之分析，除了能檢視標準設定成員之間的變異性，也可提供有用的訊息供標準設定成員參考。除此之外，對於標準設定成員自身內的衝突決定亦能提供有效率的檢視。

When conducting the standard setting, the variability of judgments between standard setting panelists is always an issue needed to be addressed. The researcher has to examine whether the variability between panelists is under the accepted range.
In addition, standard setting is a time-consuming process. It usually takes several rounds to discuss the judgments. How to provide the useful feedback for panelists to review their judgments is crucial. In this study, the many facet Rasch model was applied on the Yes/No Angoff standard setting procedure. The result shows that the many facet Rasch model has advantages on examining the variability between panelists. It also provides useful feedback to review the internal conflict decisions within each panelist.