標準設定效度驗證之探究--以大型數學學習成就評量為例

Validating the Performance Standards for Cut Scores in a Large-Scale Mathematics Assessment

本研究旨在探討標準設定的效度，就程序性、內部性與外部性等方面進行表現標準的效度驗證，一方面透過類推性理論與其他重要規準（criteria），提供最終標準的內部性效度證據；另一方面，探究整體標準設定過程之品質，列舉程序性與外部性證據，以支持最終標準具備良好效度。本研究以一項大型數學學習成就評量為場景，為之設定基礎（basic）、精熟（proficient）、進階（advanced）等成就水準，設定材料為四、六年級評量的104個選擇題，延攬16 名具備國小數學教材編纂或教學經驗之專家、教師組成設定小組，進行標準設定活動。在標準設定方法部分，採行修正的Angoff 法設計標準設定程序，主要參考NAEP（Allen， Jenkins， Kulick， & Zelenak， 1997）、Reckase（1998， 2000）以及Hambleton（2001）所建議之方式與準則進行之，針對各個成就水準對應之通過分數進行標準設定之效度驗證，提出相關證據與效標來呈現表現標準具備的良好效度。本研究結果為，四年級標準設定結果呈現良好的內部性與程序性證據，可支持其表現標準之效度，惟在外部性證據上稍有瑕疵；六年級表現標準則在內部性、外\r部性與程序性方面，均獲致理想的效度證據支持。

The purpose of the study was to provide the validation of the performance standards for cut scores in a large-scale mathematics assessment. In order to testify and examine the improvement of consistency， the study applied different procedures and approaches to the modified Angoff standard setting method， including feedbacks with empirical difficulties and results information from previous setting round. A generalizabilitystudy was performed to investigate the sources and magnitude of the variation of results in terms of several facets item， panelist， and item difficulty.The important conclusions are summarized below:1.Cut scores were gradually and logically increased in accordance with the achievementof basic， proficient， and advanced. However， item difficulty had a great impact on the judgments of panelists. Therefore， the necessity of multiple rounds， discussion and feedbacks in a multi-stage standard setting process was guaranteed.2.As the setting rounds passed， the inconsistency between panelists was removed or adjusted， while error in the performance standards was declined to the lowest at the final round. The accuracy and consistency of cut scores in both grades were moderate.3.Finally， evidence of procedural and internal focus showed that the validity of performance standards at each achievement level was built and proved. Also， it was suggested that evidence of external focus needs to be strengthened.