標準設定之效度評估-以2010年臺灣八年級學生國語文學習成就資料為例

Exploring Standard Setting in Chinese for the 2010 Taiwan Assessment of Student Achievement in Grade Eight

本研究是使用臺灣學生學習成就資料庫2010年收集之臺灣八年級學生實徵數據，進行八年級國語文學習成就的標準設定。本研究經學科專家確立政策定義及國語文表現標準描述後，邀請北、中、南、東及離島等區之19名標準設定成員，使用Yes/No Angoff方法執行標準設定，從效度的內部證據、效度的過程證據和效度的外部證據，探討TASA國語文八年級標準設定之適切性。本研究從極端判斷值、標準設定成員內設定結果的一致性和標準設定成員間設定結果的一致性評估效度的內部證據，本研究內部證據來源顯示本研究之標準設定結果具有一致性。從標準設定技術的選擇及執行，包含會議執行方式之評估、標準設定成員的選擇、標準設定方法、回饋訊息的影響等面向進行效度的過程證據評估。整體而言，八年級國語文科標準設定具有相當水準的過程效度。效度的外部證據檢視方面，本研究從切截分數設定與分類的一致性，Angoff法對於基礎水準切截分數設定較低，而在進階水準切截分數設定是較高的，但以Angoff方法為基礎，其他三種方法與Angoff方法之分類具有一致性。本研究能為實務應用者提供良好參考範例。

The purpose of this study is to conduct standard setting in Chinese for the 2010 Taiwan Assessment of Student Achievement (TASA) of grade eight. The specific definitions of the basic, proficient, and advanced achievement levels are defined by the subject matter experts. The nineteen panelists coming from north, middle, south, and east areas of Taiwan are invited to conduct the standard setting procedures with Yes/No Angoff method. This study evaluates the internal evidence for validity using the extreme value method, standard setting consistency within and between panelists. Such internal evidence then reports that the results of the standard settings in this study are consistent. Meanwhile, we also evaluate the procedural evidence of validity in the choice and execution of the standard settings which collect the assessment of panel meeting, the recruitment of panelists, standard setting methods, and the impact of feedbacks. And the procedural validity of the standard setting in Chinese for grade eight performs quite well as an entirety. Finally, the external evidence of validity is collected from the consistency between the cut-scores setting and the classification. Yes/No Angoff’s method may tend to set low cut-scores for the basic level while it poses high cut-scores for the advanced level. Based on the Yes/No Angoff method, its classification would be consistent with the three other methods. Based on the internal, procedural, and external evidences, the standard setting procedures for the 2010 TASA of gradeeight has its validity. This study shall provide a great reference for those who wish to conduct standard setting methods into practical use.