大學質性評鑑的挑戰：系所與外部專家評鑑觀點的一致性分析

CHALLENGES OF QUALITATIVE EVALUATION: DIFFERENT VIEWS OF REVIEWERS AND UNIVERSITY STAFF IN THE PROGRAM ACCREDITATION OF HIGHER EDUCATION

研究目的：質性評鑑受到質性研究的理念影響，質性研究的信實度已從過去客觀中立事實的認定，轉而追求詮釋與推論的確實性與可靠性，亦即質性研究者能從被研究者的立場詮釋，使雙方意見具有一致性。我國系所評鑑採行認可制，由熟悉大學治理的專家進行質性品質的判斷，以確保高等教育品質。雖然系所評鑑運用了質性研究的理念，以資料的三角校正等方法設計評鑑流程，然而，質性評鑑的品質仍然受到各界關注。由於評鑑專家是在社會情境脈落之下進行評鑑，專家本身的社會文化及個人信念，有可能帶入評鑑過程中。為尊重受評系所的意見，我國評鑑設計了申復機制，系所可藉由申復讓專家重新考量。本研究的目的是探討系所自評與外部專家評鑑觀點之異同，從訪評意見、學校申復、及申復回覆的意見往返，探討學校與外部專家意見的一致性，了解大學質性評鑑的挑戰。研究設計／方法／取徑：本研究運用內容分析法，分析2006至2010年第一週期系所評鑑1,156個班制的申復報告及其相關回覆。研究發現或結論：研究發現，系所申復報告主要說明大學提供資訊或質疑專家引用資訊的確實性（佔68.4%），次要是質疑評鑑結論與決定的公正性（佔13.4%），第三是說明系所情境脈絡或質疑專家未考慮學校情境（佔12.0%）。而專家對於申復的回覆，主要說明評鑑報告的結論與決定的（佔48.4%），次要是說明資訊確實性的考量情形（佔31.0%），第三是詳細說明評鑑推論（佔12.7%）。交叉檢視申復及回覆的往返內容，結果顯示系所評鑑運用了增進質性研究信實性的方法設計評鑑流程，然而，專家在執行上仍然面臨了四項威脅：一、未能考慮情境脈絡；二、檢視的是結果而非動態歷程；三、專家價值觀的涉入；四、評鑑歷程缺乏長期投入與互動。研究原創性／價值性：良好的大學質性評鑑，需要大學與專家具備相當的評鑑專業知能，才能從根本審視學校問題，促進學校改善。本研究以分析當前大學質性評鑑的執行情形及面臨的挑戰為例，提出增進質性評鑑品質的建議，以期作為改進評鑑機制及擬訂相關政策之參考。

Purpose Qualitative research is a method of inquiry employed in the qualitative evaluation. The quality of qualitative research was shifted from emphasizing on reliability and validity based on neutral facts to trustworthiness of inferences based on the consensus of qualitative researchers and participants. Taiwanese higher education adopts accreditation approach for educational quality assurance. Having experts with deeply understanding of the field being evaluated, peer review is able to make fair judgment and comments in external evaluation to assure the higher education quality. Although the accreditation approach applies the conceptions of qualitative research methods, such as triangulation for data collection during an evaluation procedure, the quality of qualitative evaluation has been challenged. It has been noticed that reviewers often conducted evaluations in a situated context and inevitably brought their own socio-cultural history and beliefs into the evaluation process. In order to remedy this problem, an objection and appeal system has been developed in Taiwanese higher education evaluation. The institution being evaluated can file an objection if they considered that the evaluation result is inconsistent with the facts. This study aims to investigate the major issues emerging from the objection reports and the corresponding replies, as well as to examine the gap between the educational quality of peer reviewers and institutional staff based on the objection reports. It provides an exploration of challenges of qualitative evaluation of higher education. Design/methodology/approach Applying content analysis methods, this study analyzed the objection reports of 1,156 programs during the 1st cycle of the program accreditation from 2006 to 2010. Findings Results showed that the institutions being evaluated tried to alter the conclusion of the on-site-visit reports by questioning the credibility of on-site visit report and providing more evidence (68.4%), weakening the justification of the review (13.4%), and asking reviewers to consider university context (12.0%). However, reviewers often repeated the same conclusion of the evaluation in the replies (48.4%), illustrated their considerations for data incredibility and inconsistency (31.0%), and provided detailed explanation for the evaluation reasoning (12.7%). Cross analysis of the opinions between university staff and reviewers revealed that the reviewers applied similar strategies to reply the objections of institutional staff. Four threats to credibility and consistency exist: 1. The university context was not fully considered; 2. The on-site visit examined only the isolated phenomena but not dynamic processes; 3. The evaluation judgments could be influenced by personal values; 4. The conclusion was not drawn based on long- term observations and interactions. Originality/value To conduct best practice of qualitative evaluation of higher education, institutional staff and external reviewers are required to have appropriate evaluation knowledge and skills. Making reflections from the suggestions of evaluation reports, institution can find out its weakness for institutional development, and make improvements for better educational quality. Through the inquiry of this study, the challenges of qualitative evaluation can be better understood, enabling future improvement of the evaluation process and making better educational policy of higher education.