Questionnaire Pretesting Methods: A Comparison of Cognitive Interviewing and Respondent Debriefing Vis-à-vis the Study of the Adoption of Decision Support Systems by Knowledge Workers

As a vehicle for data collection, the questionnaire is a vital component in achieving higher quality in research studies. A frequent difficulty with questionnaire design is that respondents often misinterpret the questions. This difficulty has been recognized in the literature. In recent years, there has been increased emphasis on building quality into questionnaire design through pretesting. A pretest aims to ensure quality by subjecting the questionnaire to some types of evaluations to assess its ability to collect the desired data. Whether constructing a new scale or revising an existing scale, through pretesting, researchers may be able to confirm whether the scale uses clear and appropriate language, has no obvious errors or omissions, and has adequate psychometric properties. Questionnaire pretesting, therefore, is a crucial element of a good study design. Prior to the use of pretests, questionnaire problems were diagnosed mainly by the questionnaire designers (through their expertise in the subject) and the interviewers (through their experience in administering the questionnaire). Thus, detailed reports of appropriate methods to undertake pretesting are currently underrepresented. As the first of its kind, this paper compares the results of two pretesting methods – namely, cognitive interviewing and respondent debriefing – used to evaluate a series of questions from the questionnaire developed for the study of executive information systems (EIS) adoption factors by knowledge workers. We believe the protocols contribute to reducing shortfalls in pretesting guidance for researchers and practitioners. This paper also aims to inform others about the research and to encourage discussions on the topic.